Add percentage-based backup outlier filtering for heterogeneous data

Root cause: IQR filtering fails with wide distributions (mixed room counts, property types).
Example: 9,459 NIS/sqm deal (68% below average) passed IQR with k=1.0 due to wide IQR.

Solution: Dual filtering approach - IQR + percentage backup

Changes:
- Add ANALYSIS_USE_PERCENTAGE_BACKUP config (default: true)
- Add ANALYSIS_PERCENTAGE_THRESHOLD config (default: 0.5 = 50%)
- Apply percentage backup after IQR when method=iqr
- Update outlier report with percentage backup parameters
- Update CLAUDE.md docs with dual filtering explanation

Behavior:
1. Hard bounds filter (1K-100K/sqm, 100K min total)
2. IQR filter (k=1.0) - catches mild outliers
3. Percentage filter (50% from median) - catches extreme outliers
4. Deal removed if flagged by ANY method

Example: 12K/sqm deal (60% below 30K median) now caught by percentage backup
even when IQR bounds are permissive due to heterogeneous data.

Tests: All 326 tests pass, manual verification confirms extreme outliers removed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Nitzan Pomerantz
2025-11-29 14:44:14 +02:00
parent 8e591423da
commit 75ee79bbfd
3 changed files with 55 additions and 2 deletions
+36
View File
@@ -300,6 +300,36 @@ def filter_deals_for_analysis(
if is_outlier:
filters_to_remove[value_indices[i]] = True
# Step 4: Apply percentage-based backup filtering (catches extreme outliers in heterogeneous data)
# This runs in addition to IQR when enabled, providing a safety net for wide distributions
if config.analysis_use_percentage_backup and config.analysis_outlier_method == "iqr":
# Extract values for the specified metric (same logic as above)
if metric == "price_per_sqm":
values = [
deal.price_per_sqm
for i, deal in enumerate(deals)
if deal.price_per_sqm is not None and not filters_to_remove[i]
]
value_indices = [
i
for i, deal in enumerate(deals)
if deal.price_per_sqm is not None and not filters_to_remove[i]
]
elif metric == "deal_amount":
values = [deal.deal_amount for i, deal in enumerate(deals) if not filters_to_remove[i]]
value_indices = [i for i in range(len(deals)) if not filters_to_remove[i]]
else:
values = []
value_indices = []
if values:
percentage_outliers = detect_outliers_percent(
values, config.analysis_percentage_threshold
)
for i, is_outlier in enumerate(percentage_outliers):
if is_outlier:
filters_to_remove[value_indices[i]] = True
# Filter deals
filtered_deals = [deal for i, deal in enumerate(deals) if not filters_to_remove[i]]
outlier_indices = [i for i, should_remove in enumerate(filters_to_remove) if should_remove]
@@ -314,6 +344,12 @@ def filter_deals_for_analysis(
"iqr_multiplier": effective_iqr_multiplier
if config.analysis_outlier_method == "iqr"
else None,
"percentage_backup_enabled": config.analysis_use_percentage_backup
if config.analysis_outlier_method == "iqr"
else None,
"percentage_threshold": config.analysis_percentage_threshold
if config.analysis_use_percentage_backup and config.analysis_outlier_method == "iqr"
else None,
"metric": metric,
"price_per_sqm_min": config.analysis_price_per_sqm_min,
"price_per_sqm_max": config.analysis_price_per_sqm_max,