Add percentage-based backup outlier filtering for heterogeneous data
Root cause: IQR filtering fails with wide distributions (mixed room counts, property types). Example: 9,459 NIS/sqm deal (68% below average) passed IQR with k=1.0 due to wide IQR. Solution: Dual filtering approach - IQR + percentage backup Changes: - Add ANALYSIS_USE_PERCENTAGE_BACKUP config (default: true) - Add ANALYSIS_PERCENTAGE_THRESHOLD config (default: 0.5 = 50%) - Apply percentage backup after IQR when method=iqr - Update outlier report with percentage backup parameters - Update CLAUDE.md docs with dual filtering explanation Behavior: 1. Hard bounds filter (1K-100K/sqm, 100K min total) 2. IQR filter (k=1.0) - catches mild outliers 3. Percentage filter (50% from median) - catches extreme outliers 4. Deal removed if flagged by ANY method Example: 12K/sqm deal (60% below 30K median) now caught by percentage backup even when IQR bounds are permissive due to heterogeneous data. Tests: All 326 tests pass, manual verification confirms extreme outliers removed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -300,6 +300,36 @@ def filter_deals_for_analysis(
|
||||
if is_outlier:
|
||||
filters_to_remove[value_indices[i]] = True
|
||||
|
||||
# Step 4: Apply percentage-based backup filtering (catches extreme outliers in heterogeneous data)
|
||||
# This runs in addition to IQR when enabled, providing a safety net for wide distributions
|
||||
if config.analysis_use_percentage_backup and config.analysis_outlier_method == "iqr":
|
||||
# Extract values for the specified metric (same logic as above)
|
||||
if metric == "price_per_sqm":
|
||||
values = [
|
||||
deal.price_per_sqm
|
||||
for i, deal in enumerate(deals)
|
||||
if deal.price_per_sqm is not None and not filters_to_remove[i]
|
||||
]
|
||||
value_indices = [
|
||||
i
|
||||
for i, deal in enumerate(deals)
|
||||
if deal.price_per_sqm is not None and not filters_to_remove[i]
|
||||
]
|
||||
elif metric == "deal_amount":
|
||||
values = [deal.deal_amount for i, deal in enumerate(deals) if not filters_to_remove[i]]
|
||||
value_indices = [i for i in range(len(deals)) if not filters_to_remove[i]]
|
||||
else:
|
||||
values = []
|
||||
value_indices = []
|
||||
|
||||
if values:
|
||||
percentage_outliers = detect_outliers_percent(
|
||||
values, config.analysis_percentage_threshold
|
||||
)
|
||||
for i, is_outlier in enumerate(percentage_outliers):
|
||||
if is_outlier:
|
||||
filters_to_remove[value_indices[i]] = True
|
||||
|
||||
# Filter deals
|
||||
filtered_deals = [deal for i, deal in enumerate(deals) if not filters_to_remove[i]]
|
||||
outlier_indices = [i for i, should_remove in enumerate(filters_to_remove) if should_remove]
|
||||
@@ -314,6 +344,12 @@ def filter_deals_for_analysis(
|
||||
"iqr_multiplier": effective_iqr_multiplier
|
||||
if config.analysis_outlier_method == "iqr"
|
||||
else None,
|
||||
"percentage_backup_enabled": config.analysis_use_percentage_backup
|
||||
if config.analysis_outlier_method == "iqr"
|
||||
else None,
|
||||
"percentage_threshold": config.analysis_percentage_threshold
|
||||
if config.analysis_use_percentage_backup and config.analysis_outlier_method == "iqr"
|
||||
else None,
|
||||
"metric": metric,
|
||||
"price_per_sqm_min": config.analysis_price_per_sqm_min,
|
||||
"price_per_sqm_max": config.analysis_price_per_sqm_max,
|
||||
|
||||
Reference in New Issue
Block a user