Add percentage-based backup outlier filtering for heterogeneous data

Root cause: IQR filtering fails with wide distributions (mixed room counts, property types).
Example: 9,459 NIS/sqm deal (68% below average) passed IQR with k=1.0 due to wide IQR.

Solution: Dual filtering approach - IQR + percentage backup

Changes:
- Add ANALYSIS_USE_PERCENTAGE_BACKUP config (default: true)
- Add ANALYSIS_PERCENTAGE_THRESHOLD config (default: 0.5 = 50%)
- Apply percentage backup after IQR when method=iqr
- Update outlier report with percentage backup parameters
- Update CLAUDE.md docs with dual filtering explanation

Behavior:
1. Hard bounds filter (1K-100K/sqm, 100K min total)
2. IQR filter (k=1.0) - catches mild outliers
3. Percentage filter (50% from median) - catches extreme outliers
4. Deal removed if flagged by ANY method

Example: 12K/sqm deal (60% below 30K median) now caught by percentage backup
even when IQR bounds are permissive due to heterogeneous data.

Tests: All 326 tests pass, manual verification confirms extreme outliers removed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Nitzan Pomerantz
2025-11-29 14:44:14 +02:00
parent 8e591423da
commit 75ee79bbfd
3 changed files with 55 additions and 2 deletions
+11
View File
@@ -67,6 +67,15 @@ class GovmapConfig:
default_factory=lambda: int(os.getenv("ANALYSIS_MIN_DEALS_FOR_OUTLIER_DETECTION", "10"))
)
# Percentage-based backup filtering (catches extreme outliers in heterogeneous data)
analysis_use_percentage_backup: bool = field(
default_factory=lambda: os.getenv("ANALYSIS_USE_PERCENTAGE_BACKUP", "true").lower()
== "true"
)
analysis_percentage_threshold: float = field(
default_factory=lambda: float(os.getenv("ANALYSIS_PERCENTAGE_THRESHOLD", "0.5"))
)
# Hard Bounds for Price per Sqm (catches obvious data errors)
analysis_price_per_sqm_min: float = field(
default_factory=lambda: float(os.getenv("ANALYSIS_PRICE_PER_SQM_MIN", "1000"))
@@ -146,6 +155,8 @@ class GovmapConfig:
raise ValueError("analysis_iqr_multiplier must be positive")
if self.analysis_min_deals_for_outlier_detection < 0:
raise ValueError("analysis_min_deals_for_outlier_detection must be non-negative")
if self.analysis_percentage_threshold <= 0 or self.analysis_percentage_threshold >= 1:
raise ValueError("analysis_percentage_threshold must be between 0 and 1")
if self.analysis_price_per_sqm_min <= 0:
raise ValueError("analysis_price_per_sqm_min must be positive")
if self.analysis_price_per_sqm_max <= self.analysis_price_per_sqm_min: