Add percentage-based backup outlier filtering for heterogeneous data
Root cause: IQR filtering fails with wide distributions (mixed room counts, property types). Example: 9,459 NIS/sqm deal (68% below average) passed IQR with k=1.0 due to wide IQR. Solution: Dual filtering approach - IQR + percentage backup Changes: - Add ANALYSIS_USE_PERCENTAGE_BACKUP config (default: true) - Add ANALYSIS_PERCENTAGE_THRESHOLD config (default: 0.5 = 50%) - Apply percentage backup after IQR when method=iqr - Update outlier report with percentage backup parameters - Update CLAUDE.md docs with dual filtering explanation Behavior: 1. Hard bounds filter (1K-100K/sqm, 100K min total) 2. IQR filter (k=1.0) - catches mild outliers 3. Percentage filter (50% from median) - catches extreme outliers 4. Deal removed if flagged by ANY method Example: 12K/sqm deal (60% below 30K median) now caught by percentage backup even when IQR bounds are permissive due to heterogeneous data. Tests: All 326 tests pass, manual verification confirms extreme outliers removed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -67,6 +67,15 @@ class GovmapConfig:
|
||||
default_factory=lambda: int(os.getenv("ANALYSIS_MIN_DEALS_FOR_OUTLIER_DETECTION", "10"))
|
||||
)
|
||||
|
||||
# Percentage-based backup filtering (catches extreme outliers in heterogeneous data)
|
||||
analysis_use_percentage_backup: bool = field(
|
||||
default_factory=lambda: os.getenv("ANALYSIS_USE_PERCENTAGE_BACKUP", "true").lower()
|
||||
== "true"
|
||||
)
|
||||
analysis_percentage_threshold: float = field(
|
||||
default_factory=lambda: float(os.getenv("ANALYSIS_PERCENTAGE_THRESHOLD", "0.5"))
|
||||
)
|
||||
|
||||
# Hard Bounds for Price per Sqm (catches obvious data errors)
|
||||
analysis_price_per_sqm_min: float = field(
|
||||
default_factory=lambda: float(os.getenv("ANALYSIS_PRICE_PER_SQM_MIN", "1000"))
|
||||
@@ -146,6 +155,8 @@ class GovmapConfig:
|
||||
raise ValueError("analysis_iqr_multiplier must be positive")
|
||||
if self.analysis_min_deals_for_outlier_detection < 0:
|
||||
raise ValueError("analysis_min_deals_for_outlier_detection must be non-negative")
|
||||
if self.analysis_percentage_threshold <= 0 or self.analysis_percentage_threshold >= 1:
|
||||
raise ValueError("analysis_percentage_threshold must be between 0 and 1")
|
||||
if self.analysis_price_per_sqm_min <= 0:
|
||||
raise ValueError("analysis_price_per_sqm_min must be positive")
|
||||
if self.analysis_price_per_sqm_max <= self.analysis_price_per_sqm_min:
|
||||
|
||||
Reference in New Issue
Block a user