Add statistical refinement and outlier detection system

Implement configurable outlier detection and robust statistical measures to
improve analysis accuracy for real estate data. Addresses issues with data
entry errors, partial deals, and other anomalies that skew statistics.

Key Features:
- IQR-based outlier detection (moderate filtering by default, k=1.5)
- Hard bounds filtering for obvious errors (price_per_sqm, deal_amount)
- Robust volatility using IQR instead of std_dev for investment analysis
- Transparent reporting with both filtered and unfiltered statistics

Implementation:
- Add outlier_detection.py module with IQR/percent/hard bounds methods
- Add OutlierReport model and enhance DealStatistics with filtered fields
- Update calculate_deal_statistics() to support optional outlier filtering
- Update analyze_investment_potential() to use robust volatility
- Add 9 new configuration parameters for customization
- Add comprehensive test suite (24 tests) for outlier detection
- Update CLAUDE.md with usage documentation

Configuration (all via env vars):
- ANALYSIS_OUTLIER_METHOD=iqr (default, or percent/none)
- ANALYSIS_IQR_MULTIPLIER=1.5 (moderate, 3.0=conservative)
- ANALYSIS_PRICE_PER_SQM_MIN/MAX=1000/100000 (bounds in NIS/sqm)
- ANALYSIS_MIN_DEAL_AMOUNT=100000 (catches partial deals)
- ANALYSIS_USE_ROBUST_VOLATILITY=true (IQR-based CV)
- ANALYSIS_USE_ROBUST_TRENDS=true (filter before regression)

Testing:
- All existing tests pass (326 passed)
- 24 new comprehensive outlier detection tests
- Real-world scenario tests (partial deals, data errors)

Backward Compatible:
- Default behavior improves accuracy without breaking changes
- All new fields in models are optional
- Config parameters have sensible defaults

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Nitzan P
2025-11-19 23:45:41 +02:00
parent 5be68a5b04
commit b78346f3b0
7 changed files with 951 additions and 37 deletions
+43
View File
@@ -183,6 +183,30 @@ class Deal(BaseModel):
raise TypeError(f"Unsupported type for date parsing: {type(v)}")
class OutlierReport(BaseModel):
"""
Report on outliers detected and removed from analysis.
Provides transparency about what data was filtered and why,
allowing LLMs to understand data quality issues.
Attributes:
total_deals: Total number of deals before filtering
outliers_removed: Number of outliers removed
outlier_indices: Indices of removed deals in original list
method_used: Outlier detection method ("iqr", "percent", "none")
parameters: Configuration parameters used for detection
"""
total_deals: int = Field(..., description="Total deals before filtering")
outliers_removed: int = Field(..., description="Number of outliers removed")
outlier_indices: List[int] = Field(default_factory=list, description="Indices of removed deals")
method_used: str = Field(..., description="Detection method (iqr, percent, none)")
parameters: Dict[str, Any] = Field(
default_factory=dict, description="Detection parameters used"
)
class DealStatistics(BaseModel):
"""
Statistical analysis of real estate deals.
@@ -222,6 +246,25 @@ class DealStatistics(BaseModel):
# Date range
date_range: Optional[Dict[str, str]] = Field(None, description="Earliest and latest deal dates")
# Outlier detection and filtered statistics
outlier_report: Optional[OutlierReport] = Field(
None, description="Outlier detection report (if filtering was applied)"
)
# Filtered statistics (after outlier removal)
filtered_deal_count: Optional[int] = Field(
None, description="Number of deals after outlier removal"
)
filtered_price_statistics: Optional[Dict[str, float]] = Field(
None, description="Price statistics after outlier filtering"
)
filtered_area_statistics: Optional[Dict[str, float]] = Field(
None, description="Area statistics after outlier filtering"
)
filtered_price_per_sqm_statistics: Optional[Dict[str, float]] = Field(
None, description="Price/sqm statistics after outlier filtering"
)
class MarketActivityScore(BaseModel):
"""