Add statistical refinement and outlier detection system
Implement configurable outlier detection and robust statistical measures to improve analysis accuracy for real estate data. Addresses issues with data entry errors, partial deals, and other anomalies that skew statistics. Key Features: - IQR-based outlier detection (moderate filtering by default, k=1.5) - Hard bounds filtering for obvious errors (price_per_sqm, deal_amount) - Robust volatility using IQR instead of std_dev for investment analysis - Transparent reporting with both filtered and unfiltered statistics Implementation: - Add outlier_detection.py module with IQR/percent/hard bounds methods - Add OutlierReport model and enhance DealStatistics with filtered fields - Update calculate_deal_statistics() to support optional outlier filtering - Update analyze_investment_potential() to use robust volatility - Add 9 new configuration parameters for customization - Add comprehensive test suite (24 tests) for outlier detection - Update CLAUDE.md with usage documentation Configuration (all via env vars): - ANALYSIS_OUTLIER_METHOD=iqr (default, or percent/none) - ANALYSIS_IQR_MULTIPLIER=1.5 (moderate, 3.0=conservative) - ANALYSIS_PRICE_PER_SQM_MIN/MAX=1000/100000 (bounds in NIS/sqm) - ANALYSIS_MIN_DEAL_AMOUNT=100000 (catches partial deals) - ANALYSIS_USE_ROBUST_VOLATILITY=true (IQR-based CV) - ANALYSIS_USE_ROBUST_TRENDS=true (filter before regression) Testing: - All existing tests pass (326 passed) - 24 new comprehensive outlier detection tests - Real-world scenario tests (partial deals, data errors) Backward Compatible: - Default behavior improves accuracy without breaking changes - All new fields in models are optional - Config parameters have sensible defaults 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -56,6 +56,45 @@ class GovmapConfig:
|
||||
default_factory=lambda: int(os.getenv("GOVMAP_MAX_POLYGONS", "10"))
|
||||
)
|
||||
|
||||
# Outlier Detection & Statistical Refinement
|
||||
analysis_outlier_method: str = field(
|
||||
default_factory=lambda: os.getenv("ANALYSIS_OUTLIER_METHOD", "iqr")
|
||||
)
|
||||
analysis_iqr_multiplier: float = field(
|
||||
default_factory=lambda: float(os.getenv("ANALYSIS_IQR_MULTIPLIER", "1.5"))
|
||||
)
|
||||
analysis_min_deals_for_outlier_detection: int = field(
|
||||
default_factory=lambda: int(os.getenv("ANALYSIS_MIN_DEALS_FOR_OUTLIER_DETECTION", "10"))
|
||||
)
|
||||
|
||||
# Hard Bounds for Price per Sqm (catches obvious data errors)
|
||||
analysis_price_per_sqm_min: float = field(
|
||||
default_factory=lambda: float(os.getenv("ANALYSIS_PRICE_PER_SQM_MIN", "1000"))
|
||||
)
|
||||
analysis_price_per_sqm_max: float = field(
|
||||
default_factory=lambda: float(os.getenv("ANALYSIS_PRICE_PER_SQM_MAX", "100000"))
|
||||
)
|
||||
|
||||
# Hard Bounds for Deal Amount (catches partial deals)
|
||||
analysis_min_deal_amount: float = field(
|
||||
default_factory=lambda: float(os.getenv("ANALYSIS_MIN_DEAL_AMOUNT", "100000"))
|
||||
)
|
||||
|
||||
# Statistical Robustness (for investment analysis)
|
||||
analysis_use_robust_volatility: bool = field(
|
||||
default_factory=lambda: os.getenv("ANALYSIS_USE_ROBUST_VOLATILITY", "true").lower()
|
||||
== "true"
|
||||
)
|
||||
analysis_use_robust_trends: bool = field(
|
||||
default_factory=lambda: os.getenv("ANALYSIS_USE_ROBUST_TRENDS", "true").lower() == "true"
|
||||
)
|
||||
|
||||
# Reporting
|
||||
analysis_include_unfiltered_stats: bool = field(
|
||||
default_factory=lambda: os.getenv("ANALYSIS_INCLUDE_UNFILTERED_STATS", "true").lower()
|
||||
== "true"
|
||||
)
|
||||
|
||||
# User agent
|
||||
user_agent: str = field(
|
||||
default_factory=lambda: os.getenv("GOVMAP_USER_AGENT", "NadlanMCP/1.0.0")
|
||||
@@ -92,6 +131,20 @@ class GovmapConfig:
|
||||
if not self.user_agent:
|
||||
raise ValueError("user_agent cannot be empty")
|
||||
|
||||
# Validate outlier detection settings
|
||||
if self.analysis_outlier_method not in ["iqr", "percent", "none"]:
|
||||
raise ValueError("analysis_outlier_method must be one of: iqr, percent, none")
|
||||
if self.analysis_iqr_multiplier <= 0:
|
||||
raise ValueError("analysis_iqr_multiplier must be positive")
|
||||
if self.analysis_min_deals_for_outlier_detection < 0:
|
||||
raise ValueError("analysis_min_deals_for_outlier_detection must be non-negative")
|
||||
if self.analysis_price_per_sqm_min <= 0:
|
||||
raise ValueError("analysis_price_per_sqm_min must be positive")
|
||||
if self.analysis_price_per_sqm_max <= self.analysis_price_per_sqm_min:
|
||||
raise ValueError("analysis_price_per_sqm_max must be > analysis_price_per_sqm_min")
|
||||
if self.analysis_min_deal_amount <= 0:
|
||||
raise ValueError("analysis_min_deal_amount must be positive")
|
||||
|
||||
|
||||
# Global configuration instance
|
||||
_config: Optional[GovmapConfig] = None
|
||||
|
||||
Reference in New Issue
Block a user