mcp-servers/nadlan-mcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add percentage-based backup outlier filtering for heterogeneous data

Browse Source 
Root cause: IQR filtering fails with wide distributions (mixed room counts, property types).
Example: 9,459 NIS/sqm deal (68% below average) passed IQR with k=1.0 due to wide IQR.

Solution: Dual filtering approach - IQR + percentage backup

Changes:
- Add ANALYSIS_USE_PERCENTAGE_BACKUP config (default: true)
- Add ANALYSIS_PERCENTAGE_THRESHOLD config (default: 0.5 = 50%)
- Apply percentage backup after IQR when method=iqr
- Update outlier report with percentage backup parameters
- Update CLAUDE.md docs with dual filtering explanation

Behavior:
1. Hard bounds filter (1K-100K/sqm, 100K min total)
2. IQR filter (k=1.0) - catches mild outliers
3. Percentage filter (50% from median) - catches extreme outliers
4. Deal removed if flagged by ANY method

Example: 12K/sqm deal (60% below 30K median) now caught by percentage backup
even when IQR bounds are permissive due to heterogeneous data.

Tests: All 326 tests pass, manual verification confirms extreme outliers removed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Nitzan Pomerantz
2025-11-29 14:44:14 +02:00
parent 8e591423da
commit 75ee79bbfd
3 changed files with 55 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+8 -2
View File
@@ -244,6 +244,7 @@ The MCP now includes configurable outlier detection and robust statistical measu
**Key Features:**
- **IQR-based outlier detection**: Uses Interquartile Range (robust to skewed data)
- **Percentage backup filtering**: Catches extreme outliers (>50% from median) in heterogeneous data
- **Hard bounds filtering**: Removes obvious errors (price_per_sqm < 1K or > 100K NIS/sqm, deals < 100K NIS)
- **Robust volatility**: Investment analysis uses IQR instead of std_dev for stability ratings
- **Transparent reporting**: Returns both filtered and unfiltered statistics with outlier reports
@@ -255,6 +256,10 @@ ANALYSIS_OUTLIER_METHOD=iqr # Options: iqr, percent, none (default: iqr
ANALYSIS_IQR_MULTIPLIER=1.0 # 1.0=aggressive, 1.5=moderate, 3.0=conservative (default: 1.0)
ANALYSIS_MIN_DEALS_FOR_OUTLIER_DETECTION=10 # Minimum deals needed (default: 10)
# Percentage-based Backup Filtering (catches extreme outliers in heterogeneous data)
ANALYSIS_USE_PERCENTAGE_BACKUP=true # Enable percentage backup (default: true)
ANALYSIS_PERCENTAGE_THRESHOLD=0.5 # Remove deals >50% from median (default: 0.5)
# Hard Bounds (catches obvious data errors)
ANALYSIS_PRICE_PER_SQM_MIN=1000 # 1K NIS/sqm minimum (default: 1000)
ANALYSIS_PRICE_PER_SQM_MAX=100000 # 100K NIS/sqm maximum (default: 100000)
@@ -296,9 +301,10 @@ original_mean = stats.price_per_sqm_statistics["mean"]
```
**Design Principles:**
- **Enabled by default**: Aggressive filtering (k=1.0) to catch common errors & suspicious deals
- **Dual filtering approach**: IQR (k=1.0) + percentage backup (50%) catch both mild and extreme outliers
- **Handles heterogeneous data**: Percentage backup catches outliers when IQR becomes too permissive (mixed room counts, property types)
- **Transparent**: Both filtered and unfiltered statistics returned
- **Conservative with real data**: Hard bounds + IQR preserve legitimate high-end properties
- **Conservative with real data**: Hard bounds + IQR + percentage preserve legitimate high-end properties
- **Configurable**: Adjust sensitivity via environment variables or function parameters
- **MCP provides data, LLM provides intelligence**: Outlier detection improves data quality; LLM interprets results