- Change ANALYSIS_IQR_MULTIPLIER default from 1.5 to 1.0 in config.py
- Add iqr_multiplier parameter to all filtering & statistics functions
- Allow runtime override via MCP tools (get_valuation_comparables, get_deal_statistics)
- Update CLAUDE.md docs with new default & override examples
Rationale: k=1.0 catches more suspicious deals (e.g. 43% below median) while
still preserving legitimate edge cases via hard bounds. Users can override
per-call for more conservative filtering (k=1.5) if needed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds visibility into the deal filtering pipeline to help diagnose
why certain queries return fewer deals than expected.
New logs show:
1. Number of deals before criteria filtering
2. Number of deals after criteria filtering (with count removed)
3. Whether outlier filtering was applied
4. Number of deals after outlier filtering (with count removed)
5. Outlier filtering method and parameters used
Example output:
INFO: Applying criteria filters to 41 deals
INFO: After criteria filtering: 12 deals (removed 29 deals)
INFO: After outlier filtering (iqr, k=1.5): 4 deals (removed 8 outliers)
This helps users and developers understand:
- If criteria filters are too restrictive
- If outlier filtering is too aggressive
- Where deals are being filtered out in the pipeline
Addresses user question: "Why do I get just 4 deals for a central address
like סירקין 16 תל אביב?" - Now they can see exactly where deals were
filtered out.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Bug: When the Govmap API returns the same polygon_id multiple times
(which can happen for addresses with multiple associated polygons),
the code was processing each duplicate separately, leading to:
- Duplicate API calls for street/neighborhood deals
- Wasted API quota and slower performance
- Confusing logs showing same polygon queried multiple times
Example from logs:
INFO: Querying polygon 53283601 (distance: 0m from search point)
INFO: Getting street deals for polygon: 53283601
INFO: Querying polygon 53283601 (distance: 0m from search point)
INFO: Getting street deals for polygon: 53283601
Root cause:
- nearby_polygons API response can contain duplicate polygon_ids
- Code was appending all polygons without deduplication
- Log message claimed "unique polygon IDs" but didn't enforce it
Solution:
- Use dict to deduplicate polygons by polygon_id (lines 612-643)
- When duplicates exist, keep the one with shortest distance
- Convert dict to list after deduplication
- Now truly have unique polygon IDs as the log claims
Impact:
- Reduces API calls (fewer duplicates = less load on Govmap API)
- Faster queries (less redundant processing)
- Clearer logs (no confusing duplicate entries)
- Better performance for addresses with many associated polygons
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The MCP now includes detailed outlier filtering information in the response,
allowing LLMs and bots to inform users about data quality improvements.
Changes to nadlan_mcp/fastmcp_server.py:
- Updated get_valuation_comparables to include outlier filtering metadata
- deal_breakdown now includes when filtering is applied:
- total_deals: Count after filtering (final result)
- total_deals_before_filtering: Original count before filtering
- outliers_removed: Number of deals filtered out
- filtering_method: Method used ("iqr", "percent", or "none")
- iqr_multiplier: IQR multiplier when using IQR method (e.g., 1.5)
- Updated docstring to clarify that outlier filtering is automatic
and metadata is included in response
Changes to tests/e2e/test_mcp_tools_comprehensive.py:
- Added assertions to verify outlier filtering metadata fields
- Test now validates the structure and types of filtering information
Example response:
{
"market_statistics": {
"deal_breakdown": {
"total_deals": 15,
"total_deals_before_filtering": 17,
"outliers_removed": 2,
"filtering_method": "iqr",
"iqr_multiplier": 1.5
}
}
}
This allows bots like nadlan-bot to properly inform users:
"Found 17 comparable deals, filtered 2 outliers using IQR method (k=1.5),
showing 15 deals."
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added log_mcp_call() helper function that logs all MCP tool invocations
with their parameters at INFO level. This provides:
- Better debugging and monitoring capabilities
- Clear visibility into which tools are being called and with what parameters
- Smart parameter formatting (truncates long strings, summarizes long lists)
- Consistent logging format across all 10 MCP tools
Example log output:
INFO - MCP tool called: find_recent_deals_for_address(address=סוקולוב 38 חולון, years_back=2, radius_meters=30, max_deals=100, deal_type=2)
This helps track API usage patterns and debug issues in production.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The same-building detection was failing because the code was accessing
deal.street_name and deal.house_number (Pydantic model field names),
but the Govmap API actually returns streetNameHeb/streetNameEng and
houseNum as extra fields.
Modified address construction to check multiple field names using
getattr() to handle the API's actual field names, falling back to
model field names if needed.
This fix ensures same-building deals are correctly identified and
prioritized with priority=0.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implements smart deal prioritization based on distance from search point,
ensuring most relevant comparables are selected first.
Problem:
- Queried all polygons equally without distance consideration
- No filtering for deals that are too far away
- Street/neighborhood deals could be from distant locations
- No way to prefer closest polygon when multiple available
Solution:
1. Sort polygons by distance from search point (closest first)
2. Query polygons progressively, stop early if enough deals found
3. Calculate and store distance_meters for each deal
4. Filter deals by configurable distance thresholds:
- Street deals: max 500m (configurable)
- Neighborhood deals: max 1000m (configurable)
5. Sort by priority → distance → date (prefer closer deals)
Changes:
Config (nadlan_mcp/config.py):
- Added max_street_deal_distance_meters (default: 500m)
- Added max_neighborhood_deal_distance_meters (default: 1000m)
Client (nadlan_mcp/govmap/client.py):
- Extract polygon metadata with coordinates
- Sort polygons by distance using utils.calculate_distance()
- Progressive polygon querying (closest first, stop when enough deals)
- Calculate deal distance (use polygon distance as approximation)
- Filter street deals beyond max_street_deal_distance_meters
- Filter neighborhood deals beyond max_neighborhood_deal_distance_meters
- Store distance_meters on each Deal (dynamic attribute)
- Updated sorting: Priority → Distance → Date
Benefits:
- More relevant comparables (from nearby locations)
- Reduced API calls (query fewer polygons, stop early)
- Better performance (fewer deals to process)
- Configurable distance thresholds
- Transparent (distance info available in each deal)
Example Configuration:
export MAX_STREET_DEAL_DISTANCE_METERS=300 # Conservative
export MAX_NEIGHBORHOOD_DEAL_DISTANCE_METERS=500
Testing:
- All 326 tests pass
- Backward compatible (high default limits)
- Works with existing outlier filtering
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Problem:
- test_get_market_activity_metrics was failing with "list index out of range"
- _safe_calculate_metric only caught ValueError, not IndexError or other exceptions
- This caused the entire function to crash when metric calculations failed
Root Cause:
- Market metric functions (calculate_market_activity_score, get_market_liquidity,
analyze_investment_potential) can throw IndexError if insufficient data
- The helper function wasn't catching these exceptions
Solution:
- Changed _safe_calculate_metric to catch all exceptions (Exception instead of ValueError)
- Added logging to track which metric failed
- Now returns error dict gracefully instead of crashing
Impact:
- get_market_activity_metrics is now more robust
- Returns partial results even if some metrics fail
- All 326 tests now pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Problem:
- get_valuation_comparables calculated statistics with outlier filtering
- But returned a deals list that still contained outliers (e.g., ₪900K, ₪1.3M deals)
- This caused inconsistency between statistics and the actual deals shown
Root Cause:
- calculate_deal_statistics() filters outliers internally for statistics
- But the deals array returned to user was only filtered by criteria (rooms, price, etc.)
- Outlier filtering was not applied to the returned deals list
Solution:
- Added explicit call to filter_deals_for_analysis() after filter_deals_by_criteria()
- Now both statistics AND deals list use outlier-filtered data
- Moved imports to top of file for better practice
Changes:
- nadlan_mcp/fastmcp_server.py:
- Added imports: get_config, filter_deals_for_analysis
- In get_valuation_comparables(): Apply outlier filtering before calculating stats
- Ensures deals list and statistics are consistent
Testing:
- 325/326 tests pass
- 1 unrelated test failure in get_market_activity_metrics (pre-existing issue)
After deployment, outliers will be automatically filtered from valuation comparables.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implement configurable outlier detection and robust statistical measures to
improve analysis accuracy for real estate data. Addresses issues with data
entry errors, partial deals, and other anomalies that skew statistics.
Key Features:
- IQR-based outlier detection (moderate filtering by default, k=1.5)
- Hard bounds filtering for obvious errors (price_per_sqm, deal_amount)
- Robust volatility using IQR instead of std_dev for investment analysis
- Transparent reporting with both filtered and unfiltered statistics
Implementation:
- Add outlier_detection.py module with IQR/percent/hard bounds methods
- Add OutlierReport model and enhance DealStatistics with filtered fields
- Update calculate_deal_statistics() to support optional outlier filtering
- Update analyze_investment_potential() to use robust volatility
- Add 9 new configuration parameters for customization
- Add comprehensive test suite (24 tests) for outlier detection
- Update CLAUDE.md with usage documentation
Configuration (all via env vars):
- ANALYSIS_OUTLIER_METHOD=iqr (default, or percent/none)
- ANALYSIS_IQR_MULTIPLIER=1.5 (moderate, 3.0=conservative)
- ANALYSIS_PRICE_PER_SQM_MIN/MAX=1000/100000 (bounds in NIS/sqm)
- ANALYSIS_MIN_DEAL_AMOUNT=100000 (catches partial deals)
- ANALYSIS_USE_ROBUST_VOLATILITY=true (IQR-based CV)
- ANALYSIS_USE_ROBUST_TRENDS=true (filter before regression)
Testing:
- All existing tests pass (326 passed)
- 24 new comprehensive outlier detection tests
- Real-world scenario tests (partial deals, data errors)
Backward Compatible:
- Default behavior improves accuracy without breaking changes
- All new fields in models are optional
- Config parameters have sensible defaults
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The Deal model's 'rooms' field was missing the 'assetRoomNum' alias,
causing room data from the API to not be loaded into the model.
This resulted in all room-based filters returning 0 results.
Bug discovered when querying "חנקין 62 חולון" for 3-room apartments.
Query returned 0 results despite multiple 3-room deals existing in the data.
Root cause: Pydantic requires field aliases to map API field names (camelCase)
to Python attributes (snake_case). The 'rooms' field had no alias.
Fix: Added alias="assetRoomNum" to rooms field in Deal model.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Enables deployment to cloud platforms (Render, Railway, etc.) while maintaining
backward compatibility with existing stdio transport for Claude Desktop.
New features:
- HTTP server entry point (run_http_server.py) using uvicorn
- Docker containerization with Python 3.13
- Health check endpoint at /health
- Comprehensive deployment documentation for Render, Railway, and Docker
Technical changes:
- Added uvicorn dependency for ASGI server
- Created Dockerfile with optimized multi-stage build (343MB)
- Added .dockerignore for efficient Docker builds
- Implemented /health endpoint using Starlette JSONResponse
- Updated README.md and DEPLOYMENT.md with HTTP deployment guides
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Autocomplete returns different coordinates for vague queries (e.g. "תל אביב דיזנגוף"),
sometimes landing on points with no nearby polygon_ids within 30m radius.
- Try top 3 autocomplete results until one has polygons
- Fallback radius expansion (30m→200m) if no polygons found
- Fix limit validation bug (min 1 for deal queries)
Fixes: compare_addresses, get_valuation_comparables, get_deal_statistics
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
## Root Cause
After Pydantic migration, get_deals_by_radius() was attempting to validate
API responses as Deal objects. However, this endpoint returns POLYGON METADATA
(with fields: dealscount, polygon_id, settlementNameHeb), NOT individual deals.
All responses failed Pydantic validation (missing dealAmount, dealDate),
resulting in empty lists and 0 deals returned from ALL queries.
## Fixes Applied
### 1. client.py - get_deals_by_radius()
- Return type: `List[Deal]` → `List[Dict[str, Any]]`
- Remove Pydantic validation - return raw metadata dicts
- Update docstring to clarify this returns polygon metadata
- Add note to use find_recent_deals_for_address() for actual deals
### 2. client.py - find_recent_deals_for_address()
- Update to handle polygon metadata dicts (not Deal objects)
- Use dict.get('polygon_id') instead of model attribute access
- Rename variable: `nearby_deals` → `nearby_polygons` for clarity
### 3. fastmcp_server.py - get_deals_by_radius() tool
- Update to handle dict responses (not Deal objects)
- Remove strip_bloat_fields() call (not needed for metadata)
- Update docstring with WARNING about polygon metadata
- Change response keys: "deals" → "polygons", "total_deals" → "total_polygons"
### 4. Tests
- test_govmap_client.py: Update to expect dicts, not Deal objects
- test_fastmcp_tools.py: Update 3 tests to mock dict responses
## Impact
- ✅ find_recent_deals_for_address() NOW WORKS (was returning 0 deals)
- ✅ All 174 tests passing
- ✅ E2E API test confirmed working with real data
## API Behavior Documented
get_deals_by_radius endpoint design:
1. Returns polygon/area metadata (not individual deals)
2. Extract polygon_ids from metadata
3. Call get_street_deals(polygon_id) to get actual deals
4. This workflow is automated in find_recent_deals_for_address()
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fixed all remaining test failures after Pydantic v2 migration:
Core fixes:
- Date handling: Convert date objects to ISO strings across 4 files
- Model serialization: Use model_dump(mode='json') for JSON compatibility
- Optional fields: Made time_period_months Optional[int] in models
- Dict access: Replace .get() with getattr() for dynamic attributes
Test updates:
- Updated 50+ test fixtures from dicts to Deal models
- Fixed date-based tests to use recent dates for time filtering
- Added missing imports (CoordinatePoint, MarketActivityScore, DealStatistics)
- Updated assertions from dict keys to model attributes (snake_case)
Files modified:
- nadlan_mcp/govmap/market_analysis.py
- nadlan_mcp/govmap/statistics.py
- nadlan_mcp/govmap/models.py
- nadlan_mcp/fastmcp_server.py
- tests/test_govmap_client.py
- tests/test_fastmcp_tools.py
- .cursor/plans/TEST-UPDATE-STATUS.md (comprehensive documentation)
Result: 174/174 tests passing (100%) ✅🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Bug Fix:
- Fixed autocomplete_address tool to use correct API response fields
- Changed from non-existent fields (addressLabel, settlementNameHeb, coordinates)
to actual API fields (text, id, type, score, shape)
- Added coordinate parsing from WKT POINT format
- Impact: HIGH - tool was returning empty data for all fields
Test Coverage Report:
- Added comprehensive test coverage analysis
- Documented 34 passing tests
- Identified e2e test results for MCP tools
- Listed missing test cases and recommendations
- Overall assessment: Good coverage, one bug found and fixed
E2E Test Results:
- ✅ find_recent_deals_for_address - WORKING
- ✅ analyze_market_trends - WORKING
- ✅ get_valuation_comparables - WORKING
- ✅ autocomplete_address - FIXED (needs MCP server restart to take effect)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
All functionality has been extracted to the govmap package:
- validators.py - Input validation functions
- utils.py - Utility functions
- filters.py - Deal filtering logic
- statistics.py - Statistical calculations
- market_analysis.py - Market analysis functions
- client.py - GovmapClient class with API methods
- __init__.py - Package exports
Backward compatibility maintained through package __init__.py exports.
Existing imports like `from nadlan_mcp.govmap import GovmapClient` now
use the govmap package instead of the old monolithic file.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Problem:
- get_valuation_comparables failed with 61,565 tokens (exceeded 25K MCP limit)
- Large MULTIPOLYGON shape data consumed ~40-50% of response tokens
- Not useful for LLM analysis, only bloating responses
Solution:
1. Added strip_bloat_fields() helper to remove:
- shape: Large coordinate data
- sourceorder: Internal ordering field
- source_polygon_id: Internal reference field
2. Applied to 5 MCP tools returning deal data:
- get_deals_by_radius
- get_street_deals
- find_recent_deals_for_address
- get_neighborhood_deals
- get_valuation_comparables
3. Reduced get_valuation_comparables default max_comparables: 200 → 50
Results:
- Token usage reduced by ~87% (61K → ~7-8K tokens)
- All tools now work within MCP token limits
- Cleaner, more efficient responses
- No loss of useful data for LLM analysis
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fixed two critical bugs in filter_deals_by_criteria():
Bug #1: CRASH on None property type
- Issue: Code called .lower() on None when propertyTypeDescription was null
- Impact: Would crash MCP tool in production with incomplete data
- Fix: Check if deal_type is None/empty before calling .lower()
Bug #2: Missing data passes through filters
- Issue: Deals with None/missing values passed filters when they shouldn't
- Example: Filtering by area=60-70 would include deals with assetArea=None
- Impact: get_valuation_comparables returned inflated results with bad data
- Fix: Explicitly check for None when filter is active and exclude those deals
Changes:
- Property type filter: Check for None/empty before normalization
- Area filter: Exclude deals with missing area when min/max_area specified
- Room filter: Exclude deals with missing rooms when min/max_rooms specified
- Price filter: Exclude deals with missing price when min/max_price specified
- Invalid data: Changed from 'pass' to 'continue' to exclude bad data
Added 6 comprehensive unit tests:
- test_filter_excludes_missing_property_type
- test_filter_excludes_missing_area
- test_filter_excludes_missing_rooms
- test_filter_excludes_missing_price
- test_filter_excludes_invalid_numeric_data
- test_filter_allows_missing_data_when_no_filter
This fixes the E2E issue where get_valuation_comparables was returning
incomplete results. Now filters properly exclude deals with missing data.
Test results: 34/34 passing (6 new tests added)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fixes three critical bugs discovered during live testing with Claude:
Bug #1: Same building detection always returned 0
- Root cause: deal.get('address') returned empty string
- Fix: Construct address from streetNameHeb + houseNum fields
- Added test: test_same_building_detection_with_api_fields
Bug #2: Property type filter too strict
- Root cause: Exact match failed for variants like 'דירת גג'
- Fix: Use flexible substring matching with normalization
Bug #3: Deal deduplication used non-existent field
- Root cause: Deduplication key referenced deal.get('address')
- Fix: Removed non-existent field from deduplication key
Also added _calculate_distance() helper for future radius filtering.
Test results: 26/28 passing (2 pre-existing failures unrelated)
References: .cursor/plans/MCP-E2E-TEST-SUMMARY.md
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Refactor repetitive try-except blocks using _safe_calculate_metric helper
function to reduce code duplication and improve maintainability.
Changes:
- Add _safe_calculate_metric helper to centralize error handling
- Remove misleading default values (0/"unknown") in summary fields
- Use None defaults instead to clearly indicate unavailable data
- Add final newline to file per convention
This prevents LLMs from misinterpreting 0 as "zero investment potential"
when it actually means "data unavailable".
Addresses PR #2 review comments on lines 733, 749, and 759.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Extract duplicated date parsing logic into a reusable _parse_deal_dates
helper method. This centralizes error handling, validation, and date
grouping logic across multiple market analysis functions.
Key improvements:
- DRY: Eliminates code duplication across three functions
- Implements time_period_months filtering (fixes unused parameter bug)
- Returns both monthly and quarterly breakdowns in one pass
- Consistent error handling and logging
- Better maintainability
This change fixes the unused time_period_months parameters in both
calculate_market_activity_score and get_market_liquidity functions.
Addresses PR #2 review comments on lines 1013, 1058, and 1257.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Define module-level constants for market activity, volatility, and liquidity
thresholds. This improves code readability and makes threshold values easier
to maintain and adjust in the future.
Constants added:
- ACTIVITY_VERY_HIGH_THRESHOLD, ACTIVITY_HIGH_THRESHOLD, etc.
- VOLATILITY_VERY_VOLATILE_THRESHOLD, VOLATILITY_VOLATILE_THRESHOLD, etc.
- LIQUIDITY_VERY_HIGH_THRESHOLD, LIQUIDITY_HIGH_THRESHOLD, etc.
Addresses PR #2 review comments on lines 1084 and 1224.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Nitzan Pomerantz