diff --git a/.cursor/plans/PLAN.md b/.cursor/plans/PLAN.md new file mode 100644 index 0000000..bd29cf4 --- /dev/null +++ b/.cursor/plans/PLAN.md @@ -0,0 +1,472 @@ + +# Nadlan MCP Comprehensive Improvement Plan + +## Phase 1: Critical Code Quality & Reliability Fixes + +### 1.1 Error Handling & Resilience + +**Files: `nadlan_mcp/govmap.py`, `nadlan_mcp/fastmcp_server.py`** + +- Add retry logic with exponential backoff for API calls (use `tenacity` library) +- Implement rate limiting protection (track request counts, add delays) +- Standardize error handling: all GovmapClient methods should raise exceptions (not return empty lists) +- Add proper input validation/sanitization for all functions +- Add timeout configuration for HTTP requests + +**Issues identified:** + +- Line 111-140 in `govmap.py`: `get_deals_by_radius` catches JSONDecodeError and returns `[]` instead of raising +- Line 180-185 in `govmap.py`: `get_street_deals` catches JSONDecodeError and returns `[]` instead of raising +- Similar pattern in `get_neighborhood_deals` +- No retry logic anywhere +- No rate limiting protection + +### 1.2 Configuration Management + +**New file: `nadlan_mcp/config.py`** + +- Create configuration class with all settings (API base URL, timeouts, retry counts, rate limits) +- Support environment variables for configuration +- Add validation for configuration values +- Document all configuration options + +**Currently hard-coded values:** + +- API base URL in `govmap.py:19` +- Default timeouts, radii, limits scattered throughout +- No environment variable support beyond basic setup + +### 1.3 Documentation Alignment + +**Files: `USECASES.md`, `README.md`** + +- Update USECASES.md to clearly mark amenity scoring as "Future Feature" with roadmap +- Add architecture diagram showing MCP → Govmap API flow +- Create ARCHITECTURE.md documenting design decisions +- Update README.md with current accurate feature list +- Add API limitations and rate limit documentation + +**Misalignments found:** + +- USECASES.md lines 23-28, 51-56, 66-69 describe amenity features that don't exist +- README mentions features not fully implemented + +## Phase 2: Missing Core Functionality + +### 2.1 Property Valuation Data Provision + +**Updates to `nadlan_mcp/govmap.py`, Add tools to `fastmcp_server.py`** + +**Purpose:** Provide comprehensive data for LLM to perform its own valuation analysis + +**New Functions:** + +- `get_comparable_properties(address, filters)` - find similar properties with flexible filtering +- `calculate_simple_statistics(deals)` - basic math helper for price/sqm calculations (optional utility) + +**Tools for LLM:** + +- `get_valuation_comparables` - return detailed comparable deals filtered by criteria + - Filter by: property type, room count, area range, floor range, time period + - Include: full deal history, neighborhood data, price trends + - Let LLM analyze and estimate value +- `get_deal_statistics` - simple statistical aggregations (mean, median, percentiles) + - For when LLM needs quick calculations on large datasets + +**Implementation:** + +- Enhanced filtering on existing deal data +- Statistical helper functions (no ML, no predictions) +- Rich contextual data for LLM decision-making +- LLM does the valuation, MCP provides the data + +### 2.2 Market Activity & Investment Analysis + +**New file: `nadlan_mcp/market_analysis.py`, Add tools to `fastmcp_server.py`** + +**New Functions:** + +- `calculate_market_activity_score(address, time_period)` - volume and velocity metrics +- `analyze_investment_potential(address, criteria)` - ROI, appreciation, risk metrics +- `get_market_liquidity(neighborhood)` - time-to-sell, supply/demand + +**Tools for LLM:** + +- `get_market_activity_metrics` - deal volume, velocity, inventory levels +- `analyze_investment_opportunity` - comprehensive investment analysis +- `get_neighborhood_market_health` - market health indicators + +**Metrics to include:** + +- Deal volume trends (deals per month) +- Average time on market (if available) +- Price appreciation rates +- Supply/demand indicators (active listings vs deals) +- Neighborhood investment score (0-100) + +### 2.3 Enhanced Deal Filtering & Search + +**Updates to `nadlan_mcp/govmap.py`** + +**New Functions:** + +- `filter_deals_by_criteria(deals, criteria)` - filter by rooms, area, price range, property type +- `search_deals_with_filters(address, filters, years_back)` - combined search with filters +- `get_deals_by_property_type(address, property_type, years_back)` - specialized property type search + +**Add to existing functions:** + +- Property type filtering (apartment, house, penthouse, etc.) +- Room count filtering (2-room, 3-room, etc.) +- Price range filtering +- Area range filtering +- Floor range filtering + +## Phase 3: Architecture & Data Model Improvements + +### 3.1 Create Structured Data Models + +**New file: `nadlan_mcp/models.py`** + +Create Pydantic models for: + +- `Deal` - standardized deal object +- `Address` - address with coordinates and metadata +- `MarketAnalysis` - market trend analysis results +- `PropertyValuation` - valuation estimates + +**Benefits:** + +- Type safety and validation +- Consistent JSON serialization +- Better IDE support +- Easier testing + +### 3.2 Separate Concerns + +**Refactor existing code:** + +- Move analysis logic from `fastmcp_server.py` into dedicated modules +- Create `nadlan_mcp/api_client.py` for pure API interactions +- Create `nadlan_mcp/analyzers/` package for analysis functions +- Keep `fastmcp_server.py` thin - just MCP tool definitions + +**Current issues:** + +- `fastmcp_server.py` has 537 lines mixing API calls, analysis, and MCP definitions +- Business logic scattered across client and server +- Hard to test and maintain + +### 3.3 Improve Tool Design for LLM Consumption + +**Update all tools in `fastmcp_server.py`** + +Each tool should offer: + +- **Structured mode** (default): Return full structured data for LLM processing +- **Summarized mode** (optional): Return analyzed/summarized data for simple queries +- Clear parameter documentation with examples +- Consistent error responses + +**Parameter pattern:** + +```python +@mcp.tool() +def find_recent_deals_for_address( + address: str, + years_back: int = 2, + summarized_response: bool = False # Default: structured/detailed +) -> str: + """ + summarized_response: + - False (default): Full deal data with all fields for LLM processing + - True: Statistical summary with key insights only + """ +``` + +## Phase 4: Testing & Quality Assurance + +### 4.1 Expand Test Coverage + +**Updates to `tests/test_govmap_client.py`, new test files** + +Add: + +- Integration tests for complete workflows (mark with `@pytest.mark.integration`) +- Edge case tests (empty results, malformed data, network errors) +- Parametrized tests for different address formats +- Tests for new functionality (valuation data, market analysis) + +**Current coverage gaps:** + +- No tests for `analyze_market_trends` +- No tests for `compare_addresses` +- No tests for error recovery +- No tests for `_is_same_building` logic + +### 4.2 Add Validation Tests + +**New file: `tests/test_validation.py`** + +- Test input validation (invalid addresses, negative numbers, etc.) +- Test configuration validation +- Test data model validation (when Pydantic models added) + +### 4.3 Mock External APIs + +**Updates to `tests/conftest.py`** + +- Create comprehensive fixtures for API responses +- Add vcr.py for recording/replaying API calls in tests + +## Phase 5: Documentation & Developer Experience + +### 5.1 Create Comprehensive Documentation + +**New files:** + +- `ARCHITECTURE.md` - system design, data flow, component interactions +- `DEPLOYMENT.md` - deployment guide for production use +- `CONTRIBUTING.md` - contribution guidelines +- `API_REFERENCE.md` - detailed API documentation with examples +- `CLAUDE.md` - instructions for AI coding agents implementing future tasks + - Task breakdown format + - Code standards and patterns + - Testing requirements + - Common pitfalls and solutions + +### 5.2 Improve Code Documentation + +**All Python files:** + +- Add module-level docstrings explaining purpose +- Add comprehensive function docstrings with examples +- Add type hints to all functions (including returns) +- Add inline comments for complex logic + +### 5.3 Add Usage Examples + +**New directory: `examples/`** + +Create example scripts: + +- `examples/basic_search.py` - simple address search +- `examples/market_analysis.py` - comprehensive market analysis +- `examples/investment_analysis.py` - investment opportunity analysis +- `examples/llm_integration.py` - example LLM agent using the MCP + +### 5.4 Update USECASES.md + +**File: `USECASES.md`** + +- Add status indicators for each feature (✅ Implemented, 🚧 In Progress, 📋 Planned) +- Add implementation priority levels +- Add expected completion timeline +- Add links to relevant documentation + +## Phase 6: Cleanup & Optimization + +### 6.1 Remove Redundant Code + +**Files to clean:** + +- Delete or integrate `mcp_server_concept.py` (appears to be duplicate/old code) +- Remove unused imports +- Remove dead code +- Consolidate duplicate logic + +### 6.2 Code Style & Linting + +**New file: `.pre-commit-config.yaml`** + +- Setup pre-commit hooks (black, isort, flake8, mypy) +- Format all code consistently +- Fix all linter warnings +- Add type checking with mypy + +### 6.3 Dependency Management + +**Updates to `requirements.txt`** + +Current dependencies are minimal. Add: + +- `tenacity` - for retry logic +- `pydantic` - for data models +- `python-dotenv` - already there but ensure proper use +- `httpx` - for async HTTP (future) +- Pin all versions with upper bounds for stability + +Split into: + +- `requirements.txt` - production dependencies +- `requirements-dev.txt` - development dependencies (pytest, black, mypy, etc.) + +## Phase 7: Future Enhancements (Backlog) + +### 7.1 Amenity Scoring System (Future Feature) + +**New files: `nadlan_mcp/amenities.py`, Add tools to `fastmcp_server.py`** + +Implement comprehensive amenity scoring using multiple data sources: + +**Data Sources:** + +- Google Places API / OpenStreetMap - basic amenity locations +- Ministry of Education - school rankings and quality metrics +- Ministry of Health - healthcare facility ratings +- CBS (Central Bureau of Statistics) - demographic data +- Public transport APIs - station locations and frequency + +**New Functions:** + +- `get_nearby_amenities(coordinates, radius, amenity_types)` - retrieve raw amenity data +- `get_school_quality_data(schools)` - fetch education quality metrics +- `calculate_amenity_score(address, weights)` - score with quality-weighted proximity +- `compare_amenity_scores(addresses)` - multi-address comparison + +**Tools for LLM:** + +- `get_address_amenity_rating` - comprehensive amenity analysis +- `compare_addresses_by_amenities` - side-by-side comparison +- `find_amenities_near_address` - raw amenity list with quality data + +**Implementation approach:** + +- Combine proximity with quality metrics (not just distance) +- Weight by category importance (configurable) +- Include static data sources for institutional quality +- Return both raw data and computed scores + +### 7.2 Caching System (Future - Not in MVP) + +**Task: In-Memory Caching** + +- Add TTL-based caching for API responses +- Cache autocomplete results (1 hour TTL) +- Cache deal results (30 minute TTL) +- Implement cache invalidation strategy +- Add cache statistics/monitoring + +**Task: Production-Ready Caching** + +- Redis integration for distributed caching +- Cache warming strategies +- Cache synchronization across instances +- Persistent cache storage +- Cache versioning for schema changes + +### 7.3 Performance Optimizations (Future) + +**Task: Async/Parallel Processing** + +- Convert to async/await for concurrent API calls +- Use `httpx` for async HTTP requests +- Parallel processing for multi-polygon queries +- Batch processing optimizations + +**Task: Database Integration** + +- SQLite for local development +- PostgreSQL for production +- Historical data tracking +- Faster query performance for trends +- Data warehouse for analytics + +### 7.4 Multi-language Support (Future) + +**Enhance language support:** + +- Full English address support (currently Hebrew-focused) +- Translation services for property descriptions +- Language detection and auto-switching +- Multi-language documentation + +### 7.5 Advanced Valuation Helper (Future) + +**Optional calculation helper for LLM:** + +**New Function:** + +- `calculate_valuation_from_comparables(comparables, weights, adjustments)` - mathematical valuation helper + - Takes LLM-selected comparable deals + - Applies LLM-specified weights and adjustments + - Returns calculated price estimate with breakdown + - Pure calculation (no intelligence), LLM provides the logic + +**Purpose:** + +- LLM selects comparables and determines methodology +- Function just does the math reliably +- Returns detailed calculation breakdown for transparency + +## Implementation Priority + +**High Priority (MVP):** + +1. Phase 1: Code Quality & Reliability (critical for stability) +2. Phase 2: Missing Core Functionality (align with use cases) +3. Phase 6.1: Cleanup redundant code + +**Medium Priority (Post-MVP):** + +4. Phase 3: Architecture improvements (foundation for scaling) +5. Phase 4: Testing expansion (ensure quality) +6. Phase 5: Documentation (user experience) + +**Low Priority (Future):** + +7. Phase 6.2-6.3: Polish & optimization +8. Phase 7: Future enhancements (amenities, caching, async, multi-language) + +## Success Criteria + +- ✅ All use cases in USECASES.md have corresponding implemented functions +- ✅ LLM can effectively use tools for all stated use cases +- ✅ Code is maintainable with <100 lines per function +- ✅ Test coverage >80% for core functionality +- ✅ All API calls have proper error handling and retries +- ✅ Documentation is comprehensive and up-to-date +- ✅ No duplicate or dead code +- ✅ Type hints on all public functions +- ✅ Configuration is externalized and documented + +## Design Principles + +- **MCP provides data, LLM provides intelligence**: MCP retrieves and structures data; LLM analyzes, decides, and estimates +- **Structured by default**: Default responses should be detailed/structured for LLM processing +- **Optional summarization**: Add `summarized_response` parameter where helpful +- **No predictions in MVP**: Statistical aggregations yes, ML predictions no +- **External APIs for future**: Amenity scoring with quality metrics is a future enhancement +- **Quality over distance**: When amenities are added, include quality metrics not just proximity + +## Notes + +- No load testing on external Govmap API (as requested) +- Focus on LLM-friendly tool design: comprehensive data provision for LLM analysis +- Amenity scoring with quality data (Ministry of Education, etc.) is future feature +- Property valuation done by LLM, not MCP - MCP just provides comparable deals data +- Caching is explicitly not in MVP scope +- Keep functions focused: one function = one clear purpose +- Always return structured data that LLMs can easily process +- Use `summarized_response: bool = False` for optional summarization + +### To-dos + +- [ ] Add retry logic, rate limiting, and standardize error handling in govmap.py and fastmcp_server.py +- [ ] Create config.py with externalized configuration and environment variable support +- [ ] Update USECASES.md and README.md to reflect actual current state and future roadmap +- [ ] Create tools for providing comparable deals and statistical data for LLM valuation +- [ ] Create market activity scoring and investment analysis functions +- [ ] Add comprehensive deal filtering by property type, rooms, price, area, floor +- [ ] Create Pydantic models for Deal, Address, MarketAnalysis, PropertyValuation +- [ ] Separate concerns: create api_client.py, analyzers package, thin fastmcp_server.py +- [ ] Update all tools with summarized_response boolean parameter and consistent responses +- [ ] Add integration tests, edge cases, parametrized tests for all functionality +- [ ] Create test_validation.py with input validation and data model tests +- [ ] Create ARCHITECTURE.md, DEPLOYMENT.md, CONTRIBUTING.md, API_REFERENCE.md, CLAUDE.md +- [ ] Add module docstrings, comprehensive function docs, type hints, inline comments +- [ ] Create examples/ directory with scripts showing all major use cases +- [ ] Add status indicators, priority levels, timeline to USECASES.md +- [ ] Delete/integrate mcp_server_concept.py, remove unused imports, consolidate duplicates +- [ ] Setup pre-commit hooks, format all code, fix linter warnings, add mypy +- [ ] Update requirements.txt with new dependencies, create requirements-dev.txt, pin versions \ No newline at end of file