mcp-servers/nadlan-mcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7780d89bb42062c6482147128435bded4d887594
nadlan-mcp/.cursor/plans/PLAN.md
T
Nitzan Pomerantz 7780d89bb4 Update tasks and plan
2025-10-19 01:02:36 +03:00

482 lines
16 KiB
Markdown
Raw Blame History

<!-- f18f82e2-05d6-4de1-bb9b-9ca39dca90fc 9249b19a-20d4-4473-be7a-f0b5bf9a0301 -->
# Nadlan MCP Comprehensive Improvement Plan
## Phase 1: Critical Code Quality & Reliability Fixes
### 1.1 Error Handling & Resilience
**Files: `nadlan_mcp/govmap.py`, `nadlan_mcp/fastmcp_server.py`**
- Add retry logic with exponential backoff for API calls (use `tenacity` library)
- Implement rate limiting protection (track request counts, add delays)
- Standardize error handling: all GovmapClient methods should raise exceptions (not return empty lists)
- Add proper input validation/sanitization for all functions
- Add timeout configuration for HTTP requests
**Issues identified:**
- Line 111-140 in `govmap.py`: `get_deals_by_radius` catches JSONDecodeError and returns `[]` instead of raising
- Line 180-185 in `govmap.py`: `get_street_deals` catches JSONDecodeError and returns `[]` instead of raising
- Similar pattern in `get_neighborhood_deals`
- No retry logic anywhere
- No rate limiting protection
### 1.2 Configuration Management
**New file: `nadlan_mcp/config.py`**
- Create configuration class with all settings (API base URL, timeouts, retry counts, rate limits)
- Support environment variables for configuration
- Add validation for configuration values
- Document all configuration options
**Currently hard-coded values:**
- API base URL in `govmap.py:19`
- Default timeouts, radii, limits scattered throughout
- No environment variable support beyond basic setup
### 1.3 Documentation Alignment
**Files: `USECASES.md`, `README.md`**
- Update USECASES.md to clearly mark amenity scoring as "Future Feature" with roadmap
- Add architecture diagram showing MCP → Govmap API flow
- Create ARCHITECTURE.md documenting design decisions
- Update README.md with current accurate feature list
- Add API limitations and rate limit documentation
**Misalignments found:**
- USECASES.md lines 23-28, 51-56, 66-69 describe amenity features that don't exist
- README mentions features not fully implemented
## Phase 2: Missing Core Functionality
### 2.1 Property Valuation Data Provision
**Updates to `nadlan_mcp/govmap.py`, Add tools to `fastmcp_server.py`**
**Purpose:** Provide comprehensive data for LLM to perform its own valuation analysis
**New Functions:**
- `get_comparable_properties(address, filters)` - find similar properties with flexible filtering
- `calculate_simple_statistics(deals)` - basic math helper for price/sqm calculations (optional utility)
**Tools for LLM:**
- `get_valuation_comparables` - return detailed comparable deals filtered by criteria
- Filter by: property type, room count, area range, floor range, time period
- Include: full deal history, neighborhood data, price trends
- Let LLM analyze and estimate value
- `get_deal_statistics` - simple statistical aggregations (mean, median, percentiles)
- For when LLM needs quick calculations on large datasets
**Implementation:**
- Enhanced filtering on existing deal data
- Statistical helper functions (no ML, no predictions)
- Rich contextual data for LLM decision-making
- LLM does the valuation, MCP provides the data
### 2.2 Market Activity & Investment Analysis
**New file: `nadlan_mcp/market_analysis.py`, Add tools to `fastmcp_server.py`**
**New Functions:**
- `calculate_market_activity_score(address, time_period)` - volume and velocity metrics
- `analyze_investment_potential(address, criteria)` - ROI, appreciation, risk metrics
- `get_market_liquidity(neighborhood)` - time-to-sell, supply/demand
**Tools for LLM:**
- `get_market_activity_metrics` - deal volume, velocity, inventory levels
- `analyze_investment_opportunity` - comprehensive investment analysis
- `get_neighborhood_market_health` - market health indicators
**Metrics to include:**
- Deal volume trends (deals per month)
- Average time on market (if available)
- Price appreciation rates
- Supply/demand indicators (active listings vs deals)
- Neighborhood investment score (0-100)
### 2.3 Enhanced Deal Filtering & Search
**Updates to `nadlan_mcp/govmap.py`**
**New Functions:**
- `filter_deals_by_criteria(deals, criteria)` - filter by rooms, area, price range, property type
- `search_deals_with_filters(address, filters, years_back)` - combined search with filters
- `get_deals_by_property_type(address, property_type, years_back)` - specialized property type search
**Add to existing functions:**
- Property type filtering (apartment, house, penthouse, etc.)
- Room count filtering (2-room, 3-room, etc.)
- Price range filtering
- Area range filtering
- Floor range filtering
## Phase 3: Architecture & Data Model Improvements
### 3.1 Create Structured Data Models
**New file: `nadlan_mcp/models.py`**
Create Pydantic models for:
- `Deal` - standardized deal object
- `Address` - address with coordinates and metadata
- `MarketAnalysis` - market trend analysis results
- `PropertyValuation` - valuation estimates
**Benefits:**
- Type safety and validation
- Consistent JSON serialization
- Better IDE support
- Easier testing
### 3.2 Separate Concerns
**Refactor existing code:**
- Move analysis logic from `fastmcp_server.py` into dedicated modules
- Create `nadlan_mcp/api_client.py` for pure API interactions
- Create `nadlan_mcp/analyzers/` package for analysis functions
- Keep `fastmcp_server.py` thin - just MCP tool definitions
**Current issues:**
- `fastmcp_server.py` has 537 lines mixing API calls, analysis, and MCP definitions
- Business logic scattered across client and server
- Hard to test and maintain
### 3.3 Improve Tool Design for LLM Consumption
**Update all tools in `fastmcp_server.py`**
Each tool should offer:
- **Structured mode** (default): Return full structured data for LLM processing
- **Summarized mode** (optional): Return analyzed/summarized data for simple queries
- Clear parameter documentation with examples
- Consistent error responses
**Parameter pattern:**
```python
@mcp.tool()
def find_recent_deals_for_address(
address: str,
years_back: int = 2,
summarized_response: bool = False # Default: structured/detailed
) -> str:
"""
summarized_response:
- False (default): Full deal data with all fields for LLM processing
- True: Statistical summary with key insights only
"""
```
## Phase 4: Testing & Quality Assurance
### 4.1 Expand Test Coverage
**Updates to `tests/test_govmap_client.py`, new test files**
Add:
- Integration tests for complete workflows (mark with `@pytest.mark.integration`)
- Edge case tests (empty results, malformed data, network errors)
- Parametrized tests for different address formats
- Tests for new functionality (valuation data, market analysis)
**Current coverage gaps:**
- No tests for `analyze_market_trends`
- No tests for `compare_addresses`
- No tests for error recovery
- No tests for `_is_same_building` logic
### 4.2 Add Validation Tests
**New file: `tests/test_validation.py`**
- Test input validation (invalid addresses, negative numbers, etc.)
- Test configuration validation
- Test data model validation (when Pydantic models added)
### 4.3 Mock External APIs
**Updates to `tests/conftest.py`**
- Create comprehensive fixtures for API responses
- Add vcr.py for recording/replaying API calls in tests
## Phase 5: Documentation & Developer Experience
### 5.1 Create Comprehensive Documentation
**New files:**
- `ARCHITECTURE.md` - system design, data flow, component interactions
- `DEPLOYMENT.md` - deployment guide for production use
- `CONTRIBUTING.md` - contribution guidelines
- `API_REFERENCE.md` - detailed API documentation with examples
- `CLAUDE.md` - instructions for AI coding agents implementing future tasks
- Task breakdown format
- Code standards and patterns
- Testing requirements
- Common pitfalls and solutions
### 5.2 Improve Code Documentation
**All Python files:**
- Add module-level docstrings explaining purpose
- Add comprehensive function docstrings with examples
- Add type hints to all functions (including returns)
- Add inline comments for complex logic
### 5.3 Add Usage Examples
**New directory: `examples/`**
Create example scripts:
- `examples/basic_search.py` - simple address search
- `examples/market_analysis.py` - comprehensive market analysis
- `examples/investment_analysis.py` - investment opportunity analysis
- `examples/llm_integration.py` - example LLM agent using the MCP
### 5.4 Update USECASES.md
**File: `USECASES.md`**
- Add status indicators for each feature (✅ Implemented, 🚧 In Progress, 📋 Planned)
- Add implementation priority levels
- Add expected completion timeline
- Add links to relevant documentation
## Phase 6: Cleanup & Optimization
### 6.1 Remove Redundant Code
**Files to clean:**
- Delete or integrate `mcp_server_concept.py` (appears to be duplicate/old code)
- Remove unused imports
- Remove dead code
- Consolidate duplicate logic
### 6.2 Code Style & Linting
**New file: `.pre-commit-config.yaml`**
- Setup pre-commit hooks (black, isort, flake8, mypy)
- Format all code consistently
- Fix all linter warnings
- Add type checking with mypy
### 6.3 Dependency Management
**Updates to `requirements.txt`**
Current dependencies are minimal. Add:
- `tenacity` - for retry logic
- `pydantic` - for data models
- `python-dotenv` - already there but ensure proper use
- `httpx` - for async HTTP (future)
- Pin all versions with upper bounds for stability
Split into:
- `requirements.txt` - production dependencies
- `requirements-dev.txt` - development dependencies (pytest, black, mypy, etc.)
## Phase 7: Future Enhancements (Backlog)
### 7.1 Amenity Scoring System (Future Feature)
**New files: `nadlan_mcp/amenities.py`, Add tools to `fastmcp_server.py`**
Implement comprehensive amenity scoring using multiple data sources:
**Data Sources:**
- Google Places API / OpenStreetMap - basic amenity locations
- Ministry of Education - school rankings and quality metrics
- Ministry of Health - healthcare facility ratings
- CBS (Central Bureau of Statistics) - demographic data
- Public transport APIs - station locations and frequency
**New Functions:**
- `get_nearby_amenities(coordinates, radius, amenity_types)` - retrieve raw amenity data
- `get_school_quality_data(schools)` - fetch education quality metrics
- `calculate_amenity_score(address, weights)` - score with quality-weighted proximity
- `compare_amenity_scores(addresses)` - multi-address comparison
**Tools for LLM:**
- `get_address_amenity_rating` - comprehensive amenity analysis
- `compare_addresses_by_amenities` - side-by-side comparison
- `find_amenities_near_address` - raw amenity list with quality data
**Implementation approach:**
- Combine proximity with quality metrics (not just distance)
- Weight by category importance (configurable)
- Include static data sources for institutional quality
- Return both raw data and computed scores
### 7.2 Caching System (Future - Not in MVP)
**Task: In-Memory Caching**
- Add TTL-based caching for API responses
- Cache autocomplete results (1 hour TTL)
- Cache deal results (30 minute TTL)
- Implement cache invalidation strategy
- Add cache statistics/monitoring
**Task: Production-Ready Caching**
- Redis integration for distributed caching
- Cache warming strategies
- Cache synchronization across instances
- Persistent cache storage
- Cache versioning for schema changes
### 7.3 Performance Optimizations (Future)
**Task: Async/Parallel Processing**
- Convert to async/await for concurrent API calls
- Use `httpx` for async HTTP requests
- Parallel processing for multi-polygon queries
- Batch processing optimizations
**Task: Database Integration**
- SQLite for local development
- PostgreSQL for production
- Historical data tracking
- Faster query performance for trends
- Data warehouse for analytics
### 7.4 Multi-language Support (Future)
**Enhance language support:**
- Full English address support (currently Hebrew-focused)
- Translation services for property descriptions
- Language detection and auto-switching
- Multi-language documentation
### 7.5 Advanced Valuation Helper (Future)
**Optional calculation helper for LLM:**
**New Function:**
- `calculate_valuation_from_comparables(comparables, weights, adjustments)` - mathematical valuation helper
- Takes LLM-selected comparable deals
- Applies LLM-specified weights and adjustments
- Returns calculated price estimate with breakdown
- Pure calculation (no intelligence), LLM provides the logic
**Purpose:**
- LLM selects comparables and determines methodology
- Function just does the math reliably
- Returns detailed calculation breakdown for transparency
## Implementation Priority
**High Priority (MVP):**
1. Phase 1: Code Quality & Reliability (critical for stability)
2. Phase 2: Missing Core Functionality (align with use cases)
3. Phase 6.1: Cleanup redundant code
**Medium Priority (Post-MVP):**
4. Phase 3: Architecture improvements (foundation for scaling)
5. Phase 4: Testing expansion (ensure quality)
6. Phase 5: Documentation (user experience)
**Low Priority (Future):**
7. Phase 6.2-6.3: Polish & optimization
8. Phase 7: Future enhancements (amenities, caching, async, multi-language)
## Success Criteria
- ✅ All use cases in USECASES.md have corresponding implemented functions
- ✅ LLM can effectively use tools for all stated use cases
- ✅ Code is maintainable with <100 lines per function
- ✅ Test coverage >80% for core functionality
- ✅ All API calls have proper error handling and retries
- ✅ Documentation is comprehensive and up-to-date
- ✅ No duplicate or dead code
- ✅ Type hints on all public functions
- ✅ Configuration is externalized and documented
## Design Principles
- **MCP provides data, LLM provides intelligence**: MCP retrieves and structures data; LLM analyzes, decides, and estimates
- **Structured by default**: Default responses should be detailed/structured for LLM processing
- **Optional summarization**: Add `summarized_response` parameter where helpful
- **No predictions in MVP**: Statistical aggregations yes, ML predictions no
- **External APIs for future**: Amenity scoring with quality metrics is a future enhancement
- **Quality over distance**: When amenities are added, include quality metrics not just proximity
## Notes
- No load testing on external Govmap API (as requested)
- Focus on LLM-friendly tool design: comprehensive data provision for LLM analysis
- Amenity scoring with quality data (Ministry of Education, etc.) is future feature
- Property valuation done by LLM, not MCP - MCP just provides comparable deals data
- Caching is explicitly not in MVP scope
- Keep functions focused: one function = one clear purpose
- Always return structured data that LLMs can easily process
- Use `summarized_response: bool = False` for optional summarization
## Implementation Tracking
**For detailed task tracking and current progress, see [TASKS.md](../../TASKS.md)**
### Completed (Phase 1)
- ✅ Configuration management system with environment variable support
- ✅ Retry logic with exponential backoff (manual implementation)
- ✅ Rate limiting protection
- ✅ Standardized error handling (raise exceptions)
- ✅ Comprehensive input validation
- ✅ Updated dependencies with version pinning
- ✅ Created requirements-dev.txt
- ✅ Updated USECASES.md with status indicators
- ✅ Created ARCHITECTURE.md
- ✅ Deleted redundant code files
### Completed (Phase 2.1)
- ✅ Implemented `filter_deals_by_criteria()` with comprehensive filtering
- ✅ Implemented `calculate_deal_statistics()` with statistical aggregations
- ✅ Added `get_valuation_comparables` MCP tool
- ✅ Added `get_deal_statistics` MCP tool
### In Progress
- 🚧 Phase 2.2: Market Activity & Investment Analysis
- 🚧 Phase 2.3: Enhanced Deal Filtering (partially complete)
### Next Up
- Phase 3: Architecture Improvements (data models, separation of concerns)
- Phase 4: Testing Expansion
- Phase 5: Complete Documentation
Reference in New Issue View Git Blame Copy Permalink