diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..fb268f7 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,490 @@ +# Nadlan-MCP Architecture + +## Overview + +Nadlan-MCP is a Model Context Protocol (MCP) server that provides Israeli real estate data to AI agents (LLMs). The architecture follows a clean separation of concerns with three main layers: + +1. **MCP Tools Layer** - Exposes functions to LLM clients +2. **Business Logic Layer** - Data analysis and processing +3. **API Client Layer** - Communicates with Govmap API + +## Design Principles + +### 1. MCP Provides Data, LLM Provides Intelligence + +The server's role is to retrieve, structure, and optionally summarize data. Complex analysis, decision-making, and predictions are left to the LLM client. + +**Example:** +- ✅ MCP: Provides comparable property deals with filters +- ✅ LLM: Analyzes comparables and estimates property value +- ❌ MCP: Calculates ML-based property valuation + +### 2. Structured by Default + +All tools return detailed, structured data by default. Optional `summarized_response` parameter provides condensed summaries when needed. + +### 3. Reliability First + +- Automatic retry with exponential backoff +- Rate limiting to respect API limits +- Comprehensive input validation +- Detailed error messages + +### 4. Configuration Over Hard-coding + +All settings (timeouts, retries, rate limits) are configurable via environment variables or code. + +## System Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ LLM Client (AI Agent) │ +│ (Claude, GPT, etc.) │ +└────────────────────────────┬────────────────────────────────┘ + │ MCP Protocol + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ MCP Tools Layer │ +│ ┌───────────────┐ ┌─────────────────┐ ┌──────────────┐ │ +│ │autocomplete │ │find_recent_deals│ │analyze_market│ │ +│ │_address │ │_for_address │ │_trends │ │ +│ └───────┬───────┘ └────────┬────────┘ └──────┬───────┘ │ +│ │ │ │ │ +│ ┌───────┴───────────────────┴────────────────────┴──────┐ │ +│ │ fastmcp_server.py (Tool Definitions) │ │ +│ └───────────────────────────┬───────────────────────────┘ │ +└────────────────────────────┬─┴────────────────────────────┘ + │ + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ Business Logic Layer │ +│ ┌────────────────────┐ ┌──────────────────────────────┐ │ +│ │ Market Analysis │ │ Deal Processing & Filtering │ │ +│ │ (analyze_market │ │ (_is_same_building, etc.) │ │ +│ │ _trends logic) │ │ │ │ +│ └────────────────────┘ └──────────────────────────────┘ │ +│ ┌────────────────────────────────────────────────────────┐ │ +│ │ govmap.py (GovmapClient) │ │ +│ │ • Validation • Retry Logic • Rate Limiting │ │ +│ └─────────────────────────┬──────────────────────────────┘ │ +└───────────────────────────┬┴──────────────────────────────┘ + │ HTTP/JSON + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ Govmap API Layer │ +│ (Israeli Government Real Estate API) │ +│ ┌───────────────┐ ┌────────────┐ ┌──────────────────┐ │ +│ │ Autocomplete │ │ Deals by │ │ Street/ │ │ +│ │ Endpoint │ │ Radius │ │ Neighborhood │ │ +│ │ │ │ Endpoint │ │ Deals Endpoints │ │ +│ └───────────────┘ └────────────┘ └──────────────────┘ │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Component Details + +### Configuration Layer (`config.py`) + +**Purpose:** Centralized configuration management + +**Key Components:** +- `GovmapConfig` - Dataclass with all configuration settings +- Environment variable support for all settings +- Validation on initialization + +**Configuration Options:** +- API settings (base URL, user agent) +- Timeout settings (connect, read) +- Retry settings (max retries, backoff) +- Rate limiting (requests per second) +- Default search parameters + +**Usage:** +```python +from nadlan_mcp.config import get_config, set_config, GovmapConfig + +# Use global config +config = get_config() + +# Override config +custom_config = GovmapConfig(max_retries=5, requests_per_second=10.0) +set_config(custom_config) +``` + +### API Client Layer (`govmap.py`) + +**Purpose:** Communicate with Govmap API with reliability features + +**Key Components:** + +#### GovmapClient Class + +**Responsibilities:** +- Make HTTP requests to Govmap API +- Implement retry logic with exponential backoff +- Enforce rate limiting +- Validate inputs +- Parse and validate responses + +**Key Methods:** +- `autocomplete_address()` - Search for addresses +- `get_gush_helka()` - Get block/parcel data +- `get_deals_by_radius()` - Get nearby deals +- `get_street_deals()` - Get street-level deals +- `get_neighborhood_deals()` - Get neighborhood deals +- `find_recent_deals_for_address()` - Main comprehensive search + +**Reliability Features:** +1. **Retry Logic** - Exponential backoff on failures +2. **Rate Limiting** - Tracks request times, enforces delays +3. **Input Validation** - Validates all inputs before API calls +4. **Timeouts** - Configurable connect and read timeouts + +### MCP Tools Layer (`fastmcp_server.py`) + +**Purpose:** Expose functionality to LLM clients via MCP protocol + +**Key Components:** +- FastMCP server instance +- Tool definitions (decorated functions) +- JSON response formatting +- Error handling for user-friendly messages + +**Tool Design Pattern:** +```python +@mcp.tool() +def tool_name(param: type, summarized_response: bool = False) -> str: + """ + Tool description for LLM. + + Args: + param: Parameter description + summarized_response: + - False (default): Full structured data for LLM processing + - True: Condensed summary with key insights + + Returns: + JSON string with data or summary + """ + # 1. Call business logic / API client + # 2. Format response + # 3. Return JSON string +``` + +## Data Flow + +### Example: Find Recent Deals for Address + +``` +1. LLM Request + └─> "Find deals for Sokolov 38 Holon in last 2 years" + +2. MCP Tool: find_recent_deals_for_address() + ├─> Validate inputs + └─> Call GovmapClient + +3. GovmapClient Workflow: + ├─> autocomplete_address("סוקולוב 38 חולון") + │ ├─> Rate limit check + │ ├─> HTTP POST (with retry) + │ └─> Parse coordinates + │ + ├─> get_deals_by_radius(coordinates, 30m) + │ ├─> Rate limit check + │ ├─> HTTP GET (with retry) + │ └─> Extract polygon_ids + │ + ├─> For each polygon_id: + │ ├─> get_street_deals(polygon_id) + │ └─> get_neighborhood_deals(polygon_id) + │ + ├─> Filter & Prioritize: + │ ├─> Same building deals (priority 0) + │ ├─> Street deals (priority 1) + │ └─> Neighborhood deals (priority 2) + │ + └─> Sort by priority then date + +4. Format Response + └─> JSON with deals + metadata + +5. Return to LLM + └─> LLM analyzes and responds to user +``` + +## Error Handling Strategy + +### Validation Errors (ValueError) +- Invalid address format +- Negative/zero parameters +- Out-of-range values +- **Action:** Immediate failure with clear message + +### Network Errors (requests.RequestException) +- Connection failures +- Timeouts +- HTTP errors +- **Action:** Retry with exponential backoff (up to configured max_retries) + +### API Response Errors +- Invalid JSON +- Unexpected format +- Missing required fields +- **Action:** Raise ValueError with specific details + +### Rate Limiting +- Track last request time +- Sleep if necessary before request +- **Action:** Transparent to caller, automatic + +## Performance Considerations + +### Current Implementation + +**Optimizations:** +- Connection pooling (requests.Session) +- Rate limiting prevents API throttling +- Early validation reduces unnecessary API calls +- Polygon deduplication in find_recent_deals_for_address() + +**Limitations:** +- Synchronous (blocking) API calls +- No caching (MVP decision) +- Sequential polygon queries + +### Future Optimizations (Not in MVP) + +**Phase 1: In-Memory Caching** +- Cache autocomplete results (1 hour TTL) +- Cache deal results (30 minute TTL) +- Reduce API load by 60-80% + +**Phase 2: Async/Parallel** +- Convert to async/await +- Parallel polygon queries +- 3-5x faster for multi-polygon searches + +**Phase 3: Production Caching** +- Redis for distributed caching +- Cache warming strategies +- Cross-instance consistency + +## Security & Rate Limiting + +### Rate Limiting + +**Current:** 5 requests/second (configurable) + +**Rationale:** +- Respects Govmap API (no published limits, being conservative) +- Prevents accidental DoS +- Balances responsiveness with safety + +**Implementation:** +- Tracks last request timestamp +- Sleeps if interval too short +- Per-instance (not distributed) + +### Input Validation + +All user inputs are validated before use: +- Address strings: length, type, non-empty +- Coordinates: numeric, reasonable bounds +- Integers: positive, max values +- Deal types: enum validation (1 or 2) + +**Prevents:** +- Injection attacks +- API errors from malformed requests +- Expensive/dangerous operations + +### API Key Management + +**Current:** No API keys required (public API) + +**Future:** If API keys added: +- Store in environment variables only +- Never log or expose in responses +- Rotate regularly + +## Testing Strategy + +### Unit Tests +- Mock API responses +- Test validation logic +- Test error handling +- Test retry logic + +### Integration Tests +- Real API calls (marked `@pytest.mark.integration`) +- VCR.py for recording/replaying +- Test complete workflows + +### Validation Tests +- Invalid inputs +- Edge cases +- Boundary conditions + +## Configuration Options + +### Environment Variables + +```bash +# API Settings +GOVMAP_BASE_URL=https://www.govmap.gov.il/api/ +GOVMAP_USER_AGENT=NadlanMCP/1.0.0 + +# Timeouts (seconds) +GOVMAP_CONNECT_TIMEOUT=10 +GOVMAP_READ_TIMEOUT=30 + +# Retry Settings +GOVMAP_MAX_RETRIES=3 +GOVMAP_RETRY_MIN_WAIT=1 +GOVMAP_RETRY_MAX_WAIT=10 + +# Rate Limiting +GOVMAP_REQUESTS_PER_SECOND=5.0 + +# Defaults +GOVMAP_DEFAULT_RADIUS=50 +GOVMAP_DEFAULT_YEARS_BACK=2 +GOVMAP_DEFAULT_DEAL_LIMIT=100 +``` + +### Tuning Guidelines + +**High Reliability (Production):** +- `MAX_RETRIES=5` +- `RETRY_MAX_WAIT=30` +- `REQUESTS_PER_SECOND=3.0` + +**High Performance (Development):** +- `MAX_RETRIES=2` +- `RETRY_MAX_WAIT=5` +- `REQUESTS_PER_SECOND=10.0` + +**Conservative (Shared API):** +- `MAX_RETRIES=3` +- `RETRY_MAX_WAIT=10` +- `REQUESTS_PER_SECOND=2.0` + +## Future Architecture Evolution + +### Phase 2: Data Models (Pydantic) + +Add structured models for type safety: +```python +class Deal(BaseModel): + deal_id: str + address: str + deal_amount: float + asset_area: float + price_per_sqm: float + deal_date: datetime + property_type: str + # ... +``` + +### Phase 3: Separation of Concerns + +``` +nadlan_mcp/ +├── api_client.py # Pure API communication +├── analyzers/ +│ ├── market.py # Market analysis logic +│ ├── valuation.py # Valuation helpers +│ └── filtering.py # Deal filtering +├── models.py # Pydantic models +└── fastmcp_server.py # Thin MCP tool definitions +``` + +### Phase 4: Database Layer (Optional) + +For historical tracking and faster queries: +``` +├── database/ +│ ├── models.py # SQLAlchemy models +│ ├── crud.py # CRUD operations +│ └── migrations/ # Alembic migrations +``` + +## Deployment + +### Development +```bash +python run_fastmcp_server.py +``` + +### Production (MCP Client) +```json +{ + "servers": { + "nadlan-mcp": { + "command": "python", + "args": ["/path/to/nadlan-mcp/run_fastmcp_server.py"], + "env": { + "GOVMAP_REQUESTS_PER_SECOND": "3.0", + "GOVMAP_MAX_RETRIES": "5" + } + } + } +} +``` + +## API Limitations + +### Govmap API Constraints + +**Known Limitations:** +- No published rate limits (being conservative with 5 req/sec) +- No authentication required (public API) +- Data freshness: Updated periodically (not real-time) +- Coverage: Official Israeli government records only + +**Best Practices:** +- Use appropriate time ranges (2-5 years recommended) +- Limit radius searches (under 1000m for performance) +- Cache results when possible (future feature) +- Respect rate limiting + +### MCP Protocol Constraints + +**Token Limits:** +- Large deal lists can exceed LLM context windows +- Use `summarized_response=True` for large datasets +- Default limits (50-100 deals) are token-optimized + +## Monitoring & Debugging + +### Logging Levels + +**INFO:** Normal operations +- Requests being made +- Successful responses + +**WARNING:** Recoverable issues +- Retrying after failures +- Rate limiting delays +- Validation warnings + +**ERROR:** Failures +- Exhausted retries +- Invalid responses +- Configuration errors + +### Debug Mode + +Enable debug logging: +```python +import logging +logging.basicConfig(level=logging.DEBUG) +``` + +## References + +- [Govmap API](https://www.govmap.gov.il/) +- [MCP Protocol](https://modelcontextprotocol.io/) +- [FastMCP Documentation](https://github.com/jlowin/fastmcp) +- [Israeli Real Estate Data](https://data.gov.il/) +