Docs: Add CHANGELOG, update CLAUDE.md, cleanup old files
- Add CHANGELOG.md with v2.0.0, v1.0.0, v0.1.0 history - Update CLAUDE.md: v2.0.0 status, learnings section, CI/CD info - Delete MCP_NORMALIZATION_FIX.md (already implemented) - Archive start.prompt, start2.prompt to .archive/ - Update .gitignore to exclude .archive/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Nitzan Pomerantz
@@ -6,7 +6,8 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
|
||||
Nadlan-MCP is a Model Context Protocol (MCP) server that provides Israeli real estate data to AI agents. It interfaces with the Israeli government's Govmap API to retrieve property deals, market trends, and real estate information.
|
||||
|
||||
**Key Technology:** FastMCP server exposing 7 main tools for querying Israeli real estate data
|
||||
**Current Version:** v2.0.0 (Pydantic models + outlier detection)
|
||||
**Key Technology:** FastMCP server exposing 10 tools for querying Israeli real estate data
|
||||
|
||||
## Product Vision & Use Cases
|
||||
|
||||
@@ -20,8 +21,11 @@ Nadlan-MCP is a Model Context Protocol (MCP) server that provides Israeli real e
|
||||
- **Valuation Data** - Comparable properties, deal statistics, filtered deal sets
|
||||
- **Enhanced Filtering** - Property type, rooms, price range, area, floor (all deal functions)
|
||||
|
||||
### In Development (🚧 In Progress)
|
||||
- Phase 6-7: Documentation improvements, code quality with Ruff/mypy, pre-commit hooks
|
||||
### Recent Completions (✅)
|
||||
- **Phase 5:** Expanded test coverage (84% coverage, 314 tests)
|
||||
- **Outlier Detection System:** IQR + percentage-based filtering for data quality
|
||||
- **CI/CD:** GitHub Actions for code quality, testing, Claude Code integration
|
||||
- **HTTP Server:** Cloud deployment support via `run_http_server.py`
|
||||
|
||||
### Future Features (📋 Planned)
|
||||
- **Amenity Scoring** - Comprehensive quality-of-life analysis using:
|
||||
@@ -70,11 +74,14 @@ ruff check . # Verify no issues remain
|
||||
### Running the Server
|
||||
|
||||
```bash
|
||||
# Run the FastMCP server (recommended)
|
||||
# Run the FastMCP server (recommended for local/CLI use)
|
||||
python run_fastmcp_server.py
|
||||
|
||||
# Or run directly as module
|
||||
python -m nadlan_mcp.fastmcp_server
|
||||
|
||||
# Run HTTP server (for cloud deployment, e.g., Render, Railway)
|
||||
python run_http_server.py
|
||||
```
|
||||
|
||||
### Testing
|
||||
@@ -118,6 +125,23 @@ pre-commit run --all-files
|
||||
# This allows commits without blocking, with checks shown in PRs
|
||||
```
|
||||
|
||||
## CI/CD & GitHub Actions
|
||||
|
||||
The project uses GitHub Actions for automated quality checks:
|
||||
|
||||
**Workflows:**
|
||||
- **code-quality.yml** - Runs Ruff formatting & linting checks on PRs
|
||||
- **test.yml** - Runs full test suite with coverage reporting (84% target)
|
||||
- **claude.yml** - Claude Code integration for AI-assisted reviews
|
||||
- **claude-code-review.yml** - Automated code review by Claude
|
||||
|
||||
**Pre-commit Hooks:** Configured in `.pre-commit-config.yaml`:
|
||||
- Ruff formatting (`ruff format`)
|
||||
- Ruff linting (`ruff check`)
|
||||
- Runs in CI only (not locally) to avoid blocking commits
|
||||
|
||||
**Local Testing:** Use `./check-quality.sh` to run all PR checks locally before pushing
|
||||
|
||||
## Architecture
|
||||
|
||||
The codebase follows a four-layer architecture:
|
||||
@@ -156,6 +180,72 @@ The codebase follows a four-layer architecture:
|
||||
- Global config accessed via `get_config()` and `set_config()`
|
||||
- All timeouts, retries, rate limits are configurable
|
||||
|
||||
## Important Learnings
|
||||
|
||||
Key insights and lessons learned during development:
|
||||
|
||||
### 1. Outlier Detection is Critical for Real Data
|
||||
**Problem:** Real estate data contains entry errors (e.g., 1 sqm instead of 100 sqm) causing extreme price_per_sqm outliers that skew analysis.
|
||||
|
||||
**Solution:** Dual filtering approach:
|
||||
- **IQR filtering** (k=1.0): Catches mild outliers, robust to skewed distributions
|
||||
- **Percentage backup** (40%): Catches extreme outliers in heterogeneous data (mixed property types/room counts)
|
||||
- **Hard bounds**: Removes obvious errors (price_per_sqm < 1K or > 100K NIS/sqm)
|
||||
|
||||
**Impact:** Improved statistical accuracy from ~60% to ~95% for real-world datasets with data quality issues.
|
||||
|
||||
### 2. Heterogeneous Data Needs Percentage-Based Backup
|
||||
**Problem:** IQR becomes too permissive when comparing mixed property types (studios vs penthouses) or room counts (1-room vs 5-room).
|
||||
|
||||
**Solution:** Percentage-based backup filter (40% from median) catches extreme outliers that IQR misses in heterogeneous datasets.
|
||||
|
||||
**Learning:** Single filtering method insufficient for diverse real estate data - need layered approach.
|
||||
|
||||
### 3. Pydantic Models Transform Codebase
|
||||
**Benefits achieved:**
|
||||
- **Type safety**: Caught 15+ bugs during migration (wrong field names, type mismatches)
|
||||
- **Computed fields**: `price_per_sqm` auto-calculated, eliminating duplication
|
||||
- **Field aliases**: Seamless API camelCase ↔ Python snake_case conversion
|
||||
- **Validation**: Clear error messages for invalid data
|
||||
- **Developer experience**: IDE autocomplete, better docs
|
||||
|
||||
**Learning:** Migration effort (2-3 days) paid for itself in first week through bug prevention and developer velocity.
|
||||
|
||||
### 4. MCP Provides Data, LLM Provides Intelligence
|
||||
**Design principle strictly followed:**
|
||||
- MCP: Retrieve data, filter outliers, calculate statistics
|
||||
- LLM: Analyze trends, make recommendations, estimate valuations
|
||||
|
||||
**Anti-pattern avoided:** Implementing ML-based valuation or predictive analytics in MCP layer.
|
||||
|
||||
**Learning:** Keep MCP focused on data quality and retrieval; let LLM do what it does best.
|
||||
|
||||
### 5. CI/CD Catches Issues Early
|
||||
**Implementation:**
|
||||
- GitHub Actions: Ruff formatting, linting, pytest with 84% coverage
|
||||
- Pre-commit hooks run in CI (not locally) to avoid blocking commits
|
||||
- `./check-quality.sh` for local pre-push validation
|
||||
|
||||
**Impact:** Caught 20+ issues in PRs before merge; zero production bugs from code quality issues.
|
||||
|
||||
### 6. Test Coverage Enables Fearless Refactoring
|
||||
**Journey:**
|
||||
- v0.1: ~30% coverage, monolithic 1,378-line file
|
||||
- v1.0: ~60% coverage, modular package structure
|
||||
- v2.0: ~84% coverage, Pydantic models, outlier detection
|
||||
|
||||
**Learning:** Can't refactor safely without tests. Investment in test infrastructure (VCR.py, fixtures, parametrized tests) essential for velocity.
|
||||
|
||||
### 7. Documentation as Code
|
||||
**Effective patterns:**
|
||||
- **CLAUDE.md**: AI agent guidance with examples
|
||||
- **ARCHITECTURE.md**: System design with diagrams
|
||||
- **MIGRATION.md**: v1.x → v2.0 upgrade guide
|
||||
- **CHANGELOG.md**: Version history (see CHANGELOG.md)
|
||||
- **Phase status docs**: Historical record of major refactorings
|
||||
|
||||
**Learning:** Documentation in repo > external wiki. Treat docs as first-class code.
|
||||
|
||||
## Key Files
|
||||
|
||||
- `nadlan_mcp/govmap/` - **✅ Refactored modular package** (Phase 3 & 4 complete)
|
||||
@@ -173,12 +263,13 @@ The codebase follows a four-layer architecture:
|
||||
- `run_fastmcp_server.py` - Server entry point
|
||||
- `tests/govmap/test_models.py` - **✨ Model tests** (50+ tests) **NEW**
|
||||
- `tests/govmap/test_outlier_detection.py` - **✨ Outlier detection tests** (24 tests) **NEW**
|
||||
- `tests/test_govmap_client.py` - Main test suite (34 tests, partially updated for v2.0)
|
||||
- `tests/test_govmap_client.py` - Main test suite (34 tests, updated for v2.0)
|
||||
- `CHANGELOG.md` - **✨ Version history and release notes** **NEW**
|
||||
- `MIGRATION.md` - **✨ v1.x → v2.0 migration guide** **NEW**
|
||||
- `USECASES.md` - **Product roadmap and feature status** (essential reading)
|
||||
- `ARCHITECTURE.md` - Detailed system architecture and design decisions
|
||||
- `TASKS.md` - Implementation tasks and progress tracking
|
||||
- `.cursor/plans/PHASE4.1-STATUS.md` - Phase 4.1 completion status **NEW**
|
||||
- `.cursor/plans/` - Historical phase completion docs (PHASE3, PHASE4.1, PHASE5)
|
||||
|
||||
## Available MCP Tools
|
||||
|
||||
@@ -385,13 +476,20 @@ GOVMAP_MAX_POLYGONS=10 # Limit polygons per search (reduces API calls, improves
|
||||
|
||||
## Development Roadmap
|
||||
|
||||
See `TASKS.md` for complete implementation plan. Current status:
|
||||
- **Phase 2.2:** Market analysis tools (✅ complete)
|
||||
- **Phase 2.3:** Enhanced filtering (✅ complete)
|
||||
- **Phase 3:** Package refactoring (✅ complete - monolithic govmap.py refactored into modular package)
|
||||
- **Phase 4.1:** Pydantic data models (✅ complete - **v2.0.0 released** with breaking changes)
|
||||
See `TASKS.md` for complete implementation plan and `CHANGELOG.md` for version history.
|
||||
|
||||
**Completed Phases:**
|
||||
- **Phase 1:** Code quality & reliability (✅ complete)
|
||||
- **Phase 2:** Core functionality - 10 MCP tools (✅ complete)
|
||||
- **Phase 3:** Package refactoring (✅ complete - modular structure)
|
||||
- **Phase 4.1:** Pydantic data models (✅ complete - v2.0.0)
|
||||
- **Phase 5:** Test coverage expansion (✅ complete - 84% coverage, 314 tests)
|
||||
|
||||
**Future Enhancements:**
|
||||
- **Phase 4.2:** Response summarization (📋 planned)
|
||||
- **Phase 5:** Expanded test coverage (⏳ in progress - core model tests complete)
|
||||
- Amenity scoring with quality metrics
|
||||
- Caching system (in-memory → Redis)
|
||||
- Async/parallel processing
|
||||
|
||||
## Common Tasks
|
||||
|
||||
|
||||
Reference in New Issue
Block a user