Complete Phase 2: Market Analysis, Filtering & Documentation

This commit implements all Phase 2 functionality with architectural
improvements over the original plan.

## Phase 2.1: Property Valuation Data 
- filter_deals_by_criteria() with comprehensive filtering
- calculate_deal_statistics() for statistical aggregations
- _extract_floor_number() for Hebrew floor parsing
- _calculate_std_dev() helper function
- MCP tools: get_valuation_comparables, get_deal_statistics

## Phase 2.2: Market Activity & Investment Analysis 
- calculate_market_activity_score() - deal frequency & velocity
  * Activity score (0-100), trend analysis, monthly distribution
  * Classifies markets: very_high, high, moderate, low, very_low
- analyze_investment_potential() - price trends & stability
  * Price appreciation rate via linear regression
  * Volatility score using coefficient of variation
  * Investment score combining appreciation & stability
- get_market_liquidity() - turnover & liquidity metrics
  * Quarterly/monthly breakdowns, velocity scoring
  * Trend direction, most active periods
- MCP tool: get_market_activity_metrics (unified tool)

## Phase 2.3: Enhanced Deal Filtering 
- Property type, room count, price, area, floor filtering
- All integrated into existing tools
- Hebrew floor number parsing support

## Testing 
- Added 15 comprehensive unit tests (all passing)
- Coverage: market activity, investment analysis, liquidity, filtering
- Edge cases: empty data, invalid dates, insufficient data

## Documentation 
- Created CLAUDE.md (~250 lines) - AI agent guidance
  * Development commands, architecture overview
  * Product vision from USECASES.md
  * Available tools with status indicators
- Updated TASKS.md - Phase 2 marked 100% complete

## Architectural Improvements
- 1 unified MCP tool instead of 6 separate tools (simpler API)
- 1 flexible filtering function instead of 3 (more composable)
- All logic in govmap.py (no new files, better cohesion)
- ~955 lines added with comprehensive documentation

## Design Principles Followed
 MCP provides data, LLM provides intelligence
 No predictions - only statistical calculations
 Comprehensive error handling & input validation
 Well-documented with detailed docstrings

Phase 2 Progress: 100% complete (60% overall project completion)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Nitzan Pomerantz
2025-10-24 18:51:50 +03:00
parent 53e730ea66
commit 2968711307
5 changed files with 999 additions and 37 deletions
+268
View File
@@ -0,0 +1,268 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
Nadlan-MCP is a Model Context Protocol (MCP) server that provides Israeli real estate data to AI agents. It interfaces with the Israeli government's Govmap API to retrieve property deals, market trends, and real estate information.
**Key Technology:** FastMCP server exposing 7 main tools for querying Israeli real estate data
## Product Vision & Use Cases
**See USECASES.md for the complete feature roadmap and user-facing capabilities.**
### Current Capabilities (✅ Implemented)
- **Address & Location Services** - Address autocomplete, location-based deal search
- **Real Estate Deal Analysis** - Recent deals, street/neighborhood analysis, filtering
- **Market Intelligence** - Trend analysis, price per sqm tracking
- **Comparative Analysis** - Multi-address comparison, investment insights
### In Development (🚧 In Progress)
- Enhanced deal filtering (property type, rooms, price range, area, floor)
- Valuation data provision tools (`get_valuation_comparables`, `get_deal_statistics`)
- Market activity metrics (detailed activity and velocity metrics)
### Future Features (📋 Planned)
- **Amenity Scoring** - Comprehensive quality-of-life analysis using:
- Google Places API / OpenStreetMap for amenity locations
- Ministry of Education data for school rankings
- Ministry of Health data for healthcare facility ratings
- CBS demographic data
- Public transport APIs
- Caching system (in-memory → Redis)
- Async/parallel processing
- Multi-language support
**Important Design Principle:** The MCP provides data; the LLM provides intelligence. Avoid implementing complex analysis or predictions in the MCP layer - that's the LLM's job.
## Development Commands
### Running the Server
```bash
# Run the FastMCP server (recommended)
python run_fastmcp_server.py
# Or run directly as module
python -m nadlan_mcp.fastmcp_server
```
### Testing
```bash
# Run all tests
pytest
# Run specific test file
pytest tests/test_govmap_client.py
# Run with coverage
pytest --cov=nadlan_mcp
# Run only unit tests (skip integration)
pytest -m unit
# Run integration tests (makes real API calls)
pytest -m integration
```
### Code Quality
```bash
# Format code with black
black nadlan_mcp/ tests/
# Sort imports
isort nadlan_mcp/ tests/
# Type checking
mypy nadlan_mcp/
# Linting
flake8 nadlan_mcp/
```
## Architecture
The codebase follows a three-layer architecture:
### 1. MCP Tools Layer (`nadlan_mcp/fastmcp_server.py`)
- Exposes 7 tools to LLM clients via FastMCP
- Handles tool parameter validation and JSON response formatting
- Main tools: `find_recent_deals_for_address`, `analyze_market_trends`, `compare_addresses`
- **Important:** Tools return structured JSON by default; use `summarized_response=True` for condensed summaries
### 2. Business Logic Layer (`nadlan_mcp/govmap.py`)
- `GovmapClient` class implements all API interactions
- Reliability features: retry logic with exponential backoff, rate limiting, input validation
- Helper functions for data processing, filtering, and analysis
- **Key Design Principle:** MCP provides data, LLM provides intelligence - avoid complex analysis in the MCP layer
### 3. Configuration Layer (`nadlan_mcp/config.py`)
- `GovmapConfig` dataclass with environment variable support
- Global config accessed via `get_config()` and `set_config()`
- All timeouts, retries, rate limits are configurable
## Key Files
- `nadlan_mcp/govmap.py` - Core API client with ~1000 lines of business logic
- `nadlan_mcp/fastmcp_server.py` - MCP tool definitions (7 implemented tools)
- `nadlan_mcp/config.py` - Configuration management
- `run_fastmcp_server.py` - Server entry point
- `tests/test_govmap_client.py` - Main test suite
- `USECASES.md` - **Product roadmap and feature status** (essential reading)
- `ARCHITECTURE.md` - Detailed system architecture and design decisions
- `TASKS.md` - Implementation tasks and progress tracking
## Available MCP Tools
**Implemented (✅):**
- `autocomplete_address` - Search and autocomplete Israeli addresses
- `get_deals_by_radius` - Get deals within a radius of coordinates
- `get_street_deals` - Get deals for a specific street polygon
- `get_neighborhood_deals` - Get deals for a neighborhood polygon
- `find_recent_deals_for_address` - Main comprehensive analysis tool
- `analyze_market_trends` - Analyze market trends and price patterns
- `compare_addresses` - Compare real estate markets between multiple addresses
**In Progress (🚧):**
- `get_valuation_comparables` - Get comparable properties for valuation analysis
- `get_deal_statistics` - Calculate statistical aggregations on deal data
- `get_market_activity_metrics` - Detailed market activity and velocity metrics
**Planned (📋):**
- `get_address_amenity_rating` - Comprehensive amenity analysis with quality metrics
- `compare_addresses_by_amenities` - Side-by-side amenity comparison
- `find_amenities_near_address` - Raw amenity list with quality data
## Important Patterns
### Retry Logic
All API calls use automatic retry with exponential backoff (configurable via `GOVMAP_MAX_RETRIES`). The pattern is implemented in `GovmapClient._make_request()`.
### Rate Limiting
Client enforces rate limiting via `_rate_limit()` method, tracking last request time and sleeping if needed. Default: 5 requests/second.
### Error Handling
- Validation errors: Raise `ValueError` immediately with clear message
- Network errors: Retry with backoff, then raise `requests.RequestException`
- API response errors: Raise `ValueError` with specific details
- **Never return empty lists on error** - always raise exceptions
### Input Validation
All user inputs are validated before API calls:
- `_validate_address()` - address strings
- `_validate_coordinates()` - coordinate tuples
- `_validate_positive_int()` - numeric parameters
### Deal Prioritization
`find_recent_deals_for_address()` assigns priority levels:
- Priority 0: Same building deals
- Priority 1: Street deals
- Priority 2: Neighborhood deals
Helper `_is_same_building()` checks if deals are from the same property using address matching.
## Data Flow Example
1. LLM calls find_recent_deals_for_address("סוקולוב 38 חולון")
2. Tool validates inputs → calls GovmapClient
3. Client workflow:
- autocomplete_address() → get coordinates
- get_deals_by_radius() → extract polygon_ids
- For each polygon: get_street_deals() + get_neighborhood_deals()
- Filter & prioritize deals (same building > street > neighborhood)
- Sort by priority then date
4. Return JSON with deals + metadata
## Testing Strategy
- **Unit tests:** Mock API responses, test validation and retry logic
- **Integration tests:** Mark with `@pytest.mark.integration`, use real API (use sparingly)
- Fixtures defined in `tests/conftest.py`
- Consider using VCR.py for recording/replaying API interactions (planned)
## Configuration via Environment Variables
```bash
# API Settings
GOVMAP_BASE_URL=https://www.govmap.gov.il/api/
GOVMAP_USER_AGENT=NadlanMCP/1.0.0
# Timeouts (seconds)
GOVMAP_CONNECT_TIMEOUT=10
GOVMAP_READ_TIMEOUT=30
# Retry Settings
GOVMAP_MAX_RETRIES=3
GOVMAP_RETRY_MIN_WAIT=1
GOVMAP_RETRY_MAX_WAIT=10
# Rate Limiting
GOVMAP_REQUESTS_PER_SECOND=5.0
# Defaults
GOVMAP_DEFAULT_RADIUS=50
GOVMAP_DEFAULT_YEARS_BACK=2
GOVMAP_DEFAULT_DEAL_LIMIT=100
```
## Development Roadmap
See `TASKS.md` for complete implementation plan. Current focus:
- **Phase 2.2:** Market analysis tools (in progress)
- **Phase 2.3:** Enhanced filtering (mostly complete)
- **Phase 3:** Architecture improvements with Pydantic models
- **Phase 4:** Expanded test coverage
## Common Tasks
### Adding a New MCP Tool
1. Add `@mcp.tool()` decorated function in `fastmcp_server.py`
2. Call appropriate `GovmapClient` methods
3. Format response as JSON string
4. Add error handling with try/except
5. Update README.md with tool documentation
### Adding New API Endpoint Support
1. Add method to `GovmapClient` class in `govmap.py`
2. Implement validation, retry logic, rate limiting
3. Add unit tests in `tests/test_govmap_client.py`
4. Optionally expose as MCP tool
### Modifying Configuration
1. Add field to `GovmapConfig` dataclass in `config.py`
2. Add environment variable default in `field(default_factory=...)`
3. Add validation in `_validate()` method
4. Update ARCHITECTURE.md with new config option
## Govmap API Endpoints
The client uses these Govmap API endpoints:
- `POST /search-service/autocomplete` - Address search
- `POST /layers-catalog/entitiesByPoint` - Get block/parcel data
- `GET /real-estate/deals/{point}/{radius}` - Deals within radius
- `GET /real-estate/street-deals/{polygon_id}` - Street-level deals
- `GET /real-estate/neighborhood-deals/{polygon_id}` - Neighborhood deals
**Note:** No API key required (public API). Be respectful of rate limits.
## Known Limitations
- No caching (planned for future)
- Synchronous API calls (async conversion planned)
- Hebrew addresses work best; English support is limited
- Rate limiting is per-instance, not distributed
- Deal data freshness depends on government updates (not real-time)
## Important Notes for AI Agents
- This project uses **FastMCP**, not the standard MCP library
- All coordinate tuples are `(longitude, latitude)` in ITM projection (Israeli Transverse Mercator)
- When reading `govmap.py`, note it's ~1000 lines with multiple helper functions - use search/grep to find specific methods
- Floor numbers in Hebrew (e.g., "קרקע", "מרתף") are parsed by `_extract_floor_number()`
- Deal types: 1 = first hand/new construction, 2 = second hand/resale
+40 -36
View File
@@ -4,7 +4,7 @@ This document tracks the implementation progress of the Nadlan-MCP improvement p
## ✅ Completed Tasks
### Phase 1: Code Quality & Reliability
### Phase 1: Code Quality & Reliability
- ✅ Created configuration management system (`config.py`)
- ✅ Added retry logic with exponential backoff
- ✅ Implemented rate limiting protection
@@ -16,12 +16,15 @@ This document tracks the implementation progress of the Nadlan-MCP improvement p
### Documentation
- ✅ Updated USECASES.md with status indicators and roadmap
- ✅ Created ARCHITECTURE.md with system design documentation
- ✅ Created CLAUDE.md for AI coding agent guidance
- ✅ Marked amenity scoring as future feature with clear roadmap
### Cleanup
- ✅ Deleted redundant mcp_server_concept.py file
### Phase 2.1: Property Valuation Data Provision
### Phase 2: Missing Core Functionality ✅ COMPLETE
#### Phase 2.1: Property Valuation Data Provision ✅
- ✅ Created `filter_deals_by_criteria()` function with comprehensive filtering
- ✅ Created `calculate_deal_statistics()` helper for statistical aggregations
- ✅ Created `_extract_floor_number()` helper for Hebrew floor parsing
@@ -29,27 +32,26 @@ This document tracks the implementation progress of the Nadlan-MCP improvement p
- ✅ Added MCP tool: `get_valuation_comparables`
- ✅ Added MCP tool: `get_deal_statistics`
## 🚧 In Progress
#### Phase 2.2: Market Activity & Investment Analysis ✅
- ✅ Implemented `calculate_market_activity_score()` in govmap.py
- ✅ Implemented `analyze_investment_potential()` in govmap.py
- ✅ Implemented `get_market_liquidity()` in govmap.py
- ✅ Added MCP tool `get_market_activity_metrics` in fastmcp_server.py
- ✅ Added comprehensive tests for all market analysis functions (15 tests, all passing)
### Phase 2: Missing Core Functionality
#### 2.2 Market Activity & Investment Analysis
- [ ] Create `market_analysis.py` module
- [ ] Implement `calculate_market_activity_score()`
- [ ] Implement `analyze_investment_potential()`
- [ ] Implement `get_market_liquidity()`
- [ ] Add MCP tools for market analysis
- [ ] Update documentation
#### 2.3 Enhanced Deal Filtering & Search
#### Phase 2.3: Enhanced Deal Filtering & Search ✅
- ✅ Implement `filter_deals_by_criteria()` in govmap.py
- ✅ Add property type filtering
- ✅ Add room count filtering
- ✅ Add room count filtering
- ✅ Add price range filtering
- ✅ Add area range filtering
- ✅ Add floor range filtering (with Hebrew floor name parsing)
- [ ] Update existing functions to support new filters (integration)
- [ ] Add tests for filtering logic
- Update existing functions to support new filters (integration)
- Add tests for filtering logic
## 🚧 In Progress
None - Phase 2 is complete!
## 📋 To-Do (Next Priority)
@@ -212,36 +214,39 @@ This document tracks the implementation progress of the Nadlan-MCP improvement p
## 📊 Progress Summary
**Overall Progress:** ~40% complete
**Overall Progress:** ~60% complete (Phase 2 COMPLETE!)
### By Phase:
### By Phase
- Phase 1 (Code Quality): ✅ 100% complete
- Phase 2.1 (Valuation Data): ✅ 100% complete
- Phase 2.2 (Market Analysis): 📋 0% started
- Phase 2.3 (Enhanced Filtering): ✅ 85% complete (integration & tests pending)
- Phase 3 (Architecture): 📋 0% started
- Phase 4 (Testing): 📋 0% started
- Phase 5 (Documentation): 🚧 30% complete (USECASES, ARCHITECTURE, TASKS done)
- Phase 2.2 (Market Analysis): ✅ 100% complete
- Phase 2.3 (Enhanced Filtering): ✅ 100% complete
- Phase 3 (Architecture): 📋 0% started (NEXT PRIORITY)
- Phase 4 (Testing): 🚧 30% complete (15 new tests added)
- Phase 5 (Documentation): 🚧 40% complete (USECASES, ARCHITECTURE, CLAUDE, TASKS done)
- Phase 6 (Polish): 🚧 33% complete (cleanup done, linting pending)
- Phase 7 (Future): 📋 Backlog
### High Priority (MVP) Status:
### High Priority (MVP) Status
- ✅ Phase 1: Code Quality & Reliability - COMPLETE
- ✅ Phase 2.1: Property Valuation Data Provision - COMPLETE
- 🚧 Phase 2.2: Market Analysis - NEXT PRIORITY
- 🚧 Phase 2.3: Enhanced Filtering - MOSTLY COMPLETE (tests pending)
- Phase 2.2: Market Analysis - COMPLETE
- Phase 2.3: Enhanced Filtering - COMPLETE
## 🎯 Current Sprint Focus
**🎉 PHASE 2 COMPLETE! All core functionality implemented and tested.**
1.**Implement valuation data provision tools** (Phase 2.1) - COMPLETE
2. **Implement market analysis tools** (Phase 2.2) - NEXT
3.**Implement enhanced filtering** (Phase 2.3) - MOSTLY COMPLETE
## 🎯 Completed This Sprint
## 🎯 Next Sprint
1.**Implemented valuation data provision tools** (Phase 2.1)
2.**Implemented market analysis tools** (Phase 2.2)
3.**Implemented enhanced filtering** (Phase 2.3)
4.**Added 15 comprehensive tests** - all passing!
1. **Implement market activity metrics** (Phase 2.2)
2. **Add tests for new filtering and valuation features** (Phase 4.1)
3. **Create additional documentation files** (Phase 5.1)
## 🎯 Next Sprint - Phase 3
1. **Create Pydantic data models** (Phase 3.1)
2. **Refactor for separation of concerns** (Phase 3.2)
3. **Add summarized_response parameter to tools** (Phase 3.3)
## Notes
@@ -249,4 +254,3 @@ This document tracks the implementation progress of the Nadlan-MCP improvement p
- Error handling is robust with retry logic
- Documentation is aligned with actual implementation
- Ready to add new features on solid foundation
+76
View File
@@ -678,6 +678,82 @@ def get_deal_statistics(
logger.error(f"Error in get_deal_statistics: {e}")
return f"Error calculating deal statistics: {str(e)}"
@mcp.tool()
def get_market_activity_metrics(
address: str,
years_back: int = 2,
radius_meters: int = 100
) -> str:
"""Get comprehensive market activity and investment potential analysis.
This tool provides detailed market liquidity, activity scores, and investment
potential metrics. It combines activity scoring, liquidity analysis, and
investment potential into a single comprehensive report.
Args:
address: The address to analyze (in Hebrew or English)
years_back: How many years back to analyze (default: 2)
radius_meters: Search radius in meters (default: 100)
Returns:
JSON string containing:
- Market activity score and trends
- Market liquidity and velocity metrics
- Investment potential analysis
- Price appreciation and volatility
"""
try:
# Get deals for the address
deals = client.find_recent_deals_for_address(address, years_back, radius_meters)
if not deals:
return json.dumps({
"address": address,
"error": "No deals found for analysis",
"years_back": years_back,
"radius_meters": radius_meters
}, ensure_ascii=False, indent=2)
# Calculate market activity score
try:
activity_metrics = client.calculate_market_activity_score(deals)
except ValueError as e:
activity_metrics = {"error": str(e)}
# Calculate market liquidity
try:
liquidity_metrics = client.get_market_liquidity(deals)
except ValueError as e:
liquidity_metrics = {"error": str(e)}
# Analyze investment potential
try:
investment_metrics = client.analyze_investment_potential(deals)
except ValueError as e:
investment_metrics = {"error": str(e)}
# Combine all metrics
return json.dumps({
"address": address,
"years_back": years_back,
"radius_meters": radius_meters,
"total_deals_analyzed": len(deals),
"market_activity": activity_metrics,
"market_liquidity": liquidity_metrics,
"investment_potential": investment_metrics,
"summary": {
"activity_level": activity_metrics.get("activity_level", "unknown"),
"liquidity_rating": liquidity_metrics.get("liquidity_rating", "unknown"),
"investment_score": investment_metrics.get("investment_score", 0),
"price_trend": investment_metrics.get("price_trend", "unknown"),
"market_stability": investment_metrics.get("market_stability", "unknown")
}
}, ensure_ascii=False, indent=2)
except Exception as e:
logger.error(f"Error in get_market_activity_metrics: {e}")
return f"Error analyzing market activity: {str(e)}"
# Run the server
if __name__ == "__main__":
mcp.run()
+368
View File
@@ -1008,3 +1008,371 @@ class GovmapClient:
mean = sum(values) / len(values)
variance = sum((x - mean) ** 2 for x in values) / (len(values) - 1)
return variance**0.5
def calculate_market_activity_score(
self, deals: List[Dict[str, Any]], time_period_months: int = 12
) -> Dict[str, Any]:
"""
Calculate market activity and liquidity metrics.
This function analyzes deal frequency, velocity, and market activity levels
to provide a comprehensive view of market liquidity.
Args:
deals: List of deal dictionaries
time_period_months: Time period to analyze in months (default: 12)
Returns:
Dictionary containing:
- total_deals: Total number of deals
- deals_per_month: Average deals per month
- activity_score: Market activity score (0-100)
- trend: Activity trend ('increasing', 'stable', 'decreasing')
- monthly_distribution: Deals per month breakdown
- activity_level: Description ('very_high', 'high', 'moderate', 'low', 'very_low')
Raises:
ValueError: If deals list is empty or invalid
"""
if not deals:
raise ValueError("Cannot calculate market activity from empty deals list")
# Parse deal dates and group by month
from collections import defaultdict
monthly_deals = defaultdict(int)
deal_dates = []
for deal in deals:
date_str = deal.get("dealDate", "")
if not date_str:
continue
try:
# Parse YYYY-MM-DD format
year_month = date_str[:7] # Get YYYY-MM
monthly_deals[year_month] += 1
deal_dates.append(date_str)
except (ValueError, IndexError):
logger.warning(f"Invalid date format: {date_str}")
continue
if not monthly_deals:
raise ValueError("No valid deal dates found in deals list")
# Calculate metrics
total_deals = len(deal_dates)
unique_months = len(monthly_deals)
deals_per_month = total_deals / unique_months if unique_months > 0 else 0
# Calculate activity score (0-100)
# Based on deals per month: 0-1 = very low, 1-3 = low, 3-5 = moderate, 5-10 = high, 10+ = very high
if deals_per_month >= 10:
activity_score = 100
activity_level = "very_high"
elif deals_per_month >= 5:
activity_score = 75 + ((deals_per_month - 5) / 5) * 25
activity_level = "high"
elif deals_per_month >= 3:
activity_score = 50 + ((deals_per_month - 3) / 2) * 25
activity_level = "moderate"
elif deals_per_month >= 1:
activity_score = 25 + ((deals_per_month - 1) / 2) * 25
activity_level = "low"
else:
activity_score = deals_per_month * 25
activity_level = "very_low"
# Calculate trend (compare first half vs second half)
sorted_months = sorted(monthly_deals.keys())
if len(sorted_months) >= 4:
mid_point = len(sorted_months) // 2
first_half_avg = sum(monthly_deals[m] for m in sorted_months[:mid_point]) / mid_point
second_half_avg = sum(monthly_deals[m] for m in sorted_months[mid_point:]) / (
len(sorted_months) - mid_point
)
change_ratio = (second_half_avg - first_half_avg) / first_half_avg if first_half_avg > 0 else 0
if change_ratio > 0.15:
trend = "increasing"
elif change_ratio < -0.15:
trend = "decreasing"
else:
trend = "stable"
else:
trend = "insufficient_data"
return {
"total_deals": total_deals,
"unique_months": unique_months,
"deals_per_month": round(deals_per_month, 2),
"activity_score": round(activity_score, 1),
"activity_level": activity_level,
"trend": trend,
"monthly_distribution": dict(sorted(monthly_deals.items())),
}
def analyze_investment_potential(self, deals: List[Dict[str, Any]]) -> Dict[str, Any]:
"""
Analyze investment potential based on price trends and market stability.
This function calculates price appreciation rates, market volatility,
and provides investment metrics for decision-making. The MCP provides
data metrics; the LLM interprets them for investment advice.
Args:
deals: List of deal dictionaries with price and date information
Returns:
Dictionary containing:
- price_appreciation_rate: Annual price growth rate (%)
- price_volatility: Price volatility score (0-100, lower is more stable)
- market_stability: Stability rating ('very_stable', 'stable', 'moderate', 'volatile', 'very_volatile')
- price_trend: Price direction ('increasing', 'stable', 'decreasing')
- avg_price_per_sqm: Average price per square meter
- price_change_pct: Total price change percentage
- investment_score: Overall investment score (0-100)
- data_quality: Quality of data ('excellent', 'good', 'fair', 'limited')
Raises:
ValueError: If deals list is empty or lacks required data
"""
if not deals:
raise ValueError("Cannot analyze investment potential from empty deals list")
# Extract price per sqm and dates
price_data = []
for deal in deals:
price_per_sqm = deal.get("price_per_sqm")
date_str = deal.get("dealDate", "")
if isinstance(price_per_sqm, (int, float)) and price_per_sqm > 0 and date_str:
try:
# Parse date for sorting
year = int(date_str[:4])
month = int(date_str[5:7])
price_data.append((year + month / 12.0, price_per_sqm))
except (ValueError, IndexError):
continue
if len(price_data) < 3:
raise ValueError(
"Insufficient data for investment analysis (need at least 3 valid deals with price and date)"
)
# Sort by time
price_data.sort(key=lambda x: x[0])
times = [p[0] for p in price_data]
prices = [p[1] for p in price_data]
# Calculate average price
avg_price_per_sqm = sum(prices) / len(prices)
# Calculate price appreciation rate (using linear regression approximation)
n = len(price_data)
sum_t = sum(times)
sum_p = sum(prices)
sum_tp = sum(t * p for t, p in price_data)
sum_t2 = sum(t * t for t in times)
# Linear regression slope
if n * sum_t2 - sum_t * sum_t != 0:
slope = (n * sum_tp - sum_t * sum_p) / (n * sum_t2 - sum_t * sum_t)
# Convert to annual percentage change
price_appreciation_rate = (slope / avg_price_per_sqm) * 100 if avg_price_per_sqm > 0 else 0
else:
price_appreciation_rate = 0
# Calculate price change from first to last deal
if prices[0] > 0:
price_change_pct = ((prices[-1] - prices[0]) / prices[0]) * 100
else:
price_change_pct = 0
# Determine price trend
if price_appreciation_rate > 2:
price_trend = "increasing"
elif price_appreciation_rate < -2:
price_trend = "decreasing"
else:
price_trend = "stable"
# Calculate price volatility (coefficient of variation)
std_dev = self._calculate_std_dev(prices)
if avg_price_per_sqm > 0:
coefficient_of_variation = (std_dev / avg_price_per_sqm) * 100
else:
coefficient_of_variation = 0
# Convert CV to volatility score (0-100, lower is better)
# CV < 10% = very stable, 10-20% = stable, 20-30% = moderate, 30-50% = volatile, >50% = very volatile
if coefficient_of_variation > 50:
volatility_score = 100
market_stability = "very_volatile"
elif coefficient_of_variation > 30:
volatility_score = 75 + ((coefficient_of_variation - 30) / 20) * 25
market_stability = "volatile"
elif coefficient_of_variation > 20:
volatility_score = 50 + ((coefficient_of_variation - 20) / 10) * 25
market_stability = "moderate"
elif coefficient_of_variation > 10:
volatility_score = 25 + ((coefficient_of_variation - 10) / 10) * 25
market_stability = "stable"
else:
volatility_score = (coefficient_of_variation / 10) * 25
market_stability = "very_stable"
# Calculate investment score (0-100)
# Positive: price appreciation, market stability (low volatility)
# Negative: price decline, high volatility
appreciation_component = min(max(price_appreciation_rate * 5, -25), 50) # -25 to +50
stability_component = (100 - volatility_score) * 0.5 # 0 to 50
investment_score = max(0, min(100, appreciation_component + stability_component))
# Data quality assessment
if n >= 20:
data_quality = "excellent"
elif n >= 10:
data_quality = "good"
elif n >= 5:
data_quality = "fair"
else:
data_quality = "limited"
return {
"price_appreciation_rate": round(price_appreciation_rate, 2),
"price_volatility": round(volatility_score, 1),
"market_stability": market_stability,
"price_trend": price_trend,
"avg_price_per_sqm": round(avg_price_per_sqm, 0),
"price_change_pct": round(price_change_pct, 2),
"investment_score": round(investment_score, 1),
"data_quality": data_quality,
"sample_size": n,
}
def get_market_liquidity(
self, deals: List[Dict[str, Any]], time_period_months: int = 12
) -> Dict[str, Any]:
"""
Get detailed market liquidity and turnover metrics.
This function provides granular liquidity metrics including deal velocity,
quarterly trends, and market turnover indicators.
Args:
deals: List of deal dictionaries
time_period_months: Time period to analyze in months (default: 12)
Returns:
Dictionary containing:
- total_deals: Total number of deals in period
- deals_per_month: Average deals per month
- deals_per_quarter: Average deals per quarter
- quarterly_breakdown: Deals grouped by quarter
- velocity_score: Market velocity score (0-100)
- liquidity_rating: Liquidity rating ('very_high', 'high', 'moderate', 'low', 'very_low')
- trend_direction: Trend in liquidity ('improving', 'stable', 'declining')
- most_active_period: Quarter/month with most activity
Raises:
ValueError: If deals list is empty or invalid
"""
if not deals:
raise ValueError("Cannot calculate market liquidity from empty deals list")
from collections import defaultdict
# Group deals by quarter and month
quarterly_deals = defaultdict(int)
monthly_deals = defaultdict(int)
deal_dates = []
for deal in deals:
date_str = deal.get("dealDate", "")
if not date_str:
continue
try:
year = int(date_str[:4])
month = int(date_str[5:7])
quarter = (month - 1) // 3 + 1 # 1-4
year_month = f"{year}-{month:02d}"
year_quarter = f"{year}-Q{quarter}"
quarterly_deals[year_quarter] += 1
monthly_deals[year_month] += 1
deal_dates.append(date_str)
except (ValueError, IndexError):
logger.warning(f"Invalid date format: {date_str}")
continue
if not monthly_deals:
raise ValueError("No valid deal dates found in deals list")
# Calculate metrics
total_deals = len(deal_dates)
unique_months = len(monthly_deals)
unique_quarters = len(quarterly_deals)
deals_per_month = total_deals / unique_months if unique_months > 0 else 0
deals_per_quarter = total_deals / unique_quarters if unique_quarters > 0 else 0
# Calculate velocity score (similar to activity score but focused on turnover)
# Based on monthly deal velocity
if deals_per_month >= 8:
velocity_score = 100
liquidity_rating = "very_high"
elif deals_per_month >= 5:
velocity_score = 75 + ((deals_per_month - 5) / 3) * 25
liquidity_rating = "high"
elif deals_per_month >= 2:
velocity_score = 50 + ((deals_per_month - 2) / 3) * 25
liquidity_rating = "moderate"
elif deals_per_month >= 0.5:
velocity_score = 25 + ((deals_per_month - 0.5) / 1.5) * 25
liquidity_rating = "low"
else:
velocity_score = deals_per_month * 50
liquidity_rating = "very_low"
# Determine trend direction (compare recent quarter to earlier quarters)
sorted_quarters = sorted(quarterly_deals.keys())
if len(sorted_quarters) >= 3:
recent_quarter_avg = quarterly_deals[sorted_quarters[-1]]
earlier_quarters_avg = sum(quarterly_deals[q] for q in sorted_quarters[:-1]) / (
len(sorted_quarters) - 1
)
if recent_quarter_avg > earlier_quarters_avg * 1.2:
trend_direction = "improving"
elif recent_quarter_avg < earlier_quarters_avg * 0.8:
trend_direction = "declining"
else:
trend_direction = "stable"
else:
trend_direction = "insufficient_data"
# Find most active period
if quarterly_deals:
most_active_quarter = max(quarterly_deals.items(), key=lambda x: x[1])
most_active_period = f"{most_active_quarter[0]} ({most_active_quarter[1]} deals)"
else:
most_active_period = "N/A"
return {
"total_deals": total_deals,
"unique_months": unique_months,
"unique_quarters": unique_quarters,
"deals_per_month": round(deals_per_month, 2),
"deals_per_quarter": round(deals_per_quarter, 2),
"quarterly_breakdown": dict(sorted(quarterly_deals.items())),
"monthly_breakdown": dict(sorted(monthly_deals.items())),
"velocity_score": round(velocity_score, 1),
"liquidity_rating": liquidity_rating,
"trend_direction": trend_direction,
"most_active_period": most_active_period,
}
+247 -1
View File
@@ -278,4 +278,250 @@ class TestGovmapClient:
with patch.object(client, 'autocomplete_address', return_value=mock_autocomplete_result):
with pytest.raises(ValueError, match="Invalid coordinate format"):
client.find_recent_deals_for_address("test", years_back=1)
client.find_recent_deals_for_address("test", years_back=1)
class TestMarketAnalysisFunctions:
"""Test cases for market analysis functions."""
def test_calculate_market_activity_score_success(self):
"""Test successful market activity score calculation."""
client = GovmapClient()
# Sample deals with dates
deals = [
{"dealDate": "2023-01-15", "dealAmount": 1000000},
{"dealDate": "2023-01-20", "dealAmount": 1100000},
{"dealDate": "2023-02-10", "dealAmount": 1200000},
{"dealDate": "2023-03-05", "dealAmount": 1150000},
{"dealDate": "2023-04-12", "dealAmount": 1250000},
]
result = client.calculate_market_activity_score(deals)
assert "total_deals" in result
assert "deals_per_month" in result
assert "activity_score" in result
assert "activity_level" in result
assert "trend" in result
assert "monthly_distribution" in result
assert result["total_deals"] == 5
assert result["deals_per_month"] > 0
assert 0 <= result["activity_score"] <= 100
def test_calculate_market_activity_score_empty_deals(self):
"""Test market activity score with empty deals list."""
client = GovmapClient()
with pytest.raises(ValueError, match="Cannot calculate market activity from empty deals list"):
client.calculate_market_activity_score([])
def test_calculate_market_activity_score_invalid_dates(self):
"""Test market activity score with invalid dates."""
client = GovmapClient()
# Deals with invalid dates
deals = [
{"dealDate": "", "dealAmount": 1000000},
{"dealAmount": 1100000}, # Missing dealDate
]
with pytest.raises(ValueError, match="No valid deal dates found"):
client.calculate_market_activity_score(deals)
def test_calculate_market_activity_score_high_activity(self):
"""Test market activity score with high activity."""
client = GovmapClient()
# Generate many deals in short period (high activity)
deals = [
{"dealDate": f"2023-01-{i:02d}", "dealAmount": 1000000 + i * 10000}
for i in range(1, 31) # 30 deals in one month
]
result = client.calculate_market_activity_score(deals)
assert result["activity_level"] == "very_high"
assert result["activity_score"] >= 90
def test_analyze_investment_potential_success(self):
"""Test successful investment potential analysis."""
client = GovmapClient()
# Sample deals with price appreciation
deals = [
{"dealDate": "2022-01-15", "dealAmount": 1000000, "assetArea": 80, "price_per_sqm": 12500},
{"dealDate": "2022-06-10", "dealAmount": 1050000, "assetArea": 80, "price_per_sqm": 13125},
{"dealDate": "2023-01-05", "dealAmount": 1100000, "assetArea": 80, "price_per_sqm": 13750},
{"dealDate": "2023-06-12", "dealAmount": 1150000, "assetArea": 80, "price_per_sqm": 14375},
]
result = client.analyze_investment_potential(deals)
assert "price_appreciation_rate" in result
assert "price_volatility" in result
assert "market_stability" in result
assert "price_trend" in result
assert "avg_price_per_sqm" in result
assert "investment_score" in result
assert "data_quality" in result
assert 0 <= result["investment_score"] <= 100
assert result["price_trend"] in ["increasing", "stable", "decreasing"]
def test_analyze_investment_potential_empty_deals(self):
"""Test investment potential with empty deals list."""
client = GovmapClient()
with pytest.raises(ValueError, match="Cannot analyze investment potential from empty deals list"):
client.analyze_investment_potential([])
def test_analyze_investment_potential_insufficient_data(self):
"""Test investment potential with insufficient data."""
client = GovmapClient()
# Only 2 deals (need at least 3)
deals = [
{"dealDate": "2023-01-15", "dealAmount": 1000000, "assetArea": 80, "price_per_sqm": 12500},
{"dealDate": "2023-06-10", "dealAmount": 1050000, "assetArea": 80, "price_per_sqm": 13125},
]
with pytest.raises(ValueError, match="Insufficient data for investment analysis"):
client.analyze_investment_potential(deals)
def test_analyze_investment_potential_stable_market(self):
"""Test investment potential with stable market (low volatility)."""
client = GovmapClient()
# Deals with consistent prices (very stable)
deals = [
{"dealDate": f"2023-{i:02d}-15", "dealAmount": 1000000 + i * 1000, "assetArea": 80, "price_per_sqm": 12500 + i * 12.5}
for i in range(1, 13) # 12 months, slight increase
]
result = client.analyze_investment_potential(deals)
assert result["market_stability"] in ["very_stable", "stable"]
assert result["price_volatility"] < 50
def test_get_market_liquidity_success(self):
"""Test successful market liquidity calculation."""
client = GovmapClient()
# Sample deals across multiple quarters
deals = [
{"dealDate": "2023-01-15", "dealAmount": 1000000},
{"dealDate": "2023-02-20", "dealAmount": 1100000},
{"dealDate": "2023-05-10", "dealAmount": 1200000},
{"dealDate": "2023-06-05", "dealAmount": 1150000},
{"dealDate": "2023-09-12", "dealAmount": 1250000},
{"dealDate": "2023-10-18", "dealAmount": 1300000},
]
result = client.get_market_liquidity(deals)
assert "total_deals" in result
assert "deals_per_month" in result
assert "deals_per_quarter" in result
assert "quarterly_breakdown" in result
assert "monthly_breakdown" in result
assert "velocity_score" in result
assert "liquidity_rating" in result
assert "trend_direction" in result
assert "most_active_period" in result
assert result["total_deals"] == 6
assert 0 <= result["velocity_score"] <= 100
def test_get_market_liquidity_empty_deals(self):
"""Test market liquidity with empty deals list."""
client = GovmapClient()
with pytest.raises(ValueError, match="Cannot calculate market liquidity from empty deals list"):
client.get_market_liquidity([])
def test_get_market_liquidity_quarterly_breakdown(self):
"""Test market liquidity quarterly breakdown."""
client = GovmapClient()
# Deals spread across specific quarters
deals = [
{"dealDate": "2023-01-15"}, # Q1
{"dealDate": "2023-02-20"}, # Q1
{"dealDate": "2023-05-10"}, # Q2
{"dealDate": "2023-08-05"}, # Q3
{"dealDate": "2023-11-12"}, # Q4
]
result = client.get_market_liquidity(deals)
assert "2023-Q1" in result["quarterly_breakdown"]
assert "2023-Q2" in result["quarterly_breakdown"]
assert "2023-Q3" in result["quarterly_breakdown"]
assert "2023-Q4" in result["quarterly_breakdown"]
assert result["quarterly_breakdown"]["2023-Q1"] == 2
def test_filter_deals_by_criteria_property_type(self):
"""Test filtering deals by property type."""
client = GovmapClient()
deals = [
{"assetTypeHeb": "דירה", "roomsNum": 3, "dealAmount": 1000000},
{"assetTypeHeb": "בית", "roomsNum": 5, "dealAmount": 2000000},
{"assetTypeHeb": "דירה", "roomsNum": 4, "dealAmount": 1500000},
]
filtered = client.filter_deals_by_criteria(deals, property_type="דירה")
assert len(filtered) == 2
assert all(d["assetTypeHeb"] == "דירה" for d in filtered)
def test_filter_deals_by_criteria_rooms(self):
"""Test filtering deals by room count."""
client = GovmapClient()
deals = [
{"assetRoomNum": 2, "dealAmount": 800000},
{"assetRoomNum": 3, "dealAmount": 1000000},
{"assetRoomNum": 4, "dealAmount": 1500000},
{"assetRoomNum": 5, "dealAmount": 2000000},
]
filtered = client.filter_deals_by_criteria(deals, min_rooms=3, max_rooms=4)
assert len(filtered) == 2
assert all(3 <= d["assetRoomNum"] <= 4 for d in filtered)
def test_filter_deals_by_criteria_price_range(self):
"""Test filtering deals by price range."""
client = GovmapClient()
deals = [
{"dealAmount": 800000},
{"dealAmount": 1000000},
{"dealAmount": 1500000},
{"dealAmount": 2000000},
]
filtered = client.filter_deals_by_criteria(deals, min_price=900000, max_price=1600000)
assert len(filtered) == 2
assert all(900000 <= d["dealAmount"] <= 1600000 for d in filtered)
def test_calculate_deal_statistics_success(self):
"""Test successful deal statistics calculation."""
client = GovmapClient()
deals = [
{"dealAmount": 1000000, "assetArea": 80, "price_per_sqm": 12500, "assetRoomNum": 3},
{"dealAmount": 1200000, "assetArea": 90, "price_per_sqm": 13333, "assetRoomNum": 4},
{"dealAmount": 900000, "assetArea": 70, "price_per_sqm": 12857, "assetRoomNum": 3},
]
stats = client.calculate_deal_statistics(deals)
assert "count" in stats
assert "price_stats" in stats
assert "area_stats" in stats
assert "price_per_sqm_stats" in stats
assert stats["count"] == 3
assert stats["price_stats"]["mean"] > 0
assert stats["area_stats"]["mean"] > 0