Add detailed logging for filtering stages in get_valuation_comparables

Adds visibility into the deal filtering pipeline to help diagnose
why certain queries return fewer deals than expected.

New logs show:
1. Number of deals before criteria filtering
2. Number of deals after criteria filtering (with count removed)
3. Whether outlier filtering was applied
4. Number of deals after outlier filtering (with count removed)
5. Outlier filtering method and parameters used

Example output:
  INFO: Applying criteria filters to 41 deals
  INFO: After criteria filtering: 12 deals (removed 29 deals)
  INFO: After outlier filtering (iqr, k=1.5): 4 deals (removed 8 outliers)

This helps users and developers understand:
- If criteria filters are too restrictive
- If outlier filtering is too aggressive
- Where deals are being filtered out in the pipeline

Addresses user question: "Why do I get just 4 deals for a central address
like סירקין 16 תל אביב?" - Now they can see exactly where deals were
filtered out.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Nitzan P
2025-11-27 00:15:33 +02:00
parent 421dec37b5
commit 1bc94e39ba
+15
View File
@@ -849,6 +849,7 @@ def get_valuation_comparables(
) )
# Apply filters # Apply filters
logger.info(f"Applying criteria filters to {len(deals)} deals")
filtered_deals = client.filter_deals_by_criteria( filtered_deals = client.filter_deals_by_criteria(
deals, deals,
property_type=property_type, property_type=property_type,
@@ -861,6 +862,10 @@ def get_valuation_comparables(
min_floor=min_floor, min_floor=min_floor,
max_floor=max_floor, max_floor=max_floor,
) )
logger.info(
f"After criteria filtering: {len(filtered_deals)} deals "
f"(removed {len(deals) - len(filtered_deals)} deals)"
)
# Apply outlier filtering to remove statistical outliers # Apply outlier filtering to remove statistical outliers
config = get_config() config = get_config()
@@ -869,9 +874,19 @@ def get_valuation_comparables(
config.analysis_outlier_method != "none" config.analysis_outlier_method != "none"
and len(filtered_deals) >= config.analysis_min_deals_for_outlier_detection and len(filtered_deals) >= config.analysis_min_deals_for_outlier_detection
): ):
deals_before_outlier_filter = len(filtered_deals)
filtered_deals, outlier_report = filter_deals_for_analysis( filtered_deals, outlier_report = filter_deals_for_analysis(
filtered_deals, config, metric="price_per_sqm" filtered_deals, config, metric="price_per_sqm"
) )
logger.info(
f"After outlier filtering ({config.analysis_outlier_method}, k={config.analysis_iqr_multiplier}): "
f"{len(filtered_deals)} deals (removed {deals_before_outlier_filter - len(filtered_deals)} outliers)"
)
else:
logger.info(
f"Skipping outlier filtering: only {len(filtered_deals)} deals "
f"(minimum {config.analysis_min_deals_for_outlier_detection} required)"
)
# Calculate statistics on filtered comparables # Calculate statistics on filtered comparables
stats = client.calculate_deal_statistics(filtered_deals) stats = client.calculate_deal_statistics(filtered_deals)