Sidecar JSON supplies published_at / effective_at as 'YYYY-MM-DD' strings.
asyncpg rejects strings for DATE columns even with '::date' casts in the
SQL, so ingest now parses them via datetime.date.fromisoformat before the
INSERT. Non-string inputs (date, datetime, None, empty) are handled too.
Before: ingest crashed on any sidecar with a date, leaving the source in
failed/ while sidecarless siblings succeeded.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two bugs surfaced during Phase 1 validation on real circulars:
1. The chunker produced one giant chunk for large statutes (ספר הליקויים,
ספר המבחנים) that the section regex could not split. This exceeded
Voyage's 120K-token batch limit and the ingest failed. Add a hard
2000-token ceiling per chunk with a character-based fallback split that
preserves heading_path and section_ref.
2. ingest_source matched "existing source" by kind + COALESCE(identifier,
''), so any two sources with NULL identifier compared equal. The second
ingest would then supersede the first unrelated document. Match by
identifier only when one is provided; otherwise fall back to
(kind + title) and require the candidate to still be active.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a searchable KB for חוק הביטוח הלאומי, תקנות הביטוח הלאומי, and חוזרי
הביטוח הלאומי. Hybrid search (pgvector cosine + tsvector/trgm) fused with
Reciprocal Rank Fusion, exposed to Shira as the `search_insurance_kb` tool.
- api/services/kb/: asyncpg pool, Voyage voyage-multilingual-2 client,
per-kind chunker (statute-by-section / circular header-aware), ingest
pipeline with supersession (old source → superseded_by), hybrid search.
- api/routes/admin_kb.py: POST /admin/kb/ingest (multipart), GET /admin/kb/stats.
- mcp_server/tools/legal_kb_tools.py: `search_insurance_kb` registered under
the `legal` toolset.
- pyproject.toml: +asyncpg, +python-docx, +boto3, +python-multipart.
Infra (external): pgvector/pgvector:pg16 on the shared PG, insurance_kb DB
with hnsw + gin indexes, MinIO bucket insurance-kb, KB_DATABASE_URL in
Infisical /espocrm, VOYAGE_API_KEY/VOYAGE_MODEL/KB_DATABASE_URL on the
shira-hermes Coolify app.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chaim