Adds a rerank stage between the hybrid RRF fusion and the final top_k cut.
RRF is a heuristic over ranks — it can't tell which of two equally-ranked
candidates is actually more relevant. A cross-encoder scores each
(query, passage) pair directly and produces a true relevance order.
- voyage.rerank(query, documents, top_k) → list of {index, relevance_score}
sorted high→low. Defaults to the model in VOYAGE_RERANK_MODEL
(rerank-2.5, 32K tokens per pair, multilingual incl. Hebrew).
- search pulls top-30 from RRF, passes them to rerank, then returns the
reranker's top-k. Each returned item gets a rerank_score field.
- Fallback: if the rerank API errors (auth, 5xx, timeout), fall back to
the RRF ordering and log a warning — degraded precision beats outage.
- KB_RERANK_DISABLED=1 bypasses rerank for debugging.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sidecar JSON supplies published_at / effective_at as 'YYYY-MM-DD' strings.
asyncpg rejects strings for DATE columns even with '::date' casts in the
SQL, so ingest now parses them via datetime.date.fromisoformat before the
INSERT. Non-string inputs (date, datetime, None, empty) are handled too.
Before: ingest crashed on any sidecar with a date, leaving the source in
failed/ while sidecarless siblings succeeded.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each file in the inbox can now be accompanied by a `<filename>.meta.json`
sidecar that supplies identifier, published_at, effective_at, and
source_url. The scanner applies filename-derived defaults first and lets
the sidecar override individual fields.
- list_inbox / list_failed skip `.meta.json` so sidecars aren't ingested
as standalone documents.
- mark_processed, mark_failed, and requeue-failed move the sidecar
alongside its parent (best-effort).
- mark_failed now sanitizes the error string to ASCII before writing it
to S3 object metadata (HTTP header encoding).
- search SQL selects published_at/effective_at; the formatter shows the
full heading_path plus publication date so citations are unambiguous.
- scripts/kb_sidecar_template.json documents the expected shape.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-shot script that fetches the consolidated National Insurance Law from
Hebrew Wikisource and emits plain text with heading lines the chunker can
parse (פרק / סימן / סעיף).
Key quirks of the source markup:
- {{ח:קטע2}} carries the level name inside the title parameter ("פרק א׳:
…"), not in the template name. Emit the title verbatim and let the
chunker's existing prefix regex handle it.
- Named args (e.g. 'אחר=[פרק א׳]') must be filtered — the original
positional parser was treating them as positional titles and producing
gibberish "חלק אחר=[פרק א׳]" headings.
- No חלק level exists in this law — only פרק > סימן > סעיף.
Run as: python3 scripts/fetch_national_insurance_law.py <out_path>
Then upload the output .txt to s3://insurance-kb/inbox/law/ and invoke
POST /admin/kb/scan-inbox.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Before: chunk_statute produced heading_path = section_ref = "ס' 195".
After: heading_path walks the four Israeli-statute levels (part, chapter,
sub-chapter, section) and renders them as "חלק ט׳ — ביטוח נכות > פרק ב׳
— ועדות רפואיות > סימן א׳ > ס' 195". section_ref stays the leaf so
citations stay precise.
- New _RE_PART / _RE_CHAPTER / _RE_SUBCHAPTER regexes with Hebrew ordinal
support including multi-letter gershayim ("י״א", "ט״ז").
- _collect_markers emits all markers ordered by position; chunk_statute
walks them tracking active_part/chapter/subchapter and attaches the
full path to each section chunk.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous batching sent up to 128 inputs per request but didn't cap total
tokens. On large Hebrew statutes (ספר הליקויים, ספר המבחנים) a single
request exceeded Voyage's 120K-token-per-batch hard cap and failed with 400.
- voyage.embed now pre-computes an estimated token count per input (Hebrew
is ~0.55 tokens/char in voyage-multilingual-2) and emits multiple
requests so each stays under a 90K-token safety margin.
- chunker lowers the per-chunk ceiling to 1500 tokens and updates the
chars→tokens estimate to match the observed tokenizer behavior.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two bugs surfaced during Phase 1 validation on real circulars:
1. The chunker produced one giant chunk for large statutes (ספר הליקויים,
ספר המבחנים) that the section regex could not split. This exceeded
Voyage's 120K-token batch limit and the ingest failed. Add a hard
2000-token ceiling per chunk with a character-based fallback split that
preserves heading_path and section_ref.
2. ingest_source matched "existing source" by kind + COALESCE(identifier,
''), so any two sources with NULL identifier compared equal. The second
ingest would then supersede the first unrelated document. Match by
identifier only when one is provided; otherwise fall back to
(kind + title) and require the candidate to still be active.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves every object under failed/<kind>/ back to inbox/<kind>/ for retry
after a transient upstream failure (e.g. Voyage 401 before a key rotation).
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a MinIO-backed ingest flow: upload to s3://insurance-kb/inbox/<kind>/
via MinIO Console or any S3 client, then POST /admin/kb/scan-inbox to
process all pending files. Successful ingests move to processed/<kind>/,
failures move to failed/<kind>/ with the error stored as S3 metadata.
- api/services/kb/s3.py: boto3 client, list_inbox/fetch/move helpers.
- api/routes/admin_kb.py: GET /admin/kb/inbox, POST /admin/kb/scan-inbox.
- The kind is derived from the folder path; title/identifier default to
the filename stem and can be refined by re-ingesting with metadata.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a searchable KB for חוק הביטוח הלאומי, תקנות הביטוח הלאומי, and חוזרי
הביטוח הלאומי. Hybrid search (pgvector cosine + tsvector/trgm) fused with
Reciprocal Rank Fusion, exposed to Shira as the `search_insurance_kb` tool.
- api/services/kb/: asyncpg pool, Voyage voyage-multilingual-2 client,
per-kind chunker (statute-by-section / circular header-aware), ingest
pipeline with supersession (old source → superseded_by), hybrid search.
- api/routes/admin_kb.py: POST /admin/kb/ingest (multipart), GET /admin/kb/stats.
- mcp_server/tools/legal_kb_tools.py: `search_insurance_kb` registered under
the `legal` toolset.
- pyproject.toml: +asyncpg, +python-docx, +boto3, +python-multipart.
Infra (external): pgvector/pgvector:pg16 on the shared PG, insurance_kb DB
with hnsw + gin indexes, MinIO bucket insurance-kb, KB_DATABASE_URL in
Infisical /espocrm, VOYAGE_API_KEY/VOYAGE_MODEL/KB_DATABASE_URL on the
shira-hermes Coolify app.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 4 Python tool registrations that call EspoCRM's SmartAssistant
executeTool endpoint with the DigitalSignature plugin tools:
- send_for_signature — send document for signing with auto-resolved signers
- check_signature_status — query status of signature requests
- remind_signers — send WhatsApp reminders to pending signers
- list_pending_signatures — show all awaiting signature requests
Tools are registered under the "signature" toolset, enabled by default.
Requires DigitalSignature extension v2.0.0+ on the EspoCRM side.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the CRM's readDocument returns success:false (scanned PDFs,
image-only DOCX, corrupted files), shira-hermes now:
1. Fetches the raw file bytes via new getDocumentBytes endpoint
2. Sends them to Claude Vision through ai-gateway:
- Images (PNG/JPG/TIFF/etc): direct vision call
- PDFs: render each page at 180 DPI via PyMuPDF, vision call per page
- DOCX: extract images from word/media/, vision call per image
New tool: read_document_ocr (explicit OCR request)
Existing tool read_document auto-falls-back to OCR on extraction failure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When EspoCRM returns documents as a flat list (legacy shape) instead of
the expected {totalFiles, folders, ...} dict, list_documents crashed with
"list object has no attribute 'get'". Now handles both shapes gracefully.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Build Docker image on push to main, push to Gitea Container Registry,
and trigger Coolify redeploy. Replaces Coolify's built-in Dockerfile build
with pre-built images for faster deploys and version management.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EspoCRM stores dates in UTC. Shira was displaying them raw (e.g., 08:00 UTC
instead of 11:00 Israel time) and storing user-provided local times as UTC.
- Add _to_local() in prompt_builder to convert UTC→Asia/Jerusalem for all
displayed dates (hearings, meetings, calls, tasks, notes)
- Update normalize_date() in _helpers to convert Israel local→UTC before
sending to EspoCRM
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hatchling couldn't find the package directory (expected shira_hermes/).
Added [tool.hatch.build.targets.wheel] packages to include api/ and
mcp_server/ directories.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Security:
- Path traversal: validate user_id, case_id, skill_name against safe regex
- Input sanitization in memory.py and skills.py file path construction
Correctness:
- ISR timezone: replace hardcoded UTC+3 with ZoneInfo("Asia/Jerusalem") for DST
- save_rule: use EspoCrmApiError.status_code instead of fragile string matching
- Background tasks: store references to prevent GC before completion
- Lock race: use dict.setdefault() for atomic lock creation
- Version: health.py now matches main.py (0.2.0)
- Skill names: normalized to lowercase for dedup consistency
Build:
- Dockerfile: install from pyproject.toml instead of hardcoded pip list
- Remove unused mcp>=1.0.0 dependency from pyproject.toml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The AssistantRule entity may not exist on some CRM instances. When save_rule
gets a 404, it now falls back to saving the rule in the local user profile
(update_user_profile). Also improved update_user_profile description so
Shira is more likely to use it for user preferences.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EspoCRM may send documents as a list instead of a dict with
totalFiles. Convert list to dict format to avoid AttributeError.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Logs the first 8 chars of expected vs provided API key on 401
to diagnose auth issues from EspoCRM.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
python:3.13-slim doesn't include curl or wget, which Coolify
needs for HTTP health checks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hatchling couldn't find the package directory. Since we run the app
directly with uvicorn (not as an installed library), install deps
explicitly in Dockerfile.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
chaim