- pyproject.toml -> 0.3.0 (significant changes since 0.2.0:
full Phase 1-7 KB stack, classifier+labels, batch-upload UI,
pending-review panel, bucket rename to legal-kb)
- api/__init__.py reads version from pyproject at import time so
/api/health and OpenAPI both reflect reality without a second
hardcoded copy
- api/services/kb/voyage.py: _post_with_retry wraps embed/rerank
POSTs with exponential backoff on 429+5xx (max 5 retries, base
2s, capped 30s, +25% jitter, honors Retry-After header). Dev and
prod share a single VOYAGE_API_KEY today, so a simultaneous bulk
re-ingest on both used to fail half the calls; now the burst is
absorbed in seconds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The bucket name was a holdover from the original single-domain
prototype (Israeli National Insurance only). With multi-topic support
the same bucket now hosts files for ביטוח לאומי, דיני עבודה, פלילי,
and any future legal domains — separated by `inbox/<topic_slug>/`
prefix, not by bucket. Generic `legal-kb` name reflects that.
Only the default in `_bucket()` changes; deployments that already set
`KB_S3_BUCKET` are unaffected. Dev was migrated to `legal-kb` in the
same operation (`mc mirror` of all 221 objects, env var update,
UPDATE on `kb_source.original_path` rewriting `s3://insurance-kb/` →
`s3://legal-kb/`). Prod MinIO has the empty `legal-kb` bucket ready
for shira-hermes prod cutover.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two production-blocker fixes for the v0.8.0 batch upload UI, plus a
small endpoint to recover review state after a hard refresh.
classifier.py:
- timeout 45s → 90s and concurrency 5 → 2. ai-gateway proxies a
single Claude OAuth session so requests effectively serialize at
the gateway; with concurrency=5, the 4th and 5th files in a batch
of 5 timed out from queue waits alone (observed 2026-04-25 on
user upload of 5 circulars: jobs 10 and 13 returned empty).
- labels schema/prompt: arrays of strings via ai-gateway came back
as [{}, {}, {}] (proxy mangled items.type). Switched the param
to a comma-separated string with explicit format guidance, then
split in _normalize. _normalize is defensive — accepts both
string and list[str|dict] shapes so older callers don't break.
- Sharpened the prompt distinction between subject (free text on
the source) and labels (navigation tags), since identical
examples were causing the model to emit one and skip the other.
ingest_jobs.list_pending_batches + admin_kb GET /admin/kb/batches:
the review screen lives in JS RAM (_activeBatch). A hard refresh
wipes it, leaving uploaded jobs orphaned in awaiting_review with
no UI path back. The new endpoint groups jobs by batch_id with
status counters; the EspoCRM 'ממתינים לאישור' panel uses it to
let users resume any pending review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 6 of the multi-topic KB refactor (backend half). Subsumes the
original Task #19 design discussion (subject vs labels). EspoCRM
extension UI ships separately.
Migrations (4 new, all idempotent, transaction-wrapped):
004_kind_tool — adds 'tool' to kb_source/kb_ingest_job CHECK
005_source_description — kb_source.description + ai_classified_at
006_labels — kb_label + kb_source_label (M:N), GRANTs
007_ingest_classifier — awaiting_review status + processing_stage
+ ai_suggestions JSONB + batch_id
Two-phase ingest (api/services/kb/ingest_jobs.py):
queued → processing(stage=classifying)
→ awaiting_review ← user reviews AI output
→ processing(stage=embedding) ← after user commits
→ done | failed
process_classify_stage() pulls the file, runs parse_first_pages
(3-page text extract), calls classifier.classify (Claude tool-call
via ai-gateway, 45s timeout, 5-concurrent semaphore, fail-soft on
any error → empty suggestions), writes ai_suggestions JSONB,
transitions to awaiting_review.
commit_job() resolves user-confirmed labels (existing by slug,
new ones via slugify+ON CONFLICT DO NOTHING), transitions to
processing(embedding), schedules process_embed_stage.
process_embed_stage() runs the legacy ingest_source path with
user-edited metadata + summary as kb_source.description, then
apply_to_source for labels.
discard_job() removes a file from a batch (status=failed,
S3 deleted).
list_jobs_by_batch() — single-roundtrip review-screen load.
Labels (NEW api/services/kb/labels.py):
Hebrew → ASCII slugify (letter-by-letter map, no external dep);
CRUD; lookup_or_create_batch (race-safe); apply_to_source +
replace_for_source maintain usage_count; merge race-safely
(UPDATE assignments + ON CONFLICT DO NOTHING).
Classifier (NEW api/services/kb/classifier.py):
AsyncOpenAI to ai-gateway, Sonnet, OpenAI-style tool calling for
guaranteed JSON shape, Hebrew system prompt with 5 kind values,
citation format examples, "do not invent" rule, summary length
bounds 80-200 chars.
Routes (api/routes/admin_kb.py — 8 new on top of existing 11):
POST /admin/kb/upload-batch (multi-file, AI classify)
GET /admin/kb/batch/{batch_id} (review-screen load)
POST /admin/kb/jobs/{id}/commit (user confirms metadata)
POST /admin/kb/jobs/{id}/discard (user removes from batch)
GET /admin/kb/labels (typeahead)
POST /admin/kb/labels (explicit create)
POST /admin/kb/labels/{id}/merge (admin cleanup)
DELETE /admin/kb/labels/{id} (only if usage=0)
Public /kb/search gains optional label_id filter.
admin_sources.list_sources LEFT JOINs labels into each row's
output. update_source accepts label_ids to replace the set.
End-to-end smoke test passed: synthetic Hebrew circular →
upload-batch → background classify → awaiting_review with kind,
title, subject, summary, identifier, published_at all populated
correctly.
Refs Task Master #20 (espocrm-extensions/KnowledgeBase v0.8.0)
Subsumes: Task #19 (labels for sub-topics)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backs the Phase 4 topic-management table in the KnowledgeBase EspoCRM
extension ("ניהול" tab → "נושאים" panel). Four new admin endpoints:
GET /admin/kb/topics — list ALL topics including
is_active=false ones, with
source_count joined from kb_source
POST /admin/kb/topics — create. Validates slug shape
(^[a-z][a-z0-9-]{1,62}$) and
uniqueness — clean 400 on duplicate
rather than leaking UniqueViolation
PUT /admin/kb/topics/{id} — patch name, description,
system_prompt_addendum, is_active.
slug is intentionally NOT editable —
the S3 layout depends on it
(inbox/<topic_slug>/<kind>/...)
DELETE /admin/kb/topics/{id} — hard-delete; refuses with 409 if
any sources still belong to the
topic. Soft-delete (is_active=false)
is the normal "remove from dropdown"
path
Refs Task Master #15 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backs the Phase 3 sources table in the KnowledgeBase EspoCRM extension
("ניהול" tab). Four new admin endpoints:
GET /admin/kb/sources — list with chunk_count and
last-ingest status (LATERAL
join against kb_ingest_job),
filterable by topic_id + kind
PUT /admin/kb/sources/{id} — patch the user-editable
metadata only (title,
identifier, source_url,
published_at, effective_at);
structural fields (kind,
topic_id, checksum) are not
accepted
DELETE /admin/kb/sources/{id} — hard delete (kb_chunk cascades
via FK; the S3 object at
original_path is best-effort
deleted)
POST /admin/kb/sources/{id}/reingest — queues a kb_ingest_job tied
to the existing source_id;
background task replaces
chunks in-place, preserves
the source row and its
hand-edited metadata, and
updates the checksum
The reingest path differs from ingest_source: ingest_source creates a
new kb_source row and supersedes the previous version. Reingest keeps
the same id (so cached chunk_index references in the browser stay
valid), wipes its kb_chunk rows in a single transaction, and inserts
fresh ones from the same file.
Refs Task Master #14 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Court rulings are a distinct content type — long discussion / reasoning,
no enumerated statutory hierarchy, case identifier instead of statute id —
so they get their own kind alongside law/regulation/circular rather than
being shoehorned into one of the existing three.
Migration 003 alters the CHECK constraints on kb_source.kind and
kb_ingest_job.kind to include 'caselaw'. The chunker uses the
circulars path for caselaw too (paragraph + size split, no statutory
section regex); the case identifier is captured at upload time in
kb_source.identifier.
Updates Literal types in ingest_source, admin_kb upload, and the
kb_public SearchRequest. _KINDS in s3.py grows to four so failed/
processed/inbox listings include caselaw subdirs.
Refs Task Master #18 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 2 of the multi-topic KB refactor. Adds three admin endpoints that
back the new "ניהול" tab in the KnowledgeBase EspoCRM extension:
POST /admin/kb/upload — multipart, queues a kb_ingest_job and
fires asyncio.create_task to process
parse → chunk → embed in the
background. Returns {job_id, status}
immediately so the browser doesn't
block on slow PDFs.
GET /admin/kb/jobs — recent jobs, optionally filtered by
topic_id / status / limit.
GET /admin/kb/jobs/{id} — single-job detail for the polling UI.
Migration 002 adds kb_ingest_job (queued/processing/done/failed) with
indexes on (status, created_at) and (topic_id, created_at).
ingest_source now accepts topic_id (NULL → defaults to 1 for back-compat
with the existing scan-inbox cron path) and writes it to kb_source.
S3 layout for new uploads is topic-aware: inbox/<topic_slug>/<kind>/...
The legacy /scan-inbox path on inbox/<kind>/ is unchanged.
Refs Task Master #13 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 of the multi-domain refactor. Adds a kb_topic table seeded with
the existing 'ביטוח לאומי' domain (id=1) and pins all 6 existing
kb_source rows to it, so search/ask can be scoped per domain without
disturbing the current single-topic UX.
- scripts/migrations/001_topics.sql: idempotent migration. Creates
kb_topic, seeds national-insurance with the system_prompt_addendum
copied verbatim from kb_public.py's hardcoded text, adds nullable
kb_source.topic_id, backfills it to 1, then SET NOT NULL. GRANTs to
shira_kb mirror its existing kb_source privileges. Wrapped in BEGIN
/COMMIT with a sanity-check that raises if the backfill is incomplete.
- api/services/kb/topics.py: list_topics / get_topic / resolve_topic_id.
resolve_topic_id raises ValueError on unknown ids so the route layer
can return 400 instead of silently mixing domains.
- api/services/kb/search.py: search() + _retrieve_rrf accept topic_id;
the SQL adds AND s.topic_id = $N when present. _expand_query is
parametrized by topic name (was hardcoded "הביטוח הלאומי").
- api/routes/kb_public.py: new GET /kb/topics. /kb/search /kb/sources
/kb/ask /kb/ask/stream all accept topic_id and call resolve_topic_id.
_build_ask_runner_context loads system_prompt_addendum from the DB
instead of the hardcoded insurance string.
- mcp_server/tools/legal_kb_tools.py: tool renamed
search_insurance_kb → search_legal_kb, takes topic_id at registration
so each conversation is pinned to its caller-chosen domain. Tool
description interpolates the topic name.
- api/services/agent_runner.py: kb_topic_id + kb_topic_name flow
through to register_legal_kb_tools. Hebrew progress labels updated.
Refs Task Master #12 (espocrm-extensions/KnowledgeBase).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds POST /kb/ask/stream that returns text/event-stream. The agent
runs in a fire-and-forget task; an on_event callback wired into
AgentRunner.run pushes {type, label, ...} events into an asyncio.Queue,
and the response generator consumes them as SSE data: lines.
Event types and Hebrew labels:
- thinking — before each LLM call ("קוראת את השאלה" / "ממשיכה לחקור")
- tool_start — before each tool. For search_insurance_kb the label
includes the actual query string Shira chose, so the user can see
WHAT she's searching for ("מחפשת בבסיס הידע: '...'"). Other tools
fall back to a generic Hebrew label keyed off the function name.
- tool_done / tool_error — after each tool. For search_insurance_kb
the label echoes the first non-empty line of the tool result (which
the legal_kb tool formats as "## נמצאו N קטעים").
- writing — when the model returns text without tool calls.
- answer — final event with {text, sources, conversationId}, mirrors
the non-streaming /kb/ask response shape.
- error — runner exception; client should close the EventSource.
Heartbeats: when the queue is idle for >15s the generator yields a
": keep-alive" comment line (ignored by EventSource) so any proxy in
the path doesn't drop the connection during long agent thinks.
The original POST /kb/ask is unchanged — the streaming endpoint is
additive. Both share _build_ask_runner_context + _build_ask_messages
helpers extracted from the original handler.
Refs Task Master #1
Two minimal additions for the upcoming v0.1.11 client work:
- /kb/search hits now include chunk_index, so the client can call
/source/{id}/chunks?around=N&ctx=K to fetch the matched chunk plus K
chunks before/after as context. Required by the new search-preview
layout for text sources, which until now duplicated the matched chunk
in both panes.
- /source/{id}/chunks accepts around (chunk_index) + ctx (window radius
clamped to 0..10). When around is given, returns only the windowed
range; otherwise unchanged (returns all chunks of the source).
Refs Task Master #1
Single-shot search ranks the source whose title literally contains the
query terms way above everything else, so "מהי תקנה 37" returns 6 hits
from the תקנה 37 circular + 2 from the law and zero from ספר הליקויים
— even though the latter is the canonical medical reference for that
regulation. The ask agent already worked around this by issuing several
search_insurance_kb calls with different phrasings; that intelligence is
now also available to the public /kb/search endpoint.
Implementation:
- _retrieve_rrf extracts vector + lexical retrieval (RRF fused, no
rerank) into a reusable async helper. Each candidate now carries
chunk_id so callers can dedup when merging multiple sub-searches.
- _expand_query asks the configured Claude model (via ai-gateway) for
up to 3 alternative phrasings, with a 12s timeout. On any failure the
caller silently falls back to single-query mode.
- search() gains an `expand: bool` parameter. When true, runs the
original query plus all variants through _retrieve_rrf in parallel,
merges by chunk_id keeping the best RRF score, and reranks the union
against the ORIGINAL query so the final order reflects what the user
asked, not the variants.
- /kb/search defaults to expand=true.
Cost per call: 1 extra LLM round-trip (~1-2s on Sonnet) and 3 extra
Voyage embeds (~150ms each, parallel). Latency budget grows from ~700ms
to ~3s, acceptable for a search the user explicitly triggered.
Refs Task Master #1
_split_oversized slices at fixed char offsets and can cut a
\x00KB_PAGE:N\x00 sentinel in half — the broken prefix/suffix no longer
matches _PAGE_MARKER_RE so _strip_page_markers leaves the \x00 byte in
place. PostgreSQL's UTF-8 check then rejects the insert with
"invalid byte sequence for encoding UTF8: 0x00", and ingest of
ספר הליקויים fails on every scan. Belt-and-suspenders: strip all
remaining \x00 bytes from clean content before returning.
Refs Task Master #2
Commit 426b0d6 fixed page propagation through the Chunk reconstruction in
chunk(), but two downstream paths still collapsed every sub-chunk to the
parent's starting page:
1. _split_oversized operated on markerless content (since _as_dicts inside
chunk_statute/chunk_circular stripped markers before returning), so when
a single giant chunk was sliced into pieces — the common path for
ספר הליקויים / ספר המבחנים, which have no statute-style section markers —
every piece inherited c.page_number = parent's start page. 209/209 chunks
landed on page 1.
2. chunk_circular packed paragraphs into sub-chunks all tagged with
seg_page (the segment's opening page), with no attempt to track where
each sub-chunk actually begins within the original text. חוזר תקנה 37
(8 chunks) and חוזר נכות 1941 (4 chunks) — neither had header matches —
landed entirely on page 1.
Fixes:
- chunk_statute and chunk_circular now return list[Chunk] with markers
still embedded in content; _as_dicts is called once at the end of
chunk() so markers are stripped AFTER splitting.
- _split_oversized builds a local page map from markers in each parent
chunk's content, with a virtual (0, parent.page_number) entry so pieces
before the first real marker still inherit the correct page.
- chunk_circular tracks each paragraph's absolute offset in the original
text and computes each emitted sub-chunk's page_number from its first
paragraph's offset against the full-text page_map.
Verified on synthetic 3-page input: regulation path now emits pages 1,1,2,3
across 4 pieces (was 1,1,1,1); circular path emits 1,1,2,2,3,3 (was
1,1,1,1,1,1).
Refs Task Master #2
chunk() reconstructs Chunk objects before _split_oversized/_as_dicts and
was dropping page_number on the floor, so every chunk came out NULL
regardless of what chunk_statute computed. Also flip _as_dicts to prefer
the caller-supplied c.page_number (derived from the chunk's starting
offset) over any marker that survived inside c.content — the latter can
resolve to a later page when a section spans pages.
Verified: ספר הליקויים chunks now carry page_number matching the PDF
page where each section actually begins.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
To let the browse/ask UIs jump straight to the right PDF page:
- ingest.py: embed \x00KB_PAGE:<N>\x00 sentinels at the start of each
PDF page during _parse_pdf.
- chunker.py: _strip_page_markers removes the sentinels and records the
first page seen per chunk as page_number. Chunk dataclass + _as_dicts
propagate page_number through both chunk_statute and chunk_circular.
- ingest.py INSERT: new kb_chunk.page_number column (DB migration
applied manually).
- search.py SQL: SELECT c.page_number + s.id + s.original_path so the
rerank results carry them forward.
- legal_kb_tools.py: register_legal_kb_tools(sources_used=list) — the
tool appends {source_id, title, kind, identifier, heading_path,
section_ref, page_number, original_path} for each hit, deduped by
(source_id, page_number).
- agent_runner.py: new kb_sources_used kwarg threads the list through
to the tool registration.
- kb_public.py /kb/ask: collects sources_used during the run, filters
to PDF-only sources, and returns them alongside text.
Chunks stored before this change have page_number = NULL; PDFs need to
be re-ingested to populate them. The TXT law source is unaffected.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The browse tab in the EspoCRM KnowledgeBase extension needs the original
PDF (with columns, tables, labels) to render layout-heavy sources like
ספר הליקויים readably. The plain-text chunk view loses the structure
because the PDF itself places each word as a separate text block.
New route:
- GET /kb/source/{id}/pdf — resolves the source's `original_path`
(stored as `s3://insurance-kb/processed/<kind>/<file>.pdf`), streams
the bytes from MinIO via the existing kb_s3.fetch helper, and returns
them with Content-Type: application/pdf + inline Content-Disposition.
- 404 when the source doesn't exist.
- 409 when the source was ingested as plain text (e.g. the law from
Wikisource — its original_path ends in .txt, not .pdf).
- Response is chunked (64 KB) to avoid pinning a 2 MB buffer for every
request.
Also select `original_path` in /kb/source/{id}/chunks so the frontend
can decide whether to render the PDF viewer or the chunk list.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous block-based parser left each pymupdf block as its own line.
That's fine for prose but wrong for ספר הליקויים's cover and preface,
where each word is its own block — you got "ו אנ / מתכבד / ים" instead
of "ואנו מתכבדים". Merge blocks whose y-coordinates are within 4 pixels
into a single line, right-to-left, joined by spaces.
Content pages (where blocks already contain whole paragraphs) are
unaffected because a single block on one row stays as one row.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pymupdf's `get_text("text")` reads glyph runs in raw order, which on
multi-column Hebrew PDFs (ספר הליקויים, ספר המבחנים) interleaves column
headers, page numbers, form labels, and body paragraphs into jumbled
lines — words split to single characters on separate lines, paragraphs
lost. Switch to `get_text("blocks")` and sort each page's blocks by
(y-axis ascending, x-axis descending) so each paragraph stays intact
and the reading order is top-to-bottom, right-to-left as expected.
Side-effect: chunk content becomes readable in the browse view.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ספר הליקויים produced 698 chunks, most of them junk. A line like
"3 . . . . . . . . . . . . . . . . . . . . . . . . . . הוראות החוק" from
its table of contents was being treated as section 3. Phone numbers like
"02-6709701" became section 02. Both of these polluted the index and
broke the browse view's readability.
Tighten _RE_SECTION so it requires either the explicit prefix (סעיף /
תקנה / §) OR real text immediately after the terminator — a row of dots
or more digits disqualifies. The scraped חוק הביטוח הלאומי still chunks
correctly because the scraper emits "סעיף N. ..." with the prefix.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The KB UI's ask mode was hijacking into generic Israeli law. Asked
"מתי אפשר לבקש דיון מחדש לפי תקנה 37?" Shira answered with תקנה 37
of תקנות סדר הדין האזרחי (civil procedure) instead of תקנה 37 to the
National Insurance regulations — even though search_insurance_kb has
the correct source.
Append a short context block to the system prompt telling Shira that
every question from the KB UI is about ביטוח לאומי, that she must call
search_insurance_kb first and cite section_ref, and that she must say
so explicitly if no KB section matches.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three endpoints under /kb/*, same X-Api-Key auth as the admin and
SmartAssistant routes:
- POST /kb/search — hybrid search + rerank, returns ranked chunks as JSON.
No LLM in the path, so it's fast enough for a live UI.
- GET /kb/sources — list of active sources (law / regulation / circular)
with basic metadata, for the browse tab.
- GET /kb/source/{id}/chunks — all chunks of one source, ordered, so
the UI can show a document inline with its hierarchical headings.
- POST /kb/ask — full agent loop (same runner as SmartAssistant but with
allowed_toolsets=legal), for the ask-shira mode of the UI.
Dates are serialized to ISO strings so the frontend doesn't have to deal
with Python date objects.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the n8n worker went down and recovered, multiple queued cron
executions fired scan-inbox in parallel. Each one saw the same file in
s3://insurance-kb/inbox/, each one called ingest_source — and the DB
ended up with duplicate source rows with identical checksums created in
the same second.
Guard scan-inbox with a session-level pg_try_advisory_lock (key "KBSC").
Concurrent callers return {skipped: true} immediately. The lock is
released in a finally block so a crash mid-scan still frees the next
scheduled run.
No schema change; no change to single-caller behavior.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a rerank stage between the hybrid RRF fusion and the final top_k cut.
RRF is a heuristic over ranks — it can't tell which of two equally-ranked
candidates is actually more relevant. A cross-encoder scores each
(query, passage) pair directly and produces a true relevance order.
- voyage.rerank(query, documents, top_k) → list of {index, relevance_score}
sorted high→low. Defaults to the model in VOYAGE_RERANK_MODEL
(rerank-2.5, 32K tokens per pair, multilingual incl. Hebrew).
- search pulls top-30 from RRF, passes them to rerank, then returns the
reranker's top-k. Each returned item gets a rerank_score field.
- Fallback: if the rerank API errors (auth, 5xx, timeout), fall back to
the RRF ordering and log a warning — degraded precision beats outage.
- KB_RERANK_DISABLED=1 bypasses rerank for debugging.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sidecar JSON supplies published_at / effective_at as 'YYYY-MM-DD' strings.
asyncpg rejects strings for DATE columns even with '::date' casts in the
SQL, so ingest now parses them via datetime.date.fromisoformat before the
INSERT. Non-string inputs (date, datetime, None, empty) are handled too.
Before: ingest crashed on any sidecar with a date, leaving the source in
failed/ while sidecarless siblings succeeded.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each file in the inbox can now be accompanied by a `<filename>.meta.json`
sidecar that supplies identifier, published_at, effective_at, and
source_url. The scanner applies filename-derived defaults first and lets
the sidecar override individual fields.
- list_inbox / list_failed skip `.meta.json` so sidecars aren't ingested
as standalone documents.
- mark_processed, mark_failed, and requeue-failed move the sidecar
alongside its parent (best-effort).
- mark_failed now sanitizes the error string to ASCII before writing it
to S3 object metadata (HTTP header encoding).
- search SQL selects published_at/effective_at; the formatter shows the
full heading_path plus publication date so citations are unambiguous.
- scripts/kb_sidecar_template.json documents the expected shape.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Before: chunk_statute produced heading_path = section_ref = "ס' 195".
After: heading_path walks the four Israeli-statute levels (part, chapter,
sub-chapter, section) and renders them as "חלק ט׳ — ביטוח נכות > פרק ב׳
— ועדות רפואיות > סימן א׳ > ס' 195". section_ref stays the leaf so
citations stay precise.
- New _RE_PART / _RE_CHAPTER / _RE_SUBCHAPTER regexes with Hebrew ordinal
support including multi-letter gershayim ("י״א", "ט״ז").
- _collect_markers emits all markers ordered by position; chunk_statute
walks them tracking active_part/chapter/subchapter and attaches the
full path to each section chunk.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous batching sent up to 128 inputs per request but didn't cap total
tokens. On large Hebrew statutes (ספר הליקויים, ספר המבחנים) a single
request exceeded Voyage's 120K-token-per-batch hard cap and failed with 400.
- voyage.embed now pre-computes an estimated token count per input (Hebrew
is ~0.55 tokens/char in voyage-multilingual-2) and emits multiple
requests so each stays under a 90K-token safety margin.
- chunker lowers the per-chunk ceiling to 1500 tokens and updates the
chars→tokens estimate to match the observed tokenizer behavior.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two bugs surfaced during Phase 1 validation on real circulars:
1. The chunker produced one giant chunk for large statutes (ספר הליקויים,
ספר המבחנים) that the section regex could not split. This exceeded
Voyage's 120K-token batch limit and the ingest failed. Add a hard
2000-token ceiling per chunk with a character-based fallback split that
preserves heading_path and section_ref.
2. ingest_source matched "existing source" by kind + COALESCE(identifier,
''), so any two sources with NULL identifier compared equal. The second
ingest would then supersede the first unrelated document. Match by
identifier only when one is provided; otherwise fall back to
(kind + title) and require the candidate to still be active.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves every object under failed/<kind>/ back to inbox/<kind>/ for retry
after a transient upstream failure (e.g. Voyage 401 before a key rotation).
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a MinIO-backed ingest flow: upload to s3://insurance-kb/inbox/<kind>/
via MinIO Console or any S3 client, then POST /admin/kb/scan-inbox to
process all pending files. Successful ingests move to processed/<kind>/,
failures move to failed/<kind>/ with the error stored as S3 metadata.
- api/services/kb/s3.py: boto3 client, list_inbox/fetch/move helpers.
- api/routes/admin_kb.py: GET /admin/kb/inbox, POST /admin/kb/scan-inbox.
- The kind is derived from the folder path; title/identifier default to
the filename stem and can be refined by re-ingesting with metadata.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a searchable KB for חוק הביטוח הלאומי, תקנות הביטוח הלאומי, and חוזרי
הביטוח הלאומי. Hybrid search (pgvector cosine + tsvector/trgm) fused with
Reciprocal Rank Fusion, exposed to Shira as the `search_insurance_kb` tool.
- api/services/kb/: asyncpg pool, Voyage voyage-multilingual-2 client,
per-kind chunker (statute-by-section / circular header-aware), ingest
pipeline with supersession (old source → superseded_by), hybrid search.
- api/routes/admin_kb.py: POST /admin/kb/ingest (multipart), GET /admin/kb/stats.
- mcp_server/tools/legal_kb_tools.py: `search_insurance_kb` registered under
the `legal` toolset.
- pyproject.toml: +asyncpg, +python-docx, +boto3, +python-multipart.
Infra (external): pgvector/pgvector:pg16 on the shared PG, insurance_kb DB
with hnsw + gin indexes, MinIO bucket insurance-kb, KB_DATABASE_URL in
Infisical /espocrm, VOYAGE_API_KEY/VOYAGE_MODEL/KB_DATABASE_URL on the
shira-hermes Coolify app.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 4 Python tool registrations that call EspoCRM's SmartAssistant
executeTool endpoint with the DigitalSignature plugin tools:
- send_for_signature — send document for signing with auto-resolved signers
- check_signature_status — query status of signature requests
- remind_signers — send WhatsApp reminders to pending signers
- list_pending_signatures — show all awaiting signature requests
Tools are registered under the "signature" toolset, enabled by default.
Requires DigitalSignature extension v2.0.0+ on the EspoCRM side.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the CRM's readDocument returns success:false (scanned PDFs,
image-only DOCX, corrupted files), shira-hermes now:
1. Fetches the raw file bytes via new getDocumentBytes endpoint
2. Sends them to Claude Vision through ai-gateway:
- Images (PNG/JPG/TIFF/etc): direct vision call
- PDFs: render each page at 180 DPI via PyMuPDF, vision call per page
- DOCX: extract images from word/media/, vision call per image
New tool: read_document_ocr (explicit OCR request)
Existing tool read_document auto-falls-back to OCR on extraction failure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EspoCRM stores dates in UTC. Shira was displaying them raw (e.g., 08:00 UTC
instead of 11:00 Israel time) and storing user-provided local times as UTC.
- Add _to_local() in prompt_builder to convert UTC→Asia/Jerusalem for all
displayed dates (hearings, meetings, calls, tasks, notes)
- Update normalize_date() in _helpers to convert Israel local→UTC before
sending to EspoCRM
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Security:
- Path traversal: validate user_id, case_id, skill_name against safe regex
- Input sanitization in memory.py and skills.py file path construction
Correctness:
- ISR timezone: replace hardcoded UTC+3 with ZoneInfo("Asia/Jerusalem") for DST
- save_rule: use EspoCrmApiError.status_code instead of fragile string matching
- Background tasks: store references to prevent GC before completion
- Lock race: use dict.setdefault() for atomic lock creation
- Version: health.py now matches main.py (0.2.0)
- Skill names: normalized to lowercase for dedup consistency
Build:
- Dockerfile: install from pyproject.toml instead of hardcoded pip list
- Remove unused mcp>=1.0.0 dependency from pyproject.toml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The AssistantRule entity may not exist on some CRM instances. When save_rule
gets a 404, it now falls back to saving the rule in the local user profile
(update_user_profile). Also improved update_user_profile description so
Shira is more likely to use it for user preferences.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EspoCRM may send documents as a list instead of a dict with
totalFiles. Convert list to dict format to avoid AttributeError.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Logs the first 8 chars of expected vs provided API key on 401
to diagnose auth issues from EspoCRM.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
chaim