Two minimal additions for the upcoming v0.1.11 client work:
- /kb/search hits now include chunk_index, so the client can call
/source/{id}/chunks?around=N&ctx=K to fetch the matched chunk plus K
chunks before/after as context. Required by the new search-preview
layout for text sources, which until now duplicated the matched chunk
in both panes.
- /source/{id}/chunks accepts around (chunk_index) + ctx (window radius
clamped to 0..10). When around is given, returns only the windowed
range; otherwise unchanged (returns all chunks of the source).
Refs Task Master #1
Single-shot search ranks the source whose title literally contains the
query terms way above everything else, so "מהי תקנה 37" returns 6 hits
from the תקנה 37 circular + 2 from the law and zero from ספר הליקויים
— even though the latter is the canonical medical reference for that
regulation. The ask agent already worked around this by issuing several
search_insurance_kb calls with different phrasings; that intelligence is
now also available to the public /kb/search endpoint.
Implementation:
- _retrieve_rrf extracts vector + lexical retrieval (RRF fused, no
rerank) into a reusable async helper. Each candidate now carries
chunk_id so callers can dedup when merging multiple sub-searches.
- _expand_query asks the configured Claude model (via ai-gateway) for
up to 3 alternative phrasings, with a 12s timeout. On any failure the
caller silently falls back to single-query mode.
- search() gains an `expand: bool` parameter. When true, runs the
original query plus all variants through _retrieve_rrf in parallel,
merges by chunk_id keeping the best RRF score, and reranks the union
against the ORIGINAL query so the final order reflects what the user
asked, not the variants.
- /kb/search defaults to expand=true.
Cost per call: 1 extra LLM round-trip (~1-2s on Sonnet) and 3 extra
Voyage embeds (~150ms each, parallel). Latency budget grows from ~700ms
to ~3s, acceptable for a search the user explicitly triggered.
Refs Task Master #1
_split_oversized slices at fixed char offsets and can cut a
\x00KB_PAGE:N\x00 sentinel in half — the broken prefix/suffix no longer
matches _PAGE_MARKER_RE so _strip_page_markers leaves the \x00 byte in
place. PostgreSQL's UTF-8 check then rejects the insert with
"invalid byte sequence for encoding UTF8: 0x00", and ingest of
ספר הליקויים fails on every scan. Belt-and-suspenders: strip all
remaining \x00 bytes from clean content before returning.
Refs Task Master #2
Commit 426b0d6 fixed page propagation through the Chunk reconstruction in
chunk(), but two downstream paths still collapsed every sub-chunk to the
parent's starting page:
1. _split_oversized operated on markerless content (since _as_dicts inside
chunk_statute/chunk_circular stripped markers before returning), so when
a single giant chunk was sliced into pieces — the common path for
ספר הליקויים / ספר המבחנים, which have no statute-style section markers —
every piece inherited c.page_number = parent's start page. 209/209 chunks
landed on page 1.
2. chunk_circular packed paragraphs into sub-chunks all tagged with
seg_page (the segment's opening page), with no attempt to track where
each sub-chunk actually begins within the original text. חוזר תקנה 37
(8 chunks) and חוזר נכות 1941 (4 chunks) — neither had header matches —
landed entirely on page 1.
Fixes:
- chunk_statute and chunk_circular now return list[Chunk] with markers
still embedded in content; _as_dicts is called once at the end of
chunk() so markers are stripped AFTER splitting.
- _split_oversized builds a local page map from markers in each parent
chunk's content, with a virtual (0, parent.page_number) entry so pieces
before the first real marker still inherit the correct page.
- chunk_circular tracks each paragraph's absolute offset in the original
text and computes each emitted sub-chunk's page_number from its first
paragraph's offset against the full-text page_map.
Verified on synthetic 3-page input: regulation path now emits pages 1,1,2,3
across 4 pieces (was 1,1,1,1); circular path emits 1,1,2,2,3,3 (was
1,1,1,1,1,1).
Refs Task Master #2
chunk() reconstructs Chunk objects before _split_oversized/_as_dicts and
was dropping page_number on the floor, so every chunk came out NULL
regardless of what chunk_statute computed. Also flip _as_dicts to prefer
the caller-supplied c.page_number (derived from the chunk's starting
offset) over any marker that survived inside c.content — the latter can
resolve to a later page when a section spans pages.
Verified: ספר הליקויים chunks now carry page_number matching the PDF
page where each section actually begins.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
To let the browse/ask UIs jump straight to the right PDF page:
- ingest.py: embed \x00KB_PAGE:<N>\x00 sentinels at the start of each
PDF page during _parse_pdf.
- chunker.py: _strip_page_markers removes the sentinels and records the
first page seen per chunk as page_number. Chunk dataclass + _as_dicts
propagate page_number through both chunk_statute and chunk_circular.
- ingest.py INSERT: new kb_chunk.page_number column (DB migration
applied manually).
- search.py SQL: SELECT c.page_number + s.id + s.original_path so the
rerank results carry them forward.
- legal_kb_tools.py: register_legal_kb_tools(sources_used=list) — the
tool appends {source_id, title, kind, identifier, heading_path,
section_ref, page_number, original_path} for each hit, deduped by
(source_id, page_number).
- agent_runner.py: new kb_sources_used kwarg threads the list through
to the tool registration.
- kb_public.py /kb/ask: collects sources_used during the run, filters
to PDF-only sources, and returns them alongside text.
Chunks stored before this change have page_number = NULL; PDFs need to
be re-ingested to populate them. The TXT law source is unaffected.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The browse tab in the EspoCRM KnowledgeBase extension needs the original
PDF (with columns, tables, labels) to render layout-heavy sources like
ספר הליקויים readably. The plain-text chunk view loses the structure
because the PDF itself places each word as a separate text block.
New route:
- GET /kb/source/{id}/pdf — resolves the source's `original_path`
(stored as `s3://insurance-kb/processed/<kind>/<file>.pdf`), streams
the bytes from MinIO via the existing kb_s3.fetch helper, and returns
them with Content-Type: application/pdf + inline Content-Disposition.
- 404 when the source doesn't exist.
- 409 when the source was ingested as plain text (e.g. the law from
Wikisource — its original_path ends in .txt, not .pdf).
- Response is chunked (64 KB) to avoid pinning a 2 MB buffer for every
request.
Also select `original_path` in /kb/source/{id}/chunks so the frontend
can decide whether to render the PDF viewer or the chunk list.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous block-based parser left each pymupdf block as its own line.
That's fine for prose but wrong for ספר הליקויים's cover and preface,
where each word is its own block — you got "ו אנ / מתכבד / ים" instead
of "ואנו מתכבדים". Merge blocks whose y-coordinates are within 4 pixels
into a single line, right-to-left, joined by spaces.
Content pages (where blocks already contain whole paragraphs) are
unaffected because a single block on one row stays as one row.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pymupdf's `get_text("text")` reads glyph runs in raw order, which on
multi-column Hebrew PDFs (ספר הליקויים, ספר המבחנים) interleaves column
headers, page numbers, form labels, and body paragraphs into jumbled
lines — words split to single characters on separate lines, paragraphs
lost. Switch to `get_text("blocks")` and sort each page's blocks by
(y-axis ascending, x-axis descending) so each paragraph stays intact
and the reading order is top-to-bottom, right-to-left as expected.
Side-effect: chunk content becomes readable in the browse view.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ספר הליקויים produced 698 chunks, most of them junk. A line like
"3 . . . . . . . . . . . . . . . . . . . . . . . . . . הוראות החוק" from
its table of contents was being treated as section 3. Phone numbers like
"02-6709701" became section 02. Both of these polluted the index and
broke the browse view's readability.
Tighten _RE_SECTION so it requires either the explicit prefix (סעיף /
תקנה / §) OR real text immediately after the terminator — a row of dots
or more digits disqualifies. The scraped חוק הביטוח הלאומי still chunks
correctly because the scraper emits "סעיף N. ..." with the prefix.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The KB UI's ask mode was hijacking into generic Israeli law. Asked
"מתי אפשר לבקש דיון מחדש לפי תקנה 37?" Shira answered with תקנה 37
of תקנות סדר הדין האזרחי (civil procedure) instead of תקנה 37 to the
National Insurance regulations — even though search_insurance_kb has
the correct source.
Append a short context block to the system prompt telling Shira that
every question from the KB UI is about ביטוח לאומי, that she must call
search_insurance_kb first and cite section_ref, and that she must say
so explicitly if no KB section matches.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three endpoints under /kb/*, same X-Api-Key auth as the admin and
SmartAssistant routes:
- POST /kb/search — hybrid search + rerank, returns ranked chunks as JSON.
No LLM in the path, so it's fast enough for a live UI.
- GET /kb/sources — list of active sources (law / regulation / circular)
with basic metadata, for the browse tab.
- GET /kb/source/{id}/chunks — all chunks of one source, ordered, so
the UI can show a document inline with its hierarchical headings.
- POST /kb/ask — full agent loop (same runner as SmartAssistant but with
allowed_toolsets=legal), for the ask-shira mode of the UI.
Dates are serialized to ISO strings so the frontend doesn't have to deal
with Python date objects.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the n8n worker went down and recovered, multiple queued cron
executions fired scan-inbox in parallel. Each one saw the same file in
s3://insurance-kb/inbox/, each one called ingest_source — and the DB
ended up with duplicate source rows with identical checksums created in
the same second.
Guard scan-inbox with a session-level pg_try_advisory_lock (key "KBSC").
Concurrent callers return {skipped: true} immediately. The lock is
released in a finally block so a crash mid-scan still frees the next
scheduled run.
No schema change; no change to single-caller behavior.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a rerank stage between the hybrid RRF fusion and the final top_k cut.
RRF is a heuristic over ranks — it can't tell which of two equally-ranked
candidates is actually more relevant. A cross-encoder scores each
(query, passage) pair directly and produces a true relevance order.
- voyage.rerank(query, documents, top_k) → list of {index, relevance_score}
sorted high→low. Defaults to the model in VOYAGE_RERANK_MODEL
(rerank-2.5, 32K tokens per pair, multilingual incl. Hebrew).
- search pulls top-30 from RRF, passes them to rerank, then returns the
reranker's top-k. Each returned item gets a rerank_score field.
- Fallback: if the rerank API errors (auth, 5xx, timeout), fall back to
the RRF ordering and log a warning — degraded precision beats outage.
- KB_RERANK_DISABLED=1 bypasses rerank for debugging.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sidecar JSON supplies published_at / effective_at as 'YYYY-MM-DD' strings.
asyncpg rejects strings for DATE columns even with '::date' casts in the
SQL, so ingest now parses them via datetime.date.fromisoformat before the
INSERT. Non-string inputs (date, datetime, None, empty) are handled too.
Before: ingest crashed on any sidecar with a date, leaving the source in
failed/ while sidecarless siblings succeeded.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each file in the inbox can now be accompanied by a `<filename>.meta.json`
sidecar that supplies identifier, published_at, effective_at, and
source_url. The scanner applies filename-derived defaults first and lets
the sidecar override individual fields.
- list_inbox / list_failed skip `.meta.json` so sidecars aren't ingested
as standalone documents.
- mark_processed, mark_failed, and requeue-failed move the sidecar
alongside its parent (best-effort).
- mark_failed now sanitizes the error string to ASCII before writing it
to S3 object metadata (HTTP header encoding).
- search SQL selects published_at/effective_at; the formatter shows the
full heading_path plus publication date so citations are unambiguous.
- scripts/kb_sidecar_template.json documents the expected shape.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-shot script that fetches the consolidated National Insurance Law from
Hebrew Wikisource and emits plain text with heading lines the chunker can
parse (פרק / סימן / סעיף).
Key quirks of the source markup:
- {{ח:קטע2}} carries the level name inside the title parameter ("פרק א׳:
…"), not in the template name. Emit the title verbatim and let the
chunker's existing prefix regex handle it.
- Named args (e.g. 'אחר=[פרק א׳]') must be filtered — the original
positional parser was treating them as positional titles and producing
gibberish "חלק אחר=[פרק א׳]" headings.
- No חלק level exists in this law — only פרק > סימן > סעיף.
Run as: python3 scripts/fetch_national_insurance_law.py <out_path>
Then upload the output .txt to s3://insurance-kb/inbox/law/ and invoke
POST /admin/kb/scan-inbox.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Before: chunk_statute produced heading_path = section_ref = "ס' 195".
After: heading_path walks the four Israeli-statute levels (part, chapter,
sub-chapter, section) and renders them as "חלק ט׳ — ביטוח נכות > פרק ב׳
— ועדות רפואיות > סימן א׳ > ס' 195". section_ref stays the leaf so
citations stay precise.
- New _RE_PART / _RE_CHAPTER / _RE_SUBCHAPTER regexes with Hebrew ordinal
support including multi-letter gershayim ("י״א", "ט״ז").
- _collect_markers emits all markers ordered by position; chunk_statute
walks them tracking active_part/chapter/subchapter and attaches the
full path to each section chunk.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous batching sent up to 128 inputs per request but didn't cap total
tokens. On large Hebrew statutes (ספר הליקויים, ספר המבחנים) a single
request exceeded Voyage's 120K-token-per-batch hard cap and failed with 400.
- voyage.embed now pre-computes an estimated token count per input (Hebrew
is ~0.55 tokens/char in voyage-multilingual-2) and emits multiple
requests so each stays under a 90K-token safety margin.
- chunker lowers the per-chunk ceiling to 1500 tokens and updates the
chars→tokens estimate to match the observed tokenizer behavior.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two bugs surfaced during Phase 1 validation on real circulars:
1. The chunker produced one giant chunk for large statutes (ספר הליקויים,
ספר המבחנים) that the section regex could not split. This exceeded
Voyage's 120K-token batch limit and the ingest failed. Add a hard
2000-token ceiling per chunk with a character-based fallback split that
preserves heading_path and section_ref.
2. ingest_source matched "existing source" by kind + COALESCE(identifier,
''), so any two sources with NULL identifier compared equal. The second
ingest would then supersede the first unrelated document. Match by
identifier only when one is provided; otherwise fall back to
(kind + title) and require the candidate to still be active.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves every object under failed/<kind>/ back to inbox/<kind>/ for retry
after a transient upstream failure (e.g. Voyage 401 before a key rotation).
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a MinIO-backed ingest flow: upload to s3://insurance-kb/inbox/<kind>/
via MinIO Console or any S3 client, then POST /admin/kb/scan-inbox to
process all pending files. Successful ingests move to processed/<kind>/,
failures move to failed/<kind>/ with the error stored as S3 metadata.
- api/services/kb/s3.py: boto3 client, list_inbox/fetch/move helpers.
- api/routes/admin_kb.py: GET /admin/kb/inbox, POST /admin/kb/scan-inbox.
- The kind is derived from the folder path; title/identifier default to
the filename stem and can be refined by re-ingesting with metadata.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a searchable KB for חוק הביטוח הלאומי, תקנות הביטוח הלאומי, and חוזרי
הביטוח הלאומי. Hybrid search (pgvector cosine + tsvector/trgm) fused with
Reciprocal Rank Fusion, exposed to Shira as the `search_insurance_kb` tool.
- api/services/kb/: asyncpg pool, Voyage voyage-multilingual-2 client,
per-kind chunker (statute-by-section / circular header-aware), ingest
pipeline with supersession (old source → superseded_by), hybrid search.
- api/routes/admin_kb.py: POST /admin/kb/ingest (multipart), GET /admin/kb/stats.
- mcp_server/tools/legal_kb_tools.py: `search_insurance_kb` registered under
the `legal` toolset.
- pyproject.toml: +asyncpg, +python-docx, +boto3, +python-multipart.
Infra (external): pgvector/pgvector:pg16 on the shared PG, insurance_kb DB
with hnsw + gin indexes, MinIO bucket insurance-kb, KB_DATABASE_URL in
Infisical /espocrm, VOYAGE_API_KEY/VOYAGE_MODEL/KB_DATABASE_URL on the
shira-hermes Coolify app.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 4 Python tool registrations that call EspoCRM's SmartAssistant
executeTool endpoint with the DigitalSignature plugin tools:
- send_for_signature — send document for signing with auto-resolved signers
- check_signature_status — query status of signature requests
- remind_signers — send WhatsApp reminders to pending signers
- list_pending_signatures — show all awaiting signature requests
Tools are registered under the "signature" toolset, enabled by default.
Requires DigitalSignature extension v2.0.0+ on the EspoCRM side.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the CRM's readDocument returns success:false (scanned PDFs,
image-only DOCX, corrupted files), shira-hermes now:
1. Fetches the raw file bytes via new getDocumentBytes endpoint
2. Sends them to Claude Vision through ai-gateway:
- Images (PNG/JPG/TIFF/etc): direct vision call
- PDFs: render each page at 180 DPI via PyMuPDF, vision call per page
- DOCX: extract images from word/media/, vision call per image
New tool: read_document_ocr (explicit OCR request)
Existing tool read_document auto-falls-back to OCR on extraction failure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When EspoCRM returns documents as a flat list (legacy shape) instead of
the expected {totalFiles, folders, ...} dict, list_documents crashed with
"list object has no attribute 'get'". Now handles both shapes gracefully.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Build Docker image on push to main, push to Gitea Container Registry,
and trigger Coolify redeploy. Replaces Coolify's built-in Dockerfile build
with pre-built images for faster deploys and version management.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EspoCRM stores dates in UTC. Shira was displaying them raw (e.g., 08:00 UTC
instead of 11:00 Israel time) and storing user-provided local times as UTC.
- Add _to_local() in prompt_builder to convert UTC→Asia/Jerusalem for all
displayed dates (hearings, meetings, calls, tasks, notes)
- Update normalize_date() in _helpers to convert Israel local→UTC before
sending to EspoCRM
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hatchling couldn't find the package directory (expected shira_hermes/).
Added [tool.hatch.build.targets.wheel] packages to include api/ and
mcp_server/ directories.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Security:
- Path traversal: validate user_id, case_id, skill_name against safe regex
- Input sanitization in memory.py and skills.py file path construction
Correctness:
- ISR timezone: replace hardcoded UTC+3 with ZoneInfo("Asia/Jerusalem") for DST
- save_rule: use EspoCrmApiError.status_code instead of fragile string matching
- Background tasks: store references to prevent GC before completion
- Lock race: use dict.setdefault() for atomic lock creation
- Version: health.py now matches main.py (0.2.0)
- Skill names: normalized to lowercase for dedup consistency
Build:
- Dockerfile: install from pyproject.toml instead of hardcoded pip list
- Remove unused mcp>=1.0.0 dependency from pyproject.toml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The AssistantRule entity may not exist on some CRM instances. When save_rule
gets a 404, it now falls back to saving the rule in the local user profile
(update_user_profile). Also improved update_user_profile description so
Shira is more likely to use it for user preferences.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EspoCRM may send documents as a list instead of a dict with
totalFiles. Convert list to dict format to avoid AttributeError.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Logs the first 8 chars of expected vs provided API key on 401
to diagnose auth issues from EspoCRM.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
python:3.13-slim doesn't include curl or wget, which Coolify
needs for HTTP health checks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hatchling couldn't find the package directory. Since we run the app
directly with uvicorn (not as an installed library), install deps
explicitly in Dockerfile.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
chaim