Shira now supplies a subject, never a filename or numbering. The TEMPLATE / FREE-DOC
/ FOLDER guidance in prompt_builder instructs the model to pass title / documentSubject
as a short Hebrew subject (no date, case number, contact, extension or path separators),
and to keep the final filename of write_document_to_folder clean. Server-side enforcement
lives in SmartAssistant CaseFolderManager (rule N1).
Refs Task Master #10
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After SmartAssistant 2.10.0 introduced UserProfile and AssistantSkill
entities, the Python side kept writing only to the filesystem. So an
admin browsing the EspoCRM UI saw nothing — but the system prompt did
pull from CRM via context['userProfile'] / context['availableSkills'].
Result: edits made by Shira (via update_user_profile / skill_manage tool)
were invisible to admins; edits made by admins took effect immediately
because the prompt fetched from DB. Now the two sources agree.
* memory.py — added _sync_user_profile_to_crm() that upserts a UserProfile
row whenever update_user_profile writes the file. Best-effort with
logging; HTTP failures don't break the tool.
* skills.py — added _sync_skill_to_crm() that upserts an AssistantSkill
row from the SKILL.md frontmatter (description, triggers, version) plus
the full body in content. Called after every save_skill in skill_manage.
Read path is unchanged: the EspoCRM CaseContextBuilder / SmartAssistantService
populate context fields, and the Python prompt_builder consumes them with
in-code defaults as a fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three changes paired with SmartAssistant 2.10.0:
1. prompt_builder.py — every prompt section (tool_rules, writing_style,
legal_assistance, personality_case, personality_office, other_rules_case,
other_rules_office) now reads from context['assistantPrompts'][key]
with the existing string as a fallback default. Admins editing
AssistantPrompt rows in the EspoCRM UI immediately affect the next
conversation. A missing/inactive row falls back to the in-code default
so deleting everything in the UI never bricks Shira.
Also: the user-profile section now prefers context['userProfile']
(from the new UserProfile entity) over the explicit `user_profile`
parameter, completing the move from /opt/data/profiles/*.md to the CRM.
2. crm_tools.py:save_memory — stops POSTing directly to /Note with a
'🧠 category:' prefix. Routes through SmartAssistant/action/executeTool
so the PHP saveMemory handler actually creates a CaseMemory row.
Cause of yesterday's prod issue: CaseMemory table was empty even
though Shira was "saving" to it.
3. scripts/migrate_legacy_memory_notes.py — one-shot migration that
sweeps all existing '🧠 category: ...' Notes into CaseMemory rows
and soft-deletes the originals. Idempotent; safe to re-run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real-world failures from prod sessions today:
* Shira called save_rule, EspoCRM returned 404 ("Controller AssistantRule
does not exist"), the tool returned a fail() result — and Shira told the
user "✅ כלל חדש נשמר". Same hallucination pattern as the document-content
incident yesterday, just on a different surface. Added an explicit
VERIFY-TOOL-RESULT block that says: read the tool result, if it starts
with ❌ or mentions שגיאה/נכשל/failed/error, report the failure honestly.
* When the user named a specific file in a case ("הודעה לבית הדין-...docx")
and list_documents only showed the case root with empty subfolders,
Shira reported "not found" instead of recursing into מסמכים/התכתבויות/
כללי/אסמכתאות. Then "fixed" it by permanently set_case_folder_path-ing
the root to a subfolder, which would have broken access to every other
document in the case. Added a SUBFOLDER SEARCH section to TOOL_RULES.
* The user noted that Shira's writing uses em-dashes (— U+2014) as a
primary separator, which is a strong "this was written by AI" tell.
Added a WRITING STYLE section banning em-dash and en-dash, mandating
hyphen-minus only when truly needed, and warning against dash pileups
and other AI tells.
Refs Task Master none (these are prompt-only changes, no schema impact).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three independent bugs combined to make Shira fabricate the content of
case-46-Friedman's appeal in production (CTS case rewritten as a knee
injury, wrong dates, wrong court file number, wrong %-of-disability —
written into save_memory/Task/Meeting on prod).
* api/services/ocr.py — ocr_docx_images now reads word/document.xml first
(text content), then OCRs embedded images. Previously it only OCR'd the
word/media/* images, so a text-heavy DOCX with one signature image
returned only "[signature]" to the LLM. Verified against the same
Friedman appeal: 16633 chars / 80 paragraphs / 0 XML noise.
* mcp_server/tools/document_tools.py — _ocr_fallback now refuses to wrap
an empty/signature-only OCR result as success. If <80 useful chars or
just "[signature]", returns an explicit failure that instructs the LLM
not to describe / summarize the document.
* api/services/prompt_builder.py — TOOL_RULES adds the absolute rule
"DOCUMENT FAITHFULNESS": never quote a document not literally in the
tool result for THIS turn; never infer content from a file name; treat
dates and case numbers as especially dangerous to invent.
Refs Task Master #5, #6, #7
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the Python side of the SmartAssistant 2.9 release. Each tool maps to
either a new EspoCRM action or a generic CRM endpoint.
Document creation:
- create_document: free-form RTL DOCX with markdown body; calls
SmartAssistant/action/createDocument.
Folder management (all sandboxed to the current case's root):
- write_document_to_folder: raw upload to a path inside the case folder.
- create_subfolder: mkdir with recursive parent creation.
- rename_item / move_item: rename/move files and folders.
- mark_document_for_deletion: soft-delete — moves the file to
"מסמכים למחיקה/" with a "למחיקה - " prefix. Shira is explicitly told
she has NO physical-delete permission.
CRM gaps:
- update_case_fields: allow-listed PUT on Case (judge, court, caseType,
practiceArea, opposingParty, status, priority, etc.).
- search_entities: generic GET with EspoCRM where[] params across
Case/Contact/Account/Document/Task. Wildcard "*" maps to LIKE.
- draft_email: POST Email with status="Draft", linked to the current case.
prompt_builder additions:
- FREE DOCUMENT CREATION: don't say "I can't" — call create_document.
- FOLDER MANAGEMENT: full read/write inside the case folder only.
- FILE DELETION: always mark_document_for_deletion; never claim a physical
delete.
Pairs with SmartAssistant extension release that adds FreeDocumentGenerator,
CaseFolderManager, and 6 controller actions.
Refs Task Master #4
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wraps the tool-calling loop in asyncio.wait_for so long-running tasks
(e.g. consultation summary generation) return a clean Hebrew message
instead of being abruptly cut when PHP closes the connection at 600s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds two new tool categories — billing (10 tools) and legal_aid (6 tools)
— so Shira can drive the full Green Invoice flow: list unbilled activities,
manage charges (approve/cancel/calculate), build draft invoices from
activities, sync to Green Invoice, plus the legal-aid sub-flow (proforma,
mark-submitted, monthly 320 wrap-up, prepare-email, resubmit rejected).
Wires both registrars into agent_runner with allowed_toolsets gating, and
extends TOOL_RULES so Shira (a) shows draft totals before creating, (b)
treats send_invoice_to_green_invoice as irreversible and demands explicit
confirmation, and (c) never marks a legal-aid proforma as submitted
without an explicit request number from the user.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
(1) discard_job was deleting the S3 file even on re-ingest jobs. For a
fresh upload the s3_key points at inbox/<topic>/<kind>/<uuid>-<file>
which the user just dropped — deletable. For a re-ingest, s3_key points
at the EXISTING source's file (kb_source.original_path); deleting it
orphans the source and bricks every future re-process attempt with
"original file unavailable". User hit this on source 141: started a
re-process, hit discard, then the next click on re-process returned
HTTP 400 because the underlying PDF was gone. Now skip the S3 delete
when source_id is set.
(2) start_reingest was setting original_filename = src.title or
filename_from_key(key). When the source's title had no extension
(literally "החלטה" in this case), parse_first_pages couldn't dispatch
by suffix and returned empty text — the classifier then logged
"empty text — skipping" and returned {} for ai_suggestions. Now use
_filename_from_key(key) directly so the parser always sees a real
.pdf/.docx/.txt extension.
Source 141's PDF is permanently lost (MinIO bucket is unversioned).
Recovery options for the user: (a) upload the same PDF to MinIO at
the original key path manually, (b) delete source 141 and re-upload
as a fresh source.
Old behavior: clicking "עיבוד מחדש" on a source preserved the existing
metadata (title, identifier, dates, summary, labels) and only re-chunked
+ re-embedded the file. Sources whose original AI classification produced
weak metadata (e.g. title="החלטה" with no summary) had no path to recover
short of deleting and re-uploading.
New behavior: re-process now goes through the same two-phase pipeline as
a fresh upload — classify (AI extracts metadata) → awaiting_review (user
sees suggestions in the batch review screen and edits) → commit (kb_source
metadata is overwritten + chunks are replaced + labels are replaced).
Implementation:
• start_reingest now returns {job_id, batch_id} and pre-fills metadata_json
with the existing source's labels + summary so the form has a fallback
if the classifier times out.
• The reingest route schedules ingest_jobs.process_classify_stage instead
of admin_sources.process_reingest_job.
• commit_job branches on job.source_id: NULL → fresh upload (creates new
kb_source); set → re-ingest (UPDATE kb_source + replace chunks).
• New process_reingest_embed_stage replaces the old process_reingest_job;
it bypasses _EDITABLE_FIELDS so the user can also change kind on
re-process (e.g. caselaw uploaded as circular).
The browser opens the same batch review screen, so the UX matches uploads
exactly. Re-ingests are single-file batches.
Frontend changes (KnowledgeBase extension v0.9.0) ship in a separate repo.
Labor-law caselaw uploads (4 DOCX, 2026-04-28) timed out at exactly 90s,
landing in awaiting_review with empty ai_suggestions. A live retry of the
same payload through ai-gateway returned valid metadata in 91.8s — one
second over the cap. PDF caselaw can be larger (20K-char cap vs DOCX 6K)
and ai-gateway's serialized OAuth session lengthens the tail under batch
load, so 180s would not be enough headroom either.
240s keeps the call fail-soft (user can still fill metadata manually) but
covers realistic Hebrew legal documents. Override via KB_CLASSIFIER_TIMEOUT_S.
/opt/data on shira-hermes is now a Docker named volume on both dev and prod
(added 2026-04-27 so learned skills/profiles/cases survive redeploys). Named
volumes copy image content only on FIRST creation — after that the volume is
opaque to image updates. So a new entry added to config/skills/ in the repo
would never reach an instance that already has a populated volume.
Fix: at FastAPI startup, walk /app/config/skills/ (baked into the image,
outside the volume mount) and copy any directory whose name does not yet
exist under /opt/data/skills/. Skills already present — whether learned by
the model, edited by a user, or seeded earlier — are never touched.
- skills.py: BAKED_SKILLS_DIR derived from __file__ so it stays correct
regardless of WORKDIR; seed_skills_from_image() returns (added, skipped)
- main.py: call after ensure_data_dirs() and log the result
- Verified with a unit fixture: 2 new baked skills copied, 1 pre-existing
skill with custom content left untouched
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- pyproject.toml -> 0.3.0 (significant changes since 0.2.0:
full Phase 1-7 KB stack, classifier+labels, batch-upload UI,
pending-review panel, bucket rename to legal-kb)
- api/__init__.py reads version from pyproject at import time so
/api/health and OpenAPI both reflect reality without a second
hardcoded copy
- api/services/kb/voyage.py: _post_with_retry wraps embed/rerank
POSTs with exponential backoff on 429+5xx (max 5 retries, base
2s, capped 30s, +25% jitter, honors Retry-After header). Dev and
prod share a single VOYAGE_API_KEY today, so a simultaneous bulk
re-ingest on both used to fail half the calls; now the burst is
absorbed in seconds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The bucket name was a holdover from the original single-domain
prototype (Israeli National Insurance only). With multi-topic support
the same bucket now hosts files for ביטוח לאומי, דיני עבודה, פלילי,
and any future legal domains — separated by `inbox/<topic_slug>/`
prefix, not by bucket. Generic `legal-kb` name reflects that.
Only the default in `_bucket()` changes; deployments that already set
`KB_S3_BUCKET` are unaffected. Dev was migrated to `legal-kb` in the
same operation (`mc mirror` of all 221 objects, env var update,
UPDATE on `kb_source.original_path` rewriting `s3://insurance-kb/` →
`s3://legal-kb/`). Prod MinIO has the empty `legal-kb` bucket ready
for shira-hermes prod cutover.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two production-blocker fixes for the v0.8.0 batch upload UI, plus a
small endpoint to recover review state after a hard refresh.
classifier.py:
- timeout 45s → 90s and concurrency 5 → 2. ai-gateway proxies a
single Claude OAuth session so requests effectively serialize at
the gateway; with concurrency=5, the 4th and 5th files in a batch
of 5 timed out from queue waits alone (observed 2026-04-25 on
user upload of 5 circulars: jobs 10 and 13 returned empty).
- labels schema/prompt: arrays of strings via ai-gateway came back
as [{}, {}, {}] (proxy mangled items.type). Switched the param
to a comma-separated string with explicit format guidance, then
split in _normalize. _normalize is defensive — accepts both
string and list[str|dict] shapes so older callers don't break.
- Sharpened the prompt distinction between subject (free text on
the source) and labels (navigation tags), since identical
examples were causing the model to emit one and skip the other.
ingest_jobs.list_pending_batches + admin_kb GET /admin/kb/batches:
the review screen lives in JS RAM (_activeBatch). A hard refresh
wipes it, leaving uploaded jobs orphaned in awaiting_review with
no UI path back. The new endpoint groups jobs by batch_id with
status counters; the EspoCRM 'ממתינים לאישור' panel uses it to
let users resume any pending review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 6 of the multi-topic KB refactor (backend half). Subsumes the
original Task #19 design discussion (subject vs labels). EspoCRM
extension UI ships separately.
Migrations (4 new, all idempotent, transaction-wrapped):
004_kind_tool — adds 'tool' to kb_source/kb_ingest_job CHECK
005_source_description — kb_source.description + ai_classified_at
006_labels — kb_label + kb_source_label (M:N), GRANTs
007_ingest_classifier — awaiting_review status + processing_stage
+ ai_suggestions JSONB + batch_id
Two-phase ingest (api/services/kb/ingest_jobs.py):
queued → processing(stage=classifying)
→ awaiting_review ← user reviews AI output
→ processing(stage=embedding) ← after user commits
→ done | failed
process_classify_stage() pulls the file, runs parse_first_pages
(3-page text extract), calls classifier.classify (Claude tool-call
via ai-gateway, 45s timeout, 5-concurrent semaphore, fail-soft on
any error → empty suggestions), writes ai_suggestions JSONB,
transitions to awaiting_review.
commit_job() resolves user-confirmed labels (existing by slug,
new ones via slugify+ON CONFLICT DO NOTHING), transitions to
processing(embedding), schedules process_embed_stage.
process_embed_stage() runs the legacy ingest_source path with
user-edited metadata + summary as kb_source.description, then
apply_to_source for labels.
discard_job() removes a file from a batch (status=failed,
S3 deleted).
list_jobs_by_batch() — single-roundtrip review-screen load.
Labels (NEW api/services/kb/labels.py):
Hebrew → ASCII slugify (letter-by-letter map, no external dep);
CRUD; lookup_or_create_batch (race-safe); apply_to_source +
replace_for_source maintain usage_count; merge race-safely
(UPDATE assignments + ON CONFLICT DO NOTHING).
Classifier (NEW api/services/kb/classifier.py):
AsyncOpenAI to ai-gateway, Sonnet, OpenAI-style tool calling for
guaranteed JSON shape, Hebrew system prompt with 5 kind values,
citation format examples, "do not invent" rule, summary length
bounds 80-200 chars.
Routes (api/routes/admin_kb.py — 8 new on top of existing 11):
POST /admin/kb/upload-batch (multi-file, AI classify)
GET /admin/kb/batch/{batch_id} (review-screen load)
POST /admin/kb/jobs/{id}/commit (user confirms metadata)
POST /admin/kb/jobs/{id}/discard (user removes from batch)
GET /admin/kb/labels (typeahead)
POST /admin/kb/labels (explicit create)
POST /admin/kb/labels/{id}/merge (admin cleanup)
DELETE /admin/kb/labels/{id} (only if usage=0)
Public /kb/search gains optional label_id filter.
admin_sources.list_sources LEFT JOINs labels into each row's
output. update_source accepts label_ids to replace the set.
End-to-end smoke test passed: synthetic Hebrew circular →
upload-batch → background classify → awaiting_review with kind,
title, subject, summary, identifier, published_at all populated
correctly.
Refs Task Master #20 (espocrm-extensions/KnowledgeBase v0.8.0)
Subsumes: Task #19 (labels for sub-topics)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backs the Phase 4 topic-management table in the KnowledgeBase EspoCRM
extension ("ניהול" tab → "נושאים" panel). Four new admin endpoints:
GET /admin/kb/topics — list ALL topics including
is_active=false ones, with
source_count joined from kb_source
POST /admin/kb/topics — create. Validates slug shape
(^[a-z][a-z0-9-]{1,62}$) and
uniqueness — clean 400 on duplicate
rather than leaking UniqueViolation
PUT /admin/kb/topics/{id} — patch name, description,
system_prompt_addendum, is_active.
slug is intentionally NOT editable —
the S3 layout depends on it
(inbox/<topic_slug>/<kind>/...)
DELETE /admin/kb/topics/{id} — hard-delete; refuses with 409 if
any sources still belong to the
topic. Soft-delete (is_active=false)
is the normal "remove from dropdown"
path
Refs Task Master #15 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backs the Phase 3 sources table in the KnowledgeBase EspoCRM extension
("ניהול" tab). Four new admin endpoints:
GET /admin/kb/sources — list with chunk_count and
last-ingest status (LATERAL
join against kb_ingest_job),
filterable by topic_id + kind
PUT /admin/kb/sources/{id} — patch the user-editable
metadata only (title,
identifier, source_url,
published_at, effective_at);
structural fields (kind,
topic_id, checksum) are not
accepted
DELETE /admin/kb/sources/{id} — hard delete (kb_chunk cascades
via FK; the S3 object at
original_path is best-effort
deleted)
POST /admin/kb/sources/{id}/reingest — queues a kb_ingest_job tied
to the existing source_id;
background task replaces
chunks in-place, preserves
the source row and its
hand-edited metadata, and
updates the checksum
The reingest path differs from ingest_source: ingest_source creates a
new kb_source row and supersedes the previous version. Reingest keeps
the same id (so cached chunk_index references in the browser stay
valid), wipes its kb_chunk rows in a single transaction, and inserts
fresh ones from the same file.
Refs Task Master #14 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Court rulings are a distinct content type — long discussion / reasoning,
no enumerated statutory hierarchy, case identifier instead of statute id —
so they get their own kind alongside law/regulation/circular rather than
being shoehorned into one of the existing three.
Migration 003 alters the CHECK constraints on kb_source.kind and
kb_ingest_job.kind to include 'caselaw'. The chunker uses the
circulars path for caselaw too (paragraph + size split, no statutory
section regex); the case identifier is captured at upload time in
kb_source.identifier.
Updates Literal types in ingest_source, admin_kb upload, and the
kb_public SearchRequest. _KINDS in s3.py grows to four so failed/
processed/inbox listings include caselaw subdirs.
Refs Task Master #18 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 2 of the multi-topic KB refactor. Adds three admin endpoints that
back the new "ניהול" tab in the KnowledgeBase EspoCRM extension:
POST /admin/kb/upload — multipart, queues a kb_ingest_job and
fires asyncio.create_task to process
parse → chunk → embed in the
background. Returns {job_id, status}
immediately so the browser doesn't
block on slow PDFs.
GET /admin/kb/jobs — recent jobs, optionally filtered by
topic_id / status / limit.
GET /admin/kb/jobs/{id} — single-job detail for the polling UI.
Migration 002 adds kb_ingest_job (queued/processing/done/failed) with
indexes on (status, created_at) and (topic_id, created_at).
ingest_source now accepts topic_id (NULL → defaults to 1 for back-compat
with the existing scan-inbox cron path) and writes it to kb_source.
S3 layout for new uploads is topic-aware: inbox/<topic_slug>/<kind>/...
The legacy /scan-inbox path on inbox/<kind>/ is unchanged.
Refs Task Master #13 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 of the multi-domain refactor. Adds a kb_topic table seeded with
the existing 'ביטוח לאומי' domain (id=1) and pins all 6 existing
kb_source rows to it, so search/ask can be scoped per domain without
disturbing the current single-topic UX.
- scripts/migrations/001_topics.sql: idempotent migration. Creates
kb_topic, seeds national-insurance with the system_prompt_addendum
copied verbatim from kb_public.py's hardcoded text, adds nullable
kb_source.topic_id, backfills it to 1, then SET NOT NULL. GRANTs to
shira_kb mirror its existing kb_source privileges. Wrapped in BEGIN
/COMMIT with a sanity-check that raises if the backfill is incomplete.
- api/services/kb/topics.py: list_topics / get_topic / resolve_topic_id.
resolve_topic_id raises ValueError on unknown ids so the route layer
can return 400 instead of silently mixing domains.
- api/services/kb/search.py: search() + _retrieve_rrf accept topic_id;
the SQL adds AND s.topic_id = $N when present. _expand_query is
parametrized by topic name (was hardcoded "הביטוח הלאומי").
- api/routes/kb_public.py: new GET /kb/topics. /kb/search /kb/sources
/kb/ask /kb/ask/stream all accept topic_id and call resolve_topic_id.
_build_ask_runner_context loads system_prompt_addendum from the DB
instead of the hardcoded insurance string.
- mcp_server/tools/legal_kb_tools.py: tool renamed
search_insurance_kb → search_legal_kb, takes topic_id at registration
so each conversation is pinned to its caller-chosen domain. Tool
description interpolates the topic name.
- api/services/agent_runner.py: kb_topic_id + kb_topic_name flow
through to register_legal_kb_tools. Hebrew progress labels updated.
Refs Task Master #12 (espocrm-extensions/KnowledgeBase).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds POST /kb/ask/stream that returns text/event-stream. The agent
runs in a fire-and-forget task; an on_event callback wired into
AgentRunner.run pushes {type, label, ...} events into an asyncio.Queue,
and the response generator consumes them as SSE data: lines.
Event types and Hebrew labels:
- thinking — before each LLM call ("קוראת את השאלה" / "ממשיכה לחקור")
- tool_start — before each tool. For search_insurance_kb the label
includes the actual query string Shira chose, so the user can see
WHAT she's searching for ("מחפשת בבסיס הידע: '...'"). Other tools
fall back to a generic Hebrew label keyed off the function name.
- tool_done / tool_error — after each tool. For search_insurance_kb
the label echoes the first non-empty line of the tool result (which
the legal_kb tool formats as "## נמצאו N קטעים").
- writing — when the model returns text without tool calls.
- answer — final event with {text, sources, conversationId}, mirrors
the non-streaming /kb/ask response shape.
- error — runner exception; client should close the EventSource.
Heartbeats: when the queue is idle for >15s the generator yields a
": keep-alive" comment line (ignored by EventSource) so any proxy in
the path doesn't drop the connection during long agent thinks.
The original POST /kb/ask is unchanged — the streaming endpoint is
additive. Both share _build_ask_runner_context + _build_ask_messages
helpers extracted from the original handler.
Refs Task Master #1
Two minimal additions for the upcoming v0.1.11 client work:
- /kb/search hits now include chunk_index, so the client can call
/source/{id}/chunks?around=N&ctx=K to fetch the matched chunk plus K
chunks before/after as context. Required by the new search-preview
layout for text sources, which until now duplicated the matched chunk
in both panes.
- /source/{id}/chunks accepts around (chunk_index) + ctx (window radius
clamped to 0..10). When around is given, returns only the windowed
range; otherwise unchanged (returns all chunks of the source).
Refs Task Master #1
Single-shot search ranks the source whose title literally contains the
query terms way above everything else, so "מהי תקנה 37" returns 6 hits
from the תקנה 37 circular + 2 from the law and zero from ספר הליקויים
— even though the latter is the canonical medical reference for that
regulation. The ask agent already worked around this by issuing several
search_insurance_kb calls with different phrasings; that intelligence is
now also available to the public /kb/search endpoint.
Implementation:
- _retrieve_rrf extracts vector + lexical retrieval (RRF fused, no
rerank) into a reusable async helper. Each candidate now carries
chunk_id so callers can dedup when merging multiple sub-searches.
- _expand_query asks the configured Claude model (via ai-gateway) for
up to 3 alternative phrasings, with a 12s timeout. On any failure the
caller silently falls back to single-query mode.
- search() gains an `expand: bool` parameter. When true, runs the
original query plus all variants through _retrieve_rrf in parallel,
merges by chunk_id keeping the best RRF score, and reranks the union
against the ORIGINAL query so the final order reflects what the user
asked, not the variants.
- /kb/search defaults to expand=true.
Cost per call: 1 extra LLM round-trip (~1-2s on Sonnet) and 3 extra
Voyage embeds (~150ms each, parallel). Latency budget grows from ~700ms
to ~3s, acceptable for a search the user explicitly triggered.
Refs Task Master #1
_split_oversized slices at fixed char offsets and can cut a
\x00KB_PAGE:N\x00 sentinel in half — the broken prefix/suffix no longer
matches _PAGE_MARKER_RE so _strip_page_markers leaves the \x00 byte in
place. PostgreSQL's UTF-8 check then rejects the insert with
"invalid byte sequence for encoding UTF8: 0x00", and ingest of
ספר הליקויים fails on every scan. Belt-and-suspenders: strip all
remaining \x00 bytes from clean content before returning.
Refs Task Master #2
Commit 426b0d6 fixed page propagation through the Chunk reconstruction in
chunk(), but two downstream paths still collapsed every sub-chunk to the
parent's starting page:
1. _split_oversized operated on markerless content (since _as_dicts inside
chunk_statute/chunk_circular stripped markers before returning), so when
a single giant chunk was sliced into pieces — the common path for
ספר הליקויים / ספר המבחנים, which have no statute-style section markers —
every piece inherited c.page_number = parent's start page. 209/209 chunks
landed on page 1.
2. chunk_circular packed paragraphs into sub-chunks all tagged with
seg_page (the segment's opening page), with no attempt to track where
each sub-chunk actually begins within the original text. חוזר תקנה 37
(8 chunks) and חוזר נכות 1941 (4 chunks) — neither had header matches —
landed entirely on page 1.
Fixes:
- chunk_statute and chunk_circular now return list[Chunk] with markers
still embedded in content; _as_dicts is called once at the end of
chunk() so markers are stripped AFTER splitting.
- _split_oversized builds a local page map from markers in each parent
chunk's content, with a virtual (0, parent.page_number) entry so pieces
before the first real marker still inherit the correct page.
- chunk_circular tracks each paragraph's absolute offset in the original
text and computes each emitted sub-chunk's page_number from its first
paragraph's offset against the full-text page_map.
Verified on synthetic 3-page input: regulation path now emits pages 1,1,2,3
across 4 pieces (was 1,1,1,1); circular path emits 1,1,2,2,3,3 (was
1,1,1,1,1,1).
Refs Task Master #2
chunk() reconstructs Chunk objects before _split_oversized/_as_dicts and
was dropping page_number on the floor, so every chunk came out NULL
regardless of what chunk_statute computed. Also flip _as_dicts to prefer
the caller-supplied c.page_number (derived from the chunk's starting
offset) over any marker that survived inside c.content — the latter can
resolve to a later page when a section spans pages.
Verified: ספר הליקויים chunks now carry page_number matching the PDF
page where each section actually begins.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
To let the browse/ask UIs jump straight to the right PDF page:
- ingest.py: embed \x00KB_PAGE:<N>\x00 sentinels at the start of each
PDF page during _parse_pdf.
- chunker.py: _strip_page_markers removes the sentinels and records the
first page seen per chunk as page_number. Chunk dataclass + _as_dicts
propagate page_number through both chunk_statute and chunk_circular.
- ingest.py INSERT: new kb_chunk.page_number column (DB migration
applied manually).
- search.py SQL: SELECT c.page_number + s.id + s.original_path so the
rerank results carry them forward.
- legal_kb_tools.py: register_legal_kb_tools(sources_used=list) — the
tool appends {source_id, title, kind, identifier, heading_path,
section_ref, page_number, original_path} for each hit, deduped by
(source_id, page_number).
- agent_runner.py: new kb_sources_used kwarg threads the list through
to the tool registration.
- kb_public.py /kb/ask: collects sources_used during the run, filters
to PDF-only sources, and returns them alongside text.
Chunks stored before this change have page_number = NULL; PDFs need to
be re-ingested to populate them. The TXT law source is unaffected.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous block-based parser left each pymupdf block as its own line.
That's fine for prose but wrong for ספר הליקויים's cover and preface,
where each word is its own block — you got "ו אנ / מתכבד / ים" instead
of "ואנו מתכבדים". Merge blocks whose y-coordinates are within 4 pixels
into a single line, right-to-left, joined by spaces.
Content pages (where blocks already contain whole paragraphs) are
unaffected because a single block on one row stays as one row.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pymupdf's `get_text("text")` reads glyph runs in raw order, which on
multi-column Hebrew PDFs (ספר הליקויים, ספר המבחנים) interleaves column
headers, page numbers, form labels, and body paragraphs into jumbled
lines — words split to single characters on separate lines, paragraphs
lost. Switch to `get_text("blocks")` and sort each page's blocks by
(y-axis ascending, x-axis descending) so each paragraph stays intact
and the reading order is top-to-bottom, right-to-left as expected.
Side-effect: chunk content becomes readable in the browse view.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ספר הליקויים produced 698 chunks, most of them junk. A line like
"3 . . . . . . . . . . . . . . . . . . . . . . . . . . הוראות החוק" from
its table of contents was being treated as section 3. Phone numbers like
"02-6709701" became section 02. Both of these polluted the index and
broke the browse view's readability.
Tighten _RE_SECTION so it requires either the explicit prefix (סעיף /
תקנה / §) OR real text immediately after the terminator — a row of dots
or more digits disqualifies. The scraped חוק הביטוח הלאומי still chunks
correctly because the scraper emits "סעיף N. ..." with the prefix.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a rerank stage between the hybrid RRF fusion and the final top_k cut.
RRF is a heuristic over ranks — it can't tell which of two equally-ranked
candidates is actually more relevant. A cross-encoder scores each
(query, passage) pair directly and produces a true relevance order.
- voyage.rerank(query, documents, top_k) → list of {index, relevance_score}
sorted high→low. Defaults to the model in VOYAGE_RERANK_MODEL
(rerank-2.5, 32K tokens per pair, multilingual incl. Hebrew).
- search pulls top-30 from RRF, passes them to rerank, then returns the
reranker's top-k. Each returned item gets a rerank_score field.
- Fallback: if the rerank API errors (auth, 5xx, timeout), fall back to
the RRF ordering and log a warning — degraded precision beats outage.
- KB_RERANK_DISABLED=1 bypasses rerank for debugging.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sidecar JSON supplies published_at / effective_at as 'YYYY-MM-DD' strings.
asyncpg rejects strings for DATE columns even with '::date' casts in the
SQL, so ingest now parses them via datetime.date.fromisoformat before the
INSERT. Non-string inputs (date, datetime, None, empty) are handled too.
Before: ingest crashed on any sidecar with a date, leaving the source in
failed/ while sidecarless siblings succeeded.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each file in the inbox can now be accompanied by a `<filename>.meta.json`
sidecar that supplies identifier, published_at, effective_at, and
source_url. The scanner applies filename-derived defaults first and lets
the sidecar override individual fields.
- list_inbox / list_failed skip `.meta.json` so sidecars aren't ingested
as standalone documents.
- mark_processed, mark_failed, and requeue-failed move the sidecar
alongside its parent (best-effort).
- mark_failed now sanitizes the error string to ASCII before writing it
to S3 object metadata (HTTP header encoding).
- search SQL selects published_at/effective_at; the formatter shows the
full heading_path plus publication date so citations are unambiguous.
- scripts/kb_sidecar_template.json documents the expected shape.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Before: chunk_statute produced heading_path = section_ref = "ס' 195".
After: heading_path walks the four Israeli-statute levels (part, chapter,
sub-chapter, section) and renders them as "חלק ט׳ — ביטוח נכות > פרק ב׳
— ועדות רפואיות > סימן א׳ > ס' 195". section_ref stays the leaf so
citations stay precise.
- New _RE_PART / _RE_CHAPTER / _RE_SUBCHAPTER regexes with Hebrew ordinal
support including multi-letter gershayim ("י״א", "ט״ז").
- _collect_markers emits all markers ordered by position; chunk_statute
walks them tracking active_part/chapter/subchapter and attaches the
full path to each section chunk.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous batching sent up to 128 inputs per request but didn't cap total
tokens. On large Hebrew statutes (ספר הליקויים, ספר המבחנים) a single
request exceeded Voyage's 120K-token-per-batch hard cap and failed with 400.
- voyage.embed now pre-computes an estimated token count per input (Hebrew
is ~0.55 tokens/char in voyage-multilingual-2) and emits multiple
requests so each stays under a 90K-token safety margin.
- chunker lowers the per-chunk ceiling to 1500 tokens and updates the
chars→tokens estimate to match the observed tokenizer behavior.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two bugs surfaced during Phase 1 validation on real circulars:
1. The chunker produced one giant chunk for large statutes (ספר הליקויים,
ספר המבחנים) that the section regex could not split. This exceeded
Voyage's 120K-token batch limit and the ingest failed. Add a hard
2000-token ceiling per chunk with a character-based fallback split that
preserves heading_path and section_ref.
2. ingest_source matched "existing source" by kind + COALESCE(identifier,
''), so any two sources with NULL identifier compared equal. The second
ingest would then supersede the first unrelated document. Match by
identifier only when one is provided; otherwise fall back to
(kind + title) and require the candidate to still be active.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves every object under failed/<kind>/ back to inbox/<kind>/ for retry
after a transient upstream failure (e.g. Voyage 401 before a key rotation).
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a MinIO-backed ingest flow: upload to s3://insurance-kb/inbox/<kind>/
via MinIO Console or any S3 client, then POST /admin/kb/scan-inbox to
process all pending files. Successful ingests move to processed/<kind>/,
failures move to failed/<kind>/ with the error stored as S3 metadata.
- api/services/kb/s3.py: boto3 client, list_inbox/fetch/move helpers.
- api/routes/admin_kb.py: GET /admin/kb/inbox, POST /admin/kb/scan-inbox.
- The kind is derived from the folder path; title/identifier default to
the filename stem and can be refined by re-ingesting with metadata.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a searchable KB for חוק הביטוח הלאומי, תקנות הביטוח הלאומי, and חוזרי
הביטוח הלאומי. Hybrid search (pgvector cosine + tsvector/trgm) fused with
Reciprocal Rank Fusion, exposed to Shira as the `search_insurance_kb` tool.
- api/services/kb/: asyncpg pool, Voyage voyage-multilingual-2 client,
per-kind chunker (statute-by-section / circular header-aware), ingest
pipeline with supersession (old source → superseded_by), hybrid search.
- api/routes/admin_kb.py: POST /admin/kb/ingest (multipart), GET /admin/kb/stats.
- mcp_server/tools/legal_kb_tools.py: `search_insurance_kb` registered under
the `legal` toolset.
- pyproject.toml: +asyncpg, +python-docx, +boto3, +python-multipart.
Infra (external): pgvector/pgvector:pg16 on the shared PG, insurance_kb DB
with hnsw + gin indexes, MinIO bucket insurance-kb, KB_DATABASE_URL in
Infisical /espocrm, VOYAGE_API_KEY/VOYAGE_MODEL/KB_DATABASE_URL on the
shira-hermes Coolify app.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 4 Python tool registrations that call EspoCRM's SmartAssistant
executeTool endpoint with the DigitalSignature plugin tools:
- send_for_signature — send document for signing with auto-resolved signers
- check_signature_status — query status of signature requests
- remind_signers — send WhatsApp reminders to pending signers
- list_pending_signatures — show all awaiting signature requests
Tools are registered under the "signature" toolset, enabled by default.
Requires DigitalSignature extension v2.0.0+ on the EspoCRM side.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the CRM's readDocument returns success:false (scanned PDFs,
image-only DOCX, corrupted files), shira-hermes now:
1. Fetches the raw file bytes via new getDocumentBytes endpoint
2. Sends them to Claude Vision through ai-gateway:
- Images (PNG/JPG/TIFF/etc): direct vision call
- PDFs: render each page at 180 DPI via PyMuPDF, vision call per page
- DOCX: extract images from word/media/, vision call per image
New tool: read_document_ocr (explicit OCR request)
Existing tool read_document auto-falls-back to OCR on extraction failure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EspoCRM stores dates in UTC. Shira was displaying them raw (e.g., 08:00 UTC
instead of 11:00 Israel time) and storing user-provided local times as UTC.
- Add _to_local() in prompt_builder to convert UTC→Asia/Jerusalem for all
displayed dates (hearings, meetings, calls, tasks, notes)
- Update normalize_date() in _helpers to convert Israel local→UTC before
sending to EspoCRM
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Security:
- Path traversal: validate user_id, case_id, skill_name against safe regex
- Input sanitization in memory.py and skills.py file path construction
Correctness:
- ISR timezone: replace hardcoded UTC+3 with ZoneInfo("Asia/Jerusalem") for DST
- save_rule: use EspoCrmApiError.status_code instead of fragile string matching
- Background tasks: store references to prevent GC before completion
- Lock race: use dict.setdefault() for atomic lock creation
- Version: health.py now matches main.py (0.2.0)
- Skill names: normalized to lowercase for dedup consistency
Build:
- Dockerfile: install from pyproject.toml instead of hardcoded pip list
- Remove unused mcp>=1.0.0 dependency from pyproject.toml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The AssistantRule entity may not exist on some CRM instances. When save_rule
gets a 404, it now falls back to saving the rule in the local user profile
(update_user_profile). Also improved update_user_profile description so
Shira is more likely to use it for user preferences.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EspoCRM may send documents as a list instead of a dict with
totalFiles. Convert list to dict format to avoid AttributeError.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
chaim