Claude Vision via ai-gateway's claude-code OAuth provider runs ~140s per
rendered page (a 3-page report measured at 417s), which blows past any safe
synchronous HTTP timeout — the EspoCRM controller's 240s curl gave up with
"0 bytes received" while ai-gateway kept working.
parse-pdf-report now accepts ?mode=async: it returns {jobId, status:"pending"}
immediately and runs the parse in the background, persisting the result/error
to a file-based job store under /opt/data (survives multi-worker + restarts).
Poll GET /parse-pdf-report/jobs/{jobId}. Default sync mode is unchanged so
n8n and other server-side callers keep working.
Refs prod ai-gateway log id cee4c17c (claudeMs=417513).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bundling all report pages as 180-DPI PNG base64 in one ai-gateway
chat-completions request blew past its body limit and returned 413.
A rendered table page is mostly white, so q85 JPEG is several times
smaller while keeping the table text crisp enough for Claude Vision.
Pairs with the ai-gateway limit bump to 25mb.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shira now supplies a subject, never a filename or numbering. The TEMPLATE / FREE-DOC
/ FOLDER guidance in prompt_builder instructs the model to pass title / documentSubject
as a short Hebrew subject (no date, case number, contact, extension or path separators),
and to keep the final filename of write_document_to_folder clean. Server-side enforcement
lives in SmartAssistant CaseFolderManager (rule N1).
Refs Task Master #10
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After SmartAssistant 2.10.0 introduced UserProfile and AssistantSkill
entities, the Python side kept writing only to the filesystem. So an
admin browsing the EspoCRM UI saw nothing — but the system prompt did
pull from CRM via context['userProfile'] / context['availableSkills'].
Result: edits made by Shira (via update_user_profile / skill_manage tool)
were invisible to admins; edits made by admins took effect immediately
because the prompt fetched from DB. Now the two sources agree.
* memory.py — added _sync_user_profile_to_crm() that upserts a UserProfile
row whenever update_user_profile writes the file. Best-effort with
logging; HTTP failures don't break the tool.
* skills.py — added _sync_skill_to_crm() that upserts an AssistantSkill
row from the SKILL.md frontmatter (description, triggers, version) plus
the full body in content. Called after every save_skill in skill_manage.
Read path is unchanged: the EspoCRM CaseContextBuilder / SmartAssistantService
populate context fields, and the Python prompt_builder consumes them with
in-code defaults as a fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three changes paired with SmartAssistant 2.10.0:
1. prompt_builder.py — every prompt section (tool_rules, writing_style,
legal_assistance, personality_case, personality_office, other_rules_case,
other_rules_office) now reads from context['assistantPrompts'][key]
with the existing string as a fallback default. Admins editing
AssistantPrompt rows in the EspoCRM UI immediately affect the next
conversation. A missing/inactive row falls back to the in-code default
so deleting everything in the UI never bricks Shira.
Also: the user-profile section now prefers context['userProfile']
(from the new UserProfile entity) over the explicit `user_profile`
parameter, completing the move from /opt/data/profiles/*.md to the CRM.
2. crm_tools.py:save_memory — stops POSTing directly to /Note with a
'🧠 category:' prefix. Routes through SmartAssistant/action/executeTool
so the PHP saveMemory handler actually creates a CaseMemory row.
Cause of yesterday's prod issue: CaseMemory table was empty even
though Shira was "saving" to it.
3. scripts/migrate_legacy_memory_notes.py — one-shot migration that
sweeps all existing '🧠 category: ...' Notes into CaseMemory rows
and soft-deletes the originals. Idempotent; safe to re-run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real-world failures from prod sessions today:
* Shira called save_rule, EspoCRM returned 404 ("Controller AssistantRule
does not exist"), the tool returned a fail() result — and Shira told the
user "✅ כלל חדש נשמר". Same hallucination pattern as the document-content
incident yesterday, just on a different surface. Added an explicit
VERIFY-TOOL-RESULT block that says: read the tool result, if it starts
with ❌ or mentions שגיאה/נכשל/failed/error, report the failure honestly.
* When the user named a specific file in a case ("הודעה לבית הדין-...docx")
and list_documents only showed the case root with empty subfolders,
Shira reported "not found" instead of recursing into מסמכים/התכתבויות/
כללי/אסמכתאות. Then "fixed" it by permanently set_case_folder_path-ing
the root to a subfolder, which would have broken access to every other
document in the case. Added a SUBFOLDER SEARCH section to TOOL_RULES.
* The user noted that Shira's writing uses em-dashes (— U+2014) as a
primary separator, which is a strong "this was written by AI" tell.
Added a WRITING STYLE section banning em-dash and en-dash, mandating
hyphen-minus only when truly needed, and warning against dash pileups
and other AI tells.
Refs Task Master none (these are prompt-only changes, no schema impact).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three independent bugs combined to make Shira fabricate the content of
case-46-Friedman's appeal in production (CTS case rewritten as a knee
injury, wrong dates, wrong court file number, wrong %-of-disability —
written into save_memory/Task/Meeting on prod).
* api/services/ocr.py — ocr_docx_images now reads word/document.xml first
(text content), then OCRs embedded images. Previously it only OCR'd the
word/media/* images, so a text-heavy DOCX with one signature image
returned only "[signature]" to the LLM. Verified against the same
Friedman appeal: 16633 chars / 80 paragraphs / 0 XML noise.
* mcp_server/tools/document_tools.py — _ocr_fallback now refuses to wrap
an empty/signature-only OCR result as success. If <80 useful chars or
just "[signature]", returns an explicit failure that instructs the LLM
not to describe / summarize the document.
* api/services/prompt_builder.py — TOOL_RULES adds the absolute rule
"DOCUMENT FAITHFULNESS": never quote a document not literally in the
tool result for THIS turn; never infer content from a file name; treat
dates and case numbers as especially dangerous to invent.
Refs Task Master #5, #6, #7
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the Python side of the SmartAssistant 2.9 release. Each tool maps to
either a new EspoCRM action or a generic CRM endpoint.
Document creation:
- create_document: free-form RTL DOCX with markdown body; calls
SmartAssistant/action/createDocument.
Folder management (all sandboxed to the current case's root):
- write_document_to_folder: raw upload to a path inside the case folder.
- create_subfolder: mkdir with recursive parent creation.
- rename_item / move_item: rename/move files and folders.
- mark_document_for_deletion: soft-delete — moves the file to
"מסמכים למחיקה/" with a "למחיקה - " prefix. Shira is explicitly told
she has NO physical-delete permission.
CRM gaps:
- update_case_fields: allow-listed PUT on Case (judge, court, caseType,
practiceArea, opposingParty, status, priority, etc.).
- search_entities: generic GET with EspoCRM where[] params across
Case/Contact/Account/Document/Task. Wildcard "*" maps to LIKE.
- draft_email: POST Email with status="Draft", linked to the current case.
prompt_builder additions:
- FREE DOCUMENT CREATION: don't say "I can't" — call create_document.
- FOLDER MANAGEMENT: full read/write inside the case folder only.
- FILE DELETION: always mark_document_for_deletion; never claim a physical
delete.
Pairs with SmartAssistant extension release that adds FreeDocumentGenerator,
CaseFolderManager, and 6 controller actions.
Refs Task Master #4
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wraps the tool-calling loop in asyncio.wait_for so long-running tasks
(e.g. consultation summary generation) return a clean Hebrew message
instead of being abruptly cut when PHP closes the connection at 600s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The real Legal Aid monthly reports have multi-line cells (sub-types like
"הודעה בהתאם להחלטה" wrap to two visual lines), RTL Hebrew column ordering
that flips depending on whether a row contains long Hebrew text, and
amounts that wrap across rows ("1478.5\n4" → 1478.54). pdfplumber's
extract_tables() returned 0 rows on a real report — the v1 parser
silently produced an empty preview.
Switching to the Claude-Vision path (mirrors api/services/ocr.py): render
each PDF page via PyMuPDF, send to ai-gateway with a JSON-output prompt,
parse the structured response. Claude reads RTL Hebrew + multi-line cells
natively and produces the same JSON shape that
LegalAidPaymentReportService.ingest() expects.
`?debug=true` added to return the raw model response for diagnostics
when a future PDF format breaks extraction.
pdfplumber stays in pyproject.toml for now in case we want a non-AI
fallback later, but the runtime path no longer touches it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ports the pdfplumber-based parser from the (currently inactive) n8n
workflow legal-aid-report-ingest.json so EspoCRM's GreenInvoiceBilling
extension can do manual upload + dry-run preview of monthly Legal Aid
payment reports without depending on n8n being live.
Auth: X-Api-Key (matches admin_kb pattern).
Response shape: same JSON the existing LegalAidPaymentReportService
ingest pipeline expects, so the EspoCRM controller can hand it
straight back to ::ingest($payload, $pdf, dryRun=true).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds two new tool categories — billing (10 tools) and legal_aid (6 tools)
— so Shira can drive the full Green Invoice flow: list unbilled activities,
manage charges (approve/cancel/calculate), build draft invoices from
activities, sync to Green Invoice, plus the legal-aid sub-flow (proforma,
mark-submitted, monthly 320 wrap-up, prepare-email, resubmit rejected).
Wires both registrars into agent_runner with allowed_toolsets gating, and
extends TOOL_RULES so Shira (a) shows draft totals before creating, (b)
treats send_invoice_to_green_invoice as irreversible and demands explicit
confirmation, and (c) never marks a legal-aid proforma as submitted
without an explicit request number from the user.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
(1) discard_job was deleting the S3 file even on re-ingest jobs. For a
fresh upload the s3_key points at inbox/<topic>/<kind>/<uuid>-<file>
which the user just dropped — deletable. For a re-ingest, s3_key points
at the EXISTING source's file (kb_source.original_path); deleting it
orphans the source and bricks every future re-process attempt with
"original file unavailable". User hit this on source 141: started a
re-process, hit discard, then the next click on re-process returned
HTTP 400 because the underlying PDF was gone. Now skip the S3 delete
when source_id is set.
(2) start_reingest was setting original_filename = src.title or
filename_from_key(key). When the source's title had no extension
(literally "החלטה" in this case), parse_first_pages couldn't dispatch
by suffix and returned empty text — the classifier then logged
"empty text — skipping" and returned {} for ai_suggestions. Now use
_filename_from_key(key) directly so the parser always sees a real
.pdf/.docx/.txt extension.
Source 141's PDF is permanently lost (MinIO bucket is unversioned).
Recovery options for the user: (a) upload the same PDF to MinIO at
the original key path manually, (b) delete source 141 and re-upload
as a fresh source.
Old behavior: clicking "עיבוד מחדש" on a source preserved the existing
metadata (title, identifier, dates, summary, labels) and only re-chunked
+ re-embedded the file. Sources whose original AI classification produced
weak metadata (e.g. title="החלטה" with no summary) had no path to recover
short of deleting and re-uploading.
New behavior: re-process now goes through the same two-phase pipeline as
a fresh upload — classify (AI extracts metadata) → awaiting_review (user
sees suggestions in the batch review screen and edits) → commit (kb_source
metadata is overwritten + chunks are replaced + labels are replaced).
Implementation:
• start_reingest now returns {job_id, batch_id} and pre-fills metadata_json
with the existing source's labels + summary so the form has a fallback
if the classifier times out.
• The reingest route schedules ingest_jobs.process_classify_stage instead
of admin_sources.process_reingest_job.
• commit_job branches on job.source_id: NULL → fresh upload (creates new
kb_source); set → re-ingest (UPDATE kb_source + replace chunks).
• New process_reingest_embed_stage replaces the old process_reingest_job;
it bypasses _EDITABLE_FIELDS so the user can also change kind on
re-process (e.g. caselaw uploaded as circular).
The browser opens the same batch review screen, so the UX matches uploads
exactly. Re-ingests are single-file batches.
Frontend changes (KnowledgeBase extension v0.9.0) ship in a separate repo.
Labor-law caselaw uploads (4 DOCX, 2026-04-28) timed out at exactly 90s,
landing in awaiting_review with empty ai_suggestions. A live retry of the
same payload through ai-gateway returned valid metadata in 91.8s — one
second over the cap. PDF caselaw can be larger (20K-char cap vs DOCX 6K)
and ai-gateway's serialized OAuth session lengthens the tail under batch
load, so 180s would not be enough headroom either.
240s keeps the call fail-soft (user can still fill metadata manually) but
covers realistic Hebrew legal documents. Override via KB_CLASSIFIER_TIMEOUT_S.
/opt/data on shira-hermes is now a Docker named volume on both dev and prod
(added 2026-04-27 so learned skills/profiles/cases survive redeploys). Named
volumes copy image content only on FIRST creation — after that the volume is
opaque to image updates. So a new entry added to config/skills/ in the repo
would never reach an instance that already has a populated volume.
Fix: at FastAPI startup, walk /app/config/skills/ (baked into the image,
outside the volume mount) and copy any directory whose name does not yet
exist under /opt/data/skills/. Skills already present — whether learned by
the model, edited by a user, or seeded earlier — are never touched.
- skills.py: BAKED_SKILLS_DIR derived from __file__ so it stays correct
regardless of WORKDIR; seed_skills_from_image() returns (added, skipped)
- main.py: call after ensure_data_dirs() and log the result
- Verified with a unit fixture: 2 new baked skills copied, 1 pre-existing
skill with custom content left untouched
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
shira-hermes prod migrated to dockerimage (uuid r4xw6upm3237c63ohbye12g7,
replaces the old Dockerfile-built xqfd89...). The new app pulls
gitea.dev.marcus-law.co.il/espocrm-extensions/shira-hermes:latest, so it
needs a redeploy when CI/CD pushes a new :latest. Adding a second
"Trigger Coolify redeploy (prod)" step that calls prod Coolify with
COOLIFY_PROD_TOKEN + COOLIFY_PROD_UUID secrets (just added to repo).
Now one push to main → image build → both dev and prod auto-update.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- pyproject.toml -> 0.3.0 (significant changes since 0.2.0:
full Phase 1-7 KB stack, classifier+labels, batch-upload UI,
pending-review panel, bucket rename to legal-kb)
- api/__init__.py reads version from pyproject at import time so
/api/health and OpenAPI both reflect reality without a second
hardcoded copy
- api/services/kb/voyage.py: _post_with_retry wraps embed/rerank
POSTs with exponential backoff on 429+5xx (max 5 retries, base
2s, capped 30s, +25% jitter, honors Retry-After header). Dev and
prod share a single VOYAGE_API_KEY today, so a simultaneous bulk
re-ingest on both used to fail half the calls; now the burst is
absorbed in seconds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The bucket name was a holdover from the original single-domain
prototype (Israeli National Insurance only). With multi-topic support
the same bucket now hosts files for ביטוח לאומי, דיני עבודה, פלילי,
and any future legal domains — separated by `inbox/<topic_slug>/`
prefix, not by bucket. Generic `legal-kb` name reflects that.
Only the default in `_bucket()` changes; deployments that already set
`KB_S3_BUCKET` are unaffected. Dev was migrated to `legal-kb` in the
same operation (`mc mirror` of all 221 objects, env var update,
UPDATE on `kb_source.original_path` rewriting `s3://insurance-kb/` →
`s3://legal-kb/`). Prod MinIO has the empty `legal-kb` bucket ready
for shira-hermes prod cutover.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends scripts/fetch_national_insurance_law.py with a --regulations mode
that pulls all canonical (non-redirect) תקנות הביטוח הלאומי pages from the
MediaWiki API and writes <slug>.txt + <slug>.txt.meta.json pairs ready for
the existing scan-inbox cron.
Used to seed the live insurance KB with 92 regulations (~1,500 chunks,
~22MB Postgres) on dev. Wikisource's redirect graph dedups spelling
variants (קיצבה/קצבה, etc.) for free.
Bumps version to 0.2.0.
Refs Task Master #3
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two production-blocker fixes for the v0.8.0 batch upload UI, plus a
small endpoint to recover review state after a hard refresh.
classifier.py:
- timeout 45s → 90s and concurrency 5 → 2. ai-gateway proxies a
single Claude OAuth session so requests effectively serialize at
the gateway; with concurrency=5, the 4th and 5th files in a batch
of 5 timed out from queue waits alone (observed 2026-04-25 on
user upload of 5 circulars: jobs 10 and 13 returned empty).
- labels schema/prompt: arrays of strings via ai-gateway came back
as [{}, {}, {}] (proxy mangled items.type). Switched the param
to a comma-separated string with explicit format guidance, then
split in _normalize. _normalize is defensive — accepts both
string and list[str|dict] shapes so older callers don't break.
- Sharpened the prompt distinction between subject (free text on
the source) and labels (navigation tags), since identical
examples were causing the model to emit one and skip the other.
ingest_jobs.list_pending_batches + admin_kb GET /admin/kb/batches:
the review screen lives in JS RAM (_activeBatch). A hard refresh
wipes it, leaving uploaded jobs orphaned in awaiting_review with
no UI path back. The new endpoint groups jobs by batch_id with
status counters; the EspoCRM 'ממתינים לאישור' panel uses it to
let users resume any pending review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 6 of the multi-topic KB refactor (backend half). Subsumes the
original Task #19 design discussion (subject vs labels). EspoCRM
extension UI ships separately.
Migrations (4 new, all idempotent, transaction-wrapped):
004_kind_tool — adds 'tool' to kb_source/kb_ingest_job CHECK
005_source_description — kb_source.description + ai_classified_at
006_labels — kb_label + kb_source_label (M:N), GRANTs
007_ingest_classifier — awaiting_review status + processing_stage
+ ai_suggestions JSONB + batch_id
Two-phase ingest (api/services/kb/ingest_jobs.py):
queued → processing(stage=classifying)
→ awaiting_review ← user reviews AI output
→ processing(stage=embedding) ← after user commits
→ done | failed
process_classify_stage() pulls the file, runs parse_first_pages
(3-page text extract), calls classifier.classify (Claude tool-call
via ai-gateway, 45s timeout, 5-concurrent semaphore, fail-soft on
any error → empty suggestions), writes ai_suggestions JSONB,
transitions to awaiting_review.
commit_job() resolves user-confirmed labels (existing by slug,
new ones via slugify+ON CONFLICT DO NOTHING), transitions to
processing(embedding), schedules process_embed_stage.
process_embed_stage() runs the legacy ingest_source path with
user-edited metadata + summary as kb_source.description, then
apply_to_source for labels.
discard_job() removes a file from a batch (status=failed,
S3 deleted).
list_jobs_by_batch() — single-roundtrip review-screen load.
Labels (NEW api/services/kb/labels.py):
Hebrew → ASCII slugify (letter-by-letter map, no external dep);
CRUD; lookup_or_create_batch (race-safe); apply_to_source +
replace_for_source maintain usage_count; merge race-safely
(UPDATE assignments + ON CONFLICT DO NOTHING).
Classifier (NEW api/services/kb/classifier.py):
AsyncOpenAI to ai-gateway, Sonnet, OpenAI-style tool calling for
guaranteed JSON shape, Hebrew system prompt with 5 kind values,
citation format examples, "do not invent" rule, summary length
bounds 80-200 chars.
Routes (api/routes/admin_kb.py — 8 new on top of existing 11):
POST /admin/kb/upload-batch (multi-file, AI classify)
GET /admin/kb/batch/{batch_id} (review-screen load)
POST /admin/kb/jobs/{id}/commit (user confirms metadata)
POST /admin/kb/jobs/{id}/discard (user removes from batch)
GET /admin/kb/labels (typeahead)
POST /admin/kb/labels (explicit create)
POST /admin/kb/labels/{id}/merge (admin cleanup)
DELETE /admin/kb/labels/{id} (only if usage=0)
Public /kb/search gains optional label_id filter.
admin_sources.list_sources LEFT JOINs labels into each row's
output. update_source accepts label_ids to replace the set.
End-to-end smoke test passed: synthetic Hebrew circular →
upload-batch → background classify → awaiting_review with kind,
title, subject, summary, identifier, published_at all populated
correctly.
Refs Task Master #20 (espocrm-extensions/KnowledgeBase v0.8.0)
Subsumes: Task #19 (labels for sub-topics)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backs the Phase 4 topic-management table in the KnowledgeBase EspoCRM
extension ("ניהול" tab → "נושאים" panel). Four new admin endpoints:
GET /admin/kb/topics — list ALL topics including
is_active=false ones, with
source_count joined from kb_source
POST /admin/kb/topics — create. Validates slug shape
(^[a-z][a-z0-9-]{1,62}$) and
uniqueness — clean 400 on duplicate
rather than leaking UniqueViolation
PUT /admin/kb/topics/{id} — patch name, description,
system_prompt_addendum, is_active.
slug is intentionally NOT editable —
the S3 layout depends on it
(inbox/<topic_slug>/<kind>/...)
DELETE /admin/kb/topics/{id} — hard-delete; refuses with 409 if
any sources still belong to the
topic. Soft-delete (is_active=false)
is the normal "remove from dropdown"
path
Refs Task Master #15 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backs the Phase 3 sources table in the KnowledgeBase EspoCRM extension
("ניהול" tab). Four new admin endpoints:
GET /admin/kb/sources — list with chunk_count and
last-ingest status (LATERAL
join against kb_ingest_job),
filterable by topic_id + kind
PUT /admin/kb/sources/{id} — patch the user-editable
metadata only (title,
identifier, source_url,
published_at, effective_at);
structural fields (kind,
topic_id, checksum) are not
accepted
DELETE /admin/kb/sources/{id} — hard delete (kb_chunk cascades
via FK; the S3 object at
original_path is best-effort
deleted)
POST /admin/kb/sources/{id}/reingest — queues a kb_ingest_job tied
to the existing source_id;
background task replaces
chunks in-place, preserves
the source row and its
hand-edited metadata, and
updates the checksum
The reingest path differs from ingest_source: ingest_source creates a
new kb_source row and supersedes the previous version. Reingest keeps
the same id (so cached chunk_index references in the browser stay
valid), wipes its kb_chunk rows in a single transaction, and inserts
fresh ones from the same file.
Refs Task Master #14 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Court rulings are a distinct content type — long discussion / reasoning,
no enumerated statutory hierarchy, case identifier instead of statute id —
so they get their own kind alongside law/regulation/circular rather than
being shoehorned into one of the existing three.
Migration 003 alters the CHECK constraints on kb_source.kind and
kb_ingest_job.kind to include 'caselaw'. The chunker uses the
circulars path for caselaw too (paragraph + size split, no statutory
section regex); the case identifier is captured at upload time in
kb_source.identifier.
Updates Literal types in ingest_source, admin_kb upload, and the
kb_public SearchRequest. _KINDS in s3.py grows to four so failed/
processed/inbox listings include caselaw subdirs.
Refs Task Master #18 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 2 of the multi-topic KB refactor. Adds three admin endpoints that
back the new "ניהול" tab in the KnowledgeBase EspoCRM extension:
POST /admin/kb/upload — multipart, queues a kb_ingest_job and
fires asyncio.create_task to process
parse → chunk → embed in the
background. Returns {job_id, status}
immediately so the browser doesn't
block on slow PDFs.
GET /admin/kb/jobs — recent jobs, optionally filtered by
topic_id / status / limit.
GET /admin/kb/jobs/{id} — single-job detail for the polling UI.
Migration 002 adds kb_ingest_job (queued/processing/done/failed) with
indexes on (status, created_at) and (topic_id, created_at).
ingest_source now accepts topic_id (NULL → defaults to 1 for back-compat
with the existing scan-inbox cron path) and writes it to kb_source.
S3 layout for new uploads is topic-aware: inbox/<topic_slug>/<kind>/...
The legacy /scan-inbox path on inbox/<kind>/ is unchanged.
Refs Task Master #13 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 of the multi-domain refactor. Adds a kb_topic table seeded with
the existing 'ביטוח לאומי' domain (id=1) and pins all 6 existing
kb_source rows to it, so search/ask can be scoped per domain without
disturbing the current single-topic UX.
- scripts/migrations/001_topics.sql: idempotent migration. Creates
kb_topic, seeds national-insurance with the system_prompt_addendum
copied verbatim from kb_public.py's hardcoded text, adds nullable
kb_source.topic_id, backfills it to 1, then SET NOT NULL. GRANTs to
shira_kb mirror its existing kb_source privileges. Wrapped in BEGIN
/COMMIT with a sanity-check that raises if the backfill is incomplete.
- api/services/kb/topics.py: list_topics / get_topic / resolve_topic_id.
resolve_topic_id raises ValueError on unknown ids so the route layer
can return 400 instead of silently mixing domains.
- api/services/kb/search.py: search() + _retrieve_rrf accept topic_id;
the SQL adds AND s.topic_id = $N when present. _expand_query is
parametrized by topic name (was hardcoded "הביטוח הלאומי").
- api/routes/kb_public.py: new GET /kb/topics. /kb/search /kb/sources
/kb/ask /kb/ask/stream all accept topic_id and call resolve_topic_id.
_build_ask_runner_context loads system_prompt_addendum from the DB
instead of the hardcoded insurance string.
- mcp_server/tools/legal_kb_tools.py: tool renamed
search_insurance_kb → search_legal_kb, takes topic_id at registration
so each conversation is pinned to its caller-chosen domain. Tool
description interpolates the topic name.
- api/services/agent_runner.py: kb_topic_id + kb_topic_name flow
through to register_legal_kb_tools. Hebrew progress labels updated.
Refs Task Master #12 (espocrm-extensions/KnowledgeBase).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds POST /kb/ask/stream that returns text/event-stream. The agent
runs in a fire-and-forget task; an on_event callback wired into
AgentRunner.run pushes {type, label, ...} events into an asyncio.Queue,
and the response generator consumes them as SSE data: lines.
Event types and Hebrew labels:
- thinking — before each LLM call ("קוראת את השאלה" / "ממשיכה לחקור")
- tool_start — before each tool. For search_insurance_kb the label
includes the actual query string Shira chose, so the user can see
WHAT she's searching for ("מחפשת בבסיס הידע: '...'"). Other tools
fall back to a generic Hebrew label keyed off the function name.
- tool_done / tool_error — after each tool. For search_insurance_kb
the label echoes the first non-empty line of the tool result (which
the legal_kb tool formats as "## נמצאו N קטעים").
- writing — when the model returns text without tool calls.
- answer — final event with {text, sources, conversationId}, mirrors
the non-streaming /kb/ask response shape.
- error — runner exception; client should close the EventSource.
Heartbeats: when the queue is idle for >15s the generator yields a
": keep-alive" comment line (ignored by EventSource) so any proxy in
the path doesn't drop the connection during long agent thinks.
The original POST /kb/ask is unchanged — the streaming endpoint is
additive. Both share _build_ask_runner_context + _build_ask_messages
helpers extracted from the original handler.
Refs Task Master #1
Two minimal additions for the upcoming v0.1.11 client work:
- /kb/search hits now include chunk_index, so the client can call
/source/{id}/chunks?around=N&ctx=K to fetch the matched chunk plus K
chunks before/after as context. Required by the new search-preview
layout for text sources, which until now duplicated the matched chunk
in both panes.
- /source/{id}/chunks accepts around (chunk_index) + ctx (window radius
clamped to 0..10). When around is given, returns only the windowed
range; otherwise unchanged (returns all chunks of the source).
Refs Task Master #1
Single-shot search ranks the source whose title literally contains the
query terms way above everything else, so "מהי תקנה 37" returns 6 hits
from the תקנה 37 circular + 2 from the law and zero from ספר הליקויים
— even though the latter is the canonical medical reference for that
regulation. The ask agent already worked around this by issuing several
search_insurance_kb calls with different phrasings; that intelligence is
now also available to the public /kb/search endpoint.
Implementation:
- _retrieve_rrf extracts vector + lexical retrieval (RRF fused, no
rerank) into a reusable async helper. Each candidate now carries
chunk_id so callers can dedup when merging multiple sub-searches.
- _expand_query asks the configured Claude model (via ai-gateway) for
up to 3 alternative phrasings, with a 12s timeout. On any failure the
caller silently falls back to single-query mode.
- search() gains an `expand: bool` parameter. When true, runs the
original query plus all variants through _retrieve_rrf in parallel,
merges by chunk_id keeping the best RRF score, and reranks the union
against the ORIGINAL query so the final order reflects what the user
asked, not the variants.
- /kb/search defaults to expand=true.
Cost per call: 1 extra LLM round-trip (~1-2s on Sonnet) and 3 extra
Voyage embeds (~150ms each, parallel). Latency budget grows from ~700ms
to ~3s, acceptable for a search the user explicitly triggered.
Refs Task Master #1
_split_oversized slices at fixed char offsets and can cut a
\x00KB_PAGE:N\x00 sentinel in half — the broken prefix/suffix no longer
matches _PAGE_MARKER_RE so _strip_page_markers leaves the \x00 byte in
place. PostgreSQL's UTF-8 check then rejects the insert with
"invalid byte sequence for encoding UTF8: 0x00", and ingest of
ספר הליקויים fails on every scan. Belt-and-suspenders: strip all
remaining \x00 bytes from clean content before returning.
Refs Task Master #2
Commit 426b0d6 fixed page propagation through the Chunk reconstruction in
chunk(), but two downstream paths still collapsed every sub-chunk to the
parent's starting page:
1. _split_oversized operated on markerless content (since _as_dicts inside
chunk_statute/chunk_circular stripped markers before returning), so when
a single giant chunk was sliced into pieces — the common path for
ספר הליקויים / ספר המבחנים, which have no statute-style section markers —
every piece inherited c.page_number = parent's start page. 209/209 chunks
landed on page 1.
2. chunk_circular packed paragraphs into sub-chunks all tagged with
seg_page (the segment's opening page), with no attempt to track where
each sub-chunk actually begins within the original text. חוזר תקנה 37
(8 chunks) and חוזר נכות 1941 (4 chunks) — neither had header matches —
landed entirely on page 1.
Fixes:
- chunk_statute and chunk_circular now return list[Chunk] with markers
still embedded in content; _as_dicts is called once at the end of
chunk() so markers are stripped AFTER splitting.
- _split_oversized builds a local page map from markers in each parent
chunk's content, with a virtual (0, parent.page_number) entry so pieces
before the first real marker still inherit the correct page.
- chunk_circular tracks each paragraph's absolute offset in the original
text and computes each emitted sub-chunk's page_number from its first
paragraph's offset against the full-text page_map.
Verified on synthetic 3-page input: regulation path now emits pages 1,1,2,3
across 4 pieces (was 1,1,1,1); circular path emits 1,1,2,2,3,3 (was
1,1,1,1,1,1).
Refs Task Master #2
chunk() reconstructs Chunk objects before _split_oversized/_as_dicts and
was dropping page_number on the floor, so every chunk came out NULL
regardless of what chunk_statute computed. Also flip _as_dicts to prefer
the caller-supplied c.page_number (derived from the chunk's starting
offset) over any marker that survived inside c.content — the latter can
resolve to a later page when a section spans pages.
Verified: ספר הליקויים chunks now carry page_number matching the PDF
page where each section actually begins.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
To let the browse/ask UIs jump straight to the right PDF page:
- ingest.py: embed \x00KB_PAGE:<N>\x00 sentinels at the start of each
PDF page during _parse_pdf.
- chunker.py: _strip_page_markers removes the sentinels and records the
first page seen per chunk as page_number. Chunk dataclass + _as_dicts
propagate page_number through both chunk_statute and chunk_circular.
- ingest.py INSERT: new kb_chunk.page_number column (DB migration
applied manually).
- search.py SQL: SELECT c.page_number + s.id + s.original_path so the
rerank results carry them forward.
- legal_kb_tools.py: register_legal_kb_tools(sources_used=list) — the
tool appends {source_id, title, kind, identifier, heading_path,
section_ref, page_number, original_path} for each hit, deduped by
(source_id, page_number).
- agent_runner.py: new kb_sources_used kwarg threads the list through
to the tool registration.
- kb_public.py /kb/ask: collects sources_used during the run, filters
to PDF-only sources, and returns them alongside text.
Chunks stored before this change have page_number = NULL; PDFs need to
be re-ingested to populate them. The TXT law source is unaffected.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The browse tab in the EspoCRM KnowledgeBase extension needs the original
PDF (with columns, tables, labels) to render layout-heavy sources like
ספר הליקויים readably. The plain-text chunk view loses the structure
because the PDF itself places each word as a separate text block.
New route:
- GET /kb/source/{id}/pdf — resolves the source's `original_path`
(stored as `s3://insurance-kb/processed/<kind>/<file>.pdf`), streams
the bytes from MinIO via the existing kb_s3.fetch helper, and returns
them with Content-Type: application/pdf + inline Content-Disposition.
- 404 when the source doesn't exist.
- 409 when the source was ingested as plain text (e.g. the law from
Wikisource — its original_path ends in .txt, not .pdf).
- Response is chunked (64 KB) to avoid pinning a 2 MB buffer for every
request.
Also select `original_path` in /kb/source/{id}/chunks so the frontend
can decide whether to render the PDF viewer or the chunk list.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous block-based parser left each pymupdf block as its own line.
That's fine for prose but wrong for ספר הליקויים's cover and preface,
where each word is its own block — you got "ו אנ / מתכבד / ים" instead
of "ואנו מתכבדים". Merge blocks whose y-coordinates are within 4 pixels
into a single line, right-to-left, joined by spaces.
Content pages (where blocks already contain whole paragraphs) are
unaffected because a single block on one row stays as one row.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pymupdf's `get_text("text")` reads glyph runs in raw order, which on
multi-column Hebrew PDFs (ספר הליקויים, ספר המבחנים) interleaves column
headers, page numbers, form labels, and body paragraphs into jumbled
lines — words split to single characters on separate lines, paragraphs
lost. Switch to `get_text("blocks")` and sort each page's blocks by
(y-axis ascending, x-axis descending) so each paragraph stays intact
and the reading order is top-to-bottom, right-to-left as expected.
Side-effect: chunk content becomes readable in the browse view.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ספר הליקויים produced 698 chunks, most of them junk. A line like
"3 . . . . . . . . . . . . . . . . . . . . . . . . . . הוראות החוק" from
its table of contents was being treated as section 3. Phone numbers like
"02-6709701" became section 02. Both of these polluted the index and
broke the browse view's readability.
Tighten _RE_SECTION so it requires either the explicit prefix (סעיף /
תקנה / §) OR real text immediately after the terminator — a row of dots
or more digits disqualifies. The scraped חוק הביטוח הלאומי still chunks
correctly because the scraper emits "סעיף N. ..." with the prefix.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The KB UI's ask mode was hijacking into generic Israeli law. Asked
"מתי אפשר לבקש דיון מחדש לפי תקנה 37?" Shira answered with תקנה 37
of תקנות סדר הדין האזרחי (civil procedure) instead of תקנה 37 to the
National Insurance regulations — even though search_insurance_kb has
the correct source.
Append a short context block to the system prompt telling Shira that
every question from the KB UI is about ביטוח לאומי, that she must call
search_insurance_kb first and cite section_ref, and that she must say
so explicitly if no KB section matches.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three endpoints under /kb/*, same X-Api-Key auth as the admin and
SmartAssistant routes:
- POST /kb/search — hybrid search + rerank, returns ranked chunks as JSON.
No LLM in the path, so it's fast enough for a live UI.
- GET /kb/sources — list of active sources (law / regulation / circular)
with basic metadata, for the browse tab.
- GET /kb/source/{id}/chunks — all chunks of one source, ordered, so
the UI can show a document inline with its hierarchical headings.
- POST /kb/ask — full agent loop (same runner as SmartAssistant but with
allowed_toolsets=legal), for the ask-shira mode of the UI.
Dates are serialized to ISO strings so the frontend doesn't have to deal
with Python date objects.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the n8n worker went down and recovered, multiple queued cron
executions fired scan-inbox in parallel. Each one saw the same file in
s3://insurance-kb/inbox/, each one called ingest_source — and the DB
ended up with duplicate source rows with identical checksums created in
the same second.
Guard scan-inbox with a session-level pg_try_advisory_lock (key "KBSC").
Concurrent callers return {skipped: true} immediately. The lock is
released in a finally block so a crash mid-scan still frees the next
scheduled run.
No schema change; no change to single-caller behavior.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a rerank stage between the hybrid RRF fusion and the final top_k cut.
RRF is a heuristic over ranks — it can't tell which of two equally-ranked
candidates is actually more relevant. A cross-encoder scores each
(query, passage) pair directly and produces a true relevance order.
- voyage.rerank(query, documents, top_k) → list of {index, relevance_score}
sorted high→low. Defaults to the model in VOYAGE_RERANK_MODEL
(rerank-2.5, 32K tokens per pair, multilingual incl. Hebrew).
- search pulls top-30 from RRF, passes them to rerank, then returns the
reranker's top-k. Each returned item gets a rerank_score field.
- Fallback: if the rerank API errors (auth, 5xx, timeout), fall back to
the RRF ordering and log a warning — degraded precision beats outage.
- KB_RERANK_DISABLED=1 bypasses rerank for debugging.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sidecar JSON supplies published_at / effective_at as 'YYYY-MM-DD' strings.
asyncpg rejects strings for DATE columns even with '::date' casts in the
SQL, so ingest now parses them via datetime.date.fromisoformat before the
INSERT. Non-string inputs (date, datetime, None, empty) are handled too.
Before: ingest crashed on any sidecar with a date, leaving the source in
failed/ while sidecarless siblings succeeded.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each file in the inbox can now be accompanied by a `<filename>.meta.json`
sidecar that supplies identifier, published_at, effective_at, and
source_url. The scanner applies filename-derived defaults first and lets
the sidecar override individual fields.
- list_inbox / list_failed skip `.meta.json` so sidecars aren't ingested
as standalone documents.
- mark_processed, mark_failed, and requeue-failed move the sidecar
alongside its parent (best-effort).
- mark_failed now sanitizes the error string to ASCII before writing it
to S3 object metadata (HTTP header encoding).
- search SQL selects published_at/effective_at; the formatter shows the
full heading_path plus publication date so citations are unambiguous.
- scripts/kb_sidecar_template.json documents the expected shape.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-shot script that fetches the consolidated National Insurance Law from
Hebrew Wikisource and emits plain text with heading lines the chunker can
parse (פרק / סימן / סעיף).
Key quirks of the source markup:
- {{ח:קטע2}} carries the level name inside the title parameter ("פרק א׳:
…"), not in the template name. Emit the title verbatim and let the
chunker's existing prefix regex handle it.
- Named args (e.g. 'אחר=[פרק א׳]') must be filtered — the original
positional parser was treating them as positional titles and producing
gibberish "חלק אחר=[פרק א׳]" headings.
- No חלק level exists in this law — only פרק > סימן > סעיף.
Run as: python3 scripts/fetch_national_insurance_law.py <out_path>
Then upload the output .txt to s3://insurance-kb/inbox/law/ and invoke
POST /admin/kb/scan-inbox.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Before: chunk_statute produced heading_path = section_ref = "ס' 195".
After: heading_path walks the four Israeli-statute levels (part, chapter,
sub-chapter, section) and renders them as "חלק ט׳ — ביטוח נכות > פרק ב׳
— ועדות רפואיות > סימן א׳ > ס' 195". section_ref stays the leaf so
citations stay precise.
- New _RE_PART / _RE_CHAPTER / _RE_SUBCHAPTER regexes with Hebrew ordinal
support including multi-letter gershayim ("י״א", "ט״ז").
- _collect_markers emits all markers ordered by position; chunk_statute
walks them tracking active_part/chapter/subchapter and attaches the
full path to each section chunk.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous batching sent up to 128 inputs per request but didn't cap total
tokens. On large Hebrew statutes (ספר הליקויים, ספר המבחנים) a single
request exceeded Voyage's 120K-token-per-batch hard cap and failed with 400.
- voyage.embed now pre-computes an estimated token count per input (Hebrew
is ~0.55 tokens/char in voyage-multilingual-2) and emits multiple
requests so each stays under a 90K-token safety margin.
- chunker lowers the per-chunk ceiling to 1500 tokens and updates the
chars→tokens estimate to match the observed tokenizer behavior.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two bugs surfaced during Phase 1 validation on real circulars:
1. The chunker produced one giant chunk for large statutes (ספר הליקויים,
ספר המבחנים) that the section regex could not split. This exceeded
Voyage's 120K-token batch limit and the ingest failed. Add a hard
2000-token ceiling per chunk with a character-based fallback split that
preserves heading_path and section_ref.
2. ingest_source matched "existing source" by kind + COALESCE(identifier,
''), so any two sources with NULL identifier compared equal. The second
ingest would then supersede the first unrelated document. Match by
identifier only when one is provided; otherwise fall back to
(kind + title) and require the candidate to still be active.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves every object under failed/<kind>/ back to inbox/<kind>/ for retry
after a transient upstream failure (e.g. Voyage 401 before a key rotation).
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chaim