Phase 1 of the multi-domain refactor. Adds a kb_topic table seeded with
the existing 'ביטוח לאומי' domain (id=1) and pins all 6 existing
kb_source rows to it, so search/ask can be scoped per domain without
disturbing the current single-topic UX.
- scripts/migrations/001_topics.sql: idempotent migration. Creates
kb_topic, seeds national-insurance with the system_prompt_addendum
copied verbatim from kb_public.py's hardcoded text, adds nullable
kb_source.topic_id, backfills it to 1, then SET NOT NULL. GRANTs to
shira_kb mirror its existing kb_source privileges. Wrapped in BEGIN
/COMMIT with a sanity-check that raises if the backfill is incomplete.
- api/services/kb/topics.py: list_topics / get_topic / resolve_topic_id.
resolve_topic_id raises ValueError on unknown ids so the route layer
can return 400 instead of silently mixing domains.
- api/services/kb/search.py: search() + _retrieve_rrf accept topic_id;
the SQL adds AND s.topic_id = $N when present. _expand_query is
parametrized by topic name (was hardcoded "הביטוח הלאומי").
- api/routes/kb_public.py: new GET /kb/topics. /kb/search /kb/sources
/kb/ask /kb/ask/stream all accept topic_id and call resolve_topic_id.
_build_ask_runner_context loads system_prompt_addendum from the DB
instead of the hardcoded insurance string.
- mcp_server/tools/legal_kb_tools.py: tool renamed
search_insurance_kb → search_legal_kb, takes topic_id at registration
so each conversation is pinned to its caller-chosen domain. Tool
description interpolates the topic name.
- api/services/agent_runner.py: kb_topic_id + kb_topic_name flow
through to register_legal_kb_tools. Hebrew progress labels updated.
Refs Task Master #12 (espocrm-extensions/KnowledgeBase).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
To let the browse/ask UIs jump straight to the right PDF page:
- ingest.py: embed \x00KB_PAGE:<N>\x00 sentinels at the start of each
PDF page during _parse_pdf.
- chunker.py: _strip_page_markers removes the sentinels and records the
first page seen per chunk as page_number. Chunk dataclass + _as_dicts
propagate page_number through both chunk_statute and chunk_circular.
- ingest.py INSERT: new kb_chunk.page_number column (DB migration
applied manually).
- search.py SQL: SELECT c.page_number + s.id + s.original_path so the
rerank results carry them forward.
- legal_kb_tools.py: register_legal_kb_tools(sources_used=list) — the
tool appends {source_id, title, kind, identifier, heading_path,
section_ref, page_number, original_path} for each hit, deduped by
(source_id, page_number).
- agent_runner.py: new kb_sources_used kwarg threads the list through
to the tool registration.
- kb_public.py /kb/ask: collects sources_used during the run, filters
to PDF-only sources, and returns them alongside text.
Chunks stored before this change have page_number = NULL; PDFs need to
be re-ingested to populate them. The TXT law source is unaffected.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each file in the inbox can now be accompanied by a `<filename>.meta.json`
sidecar that supplies identifier, published_at, effective_at, and
source_url. The scanner applies filename-derived defaults first and lets
the sidecar override individual fields.
- list_inbox / list_failed skip `.meta.json` so sidecars aren't ingested
as standalone documents.
- mark_processed, mark_failed, and requeue-failed move the sidecar
alongside its parent (best-effort).
- mark_failed now sanitizes the error string to ASCII before writing it
to S3 object metadata (HTTP header encoding).
- search SQL selects published_at/effective_at; the formatter shows the
full heading_path plus publication date so citations are unambiguous.
- scripts/kb_sidecar_template.json documents the expected shape.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a searchable KB for חוק הביטוח הלאומי, תקנות הביטוח הלאומי, and חוזרי
הביטוח הלאומי. Hybrid search (pgvector cosine + tsvector/trgm) fused with
Reciprocal Rank Fusion, exposed to Shira as the `search_insurance_kb` tool.
- api/services/kb/: asyncpg pool, Voyage voyage-multilingual-2 client,
per-kind chunker (statute-by-section / circular header-aware), ingest
pipeline with supersession (old source → superseded_by), hybrid search.
- api/routes/admin_kb.py: POST /admin/kb/ingest (multipart), GET /admin/kb/stats.
- mcp_server/tools/legal_kb_tools.py: `search_insurance_kb` registered under
the `legal` toolset.
- pyproject.toml: +asyncpg, +python-docx, +boto3, +python-multipart.
Infra (external): pgvector/pgvector:pg16 on the shared PG, insurance_kb DB
with hnsw + gin indexes, MinIO bucket insurance-kb, KB_DATABASE_URL in
Infisical /espocrm, VOYAGE_API_KEY/VOYAGE_MODEL/KB_DATABASE_URL on the
shira-hermes Coolify app.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 4 Python tool registrations that call EspoCRM's SmartAssistant
executeTool endpoint with the DigitalSignature plugin tools:
- send_for_signature — send document for signing with auto-resolved signers
- check_signature_status — query status of signature requests
- remind_signers — send WhatsApp reminders to pending signers
- list_pending_signatures — show all awaiting signature requests
Tools are registered under the "signature" toolset, enabled by default.
Requires DigitalSignature extension v2.0.0+ on the EspoCRM side.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the CRM's readDocument returns success:false (scanned PDFs,
image-only DOCX, corrupted files), shira-hermes now:
1. Fetches the raw file bytes via new getDocumentBytes endpoint
2. Sends them to Claude Vision through ai-gateway:
- Images (PNG/JPG/TIFF/etc): direct vision call
- PDFs: render each page at 180 DPI via PyMuPDF, vision call per page
- DOCX: extract images from word/media/, vision call per image
New tool: read_document_ocr (explicit OCR request)
Existing tool read_document auto-falls-back to OCR on extraction failure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When EspoCRM returns documents as a flat list (legacy shape) instead of
the expected {totalFiles, folders, ...} dict, list_documents crashed with
"list object has no attribute 'get'". Now handles both shapes gracefully.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EspoCRM stores dates in UTC. Shira was displaying them raw (e.g., 08:00 UTC
instead of 11:00 Israel time) and storing user-provided local times as UTC.
- Add _to_local() in prompt_builder to convert UTC→Asia/Jerusalem for all
displayed dates (hearings, meetings, calls, tasks, notes)
- Update normalize_date() in _helpers to convert Israel local→UTC before
sending to EspoCRM
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Security:
- Path traversal: validate user_id, case_id, skill_name against safe regex
- Input sanitization in memory.py and skills.py file path construction
Correctness:
- ISR timezone: replace hardcoded UTC+3 with ZoneInfo("Asia/Jerusalem") for DST
- save_rule: use EspoCrmApiError.status_code instead of fragile string matching
- Background tasks: store references to prevent GC before completion
- Lock race: use dict.setdefault() for atomic lock creation
- Version: health.py now matches main.py (0.2.0)
- Skill names: normalized to lowercase for dedup consistency
Build:
- Dockerfile: install from pyproject.toml instead of hardcoded pip list
- Remove unused mcp>=1.0.0 dependency from pyproject.toml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The AssistantRule entity may not exist on some CRM instances. When save_rule
gets a 404, it now falls back to saving the rule in the local user profile
(update_user_profile). Also improved update_user_profile description so
Shira is more likely to use it for user preferences.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
chaim