Phase 6 of the multi-topic KB refactor (backend half). Subsumes the
original Task #19 design discussion (subject vs labels). EspoCRM
extension UI ships separately.
Migrations (4 new, all idempotent, transaction-wrapped):
004_kind_tool — adds 'tool' to kb_source/kb_ingest_job CHECK
005_source_description — kb_source.description + ai_classified_at
006_labels — kb_label + kb_source_label (M:N), GRANTs
007_ingest_classifier — awaiting_review status + processing_stage
+ ai_suggestions JSONB + batch_id
Two-phase ingest (api/services/kb/ingest_jobs.py):
queued → processing(stage=classifying)
→ awaiting_review ← user reviews AI output
→ processing(stage=embedding) ← after user commits
→ done | failed
process_classify_stage() pulls the file, runs parse_first_pages
(3-page text extract), calls classifier.classify (Claude tool-call
via ai-gateway, 45s timeout, 5-concurrent semaphore, fail-soft on
any error → empty suggestions), writes ai_suggestions JSONB,
transitions to awaiting_review.
commit_job() resolves user-confirmed labels (existing by slug,
new ones via slugify+ON CONFLICT DO NOTHING), transitions to
processing(embedding), schedules process_embed_stage.
process_embed_stage() runs the legacy ingest_source path with
user-edited metadata + summary as kb_source.description, then
apply_to_source for labels.
discard_job() removes a file from a batch (status=failed,
S3 deleted).
list_jobs_by_batch() — single-roundtrip review-screen load.
Labels (NEW api/services/kb/labels.py):
Hebrew → ASCII slugify (letter-by-letter map, no external dep);
CRUD; lookup_or_create_batch (race-safe); apply_to_source +
replace_for_source maintain usage_count; merge race-safely
(UPDATE assignments + ON CONFLICT DO NOTHING).
Classifier (NEW api/services/kb/classifier.py):
AsyncOpenAI to ai-gateway, Sonnet, OpenAI-style tool calling for
guaranteed JSON shape, Hebrew system prompt with 5 kind values,
citation format examples, "do not invent" rule, summary length
bounds 80-200 chars.
Routes (api/routes/admin_kb.py — 8 new on top of existing 11):
POST /admin/kb/upload-batch (multi-file, AI classify)
GET /admin/kb/batch/{batch_id} (review-screen load)
POST /admin/kb/jobs/{id}/commit (user confirms metadata)
POST /admin/kb/jobs/{id}/discard (user removes from batch)
GET /admin/kb/labels (typeahead)
POST /admin/kb/labels (explicit create)
POST /admin/kb/labels/{id}/merge (admin cleanup)
DELETE /admin/kb/labels/{id} (only if usage=0)
Public /kb/search gains optional label_id filter.
admin_sources.list_sources LEFT JOINs labels into each row's
output. update_source accepts label_ids to replace the set.
End-to-end smoke test passed: synthetic Hebrew circular →
upload-batch → background classify → awaiting_review with kind,
title, subject, summary, identifier, published_at all populated
correctly.
Refs Task Master #20 (espocrm-extensions/KnowledgeBase v0.8.0)
Subsumes: Task #19 (labels for sub-topics)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Court rulings are a distinct content type — long discussion / reasoning,
no enumerated statutory hierarchy, case identifier instead of statute id —
so they get their own kind alongside law/regulation/circular rather than
being shoehorned into one of the existing three.
Migration 003 alters the CHECK constraints on kb_source.kind and
kb_ingest_job.kind to include 'caselaw'. The chunker uses the
circulars path for caselaw too (paragraph + size split, no statutory
section regex); the case identifier is captured at upload time in
kb_source.identifier.
Updates Literal types in ingest_source, admin_kb upload, and the
kb_public SearchRequest. _KINDS in s3.py grows to four so failed/
processed/inbox listings include caselaw subdirs.
Refs Task Master #18 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 of the multi-domain refactor. Adds a kb_topic table seeded with
the existing 'ביטוח לאומי' domain (id=1) and pins all 6 existing
kb_source rows to it, so search/ask can be scoped per domain without
disturbing the current single-topic UX.
- scripts/migrations/001_topics.sql: idempotent migration. Creates
kb_topic, seeds national-insurance with the system_prompt_addendum
copied verbatim from kb_public.py's hardcoded text, adds nullable
kb_source.topic_id, backfills it to 1, then SET NOT NULL. GRANTs to
shira_kb mirror its existing kb_source privileges. Wrapped in BEGIN
/COMMIT with a sanity-check that raises if the backfill is incomplete.
- api/services/kb/topics.py: list_topics / get_topic / resolve_topic_id.
resolve_topic_id raises ValueError on unknown ids so the route layer
can return 400 instead of silently mixing domains.
- api/services/kb/search.py: search() + _retrieve_rrf accept topic_id;
the SQL adds AND s.topic_id = $N when present. _expand_query is
parametrized by topic name (was hardcoded "הביטוח הלאומי").
- api/routes/kb_public.py: new GET /kb/topics. /kb/search /kb/sources
/kb/ask /kb/ask/stream all accept topic_id and call resolve_topic_id.
_build_ask_runner_context loads system_prompt_addendum from the DB
instead of the hardcoded insurance string.
- mcp_server/tools/legal_kb_tools.py: tool renamed
search_insurance_kb → search_legal_kb, takes topic_id at registration
so each conversation is pinned to its caller-chosen domain. Tool
description interpolates the topic name.
- api/services/agent_runner.py: kb_topic_id + kb_topic_name flow
through to register_legal_kb_tools. Hebrew progress labels updated.
Refs Task Master #12 (espocrm-extensions/KnowledgeBase).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two minimal additions for the upcoming v0.1.11 client work:
- /kb/search hits now include chunk_index, so the client can call
/source/{id}/chunks?around=N&ctx=K to fetch the matched chunk plus K
chunks before/after as context. Required by the new search-preview
layout for text sources, which until now duplicated the matched chunk
in both panes.
- /source/{id}/chunks accepts around (chunk_index) + ctx (window radius
clamped to 0..10). When around is given, returns only the windowed
range; otherwise unchanged (returns all chunks of the source).
Refs Task Master #1
Single-shot search ranks the source whose title literally contains the
query terms way above everything else, so "מהי תקנה 37" returns 6 hits
from the תקנה 37 circular + 2 from the law and zero from ספר הליקויים
— even though the latter is the canonical medical reference for that
regulation. The ask agent already worked around this by issuing several
search_insurance_kb calls with different phrasings; that intelligence is
now also available to the public /kb/search endpoint.
Implementation:
- _retrieve_rrf extracts vector + lexical retrieval (RRF fused, no
rerank) into a reusable async helper. Each candidate now carries
chunk_id so callers can dedup when merging multiple sub-searches.
- _expand_query asks the configured Claude model (via ai-gateway) for
up to 3 alternative phrasings, with a 12s timeout. On any failure the
caller silently falls back to single-query mode.
- search() gains an `expand: bool` parameter. When true, runs the
original query plus all variants through _retrieve_rrf in parallel,
merges by chunk_id keeping the best RRF score, and reranks the union
against the ORIGINAL query so the final order reflects what the user
asked, not the variants.
- /kb/search defaults to expand=true.
Cost per call: 1 extra LLM round-trip (~1-2s on Sonnet) and 3 extra
Voyage embeds (~150ms each, parallel). Latency budget grows from ~700ms
to ~3s, acceptable for a search the user explicitly triggered.
Refs Task Master #1
To let the browse/ask UIs jump straight to the right PDF page:
- ingest.py: embed \x00KB_PAGE:<N>\x00 sentinels at the start of each
PDF page during _parse_pdf.
- chunker.py: _strip_page_markers removes the sentinels and records the
first page seen per chunk as page_number. Chunk dataclass + _as_dicts
propagate page_number through both chunk_statute and chunk_circular.
- ingest.py INSERT: new kb_chunk.page_number column (DB migration
applied manually).
- search.py SQL: SELECT c.page_number + s.id + s.original_path so the
rerank results carry them forward.
- legal_kb_tools.py: register_legal_kb_tools(sources_used=list) — the
tool appends {source_id, title, kind, identifier, heading_path,
section_ref, page_number, original_path} for each hit, deduped by
(source_id, page_number).
- agent_runner.py: new kb_sources_used kwarg threads the list through
to the tool registration.
- kb_public.py /kb/ask: collects sources_used during the run, filters
to PDF-only sources, and returns them alongside text.
Chunks stored before this change have page_number = NULL; PDFs need to
be re-ingested to populate them. The TXT law source is unaffected.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a rerank stage between the hybrid RRF fusion and the final top_k cut.
RRF is a heuristic over ranks — it can't tell which of two equally-ranked
candidates is actually more relevant. A cross-encoder scores each
(query, passage) pair directly and produces a true relevance order.
- voyage.rerank(query, documents, top_k) → list of {index, relevance_score}
sorted high→low. Defaults to the model in VOYAGE_RERANK_MODEL
(rerank-2.5, 32K tokens per pair, multilingual incl. Hebrew).
- search pulls top-30 from RRF, passes them to rerank, then returns the
reranker's top-k. Each returned item gets a rerank_score field.
- Fallback: if the rerank API errors (auth, 5xx, timeout), fall back to
the RRF ordering and log a warning — degraded precision beats outage.
- KB_RERANK_DISABLED=1 bypasses rerank for debugging.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each file in the inbox can now be accompanied by a `<filename>.meta.json`
sidecar that supplies identifier, published_at, effective_at, and
source_url. The scanner applies filename-derived defaults first and lets
the sidecar override individual fields.
- list_inbox / list_failed skip `.meta.json` so sidecars aren't ingested
as standalone documents.
- mark_processed, mark_failed, and requeue-failed move the sidecar
alongside its parent (best-effort).
- mark_failed now sanitizes the error string to ASCII before writing it
to S3 object metadata (HTTP header encoding).
- search SQL selects published_at/effective_at; the formatter shows the
full heading_path plus publication date so citations are unambiguous.
- scripts/kb_sidecar_template.json documents the expected shape.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a searchable KB for חוק הביטוח הלאומי, תקנות הביטוח הלאומי, and חוזרי
הביטוח הלאומי. Hybrid search (pgvector cosine + tsvector/trgm) fused with
Reciprocal Rank Fusion, exposed to Shira as the `search_insurance_kb` tool.
- api/services/kb/: asyncpg pool, Voyage voyage-multilingual-2 client,
per-kind chunker (statute-by-section / circular header-aware), ingest
pipeline with supersession (old source → superseded_by), hybrid search.
- api/routes/admin_kb.py: POST /admin/kb/ingest (multipart), GET /admin/kb/stats.
- mcp_server/tools/legal_kb_tools.py: `search_insurance_kb` registered under
the `legal` toolset.
- pyproject.toml: +asyncpg, +python-docx, +boto3, +python-multipart.
Infra (external): pgvector/pgvector:pg16 on the shared PG, insurance_kb DB
with hnsw + gin indexes, MinIO bucket insurance-kb, KB_DATABASE_URL in
Infisical /espocrm, VOYAGE_API_KEY/VOYAGE_MODEL/KB_DATABASE_URL on the
shira-hermes Coolify app.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chaim