The bucket name was a holdover from the original single-domain
prototype (Israeli National Insurance only). With multi-topic support
the same bucket now hosts files for ביטוח לאומי, דיני עבודה, פלילי,
and any future legal domains — separated by `inbox/<topic_slug>/`
prefix, not by bucket. Generic `legal-kb` name reflects that.
Only the default in `_bucket()` changes; deployments that already set
`KB_S3_BUCKET` are unaffected. Dev was migrated to `legal-kb` in the
same operation (`mc mirror` of all 221 objects, env var update,
UPDATE on `kb_source.original_path` rewriting `s3://insurance-kb/` →
`s3://legal-kb/`). Prod MinIO has the empty `legal-kb` bucket ready
for shira-hermes prod cutover.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 6 of the multi-topic KB refactor (backend half). Subsumes the
original Task #19 design discussion (subject vs labels). EspoCRM
extension UI ships separately.
Migrations (4 new, all idempotent, transaction-wrapped):
004_kind_tool — adds 'tool' to kb_source/kb_ingest_job CHECK
005_source_description — kb_source.description + ai_classified_at
006_labels — kb_label + kb_source_label (M:N), GRANTs
007_ingest_classifier — awaiting_review status + processing_stage
+ ai_suggestions JSONB + batch_id
Two-phase ingest (api/services/kb/ingest_jobs.py):
queued → processing(stage=classifying)
→ awaiting_review ← user reviews AI output
→ processing(stage=embedding) ← after user commits
→ done | failed
process_classify_stage() pulls the file, runs parse_first_pages
(3-page text extract), calls classifier.classify (Claude tool-call
via ai-gateway, 45s timeout, 5-concurrent semaphore, fail-soft on
any error → empty suggestions), writes ai_suggestions JSONB,
transitions to awaiting_review.
commit_job() resolves user-confirmed labels (existing by slug,
new ones via slugify+ON CONFLICT DO NOTHING), transitions to
processing(embedding), schedules process_embed_stage.
process_embed_stage() runs the legacy ingest_source path with
user-edited metadata + summary as kb_source.description, then
apply_to_source for labels.
discard_job() removes a file from a batch (status=failed,
S3 deleted).
list_jobs_by_batch() — single-roundtrip review-screen load.
Labels (NEW api/services/kb/labels.py):
Hebrew → ASCII slugify (letter-by-letter map, no external dep);
CRUD; lookup_or_create_batch (race-safe); apply_to_source +
replace_for_source maintain usage_count; merge race-safely
(UPDATE assignments + ON CONFLICT DO NOTHING).
Classifier (NEW api/services/kb/classifier.py):
AsyncOpenAI to ai-gateway, Sonnet, OpenAI-style tool calling for
guaranteed JSON shape, Hebrew system prompt with 5 kind values,
citation format examples, "do not invent" rule, summary length
bounds 80-200 chars.
Routes (api/routes/admin_kb.py — 8 new on top of existing 11):
POST /admin/kb/upload-batch (multi-file, AI classify)
GET /admin/kb/batch/{batch_id} (review-screen load)
POST /admin/kb/jobs/{id}/commit (user confirms metadata)
POST /admin/kb/jobs/{id}/discard (user removes from batch)
GET /admin/kb/labels (typeahead)
POST /admin/kb/labels (explicit create)
POST /admin/kb/labels/{id}/merge (admin cleanup)
DELETE /admin/kb/labels/{id} (only if usage=0)
Public /kb/search gains optional label_id filter.
admin_sources.list_sources LEFT JOINs labels into each row's
output. update_source accepts label_ids to replace the set.
End-to-end smoke test passed: synthetic Hebrew circular →
upload-batch → background classify → awaiting_review with kind,
title, subject, summary, identifier, published_at all populated
correctly.
Refs Task Master #20 (espocrm-extensions/KnowledgeBase v0.8.0)
Subsumes: Task #19 (labels for sub-topics)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Court rulings are a distinct content type — long discussion / reasoning,
no enumerated statutory hierarchy, case identifier instead of statute id —
so they get their own kind alongside law/regulation/circular rather than
being shoehorned into one of the existing three.
Migration 003 alters the CHECK constraints on kb_source.kind and
kb_ingest_job.kind to include 'caselaw'. The chunker uses the
circulars path for caselaw too (paragraph + size split, no statutory
section regex); the case identifier is captured at upload time in
kb_source.identifier.
Updates Literal types in ingest_source, admin_kb upload, and the
kb_public SearchRequest. _KINDS in s3.py grows to four so failed/
processed/inbox listings include caselaw subdirs.
Refs Task Master #18 (espocrm-extensions/KnowledgeBase)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each file in the inbox can now be accompanied by a `<filename>.meta.json`
sidecar that supplies identifier, published_at, effective_at, and
source_url. The scanner applies filename-derived defaults first and lets
the sidecar override individual fields.
- list_inbox / list_failed skip `.meta.json` so sidecars aren't ingested
as standalone documents.
- mark_processed, mark_failed, and requeue-failed move the sidecar
alongside its parent (best-effort).
- mark_failed now sanitizes the error string to ASCII before writing it
to S3 object metadata (HTTP header encoding).
- search SQL selects published_at/effective_at; the formatter shows the
full heading_path plus publication date so citations are unambiguous.
- scripts/kb_sidecar_template.json documents the expected shape.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves every object under failed/<kind>/ back to inbox/<kind>/ for retry
after a transient upstream failure (e.g. Voyage 401 before a key rotation).
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a MinIO-backed ingest flow: upload to s3://insurance-kb/inbox/<kind>/
via MinIO Console or any S3 client, then POST /admin/kb/scan-inbox to
process all pending files. Successful ingests move to processed/<kind>/,
failures move to failed/<kind>/ with the error stored as S3 metadata.
- api/services/kb/s3.py: boto3 client, list_inbox/fetch/move helpers.
- api/routes/admin_kb.py: GET /admin/kb/inbox, POST /admin/kb/scan-inbox.
- The kind is derived from the folder path; title/identifier default to
the filename stem and can be refined by re-ingesting with metadata.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chaim