0d678da25f
Phase 6 of the multi-topic KB refactor (backend half). Subsumes the original Task #19 design discussion (subject vs labels). EspoCRM extension UI ships separately. Migrations (4 new, all idempotent, transaction-wrapped): 004_kind_tool — adds 'tool' to kb_source/kb_ingest_job CHECK 005_source_description — kb_source.description + ai_classified_at 006_labels — kb_label + kb_source_label (M:N), GRANTs 007_ingest_classifier — awaiting_review status + processing_stage + ai_suggestions JSONB + batch_id Two-phase ingest (api/services/kb/ingest_jobs.py): queued → processing(stage=classifying) → awaiting_review ← user reviews AI output → processing(stage=embedding) ← after user commits → done | failed process_classify_stage() pulls the file, runs parse_first_pages (3-page text extract), calls classifier.classify (Claude tool-call via ai-gateway, 45s timeout, 5-concurrent semaphore, fail-soft on any error → empty suggestions), writes ai_suggestions JSONB, transitions to awaiting_review. commit_job() resolves user-confirmed labels (existing by slug, new ones via slugify+ON CONFLICT DO NOTHING), transitions to processing(embedding), schedules process_embed_stage. process_embed_stage() runs the legacy ingest_source path with user-edited metadata + summary as kb_source.description, then apply_to_source for labels. discard_job() removes a file from a batch (status=failed, S3 deleted). list_jobs_by_batch() — single-roundtrip review-screen load. Labels (NEW api/services/kb/labels.py): Hebrew → ASCII slugify (letter-by-letter map, no external dep); CRUD; lookup_or_create_batch (race-safe); apply_to_source + replace_for_source maintain usage_count; merge race-safely (UPDATE assignments + ON CONFLICT DO NOTHING). Classifier (NEW api/services/kb/classifier.py): AsyncOpenAI to ai-gateway, Sonnet, OpenAI-style tool calling for guaranteed JSON shape, Hebrew system prompt with 5 kind values, citation format examples, "do not invent" rule, summary length bounds 80-200 chars. Routes (api/routes/admin_kb.py — 8 new on top of existing 11): POST /admin/kb/upload-batch (multi-file, AI classify) GET /admin/kb/batch/{batch_id} (review-screen load) POST /admin/kb/jobs/{id}/commit (user confirms metadata) POST /admin/kb/jobs/{id}/discard (user removes from batch) GET /admin/kb/labels (typeahead) POST /admin/kb/labels (explicit create) POST /admin/kb/labels/{id}/merge (admin cleanup) DELETE /admin/kb/labels/{id} (only if usage=0) Public /kb/search gains optional label_id filter. admin_sources.list_sources LEFT JOINs labels into each row's output. update_source accepts label_ids to replace the set. End-to-end smoke test passed: synthetic Hebrew circular → upload-batch → background classify → awaiting_review with kind, title, subject, summary, identifier, published_at all populated correctly. Refs Task Master #20 (espocrm-extensions/KnowledgeBase v0.8.0) Subsumes: Task #19 (labels for sub-topics) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
61 lines
2.8 KiB
PL/PgSQL
61 lines
2.8 KiB
PL/PgSQL
-- Migration 006: label-based sub-topic grouping (kb_label many-to-many kb_source).
|
|
--
|
|
-- Task Master #20 (v0.8.0); subsumes Task #19 (the original "subject vs
|
|
-- tags" design discussion). Many legal documents touch multiple subjects
|
|
-- ("ילד נכה / אוטיזם" AND "ילד נכה / תלות בזולת"), which makes a M:N
|
|
-- relationship the right shape — a single subject column would force one
|
|
-- bucket per source and lose information.
|
|
--
|
|
-- Schema choices:
|
|
-- • Flat table (no parent_id). Slash-separated names like
|
|
-- "ילד נכה / אוטיזם" are a UI display convention only — slug is
|
|
-- the unique key.
|
|
-- • Slug constrained to ASCII (lowercase letters, digits, hyphens).
|
|
-- Hebrew name → ASCII slug via transliteration in Python (unidecode).
|
|
-- • topic_id is nullable so a label can be cross-topic (rare but real:
|
|
-- "פסיקה כללית" might apply across multiple topics).
|
|
-- • usage_count denormalized so the autocomplete dropdown doesn't
|
|
-- COUNT(*) on every keystroke. Maintained by application code on
|
|
-- INSERT/DELETE of kb_source_label (no trigger — explicit is better).
|
|
-- • ON DELETE CASCADE on both FKs in kb_source_label: deleting a
|
|
-- source nukes its assignments; deleting a label nukes all
|
|
-- assignments. We never want orphans.
|
|
--
|
|
-- Idempotent — wrapped in a single transaction.
|
|
--
|
|
-- Run as postgres superuser:
|
|
-- docker exec -i <pg-container> psql -U postgres -d insurance_kb \
|
|
-- -f /tmp/006_labels.sql
|
|
|
|
BEGIN;
|
|
|
|
CREATE TABLE IF NOT EXISTS kb_label (
|
|
id SERIAL PRIMARY KEY,
|
|
slug TEXT NOT NULL UNIQUE
|
|
CHECK (slug ~ '^[a-z0-9][a-z0-9-]*$' AND length(slug) BETWEEN 1 AND 200),
|
|
name TEXT NOT NULL,
|
|
topic_id INTEGER NULL REFERENCES kb_topic(id) ON DELETE SET NULL,
|
|
-- Free-form userName from EspoCRM's X-User-Name header. Audit only.
|
|
created_by TEXT,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
-- Maintained by app code on every kb_source_label INSERT/DELETE so the
|
|
-- autocomplete can ORDER BY usage_count DESC without a join.
|
|
usage_count INTEGER NOT NULL DEFAULT 0
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS kb_label_topic_idx ON kb_label (topic_id);
|
|
CREATE INDEX IF NOT EXISTS kb_label_usage_idx ON kb_label (usage_count DESC);
|
|
|
|
CREATE TABLE IF NOT EXISTS kb_source_label (
|
|
source_id INTEGER NOT NULL REFERENCES kb_source(id) ON DELETE CASCADE,
|
|
label_id INTEGER NOT NULL REFERENCES kb_label(id) ON DELETE CASCADE,
|
|
PRIMARY KEY (source_id, label_id)
|
|
);
|
|
CREATE INDEX IF NOT EXISTS kb_source_label_label_idx ON kb_source_label (label_id);
|
|
|
|
GRANT SELECT, INSERT, UPDATE, DELETE ON kb_label TO shira_kb;
|
|
GRANT SELECT, INSERT, UPDATE, DELETE ON kb_source_label TO shira_kb;
|
|
GRANT USAGE, SELECT, UPDATE ON SEQUENCE kb_label_id_seq TO shira_kb;
|
|
|
|
COMMIT;
|