This repository has been archived on 2026-07-19. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
chaim 0d678da25f feat(kb): backend for v0.8.0 — tool kind + labels + AI classifier + bulk upload
Phase 6 of the multi-topic KB refactor (backend half). Subsumes the
original Task #19 design discussion (subject vs labels). EspoCRM
extension UI ships separately.

Migrations (4 new, all idempotent, transaction-wrapped):
  004_kind_tool             — adds 'tool' to kb_source/kb_ingest_job CHECK
  005_source_description    — kb_source.description + ai_classified_at
  006_labels                — kb_label + kb_source_label (M:N), GRANTs
  007_ingest_classifier     — awaiting_review status + processing_stage
                              + ai_suggestions JSONB + batch_id

Two-phase ingest (api/services/kb/ingest_jobs.py):
  queued → processing(stage=classifying)
         → awaiting_review                  ← user reviews AI output
         → processing(stage=embedding)      ← after user commits
         → done | failed

  process_classify_stage() pulls the file, runs parse_first_pages
  (3-page text extract), calls classifier.classify (Claude tool-call
  via ai-gateway, 45s timeout, 5-concurrent semaphore, fail-soft on
  any error → empty suggestions), writes ai_suggestions JSONB,
  transitions to awaiting_review.

  commit_job() resolves user-confirmed labels (existing by slug,
  new ones via slugify+ON CONFLICT DO NOTHING), transitions to
  processing(embedding), schedules process_embed_stage.

  process_embed_stage() runs the legacy ingest_source path with
  user-edited metadata + summary as kb_source.description, then
  apply_to_source for labels.

  discard_job() removes a file from a batch (status=failed,
  S3 deleted).

  list_jobs_by_batch() — single-roundtrip review-screen load.

Labels (NEW api/services/kb/labels.py):
  Hebrew → ASCII slugify (letter-by-letter map, no external dep);
  CRUD; lookup_or_create_batch (race-safe); apply_to_source +
  replace_for_source maintain usage_count; merge race-safely
  (UPDATE assignments + ON CONFLICT DO NOTHING).

Classifier (NEW api/services/kb/classifier.py):
  AsyncOpenAI to ai-gateway, Sonnet, OpenAI-style tool calling for
  guaranteed JSON shape, Hebrew system prompt with 5 kind values,
  citation format examples, "do not invent" rule, summary length
  bounds 80-200 chars.

Routes (api/routes/admin_kb.py — 8 new on top of existing 11):
  POST   /admin/kb/upload-batch                  (multi-file, AI classify)
  GET    /admin/kb/batch/{batch_id}              (review-screen load)
  POST   /admin/kb/jobs/{id}/commit              (user confirms metadata)
  POST   /admin/kb/jobs/{id}/discard             (user removes from batch)
  GET    /admin/kb/labels                        (typeahead)
  POST   /admin/kb/labels                        (explicit create)
  POST   /admin/kb/labels/{id}/merge             (admin cleanup)
  DELETE /admin/kb/labels/{id}                   (only if usage=0)

Public /kb/search gains optional label_id filter.
admin_sources.list_sources LEFT JOINs labels into each row's
output. update_source accepts label_ids to replace the set.

End-to-end smoke test passed: synthetic Hebrew circular →
upload-batch → background classify → awaiting_review with kind,
title, subject, summary, identifier, published_at all populated
correctly.

Refs Task Master #20 (espocrm-extensions/KnowledgeBase v0.8.0)
Subsumes: Task #19 (labels for sub-topics)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-25 18:42:15 +00:00

61 lines
2.8 KiB
PL/PgSQL

-- Migration 006: label-based sub-topic grouping (kb_label many-to-many kb_source).
--
-- Task Master #20 (v0.8.0); subsumes Task #19 (the original "subject vs
-- tags" design discussion). Many legal documents touch multiple subjects
-- ("ילד נכה / אוטיזם" AND "ילד נכה / תלות בזולת"), which makes a M:N
-- relationship the right shape — a single subject column would force one
-- bucket per source and lose information.
--
-- Schema choices:
-- • Flat table (no parent_id). Slash-separated names like
-- "ילד נכה / אוטיזם" are a UI display convention only — slug is
-- the unique key.
-- • Slug constrained to ASCII (lowercase letters, digits, hyphens).
-- Hebrew name → ASCII slug via transliteration in Python (unidecode).
-- • topic_id is nullable so a label can be cross-topic (rare but real:
-- "פסיקה כללית" might apply across multiple topics).
-- • usage_count denormalized so the autocomplete dropdown doesn't
-- COUNT(*) on every keystroke. Maintained by application code on
-- INSERT/DELETE of kb_source_label (no trigger — explicit is better).
-- • ON DELETE CASCADE on both FKs in kb_source_label: deleting a
-- source nukes its assignments; deleting a label nukes all
-- assignments. We never want orphans.
--
-- Idempotent — wrapped in a single transaction.
--
-- Run as postgres superuser:
-- docker exec -i <pg-container> psql -U postgres -d insurance_kb \
-- -f /tmp/006_labels.sql
BEGIN;
CREATE TABLE IF NOT EXISTS kb_label (
id SERIAL PRIMARY KEY,
slug TEXT NOT NULL UNIQUE
CHECK (slug ~ '^[a-z0-9][a-z0-9-]*$' AND length(slug) BETWEEN 1 AND 200),
name TEXT NOT NULL,
topic_id INTEGER NULL REFERENCES kb_topic(id) ON DELETE SET NULL,
-- Free-form userName from EspoCRM's X-User-Name header. Audit only.
created_by TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- Maintained by app code on every kb_source_label INSERT/DELETE so the
-- autocomplete can ORDER BY usage_count DESC without a join.
usage_count INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS kb_label_topic_idx ON kb_label (topic_id);
CREATE INDEX IF NOT EXISTS kb_label_usage_idx ON kb_label (usage_count DESC);
CREATE TABLE IF NOT EXISTS kb_source_label (
source_id INTEGER NOT NULL REFERENCES kb_source(id) ON DELETE CASCADE,
label_id INTEGER NOT NULL REFERENCES kb_label(id) ON DELETE CASCADE,
PRIMARY KEY (source_id, label_id)
);
CREATE INDEX IF NOT EXISTS kb_source_label_label_idx ON kb_source_label (label_id);
GRANT SELECT, INSERT, UPDATE, DELETE ON kb_label TO shira_kb;
GRANT SELECT, INSERT, UPDATE, DELETE ON kb_source_label TO shira_kb;
GRANT USAGE, SELECT, UPDATE ON SEQUENCE kb_label_id_seq TO shira_kb;
COMMIT;