feat: Phase 2 — document upload from KB UI

Adds a "ניהול" tab with multipart file upload, async job tracking, and
a recent-jobs list with live status polling. Browser POSTs to
/KnowledgeBase/action/upload (PHP proxy), which forwards as multipart
to shira-hermes /admin/kb/upload. shira-hermes returns a job_id
immediately and processes parse/chunk/embed in a background task; the
browser polls every 2s until status hits done|failed.

New EspoCRM endpoints:
  POST /KnowledgeBase/action/upload   (multipart, $_FILES['file'])
  GET  /KnowledgeBase/action/jobs     (list, filtered by topicId/status)
  GET  /KnowledgeBase/action/job?id   (single-job detail)

The X-User-Name header is forwarded so kb_ingest_job records who
uploaded each file. Cross-topic guard in switchTopic stops polling
when the user changes topics mid-upload (the job continues server-side).

Depends on shira-hermes commit 0cb89fb (admin upload endpoints +
migration 002).

Refs Task Master #13

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-25 16:46:42 +00:00
parent dacb7f6256
commit 6781ac4e37
7 changed files with 526 additions and 7 deletions
+7 -6
View File
@@ -137,20 +137,20 @@
"id": "12",
"title": "Phase 1 — Multi-topic KB foundation (DB schema, topic-aware endpoints, topic selector)",
"description": "Generalize the KB from being insurance-only to supporting multiple legal domains within one CRM (e.g. ביטוח לאומי, דיני עבודה, דין פלילי). A 'topic' is the firm-level grouping; each kb_source belongs to exactly one topic. Search/ask are scoped to a single topic at a time, chosen from a topic dropdown the user controls. The system_prompt for /kb/ask becomes per-topic so Shira's domain context is correct (today the prompt hardcodes 'Israeli National Insurance'). Backwards-compatible migration: all 6 existing sources move to topic_id=1 named 'ביטוח לאומי' with the existing system-prompt addendum, and the UI defaults to that topic so nothing visible changes for existing users until they choose another topic.",
"status": "in-progress",
"status": "done",
"priority": "high",
"details": "DB migration (shira-hermes side, against insurance_kb DB):\n- New table kb_topic: id PK, slug TEXT UNIQUE, name TEXT NOT NULL, description TEXT, system_prompt_addendum TEXT, is_active BOOLEAN DEFAULT true, created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ DEFAULT now().\n- Seed: INSERT (id=1, slug='national-insurance', name='ביטוח לאומי', system_prompt_addendum=<copy from current /kb/ask hardcoded text>, is_active=true). RENAME the DB itself? Probably leave as 'insurance_kb' — costly to rename, no real value. Document in memory.\n- ALTER TABLE kb_source ADD COLUMN topic_id INTEGER REFERENCES kb_topic(id). Backfill: UPDATE kb_source SET topic_id = 1. Then ALTER COLUMN SET NOT NULL.\n\nshira-hermes endpoints:\n- New GET /kb/topics — public, returns active topics with id, slug, name, description (no prompt).\n- /kb/search: accepts optional topic_id (single int). When present, the SQL adds AND s.topic_id = $N. When absent (back-compat), still returns all (might want to deprecate this and require topic_id later).\n- /kb/sources, /kb/source/{id}/chunks, /kb/source/{id}/pdf: same — optional topic filter on listings, no change to single-source endpoints.\n- /kb/ask + /kb/ask/stream: accept topic_id param; load topic.system_prompt_addendum and inject into system_prompt instead of the hardcoded insurance text in _build_ask_runner_context. Pin the legal_kb tool to topic_id too (so search_insurance_kb queries are constrained — RENAME the tool to search_legal_kb and pass topic_id at registration time).\n- _expand_query in kb_search.py: parametrize the prompt by topic name (currently hardcoded 'הביטוח הלאומי בישראל'). Pass topic.name in.\n\nKnowledgeBase extension (EspoCRM):\n- New route+action GET /KnowledgeBase/action/topics → proxies /kb/topics.\n- Service.search/ask gain topic_id, controller passes through.\n- Template gets a topic <select> at the top (above the tabs); fetches /KnowledgeBase/action/topics on mount, persists selection to localStorage 'kb-topic'. Default: topic id=1 if it exists, else first active topic.\n- this.kind etc. already pass through; add this.topicId. Pass topic_id with every search/ask call.\n\nConsequences for sessionStorage replay (v0.1.9, v0.1.10): the cached _lastAsk / _lastSearch should record topic_id; on remount only replay if the current topic matches. Else show empty state.",
"testStrategy": "1) After migration: SELECT count(*) FROM kb_source WHERE topic_id IS NULL → 0; SELECT name FROM kb_topic → 'ביטוח לאומי'. 2) GET /kb/topics returns the seeded topic. 3) Search/ask with topic_id=1 returns the same 6 sources as before. 4) Add a second topic (manually for the test) and a single test source under it; verify topic_id=1 search no longer surfaces the test source and vice versa. 5) /kb/ask system prompt confirmed via logs to include the topic-specific addendum, not the hardcoded text.",
"subtasks": [],
"dependencies": [],
"createdAt": "2026-04-25T13:30:00Z",
"updatedAt": "2026-04-25T14:15:58.498Z"
"updatedAt": "2026-04-25T14:34:08.086Z"
},
{
"id": "13",
"title": "Phase 2 — Document upload from the KB UI (file picker + async ingestion + job tracking)",
"description": "Today documents enter the KB by manually putting them in the MinIO inbox/ folder and waiting for the n8n cron to call /admin/kb/scan-inbox. End users have no path to upload via the UI. Add a file-upload form in the management panel: user picks a PDF/DOCX/TXT, picks a kind (law/regulation/circular), confirms the topic, optionally fills metadata (title, identifier, dates, source_url), and clicks upload. The file lands in MinIO under inbox/<topic_slug>/<kind>/, a kb_ingest_job row is created, and a background asyncio task processes it (parse → chunk → embed → upsert kb_source). The UI polls the job status and shows progress live; on done it links to the new source in the management table.",
"status": "pending",
"status": "in-progress",
"priority": "high",
"details": "DB:\n- New table kb_ingest_job: id PK, source_id INTEGER NULL FK kb_source(id) ON DELETE SET NULL, original_filename TEXT, s3_key TEXT, kind TEXT, topic_id INTEGER NOT NULL, metadata_json JSONB, status TEXT NOT NULL CHECK (status IN ('queued','processing','done','failed')), error_message TEXT, chunks_created INTEGER, created_at TIMESTAMPTZ DEFAULT now(), started_at TIMESTAMPTZ, completed_at TIMESTAMPTZ, requested_by_user TEXT.\n- Index: (status, created_at DESC) for the queued list.\n\nshira-hermes endpoints:\n- POST /admin/kb/upload (multipart): receives file, kind, topic_id, optional title/identifier/published_at/effective_at/source_url. Validates topic exists. Writes file to s3 inbox/<topic_slug>/<kind>/<uuid-prefixed filename>. Inserts kb_ingest_job with status=queued. Schedules asyncio.create_task(_process_ingest_job(job_id)) so processing starts immediately, not on the next n8n cron tick. Returns {job_id, status:'queued'}.\n- GET /admin/kb/jobs?topic_id=&status=&limit=: list jobs, newest first.\n- GET /admin/kb/jobs/{id}: single job detail.\n- _process_ingest_job(job_id): UPDATE status='processing', started_at=now. Calls existing kb_ingest.ingest_source. On success: UPDATE status='done', source_id=<new>, chunks_created=<n>, completed_at=now. On failure: UPDATE status='failed', error_message=<exception text>. The upload doesn't block the HTTP request — the user gets the job_id immediately and polls.\n- Reuses scan-inbox advisory lock pattern so a manually-uploaded file plus an n8n cron run don't double-process.\n\nKnowledgeBase extension:\n- EspoCRM PHP proxy for multipart upload — Controllers can read $request->getParsedBody() but multipart needs special handling. Look at how other extensions do it (LegalAssistance/DigitalSignature might have examples).\n- New tab 'ניהול' (visible only to non-portal users; can also gate to admin role later). The tab opens a panel with: (a) Upload form, (b) Sources table for current topic [implemented in Phase 3], (c) Recent jobs.\n- Upload form fields: file picker, kind dropdown, topic dropdown (defaulting to currently-selected topic), optional title/identifier/dates/source_url collapse. Submit → POST KnowledgeBase/action/upload → returns job_id → switch to job-status view that polls every 2s.\n- Job status view: shows the job's status badge (queued/processing/done/failed), elapsed time, and on done: link to the source in the management table (Phase 3).\n\nFile size: enforce 50MB limit at the FastAPI route + server-side. PDFs of עשרות-שעות could approach this.",
"testStrategy": "1) Upload a small test PDF (~1MB). Job goes queued → processing → done in <30s. UI shows progress live. 2) Upload a corrupt file. Job ends in failed with a useful error_message. 3) Upload while another large ingest is running — job shows 'processing' but doesn't deadlock. 4) Refresh the page mid-upload — job status restored from sessionStorage + a fresh poll. 5) /admin/kb/jobs returns the upload's row.",
@@ -158,7 +158,8 @@
"dependencies": [
"12"
],
"createdAt": "2026-04-25T13:30:00Z"
"createdAt": "2026-04-25T13:30:00Z",
"updatedAt": "2026-04-25T16:20:11.931Z"
},
{
"id": "14",
@@ -218,9 +219,9 @@
],
"metadata": {
"version": "1.0.0",
"lastModified": "2026-04-25T14:15:58.500Z",
"lastModified": "2026-04-25T16:20:11.932Z",
"taskCount": 17,
"completedCount": 8,
"completedCount": 9,
"tags": [
"master"
]