From e7b21919c26fac5644a03ba73603b54018ad9d66 Mon Sep 17 00:00:00 2001 From: Chaim Date: Sat, 25 Apr 2026 17:01:12 +0000 Subject: [PATCH] =?UTF-8?q?feat:=20add=20'caselaw'=20(=D7=A4=D7=A1=D7=99?= =?UTF-8?q?=D7=A7=D7=94)=20kind=20alongside=20=D7=97=D7=95=D7=A7/=D7=AA?= =?UTF-8?q?=D7=A7=D7=A0=D7=95=D7=AA/=D7=97=D7=95=D7=96=D7=A8=D7=99=D7=9D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Court rulings get their own kind so users can filter search results to "פסיקה" specifically and upload case-law PDFs from the ניהול tab. PHP validators in Controller + Service accept the new kind; the search dropdown and upload kind dropdown gain a "פסיקה" option; kindHe in index.js maps caselaw → "פסיקה" so jobs list, search results, and browse view all label it consistently. Backend: depends on shira-hermes commit 16eeeff (kind validators + chunker fallback) and migration 003_caselaw_kind.sql (already applied on dev — alters CHECK constraints on kb_source.kind and kb_ingest_job.kind to include 'caselaw'). Refs Task Master #18 Co-Authored-By: Claude Opus 4.6 (1M context) --- .taskmaster/state.json | 2 +- .taskmaster/tasks/tasks.json | 34 ++++++++++++++++--- .../knowledge-base/res/templates/kb/index.tpl | 2 ++ .../knowledge-base/src/views/kb/index.js | 10 +++--- .../Controllers/KnowledgeBase.php | 4 +-- .../Services/KnowledgeBaseService.php | 6 ++-- manifest.json | 2 +- 7 files changed, 43 insertions(+), 17 deletions(-) diff --git a/.taskmaster/state.json b/.taskmaster/state.json index f3493fd..67d0cf8 100644 --- a/.taskmaster/state.json +++ b/.taskmaster/state.json @@ -2,5 +2,5 @@ "currentTag": "master", "lastSwitched": "2026-04-24T15:47:33.139Z", "branchTagMapping": {}, - "migrationNoticeShown": false + "migrationNoticeShown": true } \ No newline at end of file diff --git a/.taskmaster/tasks/tasks.json b/.taskmaster/tasks/tasks.json index 05af49e..acfba35 100644 --- a/.taskmaster/tasks/tasks.json +++ b/.taskmaster/tasks/tasks.json @@ -150,7 +150,7 @@ "id": "13", "title": "Phase 2 — Document upload from the KB UI (file picker + async ingestion + job tracking)", "description": "Today documents enter the KB by manually putting them in the MinIO inbox/ folder and waiting for the n8n cron to call /admin/kb/scan-inbox. End users have no path to upload via the UI. Add a file-upload form in the management panel: user picks a PDF/DOCX/TXT, picks a kind (law/regulation/circular), confirms the topic, optionally fills metadata (title, identifier, dates, source_url), and clicks upload. The file lands in MinIO under inbox///, a kb_ingest_job row is created, and a background asyncio task processes it (parse → chunk → embed → upsert kb_source). The UI polls the job status and shows progress live; on done it links to the new source in the management table.", - "status": "in-progress", + "status": "done", "priority": "high", "details": "DB:\n- New table kb_ingest_job: id PK, source_id INTEGER NULL FK kb_source(id) ON DELETE SET NULL, original_filename TEXT, s3_key TEXT, kind TEXT, topic_id INTEGER NOT NULL, metadata_json JSONB, status TEXT NOT NULL CHECK (status IN ('queued','processing','done','failed')), error_message TEXT, chunks_created INTEGER, created_at TIMESTAMPTZ DEFAULT now(), started_at TIMESTAMPTZ, completed_at TIMESTAMPTZ, requested_by_user TEXT.\n- Index: (status, created_at DESC) for the queued list.\n\nshira-hermes endpoints:\n- POST /admin/kb/upload (multipart): receives file, kind, topic_id, optional title/identifier/published_at/effective_at/source_url. Validates topic exists. Writes file to s3 inbox///. Inserts kb_ingest_job with status=queued. Schedules asyncio.create_task(_process_ingest_job(job_id)) so processing starts immediately, not on the next n8n cron tick. Returns {job_id, status:'queued'}.\n- GET /admin/kb/jobs?topic_id=&status=&limit=: list jobs, newest first.\n- GET /admin/kb/jobs/{id}: single job detail.\n- _process_ingest_job(job_id): UPDATE status='processing', started_at=now. Calls existing kb_ingest.ingest_source. On success: UPDATE status='done', source_id=, chunks_created=, completed_at=now. On failure: UPDATE status='failed', error_message=. The upload doesn't block the HTTP request — the user gets the job_id immediately and polls.\n- Reuses scan-inbox advisory lock pattern so a manually-uploaded file plus an n8n cron run don't double-process.\n\nKnowledgeBase extension:\n- EspoCRM PHP proxy for multipart upload — Controllers can read $request->getParsedBody() but multipart needs special handling. Look at how other extensions do it (LegalAssistance/DigitalSignature might have examples).\n- New tab 'ניהול' (visible only to non-portal users; can also gate to admin role later). The tab opens a panel with: (a) Upload form, (b) Sources table for current topic [implemented in Phase 3], (c) Recent jobs.\n- Upload form fields: file picker, kind dropdown, topic dropdown (defaulting to currently-selected topic), optional title/identifier/dates/source_url collapse. Submit → POST KnowledgeBase/action/upload → returns job_id → switch to job-status view that polls every 2s.\n- Job status view: shows the job's status badge (queued/processing/done/failed), elapsed time, and on done: link to the source in the management table (Phase 3).\n\nFile size: enforce 50MB limit at the FastAPI route + server-side. PDFs of עשרות-שעות could approach this.", "testStrategy": "1) Upload a small test PDF (~1MB). Job goes queued → processing → done in <30s. UI shows progress live. 2) Upload a corrupt file. Job ends in failed with a useful error_message. 3) Upload while another large ingest is running — job shows 'processing' but doesn't deadlock. 4) Refresh the page mid-upload — job status restored from sessionStorage + a fresh poll. 5) /admin/kb/jobs returns the upload's row.", @@ -159,7 +159,7 @@ "12" ], "createdAt": "2026-04-25T13:30:00Z", - "updatedAt": "2026-04-25T16:20:11.931Z" + "updatedAt": "2026-04-25T16:47:46.599Z" }, { "id": "14", @@ -215,13 +215,37 @@ "15" ], "createdAt": "2026-04-25T13:30:00Z" + }, + { + "id": "18", + "title": "feat(kb): add 'caselaw' (פסיקה) as a fourth kind", + "description": "Add 'caselaw' alongside law/regulation/circular across the whole stack. Touches: DB CHECK constraints (migration 003), shira-hermes Python (Literal types, _KINDS, kind validation, chunker section detection), EspoCRM Controller/Service validators, template dropdowns, JS kindHe mapping.", + "details": "DB: migration 003_caselaw_kind.sql alters CHECK on kb_source.kind and kb_ingest_job.kind to include 'caselaw'. shira-hermes: update Literal types in ingest.py, admin_kb.py, kb_public.py SearchRequest; add 'caselaw' to s3.py _KINDS; chunker.py — fall back to generic section detector (regex for 'פסק דין', 'תיק' / case number, paragraph numbers). EspoCRM: update validators in Controller postActionUpload + Service search/uploadFile; add 'caselaw' option to kind