3 Commits

Author SHA1 Message Date
chaim d051b268dd feat: Phase 3 — sources management table in 'ניהול' tab
Adds full CRUD for KB sources from the EspoCRM UI: a sortable table at
the top of the 'ניהול' tab with kind / title-identifier-updated / chunk
count / actions columns, a kind filter (חוק / תקנות / חוזרים / פסיקה),
and three per-row actions:

  ✏ Edit       — inline form with title, identifier, dates, source URL
  ↻ Reingest   — re-parses the original file in place, replacing chunks
                 while keeping the source row + hand-edited metadata
  🗑 Delete    — confirm dialog with chunk count; double confirm above
                 100 chunks; also deletes the MinIO object

Browser polls the standard /KnowledgeBase/action/job endpoint for
re-ingest progress (same banner as upload). On terminal status
(done|failed) the sources table auto-refreshes so chunk_count and
last_ingest reflect the new state. Topic switch invalidates the
per-topic admin-sources cache so a stale list doesn't bleed across
topics.

Backend: depends on shira-hermes commit 17f93b5
(GET/PUT/DELETE /admin/kb/sources + POST /reingest).

Refs Task Master #14

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-25 17:27:45 +00:00
chaim e7b21919c2 feat: add 'caselaw' (פסיקה) kind alongside חוק/תקנות/חוזרים
Court rulings get their own kind so users can filter search results
to "פסיקה" specifically and upload case-law PDFs from the ניהול tab.
PHP validators in Controller + Service accept the new kind; the search
dropdown and upload kind dropdown gain a "פסיקה" option; kindHe in
index.js maps caselaw → "פסיקה" so jobs list, search results, and
browse view all label it consistently.

Backend: depends on shira-hermes commit 16eeeff (kind validators +
chunker fallback) and migration 003_caselaw_kind.sql (already applied
on dev — alters CHECK constraints on kb_source.kind and
kb_ingest_job.kind to include 'caselaw').

Refs Task Master #18

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-25 17:01:12 +00:00
chaim 6781ac4e37 feat: Phase 2 — document upload from KB UI
Adds a "ניהול" tab with multipart file upload, async job tracking, and
a recent-jobs list with live status polling. Browser POSTs to
/KnowledgeBase/action/upload (PHP proxy), which forwards as multipart
to shira-hermes /admin/kb/upload. shira-hermes returns a job_id
immediately and processes parse/chunk/embed in a background task; the
browser polls every 2s until status hits done|failed.

New EspoCRM endpoints:
  POST /KnowledgeBase/action/upload   (multipart, $_FILES['file'])
  GET  /KnowledgeBase/action/jobs     (list, filtered by topicId/status)
  GET  /KnowledgeBase/action/job?id   (single-job detail)

The X-User-Name header is forwarded so kb_ingest_job records who
uploaded each file. Cross-topic guard in switchTopic stops polling
when the user changes topics mid-upload (the job continues server-side).

Depends on shira-hermes commit 0cb89fb (admin upload endpoints +
migration 002).

Refs Task Master #13

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-25 16:46:42 +00:00
8 changed files with 987 additions and 16 deletions
+1 -1
View File
@@ -2,5 +2,5 @@
"currentTag": "master", "currentTag": "master",
"lastSwitched": "2026-04-24T15:47:33.139Z", "lastSwitched": "2026-04-24T15:47:33.139Z",
"branchTagMapping": {}, "branchTagMapping": {},
"migrationNoticeShown": false "migrationNoticeShown": true
} }
+35 -9
View File
@@ -137,20 +137,20 @@
"id": "12", "id": "12",
"title": "Phase 1 — Multi-topic KB foundation (DB schema, topic-aware endpoints, topic selector)", "title": "Phase 1 — Multi-topic KB foundation (DB schema, topic-aware endpoints, topic selector)",
"description": "Generalize the KB from being insurance-only to supporting multiple legal domains within one CRM (e.g. ביטוח לאומי, דיני עבודה, דין פלילי). A 'topic' is the firm-level grouping; each kb_source belongs to exactly one topic. Search/ask are scoped to a single topic at a time, chosen from a topic dropdown the user controls. The system_prompt for /kb/ask becomes per-topic so Shira's domain context is correct (today the prompt hardcodes 'Israeli National Insurance'). Backwards-compatible migration: all 6 existing sources move to topic_id=1 named 'ביטוח לאומי' with the existing system-prompt addendum, and the UI defaults to that topic so nothing visible changes for existing users until they choose another topic.", "description": "Generalize the KB from being insurance-only to supporting multiple legal domains within one CRM (e.g. ביטוח לאומי, דיני עבודה, דין פלילי). A 'topic' is the firm-level grouping; each kb_source belongs to exactly one topic. Search/ask are scoped to a single topic at a time, chosen from a topic dropdown the user controls. The system_prompt for /kb/ask becomes per-topic so Shira's domain context is correct (today the prompt hardcodes 'Israeli National Insurance'). Backwards-compatible migration: all 6 existing sources move to topic_id=1 named 'ביטוח לאומי' with the existing system-prompt addendum, and the UI defaults to that topic so nothing visible changes for existing users until they choose another topic.",
"status": "in-progress", "status": "done",
"priority": "high", "priority": "high",
"details": "DB migration (shira-hermes side, against insurance_kb DB):\n- New table kb_topic: id PK, slug TEXT UNIQUE, name TEXT NOT NULL, description TEXT, system_prompt_addendum TEXT, is_active BOOLEAN DEFAULT true, created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ DEFAULT now().\n- Seed: INSERT (id=1, slug='national-insurance', name='ביטוח לאומי', system_prompt_addendum=<copy from current /kb/ask hardcoded text>, is_active=true). RENAME the DB itself? Probably leave as 'insurance_kb' — costly to rename, no real value. Document in memory.\n- ALTER TABLE kb_source ADD COLUMN topic_id INTEGER REFERENCES kb_topic(id). Backfill: UPDATE kb_source SET topic_id = 1. Then ALTER COLUMN SET NOT NULL.\n\nshira-hermes endpoints:\n- New GET /kb/topics — public, returns active topics with id, slug, name, description (no prompt).\n- /kb/search: accepts optional topic_id (single int). When present, the SQL adds AND s.topic_id = $N. When absent (back-compat), still returns all (might want to deprecate this and require topic_id later).\n- /kb/sources, /kb/source/{id}/chunks, /kb/source/{id}/pdf: same — optional topic filter on listings, no change to single-source endpoints.\n- /kb/ask + /kb/ask/stream: accept topic_id param; load topic.system_prompt_addendum and inject into system_prompt instead of the hardcoded insurance text in _build_ask_runner_context. Pin the legal_kb tool to topic_id too (so search_insurance_kb queries are constrained — RENAME the tool to search_legal_kb and pass topic_id at registration time).\n- _expand_query in kb_search.py: parametrize the prompt by topic name (currently hardcoded 'הביטוח הלאומי בישראל'). Pass topic.name in.\n\nKnowledgeBase extension (EspoCRM):\n- New route+action GET /KnowledgeBase/action/topics → proxies /kb/topics.\n- Service.search/ask gain topic_id, controller passes through.\n- Template gets a topic <select> at the top (above the tabs); fetches /KnowledgeBase/action/topics on mount, persists selection to localStorage 'kb-topic'. Default: topic id=1 if it exists, else first active topic.\n- this.kind etc. already pass through; add this.topicId. Pass topic_id with every search/ask call.\n\nConsequences for sessionStorage replay (v0.1.9, v0.1.10): the cached _lastAsk / _lastSearch should record topic_id; on remount only replay if the current topic matches. Else show empty state.", "details": "DB migration (shira-hermes side, against insurance_kb DB):\n- New table kb_topic: id PK, slug TEXT UNIQUE, name TEXT NOT NULL, description TEXT, system_prompt_addendum TEXT, is_active BOOLEAN DEFAULT true, created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ DEFAULT now().\n- Seed: INSERT (id=1, slug='national-insurance', name='ביטוח לאומי', system_prompt_addendum=<copy from current /kb/ask hardcoded text>, is_active=true). RENAME the DB itself? Probably leave as 'insurance_kb' — costly to rename, no real value. Document in memory.\n- ALTER TABLE kb_source ADD COLUMN topic_id INTEGER REFERENCES kb_topic(id). Backfill: UPDATE kb_source SET topic_id = 1. Then ALTER COLUMN SET NOT NULL.\n\nshira-hermes endpoints:\n- New GET /kb/topics — public, returns active topics with id, slug, name, description (no prompt).\n- /kb/search: accepts optional topic_id (single int). When present, the SQL adds AND s.topic_id = $N. When absent (back-compat), still returns all (might want to deprecate this and require topic_id later).\n- /kb/sources, /kb/source/{id}/chunks, /kb/source/{id}/pdf: same — optional topic filter on listings, no change to single-source endpoints.\n- /kb/ask + /kb/ask/stream: accept topic_id param; load topic.system_prompt_addendum and inject into system_prompt instead of the hardcoded insurance text in _build_ask_runner_context. Pin the legal_kb tool to topic_id too (so search_insurance_kb queries are constrained — RENAME the tool to search_legal_kb and pass topic_id at registration time).\n- _expand_query in kb_search.py: parametrize the prompt by topic name (currently hardcoded 'הביטוח הלאומי בישראל'). Pass topic.name in.\n\nKnowledgeBase extension (EspoCRM):\n- New route+action GET /KnowledgeBase/action/topics → proxies /kb/topics.\n- Service.search/ask gain topic_id, controller passes through.\n- Template gets a topic <select> at the top (above the tabs); fetches /KnowledgeBase/action/topics on mount, persists selection to localStorage 'kb-topic'. Default: topic id=1 if it exists, else first active topic.\n- this.kind etc. already pass through; add this.topicId. Pass topic_id with every search/ask call.\n\nConsequences for sessionStorage replay (v0.1.9, v0.1.10): the cached _lastAsk / _lastSearch should record topic_id; on remount only replay if the current topic matches. Else show empty state.",
"testStrategy": "1) After migration: SELECT count(*) FROM kb_source WHERE topic_id IS NULL → 0; SELECT name FROM kb_topic → 'ביטוח לאומי'. 2) GET /kb/topics returns the seeded topic. 3) Search/ask with topic_id=1 returns the same 6 sources as before. 4) Add a second topic (manually for the test) and a single test source under it; verify topic_id=1 search no longer surfaces the test source and vice versa. 5) /kb/ask system prompt confirmed via logs to include the topic-specific addendum, not the hardcoded text.", "testStrategy": "1) After migration: SELECT count(*) FROM kb_source WHERE topic_id IS NULL → 0; SELECT name FROM kb_topic → 'ביטוח לאומי'. 2) GET /kb/topics returns the seeded topic. 3) Search/ask with topic_id=1 returns the same 6 sources as before. 4) Add a second topic (manually for the test) and a single test source under it; verify topic_id=1 search no longer surfaces the test source and vice versa. 5) /kb/ask system prompt confirmed via logs to include the topic-specific addendum, not the hardcoded text.",
"subtasks": [], "subtasks": [],
"dependencies": [], "dependencies": [],
"createdAt": "2026-04-25T13:30:00Z", "createdAt": "2026-04-25T13:30:00Z",
"updatedAt": "2026-04-25T14:15:58.498Z" "updatedAt": "2026-04-25T14:34:08.086Z"
}, },
{ {
"id": "13", "id": "13",
"title": "Phase 2 — Document upload from the KB UI (file picker + async ingestion + job tracking)", "title": "Phase 2 — Document upload from the KB UI (file picker + async ingestion + job tracking)",
"description": "Today documents enter the KB by manually putting them in the MinIO inbox/ folder and waiting for the n8n cron to call /admin/kb/scan-inbox. End users have no path to upload via the UI. Add a file-upload form in the management panel: user picks a PDF/DOCX/TXT, picks a kind (law/regulation/circular), confirms the topic, optionally fills metadata (title, identifier, dates, source_url), and clicks upload. The file lands in MinIO under inbox/<topic_slug>/<kind>/, a kb_ingest_job row is created, and a background asyncio task processes it (parse → chunk → embed → upsert kb_source). The UI polls the job status and shows progress live; on done it links to the new source in the management table.", "description": "Today documents enter the KB by manually putting them in the MinIO inbox/ folder and waiting for the n8n cron to call /admin/kb/scan-inbox. End users have no path to upload via the UI. Add a file-upload form in the management panel: user picks a PDF/DOCX/TXT, picks a kind (law/regulation/circular), confirms the topic, optionally fills metadata (title, identifier, dates, source_url), and clicks upload. The file lands in MinIO under inbox/<topic_slug>/<kind>/, a kb_ingest_job row is created, and a background asyncio task processes it (parse → chunk → embed → upsert kb_source). The UI polls the job status and shows progress live; on done it links to the new source in the management table.",
"status": "pending", "status": "done",
"priority": "high", "priority": "high",
"details": "DB:\n- New table kb_ingest_job: id PK, source_id INTEGER NULL FK kb_source(id) ON DELETE SET NULL, original_filename TEXT, s3_key TEXT, kind TEXT, topic_id INTEGER NOT NULL, metadata_json JSONB, status TEXT NOT NULL CHECK (status IN ('queued','processing','done','failed')), error_message TEXT, chunks_created INTEGER, created_at TIMESTAMPTZ DEFAULT now(), started_at TIMESTAMPTZ, completed_at TIMESTAMPTZ, requested_by_user TEXT.\n- Index: (status, created_at DESC) for the queued list.\n\nshira-hermes endpoints:\n- POST /admin/kb/upload (multipart): receives file, kind, topic_id, optional title/identifier/published_at/effective_at/source_url. Validates topic exists. Writes file to s3 inbox/<topic_slug>/<kind>/<uuid-prefixed filename>. Inserts kb_ingest_job with status=queued. Schedules asyncio.create_task(_process_ingest_job(job_id)) so processing starts immediately, not on the next n8n cron tick. Returns {job_id, status:'queued'}.\n- GET /admin/kb/jobs?topic_id=&status=&limit=: list jobs, newest first.\n- GET /admin/kb/jobs/{id}: single job detail.\n- _process_ingest_job(job_id): UPDATE status='processing', started_at=now. Calls existing kb_ingest.ingest_source. On success: UPDATE status='done', source_id=<new>, chunks_created=<n>, completed_at=now. On failure: UPDATE status='failed', error_message=<exception text>. The upload doesn't block the HTTP request — the user gets the job_id immediately and polls.\n- Reuses scan-inbox advisory lock pattern so a manually-uploaded file plus an n8n cron run don't double-process.\n\nKnowledgeBase extension:\n- EspoCRM PHP proxy for multipart upload — Controllers can read $request->getParsedBody() but multipart needs special handling. Look at how other extensions do it (LegalAssistance/DigitalSignature might have examples).\n- New tab 'ניהול' (visible only to non-portal users; can also gate to admin role later). The tab opens a panel with: (a) Upload form, (b) Sources table for current topic [implemented in Phase 3], (c) Recent jobs.\n- Upload form fields: file picker, kind dropdown, topic dropdown (defaulting to currently-selected topic), optional title/identifier/dates/source_url collapse. Submit → POST KnowledgeBase/action/upload → returns job_id → switch to job-status view that polls every 2s.\n- Job status view: shows the job's status badge (queued/processing/done/failed), elapsed time, and on done: link to the source in the management table (Phase 3).\n\nFile size: enforce 50MB limit at the FastAPI route + server-side. PDFs of עשרות-שעות could approach this.", "details": "DB:\n- New table kb_ingest_job: id PK, source_id INTEGER NULL FK kb_source(id) ON DELETE SET NULL, original_filename TEXT, s3_key TEXT, kind TEXT, topic_id INTEGER NOT NULL, metadata_json JSONB, status TEXT NOT NULL CHECK (status IN ('queued','processing','done','failed')), error_message TEXT, chunks_created INTEGER, created_at TIMESTAMPTZ DEFAULT now(), started_at TIMESTAMPTZ, completed_at TIMESTAMPTZ, requested_by_user TEXT.\n- Index: (status, created_at DESC) for the queued list.\n\nshira-hermes endpoints:\n- POST /admin/kb/upload (multipart): receives file, kind, topic_id, optional title/identifier/published_at/effective_at/source_url. Validates topic exists. Writes file to s3 inbox/<topic_slug>/<kind>/<uuid-prefixed filename>. Inserts kb_ingest_job with status=queued. Schedules asyncio.create_task(_process_ingest_job(job_id)) so processing starts immediately, not on the next n8n cron tick. Returns {job_id, status:'queued'}.\n- GET /admin/kb/jobs?topic_id=&status=&limit=: list jobs, newest first.\n- GET /admin/kb/jobs/{id}: single job detail.\n- _process_ingest_job(job_id): UPDATE status='processing', started_at=now. Calls existing kb_ingest.ingest_source. On success: UPDATE status='done', source_id=<new>, chunks_created=<n>, completed_at=now. On failure: UPDATE status='failed', error_message=<exception text>. The upload doesn't block the HTTP request — the user gets the job_id immediately and polls.\n- Reuses scan-inbox advisory lock pattern so a manually-uploaded file plus an n8n cron run don't double-process.\n\nKnowledgeBase extension:\n- EspoCRM PHP proxy for multipart upload — Controllers can read $request->getParsedBody() but multipart needs special handling. Look at how other extensions do it (LegalAssistance/DigitalSignature might have examples).\n- New tab 'ניהול' (visible only to non-portal users; can also gate to admin role later). The tab opens a panel with: (a) Upload form, (b) Sources table for current topic [implemented in Phase 3], (c) Recent jobs.\n- Upload form fields: file picker, kind dropdown, topic dropdown (defaulting to currently-selected topic), optional title/identifier/dates/source_url collapse. Submit → POST KnowledgeBase/action/upload → returns job_id → switch to job-status view that polls every 2s.\n- Job status view: shows the job's status badge (queued/processing/done/failed), elapsed time, and on done: link to the source in the management table (Phase 3).\n\nFile size: enforce 50MB limit at the FastAPI route + server-side. PDFs of עשרות-שעות could approach this.",
"testStrategy": "1) Upload a small test PDF (~1MB). Job goes queued → processing → done in <30s. UI shows progress live. 2) Upload a corrupt file. Job ends in failed with a useful error_message. 3) Upload while another large ingest is running — job shows 'processing' but doesn't deadlock. 4) Refresh the page mid-upload — job status restored from sessionStorage + a fresh poll. 5) /admin/kb/jobs returns the upload's row.", "testStrategy": "1) Upload a small test PDF (~1MB). Job goes queued → processing → done in <30s. UI shows progress live. 2) Upload a corrupt file. Job ends in failed with a useful error_message. 3) Upload while another large ingest is running — job shows 'processing' but doesn't deadlock. 4) Refresh the page mid-upload — job status restored from sessionStorage + a fresh poll. 5) /admin/kb/jobs returns the upload's row.",
@@ -158,13 +158,14 @@
"dependencies": [ "dependencies": [
"12" "12"
], ],
"createdAt": "2026-04-25T13:30:00Z" "createdAt": "2026-04-25T13:30:00Z",
"updatedAt": "2026-04-25T16:47:46.599Z"
}, },
{ {
"id": "14", "id": "14",
"title": "Phase 3 — Management panel UI (sources table, edit metadata, delete, re-ingest)", "title": "Phase 3 — Management panel UI (sources table, edit metadata, delete, re-ingest)",
"description": "Today the only way to manage existing sources is via SQL on the shared PG (DELETE FROM kb_source WHERE id=…) plus an mc command on MinIO. End users need a panel where they can: see all sources for the selected topic, with chunk counts and ingestion dates; edit source metadata (title, identifier, dates, source_url); delete a source (with confirm and clear cascade messaging); re-ingest a source (re-fetch its file from MinIO processed/, drop chunks, re-run parse+chunk+embed); see the most recent ingest jobs (success + failure both). The panel sits in the same KB extension under a new tab 'ניהול'.", "description": "Today the only way to manage existing sources is via SQL on the shared PG (DELETE FROM kb_source WHERE id=…) plus an mc command on MinIO. End users need a panel where they can: see all sources for the selected topic, with chunk counts and ingestion dates; edit source metadata (title, identifier, dates, source_url); delete a source (with confirm and clear cascade messaging); re-ingest a source (re-fetch its file from MinIO processed/, drop chunks, re-run parse+chunk+embed); see the most recent ingest jobs (success + failure both). The panel sits in the same KB extension under a new tab 'ניהול'.",
"status": "pending", "status": "in-progress",
"priority": "high", "priority": "high",
"details": "shira-hermes endpoints:\n- GET /admin/kb/sources?topic_id=&kind=&limit=: list with id, kind, title, identifier, chunk_count, ingestion_date, last_ingest_status (joining kb_ingest_job).\n- PUT /admin/kb/sources/{id}: update editable fields (title, identifier, source_url, published_at, effective_at).\n- DELETE /admin/kb/sources/{id}: hard delete (cascades chunks via FK, drops processed/ files in S3 too).\n- POST /admin/kb/sources/{id}/reingest: requires source_id; finds the original file at original_path s3:// URI; creates a new kb_ingest_job row with status=queued and the existing source_id; background task replaces chunks+embeddings (does NOT change source_id, so all v0.1.11 chunk_index references remain valid IF the chunker output is stable; otherwise old _lastSearch sessionStorage will point at chunk_indexes that no longer exist — flag for testing). Should optionally just delete + re-create the source — simpler.\n\nEspoCRM client:\n- 'ניהול' tab. Top: header showing topic name + a count badge ('X מסמכים בנושא'). Below: 3 collapsed sections — uploaded jobs (collapsed if all done), sources table, recent failures.\n- Sources table: id, kind label, title, identifier, chunk_count, ingestion_date. Rightmost column: action buttons (✏ edit, 🗑 delete, ↻ re-ingest, 👁 view in browse mode).\n- Edit: opens a modal with the editable fields. PUT on save.\n- Delete: confirm modal that shows chunk_count + 'this will also remove the PDF from the search results'. Two-step confirmation if source has >100 chunks.\n- Re-ingest: confirms; submits POST /reingest; surfaces in the jobs section.\n- Use jQuery + Espo.Ui.dialog patterns consistent with the rest of the extension.\n\nACL gate: this tab visible only to users with role 'KB Admin' (define in EspoCRM if not present) or fall back to !isPortal for now.", "details": "shira-hermes endpoints:\n- GET /admin/kb/sources?topic_id=&kind=&limit=: list with id, kind, title, identifier, chunk_count, ingestion_date, last_ingest_status (joining kb_ingest_job).\n- PUT /admin/kb/sources/{id}: update editable fields (title, identifier, source_url, published_at, effective_at).\n- DELETE /admin/kb/sources/{id}: hard delete (cascades chunks via FK, drops processed/ files in S3 too).\n- POST /admin/kb/sources/{id}/reingest: requires source_id; finds the original file at original_path s3:// URI; creates a new kb_ingest_job row with status=queued and the existing source_id; background task replaces chunks+embeddings (does NOT change source_id, so all v0.1.11 chunk_index references remain valid IF the chunker output is stable; otherwise old _lastSearch sessionStorage will point at chunk_indexes that no longer exist — flag for testing). Should optionally just delete + re-create the source — simpler.\n\nEspoCRM client:\n- 'ניהול' tab. Top: header showing topic name + a count badge ('X מסמכים בנושא'). Below: 3 collapsed sections — uploaded jobs (collapsed if all done), sources table, recent failures.\n- Sources table: id, kind label, title, identifier, chunk_count, ingestion_date. Rightmost column: action buttons (✏ edit, 🗑 delete, ↻ re-ingest, 👁 view in browse mode).\n- Edit: opens a modal with the editable fields. PUT on save.\n- Delete: confirm modal that shows chunk_count + 'this will also remove the PDF from the search results'. Two-step confirmation if source has >100 chunks.\n- Re-ingest: confirms; submits POST /reingest; surfaces in the jobs section.\n- Use jQuery + Espo.Ui.dialog patterns consistent with the rest of the extension.\n\nACL gate: this tab visible only to users with role 'KB Admin' (define in EspoCRM if not present) or fall back to !isPortal for now.",
"testStrategy": "1) Open 'ניהול' tab. See the 6 existing sources for ביטוח לאומי. 2) Edit one source's title — refresh → new title sticks. 3) Re-ingest ספר הליקויים. Job appears in the jobs section processing → done. Source still searchable end-to-end after. 4) Delete one source. Confirm modal appears with chunk count. After delete: source gone from /kb/sources, /kb/search no longer surfaces it. 5) Failed re-ingest (e.g. delete the file in MinIO first) — surfaces in failures section with error_message.", "testStrategy": "1) Open 'ניהול' tab. See the 6 existing sources for ביטוח לאומי. 2) Edit one source's title — refresh → new title sticks. 3) Re-ingest ספר הליקויים. Job appears in the jobs section processing → done. Source still searchable end-to-end after. 4) Delete one source. Confirm modal appears with chunk count. After delete: source gone from /kb/sources, /kb/search no longer surfaces it. 5) Failed re-ingest (e.g. delete the file in MinIO first) — surfaces in failures section with error_message.",
@@ -172,7 +173,8 @@
"dependencies": [ "dependencies": [
"13" "13"
], ],
"createdAt": "2026-04-25T13:30:00Z" "createdAt": "2026-04-25T13:30:00Z",
"updatedAt": "2026-04-25T17:03:34.486Z"
}, },
{ {
"id": "15", "id": "15",
@@ -214,13 +216,37 @@
"15" "15"
], ],
"createdAt": "2026-04-25T13:30:00Z" "createdAt": "2026-04-25T13:30:00Z"
},
{
"id": "18",
"title": "feat(kb): add 'caselaw' (פסיקה) as a fourth kind",
"description": "Add 'caselaw' alongside law/regulation/circular across the whole stack. Touches: DB CHECK constraints (migration 003), shira-hermes Python (Literal types, _KINDS, kind validation, chunker section detection), EspoCRM Controller/Service validators, template dropdowns, JS kindHe mapping.",
"details": "DB: migration 003_caselaw_kind.sql alters CHECK on kb_source.kind and kb_ingest_job.kind to include 'caselaw'. shira-hermes: update Literal types in ingest.py, admin_kb.py, kb_public.py SearchRequest; add 'caselaw' to s3.py _KINDS; chunker.py — fall back to generic section detector (regex for 'פסק דין', 'תיק' / case number, paragraph numbers). EspoCRM: update validators in Controller postActionUpload + Service search/uploadFile; add 'caselaw' option to kind <select> in index.tpl (search box + upload form); update kindHe in index.js.",
"testStrategy": "",
"status": "done",
"dependencies": [],
"priority": "medium",
"subtasks": [],
"updatedAt": "2026-04-25T17:02:14.241Z"
},
{
"id": "19",
"title": "feat(kb): tag/label sub-topics within a topic (e.g. 'ילד נכה')",
"description": "Group multiple sources within a topic by sub-subject so users can find e.g. all 'ילד נכה' circulars together. Recommendation: many-to-many kb_label + kb_source_label so a source can carry multiple tags reflecting that legal docs commonly touch several subjects.",
"status": "pending",
"priority": "medium",
"details": "Three approaches considered:\n\n1. Free-text 'subject' column on kb_source (~1d): cheapest, typo-prone (ילד נכה / ילדים נכים → separate buckets).\n\n2. Many-to-many labels (~3d, RECOMMENDED): kb_label(id, topic_id, slug UNIQUE per topic, name) + kb_source_label(source_id, label_id, PK both). UI shows label chips on sources; click to filter. Naturally handles cross-cutting subjects (a circular tagged both 'ילד נכה' AND 'אזרח ותיק'). Needs admin UI to manage labels.\n\n3. Hierarchical sub-topics under kb_topic (~5d): clean tree, but a source can only live in one branch — wrong fit for the legal world. Also clashes with planned Phase 4 topic-CRUD UI.\n\nPragmatic path: ship #1 first (subject TEXT field on kb_source + dropdown of seen values + browse-view grouping); upgrade to #2 once we see how labels are used in practice.",
"testStrategy": "Upload 3 circulars all in 'ילד נכה' subject; verify they appear grouped together in browse view; verify search results show the subject chip; verify topic switch resets the subject filter.",
"subtasks": [],
"dependencies": [],
"createdAt": "2026-04-25T16:54:54.478470Z"
} }
], ],
"metadata": { "metadata": {
"version": "1.0.0", "version": "1.0.0",
"lastModified": "2026-04-25T14:15:58.500Z", "lastModified": "2026-04-25T17:03:34.486Z",
"taskCount": 17, "taskCount": 19,
"completedCount": 8, "completedCount": 11,
"tags": [ "tags": [
"master" "master"
] ]
@@ -20,6 +20,9 @@
<li class="{{#ifEqual mode 'browse'}}active{{/ifEqual}}"> <li class="{{#ifEqual mode 'browse'}}active{{/ifEqual}}">
<a href="#" role="button" data-action="switchMode" data-mode="browse">עיון</a> <a href="#" role="button" data-action="switchMode" data-mode="browse">עיון</a>
</li> </li>
<li class="{{#ifEqual mode 'manage'}}active{{/ifEqual}}">
<a href="#" role="button" data-action="switchMode" data-mode="manage">ניהול</a>
</li>
</ul> </ul>
{{#ifEqual mode 'search'}} {{#ifEqual mode 'search'}}
@@ -33,6 +36,7 @@
<option value="law">חוק</option> <option value="law">חוק</option>
<option value="regulation">תקנות</option> <option value="regulation">תקנות</option>
<option value="circular">חוזרים</option> <option value="circular">חוזרים</option>
<option value="caselaw">פסיקה</option>
</select> </select>
<button type="button" class="btn btn-primary" data-action="submit">חפש</button> <button type="button" class="btn btn-primary" data-action="submit">חפש</button>
<button type="button" class="btn btn-default" data-action="clearResults" <button type="button" class="btn btn-default" data-action="clearResults"
@@ -66,5 +70,94 @@
</div> </div>
{{/ifEqual}} {{/ifEqual}}
{{#ifEqual mode 'manage'}}
<div class="kb-manage" style="display:flex;flex-direction:column;gap:16px;">
<div class="panel panel-default kb-sources-panel" style="padding:12px;">
<div style="display:flex;justify-content:space-between;align-items:center;flex-wrap:wrap;gap:8px;margin-bottom:8px;">
<h4 style="margin:0;">מקורות בנושא<span class="kb-topic-name-suffix"></span></h4>
<div style="display:flex;gap:8px;align-items:center;">
<select class="form-control kb-sources-kind-filter" style="width:auto;">
<option value="">כל הסוגים</option>
<option value="law">חוק</option>
<option value="regulation">תקנות</option>
<option value="circular">חוזרים</option>
<option value="caselaw">פסיקה</option>
</select>
<button type="button" class="btn btn-default btn-sm" data-action="refreshSources" title="רענן">
<span class="glyphicon glyphicon-refresh"></span>
</button>
</div>
</div>
<div class="kb-sources-table">
<div class="text-muted">טוען מקורות…</div>
</div>
</div>
<div class="panel panel-default" style="padding:12px;">
<h4 style="margin-top:0;">העלאת מסמך חדש</h4>
<form class="kb-upload-form" enctype="multipart/form-data"
style="display:flex;flex-direction:column;gap:10px;">
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label for="kb-upload-file" class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">קובץ:</label>
<input type="file" id="kb-upload-file" name="file"
accept=".pdf,.docx,.txt"
style="flex:1 1 280px;" />
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label for="kb-upload-kind" class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">סוג:</label>
<select id="kb-upload-kind" class="form-control" name="kind"
style="flex:0 0 auto;width:200px;">
<option value="law">חוק</option>
<option value="regulation">תקנות</option>
<option value="circular">חוזר</option>
<option value="caselaw">פסיקה</option>
</select>
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label for="kb-upload-title" class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">כותרת:</label>
<input type="text" id="kb-upload-title" class="form-control" name="title"
placeholder="ברירת מחדל: שם הקובץ"
style="flex:1 1 280px;" />
</div>
<details>
<summary class="text-muted" style="cursor:pointer;">מטא-דאטה נוסף (אופציונלי)</summary>
<div style="display:flex;flex-direction:column;gap:8px;margin-top:8px;padding-right:12px;">
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">מזהה:</label>
<input type="text" class="form-control" name="identifier"
placeholder="למשל: ח'(353) 14.1.2018"
style="flex:1 1 280px;" />
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">פורסם ב:</label>
<input type="date" class="form-control" name="published_at"
style="flex:0 0 auto;width:200px;" />
<label class="text-muted" style="margin:0;font-weight:normal;">תוקף מ:</label>
<input type="date" class="form-control" name="effective_at"
style="flex:0 0 auto;width:200px;" />
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">קישור מקור:</label>
<input type="url" class="form-control" name="source_url"
placeholder="https://…"
style="flex:1 1 280px;" />
</div>
</div>
</details>
<div style="display:flex;gap:8px;align-items:center;">
<button type="button" class="btn btn-primary" data-action="submitUpload">העלה</button>
<span class="kb-upload-hint text-muted small">מקסימום 50MB. PDF / DOCX / TXT.</span>
</div>
</form>
</div>
<div class="panel panel-default" style="padding:12px;">
<h4 style="margin-top:0;">משימות אחרונות</h4>
<div class="kb-jobs-list">
<div class="text-muted">טוען…</div>
</div>
</div>
</div>
{{/ifEqual}}
<div class="kb-results" style="margin-top:12px;"></div> <div class="kb-results" style="margin-top:12px;"></div>
</div> </div>
@@ -12,6 +12,10 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
let _lastSearch = null; // {query, kind, topicId, hits, selectedIdx, completedAt} let _lastSearch = null; // {query, kind, topicId, hits, selectedIdx, completedAt}
let _topics = null; // [{id, slug, name, description, is_active}] let _topics = null; // [{id, slug, name, description, is_active}]
let _topicsPromise = null; // dedupe concurrent fetches across mounts let _topicsPromise = null; // dedupe concurrent fetches across mounts
let _activeJobId = null; // currently-polled ingest job id (across remounts)
let _activeJobTopicId = null; // topic this job belongs to (for cross-topic guard)
let _pollIntervalId = null; // setInterval handle so remounts don't double-poll
let _adminSourcesByTopic = {}; // {topicId: [...sources]} cache for the manage tab
const SS_ASK = 'kb-last-ask'; const SS_ASK = 'kb-last-ask';
const SS_SEARCH = 'kb-last-search'; const SS_SEARCH = 'kb-last-search';
@@ -117,6 +121,41 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
if (!Number.isFinite(id) || id === this.topicId) return; if (!Number.isFinite(id) || id === this.topicId) return;
this.switchTopic(id); this.switchTopic(id);
}, },
'click [data-action="submitUpload"]': function (e) {
e.preventDefault();
this.runUpload();
},
'click [data-action="refreshSources"]': function (e) {
e.preventDefault();
this._loadAdminSources(/* force */ true);
},
'change .kb-sources-kind-filter': function () {
this._renderAdminSourcesTable();
},
'click [data-action="editSource"]': function (e) {
e.preventDefault();
const id = parseInt($(e.currentTarget).data('id'), 10);
if (id) this._openSourceEditForm(id);
},
'click [data-action="cancelEdit"]': function (e) {
e.preventDefault();
this._renderAdminSourcesTable();
},
'click [data-action="saveEdit"]': function (e) {
e.preventDefault();
const id = parseInt($(e.currentTarget).data('id'), 10);
if (id) this._saveSourceEdit(id);
},
'click [data-action="deleteSource"]': function (e) {
e.preventDefault();
const id = parseInt($(e.currentTarget).data('id'), 10);
if (id) this._deleteSource(id);
},
'click [data-action="reingestSource"]': function (e) {
e.preventDefault();
const id = parseInt($(e.currentTarget).data('id'), 10);
if (id) this._reingestSource(id);
},
}, },
// Cancel any in-flight ask/search, drop the cached last-result for // Cancel any in-flight ask/search, drop the cached last-result for
@@ -190,6 +229,414 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
}); });
}, },
// Phase 2: lazy-loads the recent-jobs list and resumes polling of any
// in-flight upload job. No-op if .kb-jobs-list isn't in the DOM (i.e.
// the user isn't currently on the 'manage' tab).
_ensureManagePanelLoaded: function () {
const self = this;
const $list = this.$el.find('.kb-jobs-list');
if (!$list.length || this.topicId == null) return;
// Resume polling if we navigated away mid-upload and came back.
if (_activeJobId && _activeJobTopicId === this.topicId) {
this._startJobPolling(_activeJobId);
} else if (_activeJobId && _activeJobTopicId !== this.topicId) {
// Cross-topic guard: an upload in another topic is invisible
// here. The polling continues silently in the background.
}
// Always refresh the recent-jobs list on mount.
this._loadJobsList();
// Phase 3: load the sources table when the manage tab is open.
this._loadAdminSources();
},
_loadJobsList: function () {
const self = this;
const $list = this.$el.find('.kb-jobs-list');
if (!$list.length) return;
const topicAtFetch = this.topicId;
Espo.Ajax.getRequest('KnowledgeBase/action/jobs', {
topicId: topicAtFetch,
limit: 20,
}).then(res => {
if (self.topicId !== topicAtFetch) return; // user switched
self._renderJobsList(res && res.items || []);
}).catch(err => {
console.error('KB: failed to load jobs', err);
$list.html('<div class="text-muted">שגיאה בטעינת רשימת המשימות.</div>');
});
},
_renderJobsList: function (items) {
const $list = this.$el.find('.kb-jobs-list');
if (!$list.length) return;
if (!items.length) {
$list.html('<div class="text-muted">אין משימות עדיין.</div>');
return;
}
const statusHe = {
queued: '<span class="label label-default">בתור</span>',
processing: '<span class="label label-info">מעובד…</span>',
done: '<span class="label label-success">הושלם</span>',
failed: '<span class="label label-danger">נכשל</span>',
};
const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזר', caselaw: 'פסיקה'};
const rows = items.map(j => {
const when = j.completed_at || j.started_at || j.created_at || '';
const whenShort = String(when).slice(0, 19).replace('T', ' ');
const meta = j.metadata_json || {};
const title = this.escape(meta.title || j.original_filename || '—');
const fname = this.escape(j.original_filename || '');
const detail = j.status === 'done'
? `<span class="text-muted">${j.chunks_created || 0} קטעים</span>`
: (j.status === 'failed'
? `<span class="text-danger" title="${this.escape(j.error_message || '')}">${this.escape((j.error_message || '').slice(0, 80))}</span>`
: '');
return `<div style="display:flex;justify-content:space-between;align-items:center;border-bottom:1px solid #eee;padding:8px 0;gap:12px;">
<div style="flex:1 1 auto;min-width:0;">
<div style="font-weight:500;">${title}</div>
<div class="text-muted small">
${kindHe[j.kind] || j.kind} · ${this.escape(whenShort)}${j.requested_by_user ? ' · ' + this.escape(j.requested_by_user) : ''}
</div>
${detail ? '<div class="small">' + detail + '</div>' : ''}
</div>
<div>${statusHe[j.status] || this.escape(j.status)}</div>
</div>`;
}).join('');
$list.html(rows);
},
// ── Phase 3: sources table (Task #14) ──────────────────────────────
_loadAdminSources: function (force) {
const self = this;
const $table = this.$el.find('.kb-sources-table');
if (!$table.length || this.topicId == null) return;
const topicAtFetch = this.topicId;
// Use cache unless caller asked for a fresh fetch.
if (!force && _adminSourcesByTopic[topicAtFetch]) {
this._renderAdminSourcesTable();
return;
}
$table.html('<div class="text-muted">טוען מקורות…</div>');
Espo.Ajax.getRequest('KnowledgeBase/action/adminSources', {
topicId: topicAtFetch,
limit: 200,
}).then(res => {
if (self.topicId !== topicAtFetch) return;
_adminSourcesByTopic[topicAtFetch] = res && res.items || [];
self._renderAdminSourcesTable();
}).catch(err => {
console.error('KB: failed to load admin sources', err);
$table.html('<div class="text-muted">שגיאה בטעינת מקורות.</div>');
});
},
_renderAdminSourcesTable: function () {
const $table = this.$el.find('.kb-sources-table');
if (!$table.length || this.topicId == null) return;
const all = _adminSourcesByTopic[this.topicId] || [];
const filterKind = (this.$el.find('.kb-sources-kind-filter').val() || '').trim();
const items = filterKind ? all.filter(s => s.kind === filterKind) : all;
if (!items.length) {
$table.html('<div class="text-muted">' + (all.length ? 'אין מקורות בסוג זה.' : 'אין מקורות בנושא הזה.') + '</div>');
return;
}
const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזר', caselaw: 'פסיקה'};
const rows = items.map(s => {
const li = s.last_ingest || {};
const failedFlag = li.status === 'failed'
? `<span class="label label-danger" title="${this.escape(li.error_message || '')}">כשל בקריאה אחרונה</span>`
: '';
const updated = String(s.updated_at || s.created_at || '').slice(0, 10);
const ident = s.identifier ? this.escape(s.identifier) : '<span class="text-muted">—</span>';
return `<tr data-id="${s.id}">
<td><span class="label label-default">${kindHe[s.kind] || s.kind}</span></td>
<td>
<div class="kb-src-title" style="font-weight:500;">${this.escape(s.title || '—')}</div>
<div class="text-muted small">${ident} · ${this.escape(updated)} ${failedFlag}</div>
</td>
<td class="text-center">${s.chunk_count}</td>
<td style="white-space:nowrap;text-align:left;">
<button class="btn btn-link btn-sm" data-action="editSource" data-id="${s.id}" title="ערוך"><span class="glyphicon glyphicon-pencil"></span></button>
<button class="btn btn-link btn-sm" data-action="reingestSource" data-id="${s.id}" title="עיבוד מחדש"><span class="glyphicon glyphicon-refresh"></span></button>
<button class="btn btn-link btn-sm" data-action="deleteSource" data-id="${s.id}" title="מחק" style="color:#d9534f;"><span class="glyphicon glyphicon-trash"></span></button>
</td>
</tr>`;
}).join('');
$table.html(`
<table class="table table-condensed table-hover" style="margin:0;">
<thead>
<tr><th style="width:80px;">סוג</th><th>כותרת / מזהה / עודכן</th><th style="width:60px;text-align:center;">קטעים</th><th style="width:140px;"></th></tr>
</thead>
<tbody>${rows}</tbody>
</table>
`);
},
_findSourceById: function (id) {
const arr = _adminSourcesByTopic[this.topicId] || [];
return arr.find(s => s.id === id);
},
_openSourceEditForm: function (id) {
const src = this._findSourceById(id);
if (!src) return;
const $row = this.$el.find(`.kb-sources-table tr[data-id="${id}"]`);
if (!$row.length) return;
const safe = (v) => this.escape(v == null ? '' : String(v));
const dateOnly = (v) => v ? String(v).slice(0, 10) : '';
$row.html(`
<td colspan="4">
<div style="display:flex;flex-direction:column;gap:6px;padding:8px;background:#fafafa;border-radius:4px;">
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">כותרת:</label>
<input type="text" class="form-control" data-edit="title" value="${safe(src.title)}" style="flex:1 1 240px;" />
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">מזהה:</label>
<input type="text" class="form-control" data-edit="identifier" value="${safe(src.identifier)}" style="flex:1 1 240px;" />
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">פורסם:</label>
<input type="date" class="form-control" data-edit="published_at" value="${safe(dateOnly(src.published_at))}" style="width:170px;" />
<label class="text-muted" style="margin:0;font-weight:normal;">תוקף מ:</label>
<input type="date" class="form-control" data-edit="effective_at" value="${safe(dateOnly(src.effective_at))}" style="width:170px;" />
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">קישור:</label>
<input type="url" class="form-control" data-edit="source_url" value="${safe(src.source_url)}" style="flex:1 1 240px;" />
</div>
<div style="display:flex;gap:8px;justify-content:flex-end;margin-top:4px;">
<button class="btn btn-default btn-sm" data-action="cancelEdit">ביטול</button>
<button class="btn btn-primary btn-sm" data-action="saveEdit" data-id="${id}">שמור</button>
</div>
</div>
</td>
`);
$row.find('input[data-edit="title"]').trigger('focus');
},
_saveSourceEdit: function (id) {
const self = this;
const $row = this.$el.find(`.kb-sources-table tr[data-id="${id}"]`);
if (!$row.length) return;
const payload = {id};
$row.find('input[data-edit]').each(function () {
const k = $(this).data('edit');
payload[k] = ($(this).val() || '').trim();
});
const $btn = $row.find('[data-action="saveEdit"]');
$btn.prop('disabled', true).text('שומר…');
Espo.Ajax.postRequest('KnowledgeBase/action/updateSource', payload)
.then(updated => {
// Replace cached row in place so the table re-renders with new metadata.
const arr = _adminSourcesByTopic[self.topicId] || [];
const idx = arr.findIndex(s => s.id === id);
if (idx >= 0) {
arr[idx] = Object.assign({}, arr[idx], updated);
}
self._renderAdminSourcesTable();
})
.catch(err => {
$btn.prop('disabled', false).text('שמור');
const msg = (err && err.responseText) || 'שגיאה לא ידועה';
alert('שמירה נכשלה: ' + msg.slice(0, 200));
});
},
_deleteSource: function (id) {
const src = this._findSourceById(id);
if (!src) return;
const cn = src.chunk_count || 0;
const titleShort = (src.title || '').slice(0, 60);
let warn = `למחוק את "${titleShort}"?\n\n` +
`מקור עם ${cn} קטעים יימחק לצמיתות מחיפוש ה-KB.\n` +
`הקובץ ב-MinIO גם יימחק.`;
if (!confirm(warn)) return;
if (cn > 100) {
if (!confirm(`למקור הזה יש מעל 100 קטעים (${cn}). אישור סופי?`)) return;
}
const self = this;
Espo.Ajax.postRequest('KnowledgeBase/action/deleteSource', {id})
.then(res => {
// Drop from cache and re-render.
const arr = _adminSourcesByTopic[self.topicId] || [];
_adminSourcesByTopic[self.topicId] = arr.filter(s => s.id !== id);
self._renderAdminSourcesTable();
})
.catch(err => {
const msg = (err && err.responseText) || 'שגיאה לא ידועה';
alert('מחיקה נכשלה: ' + msg.slice(0, 200));
});
},
_reingestSource: function (id) {
const src = this._findSourceById(id);
if (!src) return;
const titleShort = (src.title || '').slice(0, 60);
if (!confirm(`לעבד מחדש את "${titleShort}"?\n\nהקטעים הקיימים יימחקו והקובץ יעובר parse + chunk + embed מחדש.\nהמטא-דאטה (כותרת, מזהה, תאריכים) נשמרים.`)) return;
const self = this;
Espo.Ajax.postRequest('KnowledgeBase/action/reingestSource', {id})
.then(res => {
if (res && res.job_id) {
// Surface in the upload-status banner + jobs list, like an upload would.
self._renderJobStatus({
original_filename: src.title || ('source #' + id),
status: res.status || 'queued',
});
self._startJobPolling(res.job_id);
// Refresh sources after the polling reports done.
}
})
.catch(err => {
const msg = (err && err.responseText) || 'שגיאה לא ידועה';
alert('עיבוד מחדש נכשל: ' + msg.slice(0, 200));
});
},
// Live status panel rendered above the upload form during processing.
_renderJobStatus: function (job) {
const $form = this.$el.find('.kb-upload-form');
if (!$form.length) return;
// Remove any previous banner so we can replace it.
this.$el.find('.kb-upload-status').remove();
const statusHe = {
queued: 'בתור',
processing: 'מעובד…',
done: 'הושלם בהצלחה',
failed: 'נכשל',
};
const klass = job.status === 'done' ? 'alert-success'
: job.status === 'failed' ? 'alert-danger'
: 'alert-info';
const filename = this.escape(job.original_filename || '');
const extra = job.status === 'done'
? ` · נוצרו ${job.chunks_created || 0} קטעים`
: (job.status === 'failed'
? ` · ${this.escape((job.error_message || '').slice(0, 200))}`
: '');
const html = `<div class="kb-upload-status alert ${klass}" style="margin-bottom:12px;">
<strong>${this.escape(filename)}</strong> — ${statusHe[job.status] || this.escape(job.status)}${extra}
</div>`;
$form.before(html);
},
_startJobPolling: function (jobId) {
const self = this;
this._stopJobPolling();
_activeJobId = jobId;
_activeJobTopicId = this.topicId;
const tick = () => {
Espo.Ajax.getRequest('KnowledgeBase/action/job', {id: jobId})
.then(job => {
if (!self.$el || !self.$el.length) return;
self._renderJobStatus(job);
if (job.status === 'done' || job.status === 'failed') {
self._stopJobPolling();
_activeJobId = null;
_activeJobTopicId = null;
// Refresh the recent-jobs list to surface the new row,
// and the admin sources table — re-ingest changed
// chunk_count + last_ingest fields, and a fresh
// upload added a new row.
self._loadJobsList();
self._loadAdminSources(/* force */ true);
}
})
.catch(err => {
console.error('KB: job poll failed', err);
// Don't kill the interval on a single transient error;
// let the next tick try again.
});
};
tick(); // immediate first read so the user sees status fast
_pollIntervalId = setInterval(tick, 2000);
},
_stopJobPolling: function () {
if (_pollIntervalId) {
clearInterval(_pollIntervalId);
_pollIntervalId = null;
}
},
runUpload: function () {
const $form = this.$el.find('.kb-upload-form');
const $file = $form.find('input[type="file"]');
const file = ($file[0] && $file[0].files && $file[0].files[0]) || null;
if (!file) {
alert('יש לבחור קובץ.');
return;
}
if (file.size > 50 * 1024 * 1024) {
alert('הקובץ גדול מ-50MB. אנא העלה קובץ קטן יותר.');
return;
}
if (this.topicId == null) {
alert('נושא לא נבחר.');
return;
}
const fd = new FormData();
fd.append('file', file, file.name);
fd.append('kind', $form.find('select[name="kind"]').val() || 'circular');
fd.append('topicId', String(this.topicId));
['title', 'identifier', 'published_at', 'effective_at', 'source_url'].forEach(name => {
const v = ($form.find(`[name="${name}"]`).val() || '').trim();
if (v) fd.append(name, v);
});
const $btn = this.$el.find('[data-action="submitUpload"]');
$btn.prop('disabled', true).text('מעלה…');
const self = this;
// Bypass Espo.Ajax.postRequest (which JSON-encodes the body) — use
// raw $.ajax so the FormData multipart body is sent intact.
$.ajax({
url: 'api/v1/KnowledgeBase/action/upload',
method: 'POST',
data: fd,
contentType: false,
processData: false,
cache: false,
}).then(res => {
$btn.prop('disabled', false).text('העלה');
// Reset the form so the user can upload another.
$form[0].reset();
if (res && res.job_id) {
self._renderJobStatus({
original_filename: file.name,
status: res.status || 'queued',
});
self._startJobPolling(res.job_id);
}
}).fail(xhr => {
$btn.prop('disabled', false).text('העלה');
let msg = 'שגיאה לא ידועה';
try {
const json = JSON.parse(xhr.responseText || '{}');
msg = json.message || json.detail || xhr.responseText || msg;
} catch (e) {
msg = xhr.responseText || msg;
}
self.$el.find('.kb-upload-status').remove();
self.$el.find('.kb-upload-form').before(
'<div class="kb-upload-status alert alert-danger" style="margin-bottom:12px;">'
+ 'העלאה נכשלה: ' + self.escape(msg.slice(0, 300))
+ '</div>'
);
});
},
_ensureTopicsLoaded: function () { _ensureTopicsLoaded: function () {
if (_topics) return Promise.resolve(_topics); if (_topics) return Promise.resolve(_topics);
if (_topicsPromise) return _topicsPromise; if (_topicsPromise) return _topicsPromise;
@@ -253,6 +700,16 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
_clear(sessionStorage, SS_SEARCH); _clear(sessionStorage, SS_SEARCH);
this._sources = null; this._sources = null;
this._sourcesPromise = null; this._sourcesPromise = null;
// An in-flight upload belongs to the topic where it started;
// stop polling it here so the new topic doesn't see a stale banner.
// The job continues processing on the server; the user can find
// it under "משימות אחרונות" if they switch back.
this._stopJobPolling();
_activeJobId = null;
_activeJobTopicId = null;
// Phase 3: a topic change invalidates the admin sources view
// (different scope) — drop the cache so afterRender re-fetches.
_adminSourcesByTopic = {};
this.stopAskProgress(); this.stopAskProgress();
this.setLoading(false); this.setLoading(false);
this.$el.find('.kb-results').empty(); this.$el.find('.kb-results').empty();
@@ -272,6 +729,7 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
if (!self.$el || !self.$el.length) return; if (!self.$el || !self.$el.length) return;
self._populateTopicPicker(); self._populateTopicPicker();
self._ensureSourcesLoaded(); self._ensureSourcesLoaded();
self._ensureManagePanelLoaded();
}; };
if (_topics) { if (_topics) {
afterTopics(); afterTopics();
@@ -590,7 +1048,7 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
return; return;
} }
const initialIdx = (typeof selectedIdx === 'number' && hits[selectedIdx]) ? selectedIdx : 0; const initialIdx = (typeof selectedIdx === 'number' && hits[selectedIdx]) ? selectedIdx : 0;
const kindHe = {law: 'חוק', regulation: 'תקנה', circular: 'חוזר'}; const kindHe = {law: 'חוק', regulation: 'תקנה', circular: 'חוזר', caselaw: 'פסיקה'};
const self = this; const self = this;
const isPdfBacked = h => h && (h.kind === 'regulation' || h.kind === 'circular'); const isPdfBacked = h => h && (h.kind === 'regulation' || h.kind === 'circular');
@@ -889,7 +1347,7 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
answerHtml = '<div style="white-space:pre-wrap;">' + this.escape(text || '') + '</div>'; answerHtml = '<div style="white-space:pre-wrap;">' + this.escape(text || '') + '</div>';
} }
const kindHe = {law: 'חוק', regulation: 'תקנה', circular: 'חוזר'}; const kindHe = {law: 'חוק', regulation: 'תקנה', circular: 'חוזר', caselaw: 'פסיקה'};
// Dedup sources by source_id (first occurrence = most relevant), // Dedup sources by source_id (first occurrence = most relevant),
// but remember every page that came up per source so users can // but remember every page that came up per source so users can
@@ -1029,8 +1487,8 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
$list.html('<div class="text-muted">אין מקורות.</div>'); $list.html('<div class="text-muted">אין מקורות.</div>');
return; return;
} }
const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזרים'}; const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזרים', caselaw: 'פסיקה'};
const grouped = {law: [], regulation: [], circular: []}; const grouped = {law: [], regulation: [], circular: [], caselaw: []};
this._sources.forEach(s => { this._sources.forEach(s => {
if (grouped[s.kind]) grouped[s.kind].push(s); if (grouped[s.kind]) grouped[s.kind].push(s);
}); });
@@ -121,4 +121,135 @@ class KnowledgeBase
$this->coerceTopicId($data->topicId ?? $data->topic_id ?? null) $this->coerceTopicId($data->topicId ?? $data->topic_id ?? null)
); );
} }
/**
* Multipart upload to shira-hermes /admin/kb/upload. We rely on PHP's
* automatic multipart parsing via $_FILES because the EspoCRM Request
* interface doesn't expose uploaded files directly. The user's
* EspoCRM identity is forwarded as `X-User-Name` so the upstream job
* row records who uploaded what.
*/
public function postActionUpload(Request $request, Response $response): array
{
$this->checkAccess();
$files = $_FILES['file'] ?? null;
if (!$files || ($files['error'] ?? UPLOAD_ERR_NO_FILE) !== UPLOAD_ERR_OK) {
throw new BadRequest('No file uploaded (form field must be `file`).');
}
$tmpPath = $files['tmp_name'] ?? '';
if (!$tmpPath || !is_uploaded_file($tmpPath)) {
throw new BadRequest('Invalid upload — tmp_name missing.');
}
// For multipart/form-data EspoCRM's getParsedBody() returns an empty
// stdClass; the form fields land in $_POST as PHP parses them.
$kind = $_POST['kind'] ?? null;
if (!in_array($kind, ['law', 'regulation', 'circular', 'caselaw'], true)) {
throw new BadRequest('kind must be one of: law, regulation, circular, caselaw.');
}
$topicId = $this->coerceTopicId($_POST['topicId'] ?? $_POST['topic_id'] ?? null);
if ($topicId === null) {
throw new BadRequest('topicId is required.');
}
$metadata = [
'title' => isset($_POST['title']) ? trim((string) $_POST['title']) : '',
'identifier' => isset($_POST['identifier']) ? trim((string) $_POST['identifier']) : '',
'published_at' => $_POST['published_at'] ?? $_POST['publishedAt'] ?? '',
'effective_at' => $_POST['effective_at'] ?? $_POST['effectiveAt'] ?? '',
'source_url' => $_POST['source_url'] ?? $_POST['sourceUrl'] ?? '',
];
return $this->getService()->uploadFile(
$tmpPath,
(string) ($files['name'] ?? 'upload'),
(string) $kind,
$topicId,
$metadata,
(string) $this->user->get('userName')
);
}
public function getActionJobs(Request $request, Response $response): array
{
$this->checkAccess();
$topicId = $this->coerceTopicId($request->getQueryParam('topicId'));
$status = $request->getQueryParam('status');
$limit = $request->getQueryParam('limit');
$limitInt = ($limit !== null && is_numeric($limit)) ? (int) $limit : 50;
return $this->getService()->listJobs($topicId, $status, $limitInt);
}
public function getActionJob(Request $request, Response $response): array
{
$this->checkAccess();
$jobId = $request->getQueryParam('id');
if (!$jobId || !is_numeric($jobId)) {
throw new BadRequest('id (numeric) is required.');
}
return $this->getService()->getJob((int) $jobId);
}
// ── Phase 3: source management (Task #14) ───────────────────────────────
public function getActionAdminSources(Request $request, Response $response): array
{
$this->checkAccess();
$topicId = $this->coerceTopicId($request->getQueryParam('topicId'));
$kind = $request->getQueryParam('kind');
$limit = $request->getQueryParam('limit');
$limitInt = ($limit !== null && is_numeric($limit)) ? (int) $limit : 200;
return $this->getService()->listAdminSources($topicId, $kind, $limitInt);
}
public function postActionUpdateSource(Request $request, Response $response): array
{
$this->checkAccess();
$body = $request->getParsedBody();
$id = $body->id ?? null;
if (!$id || !is_numeric($id)) {
throw new BadRequest('id (numeric) is required.');
}
// Whitelist what we forward — never let arbitrary body fields hit
// upstream's PUT body. Empty strings are explicitly preserved as
// null clearing (the user wiping a date or identifier).
$payload = [];
foreach (['title', 'identifier', 'source_url', 'sourceUrl', 'published_at', 'publishedAt', 'effective_at', 'effectiveAt'] as $k) {
if (property_exists($body, $k)) {
$canon = $k;
if ($k === 'sourceUrl') $canon = 'source_url';
if ($k === 'publishedAt') $canon = 'published_at';
if ($k === 'effectiveAt') $canon = 'effective_at';
$payload[$canon] = $body->$k;
}
}
return $this->getService()->updateAdminSource((int) $id, $payload);
}
public function postActionDeleteSource(Request $request, Response $response): array
{
$this->checkAccess();
$body = $request->getParsedBody();
$id = $body->id ?? null;
if (!$id || !is_numeric($id)) {
throw new BadRequest('id (numeric) is required.');
}
return $this->getService()->deleteAdminSource((int) $id);
}
public function postActionReingestSource(Request $request, Response $response): array
{
$this->checkAccess();
$body = $request->getParsedBody();
$id = $body->id ?? null;
if (!$id || !is_numeric($id)) {
throw new BadRequest('id (numeric) is required.');
}
return $this->getService()->reingestAdminSource(
(int) $id,
(string) $this->user->get('userName')
);
}
} }
@@ -38,5 +38,61 @@
"controller": "KnowledgeBase", "controller": "KnowledgeBase",
"action": "ask" "action": "ask"
} }
},
{
"route": "/KnowledgeBase/action/upload",
"method": "post",
"params": {
"controller": "KnowledgeBase",
"action": "upload"
}
},
{
"route": "/KnowledgeBase/action/jobs",
"method": "get",
"params": {
"controller": "KnowledgeBase",
"action": "jobs"
}
},
{
"route": "/KnowledgeBase/action/job",
"method": "get",
"params": {
"controller": "KnowledgeBase",
"action": "job"
}
},
{
"route": "/KnowledgeBase/action/adminSources",
"method": "get",
"params": {
"controller": "KnowledgeBase",
"action": "adminSources"
}
},
{
"route": "/KnowledgeBase/action/updateSource",
"method": "post",
"params": {
"controller": "KnowledgeBase",
"action": "updateSource"
}
},
{
"route": "/KnowledgeBase/action/deleteSource",
"method": "post",
"params": {
"controller": "KnowledgeBase",
"action": "deleteSource"
}
},
{
"route": "/KnowledgeBase/action/reingestSource",
"method": "post",
"params": {
"controller": "KnowledgeBase",
"action": "reingestSource"
}
} }
] ]
@@ -37,7 +37,7 @@ class KnowledgeBaseService
public function search(string $query, string $kind, int $topK, ?int $topicId = null): array public function search(string $query, string $kind, int $topK, ?int $topicId = null): array
{ {
$topK = max(1, min(15, $topK)); $topK = max(1, min(15, $topK));
if (!in_array($kind, ['any', 'law', 'regulation', 'circular'], true)) { if (!in_array($kind, ['any', 'law', 'regulation', 'circular', 'caselaw'], true)) {
$kind = 'any'; $kind = 'any';
} }
@@ -70,6 +70,213 @@ class KnowledgeBaseService
return $this->get($path); return $this->get($path);
} }
/**
* Phase 2: async upload to /admin/kb/upload. Forwards the temp-uploaded
* file via cURL multipart and the EspoCRM username as `X-User-Name`
* so shira-hermes can record who uploaded what.
*
* @return array{job_id:int, status:string}
*/
public function uploadFile(
string $tmpPath,
string $filename,
string $kind,
int $topicId,
array $metadata,
string $username
): array {
if (!is_readable($tmpPath)) {
throw new Error('Uploaded file is not readable on disk.');
}
if (!in_array($kind, ['law', 'regulation', 'circular', 'caselaw'], true)) {
throw new BadRequest('kind must be one of: law, regulation, circular, caselaw.');
}
$url = $this->getBaseUrl() . '/admin/kb/upload';
$apiKey = $this->getApiKey();
if (!$apiKey) {
throw new Error('SmartAssistant API key is not configured.');
}
// Build the multipart form. CURLOPT_POSTFIELDS as an array makes
// cURL set the multipart Content-Type with a generated boundary.
$fields = [
'kind' => $kind,
'topic_id' => (string) $topicId,
'file' => new \CURLFile($tmpPath, $this->guessMime($filename), $filename),
];
foreach (['title', 'identifier', 'published_at', 'effective_at', 'source_url'] as $k) {
$val = $metadata[$k] ?? '';
if ($val !== '' && $val !== null) {
$fields[$k] = (string) $val;
}
}
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $fields,
CURLOPT_HTTPHEADER => [
'Accept: application/json',
'X-Api-Key: ' . $apiKey,
'X-User-Name: ' . $username,
],
CURLOPT_RETURNTRANSFER => true,
CURLOPT_TIMEOUT => 120,
CURLOPT_CONNECTTIMEOUT => 10,
]);
$body = curl_exec($ch);
$httpCode = (int) curl_getinfo($ch, CURLINFO_HTTP_CODE);
$err = curl_error($ch);
curl_close($ch);
if ($err) {
$this->log->error("KnowledgeBase: upload transport error: {$err}");
throw new Error("Failed to reach Knowledge Base: {$err}");
}
if ($httpCode === 413) {
throw new BadRequest('File too large (max 50MB).');
}
if ($httpCode >= 400 && $httpCode < 500) {
$detail = $this->decodeDetail($body) ?: 'Bad request';
throw new BadRequest($detail);
}
if ($httpCode < 200 || $httpCode >= 300) {
$this->log->error("KnowledgeBase: upload HTTP {$httpCode}: {$body}");
throw new Error("Knowledge Base returned HTTP {$httpCode}");
}
$decoded = json_decode((string) $body, true);
if (!is_array($decoded) || !isset($decoded['job_id'])) {
throw new Error('Invalid response from Knowledge Base.');
}
return $decoded;
}
public function listJobs(?int $topicId, ?string $status, int $limit): array
{
$limit = max(1, min(200, $limit));
$qs = ['limit' => $limit];
if ($topicId !== null) {
$qs['topic_id'] = $topicId;
}
if ($status !== null && $status !== '') {
$qs['status'] = $status;
}
$path = '/admin/kb/jobs?' . http_build_query($qs);
return $this->get($path);
}
public function getJob(int $jobId): array
{
return $this->get('/admin/kb/jobs/' . $jobId);
}
// ── Phase 3: admin source management (Task #14) ─────────────────────────
public function listAdminSources(?int $topicId, ?string $kind, int $limit): array
{
$limit = max(1, min(500, $limit));
$qs = ['limit' => $limit];
if ($topicId !== null) $qs['topic_id'] = $topicId;
if ($kind !== null && $kind !== '') $qs['kind'] = $kind;
return $this->get('/admin/kb/sources?' . http_build_query($qs));
}
public function updateAdminSource(int $sourceId, array $fields): array
{
return $this->request('PUT', '/admin/kb/sources/' . $sourceId, $fields);
}
public function deleteAdminSource(int $sourceId): array
{
return $this->request('DELETE', '/admin/kb/sources/' . $sourceId, null);
}
public function reingestAdminSource(int $sourceId, string $username): array
{
// Re-ingest needs the user identity for audit; piggyback on the
// X-User-Name header that the upload route already understands.
return $this->postWithUser(
'/admin/kb/sources/' . $sourceId . '/reingest',
null,
$username
);
}
private function postWithUser(string $path, ?array $payload, string $username): array
{
$url = $this->getBaseUrl() . $path;
$apiKey = $this->getApiKey();
$headers = ['Accept: application/json', 'X-User-Name: ' . $username];
if ($apiKey) $headers[] = 'X-Api-Key: ' . $apiKey;
$ch = curl_init($url);
$opts = [
CURLOPT_CUSTOMREQUEST => 'POST',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_TIMEOUT => 180,
CURLOPT_CONNECTTIMEOUT => 10,
];
if ($payload !== null) {
$headers[] = 'Content-Type: application/json';
$opts[CURLOPT_POSTFIELDS] = json_encode($payload);
}
$opts[CURLOPT_HTTPHEADER] = $headers;
curl_setopt_array($ch, $opts);
$body = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$error = curl_error($ch);
curl_close($ch);
if ($error) {
$this->log->error("KnowledgeBase: HTTP error calling {$url}: {$error}");
throw new Error("Failed to reach Knowledge Base: {$error}");
}
if ($httpCode < 200 || $httpCode >= 300) {
$detail = $this->decodeDetail($body) ?: "HTTP {$httpCode}";
if ($httpCode >= 400 && $httpCode < 500) {
throw new BadRequest($detail);
}
throw new Error("Knowledge Base returned HTTP {$httpCode}");
}
$decoded = json_decode($body, true);
if (!is_array($decoded)) {
throw new Error('Invalid JSON from Knowledge Base.');
}
return $decoded;
}
private function guessMime(string $filename): string
{
$lower = strtolower($filename);
if (str_ends_with($lower, '.pdf')) {
return 'application/pdf';
}
if (str_ends_with($lower, '.docx')) {
return 'application/vnd.openxmlformats-officedocument.wordprocessingml.document';
}
if (str_ends_with($lower, '.txt')) {
return 'text/plain';
}
return 'application/octet-stream';
}
private function decodeDetail($body): ?string
{
if (!is_string($body)) {
return null;
}
$decoded = json_decode($body, true);
if (is_array($decoded) && isset($decoded['detail'])) {
return is_string($decoded['detail']) ? $decoded['detail'] : json_encode($decoded['detail']);
}
return null;
}
public function ask( public function ask(
string $message, string $message,
?string $conversationId, ?string $conversationId,
+1 -1
View File
@@ -1,7 +1,7 @@
{ {
"name": "KnowledgeBase", "name": "KnowledgeBase",
"module": "KnowledgeBase", "module": "KnowledgeBase",
"version": "0.3.0", "version": "0.6.0",
"acceptableVersions": [ "acceptableVersions": [
">=8.0.0" ">=8.0.0"
], ],