2 Commits

Author SHA1 Message Date
chaim e7b21919c2 feat: add 'caselaw' (פסיקה) kind alongside חוק/תקנות/חוזרים
Court rulings get their own kind so users can filter search results
to "פסיקה" specifically and upload case-law PDFs from the ניהול tab.
PHP validators in Controller + Service accept the new kind; the search
dropdown and upload kind dropdown gain a "פסיקה" option; kindHe in
index.js maps caselaw → "פסיקה" so jobs list, search results, and
browse view all label it consistently.

Backend: depends on shira-hermes commit 16eeeff (kind validators +
chunker fallback) and migration 003_caselaw_kind.sql (already applied
on dev — alters CHECK constraints on kb_source.kind and
kb_ingest_job.kind to include 'caselaw').

Refs Task Master #18

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-25 17:01:12 +00:00
chaim 6781ac4e37 feat: Phase 2 — document upload from KB UI
Adds a "ניהול" tab with multipart file upload, async job tracking, and
a recent-jobs list with live status polling. Browser POSTs to
/KnowledgeBase/action/upload (PHP proxy), which forwards as multipart
to shira-hermes /admin/kb/upload. shira-hermes returns a job_id
immediately and processes parse/chunk/embed in a background task; the
browser polls every 2s until status hits done|failed.

New EspoCRM endpoints:
  POST /KnowledgeBase/action/upload   (multipart, $_FILES['file'])
  GET  /KnowledgeBase/action/jobs     (list, filtered by topicId/status)
  GET  /KnowledgeBase/action/job?id   (single-job detail)

The X-User-Name header is forwarded so kb_ingest_job records who
uploaded each file. Cross-topic guard in switchTopic stops polling
when the user changes topics mid-upload (the job continues server-side).

Depends on shira-hermes commit 0cb89fb (admin upload endpoints +
migration 002).

Refs Task Master #13

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-25 16:46:42 +00:00
8 changed files with 559 additions and 14 deletions
+1 -1
View File
@@ -2,5 +2,5 @@
"currentTag": "master", "currentTag": "master",
"lastSwitched": "2026-04-24T15:47:33.139Z", "lastSwitched": "2026-04-24T15:47:33.139Z",
"branchTagMapping": {}, "branchTagMapping": {},
"migrationNoticeShown": false "migrationNoticeShown": true
} }
+32 -7
View File
@@ -137,20 +137,20 @@
"id": "12", "id": "12",
"title": "Phase 1 — Multi-topic KB foundation (DB schema, topic-aware endpoints, topic selector)", "title": "Phase 1 — Multi-topic KB foundation (DB schema, topic-aware endpoints, topic selector)",
"description": "Generalize the KB from being insurance-only to supporting multiple legal domains within one CRM (e.g. ביטוח לאומי, דיני עבודה, דין פלילי). A 'topic' is the firm-level grouping; each kb_source belongs to exactly one topic. Search/ask are scoped to a single topic at a time, chosen from a topic dropdown the user controls. The system_prompt for /kb/ask becomes per-topic so Shira's domain context is correct (today the prompt hardcodes 'Israeli National Insurance'). Backwards-compatible migration: all 6 existing sources move to topic_id=1 named 'ביטוח לאומי' with the existing system-prompt addendum, and the UI defaults to that topic so nothing visible changes for existing users until they choose another topic.", "description": "Generalize the KB from being insurance-only to supporting multiple legal domains within one CRM (e.g. ביטוח לאומי, דיני עבודה, דין פלילי). A 'topic' is the firm-level grouping; each kb_source belongs to exactly one topic. Search/ask are scoped to a single topic at a time, chosen from a topic dropdown the user controls. The system_prompt for /kb/ask becomes per-topic so Shira's domain context is correct (today the prompt hardcodes 'Israeli National Insurance'). Backwards-compatible migration: all 6 existing sources move to topic_id=1 named 'ביטוח לאומי' with the existing system-prompt addendum, and the UI defaults to that topic so nothing visible changes for existing users until they choose another topic.",
"status": "in-progress", "status": "done",
"priority": "high", "priority": "high",
"details": "DB migration (shira-hermes side, against insurance_kb DB):\n- New table kb_topic: id PK, slug TEXT UNIQUE, name TEXT NOT NULL, description TEXT, system_prompt_addendum TEXT, is_active BOOLEAN DEFAULT true, created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ DEFAULT now().\n- Seed: INSERT (id=1, slug='national-insurance', name='ביטוח לאומי', system_prompt_addendum=<copy from current /kb/ask hardcoded text>, is_active=true). RENAME the DB itself? Probably leave as 'insurance_kb' — costly to rename, no real value. Document in memory.\n- ALTER TABLE kb_source ADD COLUMN topic_id INTEGER REFERENCES kb_topic(id). Backfill: UPDATE kb_source SET topic_id = 1. Then ALTER COLUMN SET NOT NULL.\n\nshira-hermes endpoints:\n- New GET /kb/topics — public, returns active topics with id, slug, name, description (no prompt).\n- /kb/search: accepts optional topic_id (single int). When present, the SQL adds AND s.topic_id = $N. When absent (back-compat), still returns all (might want to deprecate this and require topic_id later).\n- /kb/sources, /kb/source/{id}/chunks, /kb/source/{id}/pdf: same — optional topic filter on listings, no change to single-source endpoints.\n- /kb/ask + /kb/ask/stream: accept topic_id param; load topic.system_prompt_addendum and inject into system_prompt instead of the hardcoded insurance text in _build_ask_runner_context. Pin the legal_kb tool to topic_id too (so search_insurance_kb queries are constrained — RENAME the tool to search_legal_kb and pass topic_id at registration time).\n- _expand_query in kb_search.py: parametrize the prompt by topic name (currently hardcoded 'הביטוח הלאומי בישראל'). Pass topic.name in.\n\nKnowledgeBase extension (EspoCRM):\n- New route+action GET /KnowledgeBase/action/topics → proxies /kb/topics.\n- Service.search/ask gain topic_id, controller passes through.\n- Template gets a topic <select> at the top (above the tabs); fetches /KnowledgeBase/action/topics on mount, persists selection to localStorage 'kb-topic'. Default: topic id=1 if it exists, else first active topic.\n- this.kind etc. already pass through; add this.topicId. Pass topic_id with every search/ask call.\n\nConsequences for sessionStorage replay (v0.1.9, v0.1.10): the cached _lastAsk / _lastSearch should record topic_id; on remount only replay if the current topic matches. Else show empty state.", "details": "DB migration (shira-hermes side, against insurance_kb DB):\n- New table kb_topic: id PK, slug TEXT UNIQUE, name TEXT NOT NULL, description TEXT, system_prompt_addendum TEXT, is_active BOOLEAN DEFAULT true, created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ DEFAULT now().\n- Seed: INSERT (id=1, slug='national-insurance', name='ביטוח לאומי', system_prompt_addendum=<copy from current /kb/ask hardcoded text>, is_active=true). RENAME the DB itself? Probably leave as 'insurance_kb' — costly to rename, no real value. Document in memory.\n- ALTER TABLE kb_source ADD COLUMN topic_id INTEGER REFERENCES kb_topic(id). Backfill: UPDATE kb_source SET topic_id = 1. Then ALTER COLUMN SET NOT NULL.\n\nshira-hermes endpoints:\n- New GET /kb/topics — public, returns active topics with id, slug, name, description (no prompt).\n- /kb/search: accepts optional topic_id (single int). When present, the SQL adds AND s.topic_id = $N. When absent (back-compat), still returns all (might want to deprecate this and require topic_id later).\n- /kb/sources, /kb/source/{id}/chunks, /kb/source/{id}/pdf: same — optional topic filter on listings, no change to single-source endpoints.\n- /kb/ask + /kb/ask/stream: accept topic_id param; load topic.system_prompt_addendum and inject into system_prompt instead of the hardcoded insurance text in _build_ask_runner_context. Pin the legal_kb tool to topic_id too (so search_insurance_kb queries are constrained — RENAME the tool to search_legal_kb and pass topic_id at registration time).\n- _expand_query in kb_search.py: parametrize the prompt by topic name (currently hardcoded 'הביטוח הלאומי בישראל'). Pass topic.name in.\n\nKnowledgeBase extension (EspoCRM):\n- New route+action GET /KnowledgeBase/action/topics → proxies /kb/topics.\n- Service.search/ask gain topic_id, controller passes through.\n- Template gets a topic <select> at the top (above the tabs); fetches /KnowledgeBase/action/topics on mount, persists selection to localStorage 'kb-topic'. Default: topic id=1 if it exists, else first active topic.\n- this.kind etc. already pass through; add this.topicId. Pass topic_id with every search/ask call.\n\nConsequences for sessionStorage replay (v0.1.9, v0.1.10): the cached _lastAsk / _lastSearch should record topic_id; on remount only replay if the current topic matches. Else show empty state.",
"testStrategy": "1) After migration: SELECT count(*) FROM kb_source WHERE topic_id IS NULL → 0; SELECT name FROM kb_topic → 'ביטוח לאומי'. 2) GET /kb/topics returns the seeded topic. 3) Search/ask with topic_id=1 returns the same 6 sources as before. 4) Add a second topic (manually for the test) and a single test source under it; verify topic_id=1 search no longer surfaces the test source and vice versa. 5) /kb/ask system prompt confirmed via logs to include the topic-specific addendum, not the hardcoded text.", "testStrategy": "1) After migration: SELECT count(*) FROM kb_source WHERE topic_id IS NULL → 0; SELECT name FROM kb_topic → 'ביטוח לאומי'. 2) GET /kb/topics returns the seeded topic. 3) Search/ask with topic_id=1 returns the same 6 sources as before. 4) Add a second topic (manually for the test) and a single test source under it; verify topic_id=1 search no longer surfaces the test source and vice versa. 5) /kb/ask system prompt confirmed via logs to include the topic-specific addendum, not the hardcoded text.",
"subtasks": [], "subtasks": [],
"dependencies": [], "dependencies": [],
"createdAt": "2026-04-25T13:30:00Z", "createdAt": "2026-04-25T13:30:00Z",
"updatedAt": "2026-04-25T14:15:58.498Z" "updatedAt": "2026-04-25T14:34:08.086Z"
}, },
{ {
"id": "13", "id": "13",
"title": "Phase 2 — Document upload from the KB UI (file picker + async ingestion + job tracking)", "title": "Phase 2 — Document upload from the KB UI (file picker + async ingestion + job tracking)",
"description": "Today documents enter the KB by manually putting them in the MinIO inbox/ folder and waiting for the n8n cron to call /admin/kb/scan-inbox. End users have no path to upload via the UI. Add a file-upload form in the management panel: user picks a PDF/DOCX/TXT, picks a kind (law/regulation/circular), confirms the topic, optionally fills metadata (title, identifier, dates, source_url), and clicks upload. The file lands in MinIO under inbox/<topic_slug>/<kind>/, a kb_ingest_job row is created, and a background asyncio task processes it (parse → chunk → embed → upsert kb_source). The UI polls the job status and shows progress live; on done it links to the new source in the management table.", "description": "Today documents enter the KB by manually putting them in the MinIO inbox/ folder and waiting for the n8n cron to call /admin/kb/scan-inbox. End users have no path to upload via the UI. Add a file-upload form in the management panel: user picks a PDF/DOCX/TXT, picks a kind (law/regulation/circular), confirms the topic, optionally fills metadata (title, identifier, dates, source_url), and clicks upload. The file lands in MinIO under inbox/<topic_slug>/<kind>/, a kb_ingest_job row is created, and a background asyncio task processes it (parse → chunk → embed → upsert kb_source). The UI polls the job status and shows progress live; on done it links to the new source in the management table.",
"status": "pending", "status": "done",
"priority": "high", "priority": "high",
"details": "DB:\n- New table kb_ingest_job: id PK, source_id INTEGER NULL FK kb_source(id) ON DELETE SET NULL, original_filename TEXT, s3_key TEXT, kind TEXT, topic_id INTEGER NOT NULL, metadata_json JSONB, status TEXT NOT NULL CHECK (status IN ('queued','processing','done','failed')), error_message TEXT, chunks_created INTEGER, created_at TIMESTAMPTZ DEFAULT now(), started_at TIMESTAMPTZ, completed_at TIMESTAMPTZ, requested_by_user TEXT.\n- Index: (status, created_at DESC) for the queued list.\n\nshira-hermes endpoints:\n- POST /admin/kb/upload (multipart): receives file, kind, topic_id, optional title/identifier/published_at/effective_at/source_url. Validates topic exists. Writes file to s3 inbox/<topic_slug>/<kind>/<uuid-prefixed filename>. Inserts kb_ingest_job with status=queued. Schedules asyncio.create_task(_process_ingest_job(job_id)) so processing starts immediately, not on the next n8n cron tick. Returns {job_id, status:'queued'}.\n- GET /admin/kb/jobs?topic_id=&status=&limit=: list jobs, newest first.\n- GET /admin/kb/jobs/{id}: single job detail.\n- _process_ingest_job(job_id): UPDATE status='processing', started_at=now. Calls existing kb_ingest.ingest_source. On success: UPDATE status='done', source_id=<new>, chunks_created=<n>, completed_at=now. On failure: UPDATE status='failed', error_message=<exception text>. The upload doesn't block the HTTP request — the user gets the job_id immediately and polls.\n- Reuses scan-inbox advisory lock pattern so a manually-uploaded file plus an n8n cron run don't double-process.\n\nKnowledgeBase extension:\n- EspoCRM PHP proxy for multipart upload — Controllers can read $request->getParsedBody() but multipart needs special handling. Look at how other extensions do it (LegalAssistance/DigitalSignature might have examples).\n- New tab 'ניהול' (visible only to non-portal users; can also gate to admin role later). The tab opens a panel with: (a) Upload form, (b) Sources table for current topic [implemented in Phase 3], (c) Recent jobs.\n- Upload form fields: file picker, kind dropdown, topic dropdown (defaulting to currently-selected topic), optional title/identifier/dates/source_url collapse. Submit → POST KnowledgeBase/action/upload → returns job_id → switch to job-status view that polls every 2s.\n- Job status view: shows the job's status badge (queued/processing/done/failed), elapsed time, and on done: link to the source in the management table (Phase 3).\n\nFile size: enforce 50MB limit at the FastAPI route + server-side. PDFs of עשרות-שעות could approach this.", "details": "DB:\n- New table kb_ingest_job: id PK, source_id INTEGER NULL FK kb_source(id) ON DELETE SET NULL, original_filename TEXT, s3_key TEXT, kind TEXT, topic_id INTEGER NOT NULL, metadata_json JSONB, status TEXT NOT NULL CHECK (status IN ('queued','processing','done','failed')), error_message TEXT, chunks_created INTEGER, created_at TIMESTAMPTZ DEFAULT now(), started_at TIMESTAMPTZ, completed_at TIMESTAMPTZ, requested_by_user TEXT.\n- Index: (status, created_at DESC) for the queued list.\n\nshira-hermes endpoints:\n- POST /admin/kb/upload (multipart): receives file, kind, topic_id, optional title/identifier/published_at/effective_at/source_url. Validates topic exists. Writes file to s3 inbox/<topic_slug>/<kind>/<uuid-prefixed filename>. Inserts kb_ingest_job with status=queued. Schedules asyncio.create_task(_process_ingest_job(job_id)) so processing starts immediately, not on the next n8n cron tick. Returns {job_id, status:'queued'}.\n- GET /admin/kb/jobs?topic_id=&status=&limit=: list jobs, newest first.\n- GET /admin/kb/jobs/{id}: single job detail.\n- _process_ingest_job(job_id): UPDATE status='processing', started_at=now. Calls existing kb_ingest.ingest_source. On success: UPDATE status='done', source_id=<new>, chunks_created=<n>, completed_at=now. On failure: UPDATE status='failed', error_message=<exception text>. The upload doesn't block the HTTP request — the user gets the job_id immediately and polls.\n- Reuses scan-inbox advisory lock pattern so a manually-uploaded file plus an n8n cron run don't double-process.\n\nKnowledgeBase extension:\n- EspoCRM PHP proxy for multipart upload — Controllers can read $request->getParsedBody() but multipart needs special handling. Look at how other extensions do it (LegalAssistance/DigitalSignature might have examples).\n- New tab 'ניהול' (visible only to non-portal users; can also gate to admin role later). The tab opens a panel with: (a) Upload form, (b) Sources table for current topic [implemented in Phase 3], (c) Recent jobs.\n- Upload form fields: file picker, kind dropdown, topic dropdown (defaulting to currently-selected topic), optional title/identifier/dates/source_url collapse. Submit → POST KnowledgeBase/action/upload → returns job_id → switch to job-status view that polls every 2s.\n- Job status view: shows the job's status badge (queued/processing/done/failed), elapsed time, and on done: link to the source in the management table (Phase 3).\n\nFile size: enforce 50MB limit at the FastAPI route + server-side. PDFs of עשרות-שעות could approach this.",
"testStrategy": "1) Upload a small test PDF (~1MB). Job goes queued → processing → done in <30s. UI shows progress live. 2) Upload a corrupt file. Job ends in failed with a useful error_message. 3) Upload while another large ingest is running — job shows 'processing' but doesn't deadlock. 4) Refresh the page mid-upload — job status restored from sessionStorage + a fresh poll. 5) /admin/kb/jobs returns the upload's row.", "testStrategy": "1) Upload a small test PDF (~1MB). Job goes queued → processing → done in <30s. UI shows progress live. 2) Upload a corrupt file. Job ends in failed with a useful error_message. 3) Upload while another large ingest is running — job shows 'processing' but doesn't deadlock. 4) Refresh the page mid-upload — job status restored from sessionStorage + a fresh poll. 5) /admin/kb/jobs returns the upload's row.",
@@ -158,7 +158,8 @@
"dependencies": [ "dependencies": [
"12" "12"
], ],
"createdAt": "2026-04-25T13:30:00Z" "createdAt": "2026-04-25T13:30:00Z",
"updatedAt": "2026-04-25T16:47:46.599Z"
}, },
{ {
"id": "14", "id": "14",
@@ -214,13 +215,37 @@
"15" "15"
], ],
"createdAt": "2026-04-25T13:30:00Z" "createdAt": "2026-04-25T13:30:00Z"
},
{
"id": "18",
"title": "feat(kb): add 'caselaw' (פסיקה) as a fourth kind",
"description": "Add 'caselaw' alongside law/regulation/circular across the whole stack. Touches: DB CHECK constraints (migration 003), shira-hermes Python (Literal types, _KINDS, kind validation, chunker section detection), EspoCRM Controller/Service validators, template dropdowns, JS kindHe mapping.",
"details": "DB: migration 003_caselaw_kind.sql alters CHECK on kb_source.kind and kb_ingest_job.kind to include 'caselaw'. shira-hermes: update Literal types in ingest.py, admin_kb.py, kb_public.py SearchRequest; add 'caselaw' to s3.py _KINDS; chunker.py — fall back to generic section detector (regex for 'פסק דין', 'תיק' / case number, paragraph numbers). EspoCRM: update validators in Controller postActionUpload + Service search/uploadFile; add 'caselaw' option to kind <select> in index.tpl (search box + upload form); update kindHe in index.js.",
"testStrategy": "",
"status": "in-progress",
"dependencies": [],
"priority": "medium",
"subtasks": [],
"updatedAt": "2026-04-25T16:49:30.816Z"
},
{
"id": "19",
"title": "feat(kb): tag/label sub-topics within a topic (e.g. 'ילד נכה')",
"description": "Group multiple sources within a topic by sub-subject so users can find e.g. all 'ילד נכה' circulars together. Recommendation: many-to-many kb_label + kb_source_label so a source can carry multiple tags reflecting that legal docs commonly touch several subjects.",
"status": "pending",
"priority": "medium",
"details": "Three approaches considered:\n\n1. Free-text 'subject' column on kb_source (~1d): cheapest, typo-prone (ילד נכה / ילדים נכים → separate buckets).\n\n2. Many-to-many labels (~3d, RECOMMENDED): kb_label(id, topic_id, slug UNIQUE per topic, name) + kb_source_label(source_id, label_id, PK both). UI shows label chips on sources; click to filter. Naturally handles cross-cutting subjects (a circular tagged both 'ילד נכה' AND 'אזרח ותיק'). Needs admin UI to manage labels.\n\n3. Hierarchical sub-topics under kb_topic (~5d): clean tree, but a source can only live in one branch — wrong fit for the legal world. Also clashes with planned Phase 4 topic-CRUD UI.\n\nPragmatic path: ship #1 first (subject TEXT field on kb_source + dropdown of seen values + browse-view grouping); upgrade to #2 once we see how labels are used in practice.",
"testStrategy": "Upload 3 circulars all in 'ילד נכה' subject; verify they appear grouped together in browse view; verify search results show the subject chip; verify topic switch resets the subject filter.",
"subtasks": [],
"dependencies": [],
"createdAt": "2026-04-25T16:54:54.478470Z"
} }
], ],
"metadata": { "metadata": {
"version": "1.0.0", "version": "1.0.0",
"lastModified": "2026-04-25T14:15:58.500Z", "lastModified": "2026-04-25T16:49:30.816Z",
"taskCount": 17, "taskCount": 18,
"completedCount": 8, "completedCount": 10,
"tags": [ "tags": [
"master" "master"
] ]
@@ -20,6 +20,9 @@
<li class="{{#ifEqual mode 'browse'}}active{{/ifEqual}}"> <li class="{{#ifEqual mode 'browse'}}active{{/ifEqual}}">
<a href="#" role="button" data-action="switchMode" data-mode="browse">עיון</a> <a href="#" role="button" data-action="switchMode" data-mode="browse">עיון</a>
</li> </li>
<li class="{{#ifEqual mode 'manage'}}active{{/ifEqual}}">
<a href="#" role="button" data-action="switchMode" data-mode="manage">ניהול</a>
</li>
</ul> </ul>
{{#ifEqual mode 'search'}} {{#ifEqual mode 'search'}}
@@ -33,6 +36,7 @@
<option value="law">חוק</option> <option value="law">חוק</option>
<option value="regulation">תקנות</option> <option value="regulation">תקנות</option>
<option value="circular">חוזרים</option> <option value="circular">חוזרים</option>
<option value="caselaw">פסיקה</option>
</select> </select>
<button type="button" class="btn btn-primary" data-action="submit">חפש</button> <button type="button" class="btn btn-primary" data-action="submit">חפש</button>
<button type="button" class="btn btn-default" data-action="clearResults" <button type="button" class="btn btn-default" data-action="clearResults"
@@ -66,5 +70,74 @@
</div> </div>
{{/ifEqual}} {{/ifEqual}}
{{#ifEqual mode 'manage'}}
<div class="kb-manage" style="display:flex;flex-direction:column;gap:16px;">
<div class="panel panel-default" style="padding:12px;">
<h4 style="margin-top:0;">העלאת מסמך חדש</h4>
<form class="kb-upload-form" enctype="multipart/form-data"
style="display:flex;flex-direction:column;gap:10px;">
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label for="kb-upload-file" class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">קובץ:</label>
<input type="file" id="kb-upload-file" name="file"
accept=".pdf,.docx,.txt"
style="flex:1 1 280px;" />
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label for="kb-upload-kind" class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">סוג:</label>
<select id="kb-upload-kind" class="form-control" name="kind"
style="flex:0 0 auto;width:200px;">
<option value="law">חוק</option>
<option value="regulation">תקנות</option>
<option value="circular">חוזר</option>
<option value="caselaw">פסיקה</option>
</select>
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label for="kb-upload-title" class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">כותרת:</label>
<input type="text" id="kb-upload-title" class="form-control" name="title"
placeholder="ברירת מחדל: שם הקובץ"
style="flex:1 1 280px;" />
</div>
<details>
<summary class="text-muted" style="cursor:pointer;">מטא-דאטה נוסף (אופציונלי)</summary>
<div style="display:flex;flex-direction:column;gap:8px;margin-top:8px;padding-right:12px;">
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">מזהה:</label>
<input type="text" class="form-control" name="identifier"
placeholder="למשל: ח'(353) 14.1.2018"
style="flex:1 1 280px;" />
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">פורסם ב:</label>
<input type="date" class="form-control" name="published_at"
style="flex:0 0 auto;width:200px;" />
<label class="text-muted" style="margin:0;font-weight:normal;">תוקף מ:</label>
<input type="date" class="form-control" name="effective_at"
style="flex:0 0 auto;width:200px;" />
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">קישור מקור:</label>
<input type="url" class="form-control" name="source_url"
placeholder="https://…"
style="flex:1 1 280px;" />
</div>
</div>
</details>
<div style="display:flex;gap:8px;align-items:center;">
<button type="button" class="btn btn-primary" data-action="submitUpload">העלה</button>
<span class="kb-upload-hint text-muted small">מקסימום 50MB. PDF / DOCX / TXT.</span>
</div>
</form>
</div>
<div class="panel panel-default" style="padding:12px;">
<h4 style="margin-top:0;">משימות אחרונות</h4>
<div class="kb-jobs-list">
<div class="text-muted">טוען…</div>
</div>
</div>
</div>
{{/ifEqual}}
<div class="kb-results" style="margin-top:12px;"></div> <div class="kb-results" style="margin-top:12px;"></div>
</div> </div>
@@ -12,6 +12,9 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
let _lastSearch = null; // {query, kind, topicId, hits, selectedIdx, completedAt} let _lastSearch = null; // {query, kind, topicId, hits, selectedIdx, completedAt}
let _topics = null; // [{id, slug, name, description, is_active}] let _topics = null; // [{id, slug, name, description, is_active}]
let _topicsPromise = null; // dedupe concurrent fetches across mounts let _topicsPromise = null; // dedupe concurrent fetches across mounts
let _activeJobId = null; // currently-polled ingest job id (across remounts)
let _activeJobTopicId = null; // topic this job belongs to (for cross-topic guard)
let _pollIntervalId = null; // setInterval handle so remounts don't double-poll
const SS_ASK = 'kb-last-ask'; const SS_ASK = 'kb-last-ask';
const SS_SEARCH = 'kb-last-search'; const SS_SEARCH = 'kb-last-search';
@@ -117,6 +120,10 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
if (!Number.isFinite(id) || id === this.topicId) return; if (!Number.isFinite(id) || id === this.topicId) return;
this.switchTopic(id); this.switchTopic(id);
}, },
'click [data-action="submitUpload"]': function (e) {
e.preventDefault();
this.runUpload();
},
}, },
// Cancel any in-flight ask/search, drop the cached last-result for // Cancel any in-flight ask/search, drop the cached last-result for
@@ -190,6 +197,214 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
}); });
}, },
// Phase 2: lazy-loads the recent-jobs list and resumes polling of any
// in-flight upload job. No-op if .kb-jobs-list isn't in the DOM (i.e.
// the user isn't currently on the 'manage' tab).
_ensureManagePanelLoaded: function () {
const self = this;
const $list = this.$el.find('.kb-jobs-list');
if (!$list.length || this.topicId == null) return;
// Resume polling if we navigated away mid-upload and came back.
if (_activeJobId && _activeJobTopicId === this.topicId) {
this._startJobPolling(_activeJobId);
} else if (_activeJobId && _activeJobTopicId !== this.topicId) {
// Cross-topic guard: an upload in another topic is invisible
// here. The polling continues silently in the background.
}
// Always refresh the recent-jobs list on mount.
this._loadJobsList();
},
_loadJobsList: function () {
const self = this;
const $list = this.$el.find('.kb-jobs-list');
if (!$list.length) return;
const topicAtFetch = this.topicId;
Espo.Ajax.getRequest('KnowledgeBase/action/jobs', {
topicId: topicAtFetch,
limit: 20,
}).then(res => {
if (self.topicId !== topicAtFetch) return; // user switched
self._renderJobsList(res && res.items || []);
}).catch(err => {
console.error('KB: failed to load jobs', err);
$list.html('<div class="text-muted">שגיאה בטעינת רשימת המשימות.</div>');
});
},
_renderJobsList: function (items) {
const $list = this.$el.find('.kb-jobs-list');
if (!$list.length) return;
if (!items.length) {
$list.html('<div class="text-muted">אין משימות עדיין.</div>');
return;
}
const statusHe = {
queued: '<span class="label label-default">בתור</span>',
processing: '<span class="label label-info">מעובד…</span>',
done: '<span class="label label-success">הושלם</span>',
failed: '<span class="label label-danger">נכשל</span>',
};
const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזר', caselaw: 'פסיקה'};
const rows = items.map(j => {
const when = j.completed_at || j.started_at || j.created_at || '';
const whenShort = String(when).slice(0, 19).replace('T', ' ');
const meta = j.metadata_json || {};
const title = this.escape(meta.title || j.original_filename || '—');
const fname = this.escape(j.original_filename || '');
const detail = j.status === 'done'
? `<span class="text-muted">${j.chunks_created || 0} קטעים</span>`
: (j.status === 'failed'
? `<span class="text-danger" title="${this.escape(j.error_message || '')}">${this.escape((j.error_message || '').slice(0, 80))}</span>`
: '');
return `<div style="display:flex;justify-content:space-between;align-items:center;border-bottom:1px solid #eee;padding:8px 0;gap:12px;">
<div style="flex:1 1 auto;min-width:0;">
<div style="font-weight:500;">${title}</div>
<div class="text-muted small">
${kindHe[j.kind] || j.kind} · ${this.escape(whenShort)}${j.requested_by_user ? ' · ' + this.escape(j.requested_by_user) : ''}
</div>
${detail ? '<div class="small">' + detail + '</div>' : ''}
</div>
<div>${statusHe[j.status] || this.escape(j.status)}</div>
</div>`;
}).join('');
$list.html(rows);
},
// Live status panel rendered above the upload form during processing.
_renderJobStatus: function (job) {
const $form = this.$el.find('.kb-upload-form');
if (!$form.length) return;
// Remove any previous banner so we can replace it.
this.$el.find('.kb-upload-status').remove();
const statusHe = {
queued: 'בתור',
processing: 'מעובד…',
done: 'הושלם בהצלחה',
failed: 'נכשל',
};
const klass = job.status === 'done' ? 'alert-success'
: job.status === 'failed' ? 'alert-danger'
: 'alert-info';
const filename = this.escape(job.original_filename || '');
const extra = job.status === 'done'
? ` · נוצרו ${job.chunks_created || 0} קטעים`
: (job.status === 'failed'
? ` · ${this.escape((job.error_message || '').slice(0, 200))}`
: '');
const html = `<div class="kb-upload-status alert ${klass}" style="margin-bottom:12px;">
<strong>${this.escape(filename)}</strong> — ${statusHe[job.status] || this.escape(job.status)}${extra}
</div>`;
$form.before(html);
},
_startJobPolling: function (jobId) {
const self = this;
this._stopJobPolling();
_activeJobId = jobId;
_activeJobTopicId = this.topicId;
const tick = () => {
Espo.Ajax.getRequest('KnowledgeBase/action/job', {id: jobId})
.then(job => {
if (!self.$el || !self.$el.length) return;
self._renderJobStatus(job);
if (job.status === 'done' || job.status === 'failed') {
self._stopJobPolling();
_activeJobId = null;
_activeJobTopicId = null;
// Refresh the recent-jobs list to surface the new row.
self._loadJobsList();
}
})
.catch(err => {
console.error('KB: job poll failed', err);
// Don't kill the interval on a single transient error;
// let the next tick try again.
});
};
tick(); // immediate first read so the user sees status fast
_pollIntervalId = setInterval(tick, 2000);
},
_stopJobPolling: function () {
if (_pollIntervalId) {
clearInterval(_pollIntervalId);
_pollIntervalId = null;
}
},
runUpload: function () {
const $form = this.$el.find('.kb-upload-form');
const $file = $form.find('input[type="file"]');
const file = ($file[0] && $file[0].files && $file[0].files[0]) || null;
if (!file) {
alert('יש לבחור קובץ.');
return;
}
if (file.size > 50 * 1024 * 1024) {
alert('הקובץ גדול מ-50MB. אנא העלה קובץ קטן יותר.');
return;
}
if (this.topicId == null) {
alert('נושא לא נבחר.');
return;
}
const fd = new FormData();
fd.append('file', file, file.name);
fd.append('kind', $form.find('select[name="kind"]').val() || 'circular');
fd.append('topicId', String(this.topicId));
['title', 'identifier', 'published_at', 'effective_at', 'source_url'].forEach(name => {
const v = ($form.find(`[name="${name}"]`).val() || '').trim();
if (v) fd.append(name, v);
});
const $btn = this.$el.find('[data-action="submitUpload"]');
$btn.prop('disabled', true).text('מעלה…');
const self = this;
// Bypass Espo.Ajax.postRequest (which JSON-encodes the body) — use
// raw $.ajax so the FormData multipart body is sent intact.
$.ajax({
url: 'api/v1/KnowledgeBase/action/upload',
method: 'POST',
data: fd,
contentType: false,
processData: false,
cache: false,
}).then(res => {
$btn.prop('disabled', false).text('העלה');
// Reset the form so the user can upload another.
$form[0].reset();
if (res && res.job_id) {
self._renderJobStatus({
original_filename: file.name,
status: res.status || 'queued',
});
self._startJobPolling(res.job_id);
}
}).fail(xhr => {
$btn.prop('disabled', false).text('העלה');
let msg = 'שגיאה לא ידועה';
try {
const json = JSON.parse(xhr.responseText || '{}');
msg = json.message || json.detail || xhr.responseText || msg;
} catch (e) {
msg = xhr.responseText || msg;
}
self.$el.find('.kb-upload-status').remove();
self.$el.find('.kb-upload-form').before(
'<div class="kb-upload-status alert alert-danger" style="margin-bottom:12px;">'
+ 'העלאה נכשלה: ' + self.escape(msg.slice(0, 300))
+ '</div>'
);
});
},
_ensureTopicsLoaded: function () { _ensureTopicsLoaded: function () {
if (_topics) return Promise.resolve(_topics); if (_topics) return Promise.resolve(_topics);
if (_topicsPromise) return _topicsPromise; if (_topicsPromise) return _topicsPromise;
@@ -253,6 +468,13 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
_clear(sessionStorage, SS_SEARCH); _clear(sessionStorage, SS_SEARCH);
this._sources = null; this._sources = null;
this._sourcesPromise = null; this._sourcesPromise = null;
// An in-flight upload belongs to the topic where it started;
// stop polling it here so the new topic doesn't see a stale banner.
// The job continues processing on the server; the user can find
// it under "משימות אחרונות" if they switch back.
this._stopJobPolling();
_activeJobId = null;
_activeJobTopicId = null;
this.stopAskProgress(); this.stopAskProgress();
this.setLoading(false); this.setLoading(false);
this.$el.find('.kb-results').empty(); this.$el.find('.kb-results').empty();
@@ -272,6 +494,7 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
if (!self.$el || !self.$el.length) return; if (!self.$el || !self.$el.length) return;
self._populateTopicPicker(); self._populateTopicPicker();
self._ensureSourcesLoaded(); self._ensureSourcesLoaded();
self._ensureManagePanelLoaded();
}; };
if (_topics) { if (_topics) {
afterTopics(); afterTopics();
@@ -590,7 +813,7 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
return; return;
} }
const initialIdx = (typeof selectedIdx === 'number' && hits[selectedIdx]) ? selectedIdx : 0; const initialIdx = (typeof selectedIdx === 'number' && hits[selectedIdx]) ? selectedIdx : 0;
const kindHe = {law: 'חוק', regulation: 'תקנה', circular: 'חוזר'}; const kindHe = {law: 'חוק', regulation: 'תקנה', circular: 'חוזר', caselaw: 'פסיקה'};
const self = this; const self = this;
const isPdfBacked = h => h && (h.kind === 'regulation' || h.kind === 'circular'); const isPdfBacked = h => h && (h.kind === 'regulation' || h.kind === 'circular');
@@ -889,7 +1112,7 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
answerHtml = '<div style="white-space:pre-wrap;">' + this.escape(text || '') + '</div>'; answerHtml = '<div style="white-space:pre-wrap;">' + this.escape(text || '') + '</div>';
} }
const kindHe = {law: 'חוק', regulation: 'תקנה', circular: 'חוזר'}; const kindHe = {law: 'חוק', regulation: 'תקנה', circular: 'חוזר', caselaw: 'פסיקה'};
// Dedup sources by source_id (first occurrence = most relevant), // Dedup sources by source_id (first occurrence = most relevant),
// but remember every page that came up per source so users can // but remember every page that came up per source so users can
@@ -1029,8 +1252,8 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
$list.html('<div class="text-muted">אין מקורות.</div>'); $list.html('<div class="text-muted">אין מקורות.</div>');
return; return;
} }
const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזרים'}; const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזרים', caselaw: 'פסיקה'};
const grouped = {law: [], regulation: [], circular: []}; const grouped = {law: [], regulation: [], circular: [], caselaw: []};
this._sources.forEach(s => { this._sources.forEach(s => {
if (grouped[s.kind]) grouped[s.kind].push(s); if (grouped[s.kind]) grouped[s.kind].push(s);
}); });
@@ -121,4 +121,74 @@ class KnowledgeBase
$this->coerceTopicId($data->topicId ?? $data->topic_id ?? null) $this->coerceTopicId($data->topicId ?? $data->topic_id ?? null)
); );
} }
/**
* Multipart upload to shira-hermes /admin/kb/upload. We rely on PHP's
* automatic multipart parsing via $_FILES because the EspoCRM Request
* interface doesn't expose uploaded files directly. The user's
* EspoCRM identity is forwarded as `X-User-Name` so the upstream job
* row records who uploaded what.
*/
public function postActionUpload(Request $request, Response $response): array
{
$this->checkAccess();
$files = $_FILES['file'] ?? null;
if (!$files || ($files['error'] ?? UPLOAD_ERR_NO_FILE) !== UPLOAD_ERR_OK) {
throw new BadRequest('No file uploaded (form field must be `file`).');
}
$tmpPath = $files['tmp_name'] ?? '';
if (!$tmpPath || !is_uploaded_file($tmpPath)) {
throw new BadRequest('Invalid upload — tmp_name missing.');
}
// For multipart/form-data EspoCRM's getParsedBody() returns an empty
// stdClass; the form fields land in $_POST as PHP parses them.
$kind = $_POST['kind'] ?? null;
if (!in_array($kind, ['law', 'regulation', 'circular', 'caselaw'], true)) {
throw new BadRequest('kind must be one of: law, regulation, circular, caselaw.');
}
$topicId = $this->coerceTopicId($_POST['topicId'] ?? $_POST['topic_id'] ?? null);
if ($topicId === null) {
throw new BadRequest('topicId is required.');
}
$metadata = [
'title' => isset($_POST['title']) ? trim((string) $_POST['title']) : '',
'identifier' => isset($_POST['identifier']) ? trim((string) $_POST['identifier']) : '',
'published_at' => $_POST['published_at'] ?? $_POST['publishedAt'] ?? '',
'effective_at' => $_POST['effective_at'] ?? $_POST['effectiveAt'] ?? '',
'source_url' => $_POST['source_url'] ?? $_POST['sourceUrl'] ?? '',
];
return $this->getService()->uploadFile(
$tmpPath,
(string) ($files['name'] ?? 'upload'),
(string) $kind,
$topicId,
$metadata,
(string) $this->user->get('userName')
);
}
public function getActionJobs(Request $request, Response $response): array
{
$this->checkAccess();
$topicId = $this->coerceTopicId($request->getQueryParam('topicId'));
$status = $request->getQueryParam('status');
$limit = $request->getQueryParam('limit');
$limitInt = ($limit !== null && is_numeric($limit)) ? (int) $limit : 50;
return $this->getService()->listJobs($topicId, $status, $limitInt);
}
public function getActionJob(Request $request, Response $response): array
{
$this->checkAccess();
$jobId = $request->getQueryParam('id');
if (!$jobId || !is_numeric($jobId)) {
throw new BadRequest('id (numeric) is required.');
}
return $this->getService()->getJob((int) $jobId);
}
} }
@@ -38,5 +38,29 @@
"controller": "KnowledgeBase", "controller": "KnowledgeBase",
"action": "ask" "action": "ask"
} }
},
{
"route": "/KnowledgeBase/action/upload",
"method": "post",
"params": {
"controller": "KnowledgeBase",
"action": "upload"
}
},
{
"route": "/KnowledgeBase/action/jobs",
"method": "get",
"params": {
"controller": "KnowledgeBase",
"action": "jobs"
}
},
{
"route": "/KnowledgeBase/action/job",
"method": "get",
"params": {
"controller": "KnowledgeBase",
"action": "job"
}
} }
] ]
@@ -37,7 +37,7 @@ class KnowledgeBaseService
public function search(string $query, string $kind, int $topK, ?int $topicId = null): array public function search(string $query, string $kind, int $topK, ?int $topicId = null): array
{ {
$topK = max(1, min(15, $topK)); $topK = max(1, min(15, $topK));
if (!in_array($kind, ['any', 'law', 'regulation', 'circular'], true)) { if (!in_array($kind, ['any', 'law', 'regulation', 'circular', 'caselaw'], true)) {
$kind = 'any'; $kind = 'any';
} }
@@ -70,6 +70,136 @@ class KnowledgeBaseService
return $this->get($path); return $this->get($path);
} }
/**
* Phase 2: async upload to /admin/kb/upload. Forwards the temp-uploaded
* file via cURL multipart and the EspoCRM username as `X-User-Name`
* so shira-hermes can record who uploaded what.
*
* @return array{job_id:int, status:string}
*/
public function uploadFile(
string $tmpPath,
string $filename,
string $kind,
int $topicId,
array $metadata,
string $username
): array {
if (!is_readable($tmpPath)) {
throw new Error('Uploaded file is not readable on disk.');
}
if (!in_array($kind, ['law', 'regulation', 'circular', 'caselaw'], true)) {
throw new BadRequest('kind must be one of: law, regulation, circular, caselaw.');
}
$url = $this->getBaseUrl() . '/admin/kb/upload';
$apiKey = $this->getApiKey();
if (!$apiKey) {
throw new Error('SmartAssistant API key is not configured.');
}
// Build the multipart form. CURLOPT_POSTFIELDS as an array makes
// cURL set the multipart Content-Type with a generated boundary.
$fields = [
'kind' => $kind,
'topic_id' => (string) $topicId,
'file' => new \CURLFile($tmpPath, $this->guessMime($filename), $filename),
];
foreach (['title', 'identifier', 'published_at', 'effective_at', 'source_url'] as $k) {
$val = $metadata[$k] ?? '';
if ($val !== '' && $val !== null) {
$fields[$k] = (string) $val;
}
}
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $fields,
CURLOPT_HTTPHEADER => [
'Accept: application/json',
'X-Api-Key: ' . $apiKey,
'X-User-Name: ' . $username,
],
CURLOPT_RETURNTRANSFER => true,
CURLOPT_TIMEOUT => 120,
CURLOPT_CONNECTTIMEOUT => 10,
]);
$body = curl_exec($ch);
$httpCode = (int) curl_getinfo($ch, CURLINFO_HTTP_CODE);
$err = curl_error($ch);
curl_close($ch);
if ($err) {
$this->log->error("KnowledgeBase: upload transport error: {$err}");
throw new Error("Failed to reach Knowledge Base: {$err}");
}
if ($httpCode === 413) {
throw new BadRequest('File too large (max 50MB).');
}
if ($httpCode >= 400 && $httpCode < 500) {
$detail = $this->decodeDetail($body) ?: 'Bad request';
throw new BadRequest($detail);
}
if ($httpCode < 200 || $httpCode >= 300) {
$this->log->error("KnowledgeBase: upload HTTP {$httpCode}: {$body}");
throw new Error("Knowledge Base returned HTTP {$httpCode}");
}
$decoded = json_decode((string) $body, true);
if (!is_array($decoded) || !isset($decoded['job_id'])) {
throw new Error('Invalid response from Knowledge Base.');
}
return $decoded;
}
public function listJobs(?int $topicId, ?string $status, int $limit): array
{
$limit = max(1, min(200, $limit));
$qs = ['limit' => $limit];
if ($topicId !== null) {
$qs['topic_id'] = $topicId;
}
if ($status !== null && $status !== '') {
$qs['status'] = $status;
}
$path = '/admin/kb/jobs?' . http_build_query($qs);
return $this->get($path);
}
public function getJob(int $jobId): array
{
return $this->get('/admin/kb/jobs/' . $jobId);
}
private function guessMime(string $filename): string
{
$lower = strtolower($filename);
if (str_ends_with($lower, '.pdf')) {
return 'application/pdf';
}
if (str_ends_with($lower, '.docx')) {
return 'application/vnd.openxmlformats-officedocument.wordprocessingml.document';
}
if (str_ends_with($lower, '.txt')) {
return 'text/plain';
}
return 'application/octet-stream';
}
private function decodeDetail($body): ?string
{
if (!is_string($body)) {
return null;
}
$decoded = json_decode($body, true);
if (is_array($decoded) && isset($decoded['detail'])) {
return is_string($decoded['detail']) ? $decoded['detail'] : json_encode($decoded['detail']);
}
return null;
}
public function ask( public function ask(
string $message, string $message,
?string $conversationId, ?string $conversationId,
+1 -1
View File
@@ -1,7 +1,7 @@
{ {
"name": "KnowledgeBase", "name": "KnowledgeBase",
"module": "KnowledgeBase", "module": "KnowledgeBase",
"version": "0.3.0", "version": "0.5.0",
"acceptableVersions": [ "acceptableVersions": [
">=8.0.0" ">=8.0.0"
], ],