chore(tm): file 8 new tasks — 3 carry-over + 5-phase multi-topic plan
Captures the open-but-not-blocking items from the v0.2.1 sessions plus the comprehensive plan for generalizing the KB beyond ביטוח לאומי. #9-11: carry-over polish noted at the v0.2.1 close - #9 SSE rejoin (in-flight stream survives F5 / tab close) - #10 n8n release-asset auto-attach has been failing for every release from v0.1.8 through v0.2.1; manual curl uploads as workaround - #11 ask mode source picker hides text-source citations (Wikisource law) because /kb/ask filters sources_used to PDFs only #12-16: multi-topic KB plan (next session) - #12 Phase 1 — DB schema with kb_topic + kb_source.topic_id, /kb/topics endpoint, topic-aware system prompts, topic dropdown in UI; backwards-compat by seeding topic_id=1 ביטוח לאומי and migrating the 6 existing sources to it - #13 Phase 2 — file upload from the management panel, async ingestion via kb_ingest_job table, live job-status polling - #14 Phase 3 — sources management UI: edit metadata, delete (with cascade messaging), re-ingest, jobs/failures view - #15 Phase 4 — topic CRUD UI for admins (add דיני עבודה, דין פלילי, etc., edit prompts, soft-disable) - #16 Phase 5 (optional) — per-topic ACL by EspoCRM role with view vs manage permissions Phases 1-3 are the must-ship subset for what the user asked. Phase 4 is admin convenience. Phase 5 is scoped only if firms ask for it.
This commit is contained in:
@@ -97,11 +97,108 @@
|
||||
"dependencies": [],
|
||||
"createdAt": "2026-04-25T10:00:00Z"
|
||||
}
|
||||
,
|
||||
{
|
||||
"id": 9,
|
||||
"title": "feat(kb/ask): rejoin in-flight SSE stream after browser reload",
|
||||
"description": "When the user F5s or closes+reopens the tab while Shira is thinking, the asyncio task in shira-hermes keeps running (tokens burn) but the EventSource is gone — there's no way to reattach. After reload, the v0.1.9 sessionStorage replay only shows the LAST completed answer, not the live in-flight stream. Build a rejoinable SSE channel: POST /kb/ask/stream allocates a request_id and starts the runner; the SSE response writes events into an in-memory store keyed by request_id AND streams them. New GET /kb/ask/stream/{request_id} reconnects: replays accumulated events first, then continues with new ones as they arrive. Garbage-collect after 5 minutes past the answer event.",
|
||||
"status": "pending",
|
||||
"priority": "normal",
|
||||
"details": "Architecture: in-process LRU dict {request_id -> {events:list, queue:asyncio.Queue, done:bool, last_used:ts}}. POST returns Set-Cookie or response header X-Request-Id; client stores it in _activeAsk.requestId + sessionStorage. afterRender: if _activeAsk has requestId and no live es, open EventSource to /kb/ask/stream/{requestId}. Server: re-emit all stored events on reconnect, then drain queue. Two-process safety: this works only because shira-hermes runs as a single FastAPI process — if we ever scale horizontally we'll need Redis pub/sub or sticky sessions. Note that for now it is single-process.",
|
||||
"testStrategy": "1) Ask Shira a long question. 2) F5 after 10s. 3) After reload, KB tab should resume showing live events including those that arrived during the reload window. 4) Answer event arrives — refresh again still shows it (sessionStorage path, unchanged from v0.1.9). 5) After 5 minutes, GET /kb/ask/stream/{old_request_id} returns 410 Gone.",
|
||||
"subtasks": [],
|
||||
"dependencies": [],
|
||||
"createdAt": "2026-04-25T13:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"title": "ops: investigate n8n release-asset auto-attach failures",
|
||||
"description": "The n8n workflow `Git: Gitea Push - Auto-Release + Mattermost` (id IcF30v1Pw3wbucbu, per ~/CLAUDE.md) is supposed to attach the built .zip as a release asset within ~12s of a tag push. Across every KnowledgeBase release in this period (v0.1.8 through v0.2.1) the workflow did NOT attach the zip — every release was published with zero assets and I had to upload manually via curl + the Gitea API. Investigate the workflow's recent execution history, find the failing step, and either fix the n8n workflow or document a better fallback.",
|
||||
"status": "pending",
|
||||
"priority": "low",
|
||||
"details": "Test pattern: after `mcp__gitea-dev__create_release`, poll GET /api/v1/repos/.../releases/{id}/assets every 10s for ~60s. Currently always returns 0. The 'Check Existing Release → Release Exists?' branch works (skips creating a duplicate release). The asset-upload branch is what's silently failing. Possible causes: (a) the n8n workflow doesn't have the zip artifact (zip is built ad-hoc with `bash build.sh` from the workspace, not in CI), (b) bad credential to Gitea API, (c) the release-creation event doesn't trigger that node. Note this workflow is for OTHER extensions too — if it's broken for all of them, fixing helps the whole release pipeline.",
|
||||
"testStrategy": "After fix: push v0.X.Y tag to KnowledgeBase via mcp__gitea-dev__create_release; verify the zip appears at GET /releases/{id}/assets within 30s without manual upload.",
|
||||
"subtasks": [],
|
||||
"dependencies": [],
|
||||
"createdAt": "2026-04-25T13:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"title": "feat(kb/ask): show text-source citations in the ask mode source picker",
|
||||
"description": "/kb/ask currently filters sources_used through _filter_pdf_sources (kb_public.py) before returning, dropping any source whose original_path doesn't end with .pdf. The Wikisource law (source 5) is therefore invisible to the user even when Shira explicitly cites a section from it. With the v0.1.11 'section in context' preview that already exists for search mode, ask mode should be able to show the same. Drop the filter, and when the user clicks a non-PDF source pill, route to the same _renderSectionContext flow that search uses (factor it into a shared helper).",
|
||||
"status": "pending",
|
||||
"priority": "normal",
|
||||
"details": "Server (shira-hermes): drop _filter_pdf_sources in both /kb/ask and /kb/ask/stream answer-event payload. Either return all sources, or re-introduce a different filter that keeps text sources but strips ones with no source_id. Client: in renderAskAnswer the source-pill click handler currently always rebuilds the iframe; route through showSearchPreview when the source is text-only. Probably easiest to extract _renderSectionContext / _renderSectionContextPlaceholder into a shared helper that both modes call. Need to add chunk_index to the sources returned from /kb/ask too — currently only on /kb/search hits.",
|
||||
"testStrategy": "Ask 'מהי תקנה 36' — Shira's answer cites both תקנה 37 (PDF) and ס׳ 36 of the law (text). Source picker shows both. Click PDF pill → iframe at the cited page. Click law pill → section in context (matched section + 2 neighbors + Wikisource button) — same look as search mode.",
|
||||
"subtasks": [],
|
||||
"dependencies": [],
|
||||
"createdAt": "2026-04-25T13:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"title": "Phase 1 — Multi-topic KB foundation (DB schema, topic-aware endpoints, topic selector)",
|
||||
"description": "Generalize the KB from being insurance-only to supporting multiple legal domains within one CRM (e.g. ביטוח לאומי, דיני עבודה, דין פלילי). A 'topic' is the firm-level grouping; each kb_source belongs to exactly one topic. Search/ask are scoped to a single topic at a time, chosen from a topic dropdown the user controls. The system_prompt for /kb/ask becomes per-topic so Shira's domain context is correct (today the prompt hardcodes 'Israeli National Insurance'). Backwards-compatible migration: all 6 existing sources move to topic_id=1 named 'ביטוח לאומי' with the existing system-prompt addendum, and the UI defaults to that topic so nothing visible changes for existing users until they choose another topic.",
|
||||
"status": "pending",
|
||||
"priority": "high",
|
||||
"details": "DB migration (shira-hermes side, against insurance_kb DB):\n- New table kb_topic: id PK, slug TEXT UNIQUE, name TEXT NOT NULL, description TEXT, system_prompt_addendum TEXT, is_active BOOLEAN DEFAULT true, created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ DEFAULT now().\n- Seed: INSERT (id=1, slug='national-insurance', name='ביטוח לאומי', system_prompt_addendum=<copy from current /kb/ask hardcoded text>, is_active=true). RENAME the DB itself? Probably leave as 'insurance_kb' — costly to rename, no real value. Document in memory.\n- ALTER TABLE kb_source ADD COLUMN topic_id INTEGER REFERENCES kb_topic(id). Backfill: UPDATE kb_source SET topic_id = 1. Then ALTER COLUMN SET NOT NULL.\n\nshira-hermes endpoints:\n- New GET /kb/topics — public, returns active topics with id, slug, name, description (no prompt).\n- /kb/search: accepts optional topic_id (single int). When present, the SQL adds AND s.topic_id = $N. When absent (back-compat), still returns all (might want to deprecate this and require topic_id later).\n- /kb/sources, /kb/source/{id}/chunks, /kb/source/{id}/pdf: same — optional topic filter on listings, no change to single-source endpoints.\n- /kb/ask + /kb/ask/stream: accept topic_id param; load topic.system_prompt_addendum and inject into system_prompt instead of the hardcoded insurance text in _build_ask_runner_context. Pin the legal_kb tool to topic_id too (so search_insurance_kb queries are constrained — RENAME the tool to search_legal_kb and pass topic_id at registration time).\n- _expand_query in kb_search.py: parametrize the prompt by topic name (currently hardcoded 'הביטוח הלאומי בישראל'). Pass topic.name in.\n\nKnowledgeBase extension (EspoCRM):\n- New route+action GET /KnowledgeBase/action/topics → proxies /kb/topics.\n- Service.search/ask gain topic_id, controller passes through.\n- Template gets a topic <select> at the top (above the tabs); fetches /KnowledgeBase/action/topics on mount, persists selection to localStorage 'kb-topic'. Default: topic id=1 if it exists, else first active topic.\n- this.kind etc. already pass through; add this.topicId. Pass topic_id with every search/ask call.\n\nConsequences for sessionStorage replay (v0.1.9, v0.1.10): the cached _lastAsk / _lastSearch should record topic_id; on remount only replay if the current topic matches. Else show empty state.",
|
||||
"testStrategy": "1) After migration: SELECT count(*) FROM kb_source WHERE topic_id IS NULL → 0; SELECT name FROM kb_topic → 'ביטוח לאומי'. 2) GET /kb/topics returns the seeded topic. 3) Search/ask with topic_id=1 returns the same 6 sources as before. 4) Add a second topic (manually for the test) and a single test source under it; verify topic_id=1 search no longer surfaces the test source and vice versa. 5) /kb/ask system prompt confirmed via logs to include the topic-specific addendum, not the hardcoded text.",
|
||||
"subtasks": [],
|
||||
"dependencies": [],
|
||||
"createdAt": "2026-04-25T13:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"title": "Phase 2 — Document upload from the KB UI (file picker + async ingestion + job tracking)",
|
||||
"description": "Today documents enter the KB by manually putting them in the MinIO inbox/ folder and waiting for the n8n cron to call /admin/kb/scan-inbox. End users have no path to upload via the UI. Add a file-upload form in the management panel: user picks a PDF/DOCX/TXT, picks a kind (law/regulation/circular), confirms the topic, optionally fills metadata (title, identifier, dates, source_url), and clicks upload. The file lands in MinIO under inbox/<topic_slug>/<kind>/, a kb_ingest_job row is created, and a background asyncio task processes it (parse → chunk → embed → upsert kb_source). The UI polls the job status and shows progress live; on done it links to the new source in the management table.",
|
||||
"status": "pending",
|
||||
"priority": "high",
|
||||
"details": "DB:\n- New table kb_ingest_job: id PK, source_id INTEGER NULL FK kb_source(id) ON DELETE SET NULL, original_filename TEXT, s3_key TEXT, kind TEXT, topic_id INTEGER NOT NULL, metadata_json JSONB, status TEXT NOT NULL CHECK (status IN ('queued','processing','done','failed')), error_message TEXT, chunks_created INTEGER, created_at TIMESTAMPTZ DEFAULT now(), started_at TIMESTAMPTZ, completed_at TIMESTAMPTZ, requested_by_user TEXT.\n- Index: (status, created_at DESC) for the queued list.\n\nshira-hermes endpoints:\n- POST /admin/kb/upload (multipart): receives file, kind, topic_id, optional title/identifier/published_at/effective_at/source_url. Validates topic exists. Writes file to s3 inbox/<topic_slug>/<kind>/<uuid-prefixed filename>. Inserts kb_ingest_job with status=queued. Schedules asyncio.create_task(_process_ingest_job(job_id)) so processing starts immediately, not on the next n8n cron tick. Returns {job_id, status:'queued'}.\n- GET /admin/kb/jobs?topic_id=&status=&limit=: list jobs, newest first.\n- GET /admin/kb/jobs/{id}: single job detail.\n- _process_ingest_job(job_id): UPDATE status='processing', started_at=now. Calls existing kb_ingest.ingest_source. On success: UPDATE status='done', source_id=<new>, chunks_created=<n>, completed_at=now. On failure: UPDATE status='failed', error_message=<exception text>. The upload doesn't block the HTTP request — the user gets the job_id immediately and polls.\n- Reuses scan-inbox advisory lock pattern so a manually-uploaded file plus an n8n cron run don't double-process.\n\nKnowledgeBase extension:\n- EspoCRM PHP proxy for multipart upload — Controllers can read $request->getParsedBody() but multipart needs special handling. Look at how other extensions do it (LegalAssistance/DigitalSignature might have examples).\n- New tab 'ניהול' (visible only to non-portal users; can also gate to admin role later). The tab opens a panel with: (a) Upload form, (b) Sources table for current topic [implemented in Phase 3], (c) Recent jobs.\n- Upload form fields: file picker, kind dropdown, topic dropdown (defaulting to currently-selected topic), optional title/identifier/dates/source_url collapse. Submit → POST KnowledgeBase/action/upload → returns job_id → switch to job-status view that polls every 2s.\n- Job status view: shows the job's status badge (queued/processing/done/failed), elapsed time, and on done: link to the source in the management table (Phase 3).\n\nFile size: enforce 50MB limit at the FastAPI route + server-side. PDFs of עשרות-שעות could approach this.",
|
||||
"testStrategy": "1) Upload a small test PDF (~1MB). Job goes queued → processing → done in <30s. UI shows progress live. 2) Upload a corrupt file. Job ends in failed with a useful error_message. 3) Upload while another large ingest is running — job shows 'processing' but doesn't deadlock. 4) Refresh the page mid-upload — job status restored from sessionStorage + a fresh poll. 5) /admin/kb/jobs returns the upload's row.",
|
||||
"subtasks": [],
|
||||
"dependencies": [12],
|
||||
"createdAt": "2026-04-25T13:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"title": "Phase 3 — Management panel UI (sources table, edit metadata, delete, re-ingest)",
|
||||
"description": "Today the only way to manage existing sources is via SQL on the shared PG (DELETE FROM kb_source WHERE id=…) plus an mc command on MinIO. End users need a panel where they can: see all sources for the selected topic, with chunk counts and ingestion dates; edit source metadata (title, identifier, dates, source_url); delete a source (with confirm and clear cascade messaging); re-ingest a source (re-fetch its file from MinIO processed/, drop chunks, re-run parse+chunk+embed); see the most recent ingest jobs (success + failure both). The panel sits in the same KB extension under a new tab 'ניהול'.",
|
||||
"status": "pending",
|
||||
"priority": "high",
|
||||
"details": "shira-hermes endpoints:\n- GET /admin/kb/sources?topic_id=&kind=&limit=: list with id, kind, title, identifier, chunk_count, ingestion_date, last_ingest_status (joining kb_ingest_job).\n- PUT /admin/kb/sources/{id}: update editable fields (title, identifier, source_url, published_at, effective_at).\n- DELETE /admin/kb/sources/{id}: hard delete (cascades chunks via FK, drops processed/ files in S3 too).\n- POST /admin/kb/sources/{id}/reingest: requires source_id; finds the original file at original_path s3:// URI; creates a new kb_ingest_job row with status=queued and the existing source_id; background task replaces chunks+embeddings (does NOT change source_id, so all v0.1.11 chunk_index references remain valid IF the chunker output is stable; otherwise old _lastSearch sessionStorage will point at chunk_indexes that no longer exist — flag for testing). Should optionally just delete + re-create the source — simpler.\n\nEspoCRM client:\n- 'ניהול' tab. Top: header showing topic name + a count badge ('X מסמכים בנושא'). Below: 3 collapsed sections — uploaded jobs (collapsed if all done), sources table, recent failures.\n- Sources table: id, kind label, title, identifier, chunk_count, ingestion_date. Rightmost column: action buttons (✏ edit, 🗑 delete, ↻ re-ingest, 👁 view in browse mode).\n- Edit: opens a modal with the editable fields. PUT on save.\n- Delete: confirm modal that shows chunk_count + 'this will also remove the PDF from the search results'. Two-step confirmation if source has >100 chunks.\n- Re-ingest: confirms; submits POST /reingest; surfaces in the jobs section.\n- Use jQuery + Espo.Ui.dialog patterns consistent with the rest of the extension.\n\nACL gate: this tab visible only to users with role 'KB Admin' (define in EspoCRM if not present) or fall back to !isPortal for now.",
|
||||
"testStrategy": "1) Open 'ניהול' tab. See the 6 existing sources for ביטוח לאומי. 2) Edit one source's title — refresh → new title sticks. 3) Re-ingest ספר הליקויים. Job appears in the jobs section processing → done. Source still searchable end-to-end after. 4) Delete one source. Confirm modal appears with chunk count. After delete: source gone from /kb/sources, /kb/search no longer surfaces it. 5) Failed re-ingest (e.g. delete the file in MinIO first) — surfaces in failures section with error_message.",
|
||||
"subtasks": [],
|
||||
"dependencies": [13],
|
||||
"createdAt": "2026-04-25T13:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"title": "Phase 4 — Topic CRUD UI (admins can add/edit/disable topics from the panel)",
|
||||
"description": "Once Phase 1 ships there will be a single seeded topic ('ביטוח לאומי'). For Klear and other firms to actually use the multi-topic capability they need a UI to create new topics — דיני עבודה, דין פלילי, נדל\"ן — without anyone running SQL. Add a topics-management section in the 'ניהול' tab: list active+inactive topics, add new (slug + name + system_prompt_addendum), edit (rename, change prompt addendum), soft-delete (sets is_active=false). Soft-delete keeps the data but hides the topic from the user-facing dropdown.",
|
||||
"status": "pending",
|
||||
"priority": "normal",
|
||||
"details": "shira-hermes endpoints:\n- GET /admin/kb/topics: list including inactive ones (with source counts).\n- POST /admin/kb/topics: create. Validates slug (lowercase, hyphens only, unique). Returns created row.\n- PUT /admin/kb/topics/{id}: update name, description, system_prompt_addendum, is_active.\n- DELETE /admin/kb/topics/{id}: 409 Conflict if topic has sources; otherwise hard-delete. Soft-delete (is_active=false) is the normal path.\n\nEspoCRM client:\n- New section in 'ניהול' tab: 'נושאים' table — slug, name, source count, is_active toggle, edit button.\n- Add new: modal with slug, name, description, prompt-addendum textarea (pre-populated with a generic template like 'You are answering legal questions about <topic>. Cite sections explicitly when possible. Never answer from generic legal training if the KB has a matching section').\n- Edit: same modal pre-filled. Saving updates the topic; if name changed, the topic dropdown refreshes.\n- The 'ניהול' tab itself only shows the topics section if user is admin. Regular users see only sources management for the topics they have access to.\n\nDefault topic on first install of multi-topic version: still id=1 'ביטוח לאומי'. New installs get the same seed.",
|
||||
"testStrategy": "1) Create topic 'דיני עבודה' (slug 'employment-law'). 2) Verify it appears in GET /kb/topics within seconds. 3) The user-facing topic dropdown lists it. 4) Upload a PDF under it (Phase 2 flow), verify search/ask scoped to it works. 5) Edit prompt addendum — /kb/ask answers in context of employment law. 6) Soft-delete the test topic — disappears from user dropdown but kept in admin list.",
|
||||
"subtasks": [],
|
||||
"dependencies": [12],
|
||||
"createdAt": "2026-04-25T13:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"title": "Phase 5 (optional) — Per-topic ACL by EspoCRM role",
|
||||
"description": "Not every user in a multi-topic firm should see every topic. Example: only the criminal-law department needs the criminal KB; the family-law team shouldn't. Wire EspoCRM's existing role/team model to the KB topics: each topic can be restricted to specific roles, with two permission levels — 'view' (use search/ask) and 'manage' (upload/edit/delete). Without role mapping, all non-portal users see all topics (current behavior). With mapping, the topic dropdown filters to topics the user has at least 'view' on, and the management UI gates 'manage' actions to users with the manage permission for that topic.",
|
||||
"status": "pending",
|
||||
"priority": "low",
|
||||
"details": "DB: new table kb_topic_acl: topic_id FK, role_name TEXT, permission TEXT CHECK in ('view', 'manage'), PRIMARY KEY (topic_id, role_name, permission).\n\nshira-hermes endpoints:\n- /kb/topics: filter by user's roles. The user's role list comes from the EspoCRM proxy (sent in a header or JWT claim — needs design).\n- /admin/kb/* actions: enforce 'manage' permission per source's topic_id.\n\nEspoCRM client:\n- Topic dropdown: only shows topics the user has 'view' on.\n- Management tab: only visible if user has 'manage' on at least one topic.\n- Edit/delete buttons in the sources table: visible only when user has 'manage' on that source's topic.\n\nThis is the most-likely-to-skip phase if the user's firm is small and everyone needs everything. Worth scoping out depth before committing.",
|
||||
"testStrategy": "Set up two test users: lawyer_a with role 'criminal-law', lawyer_b with role 'employment-law'. Topics: 'דין פלילי' restricted to 'criminal-law' role only, 'דיני עבודה' restricted to 'employment-law'. lawyer_a sees only the criminal topic; lawyer_b only employment. Cross-user attempts return 403.",
|
||||
"subtasks": [],
|
||||
"dependencies": [12, 15],
|
||||
"createdAt": "2026-04-25T13:30:00Z"
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"created": "2026-04-24T15:47:00Z",
|
||||
"updated": "2026-04-25T10:30:00Z",
|
||||
"description": "KnowledgeBase extension tasks"
|
||||
"updated": "2026-04-25T13:30:00Z",
|
||||
"description": "KnowledgeBase extension tasks. Tasks 1-8 = built and shipped (v0.1.7 → v0.2.1). Tasks 9-11 = carry-over polish from the v0.2.1 sessions. Tasks 12-16 = comprehensive plan for generalizing the single-domain (ביטוח לאומי) KB into a multi-domain KB with self-service document upload + management UI. Phases 1-3 are the must-ship subset; Phase 4 is admin nice-to-have; Phase 5 is optional ACL scoping."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user