Generalize the Knowledge Base from insurance-only to multi-domain.
Introduces kb_topic table (shira-hermes), topic-aware endpoints
(/kb/topics, /kb/search, /kb/ask, /kb/ask/stream all accept topic_id),
a topic picker <select> in the EspoCRM UI (localStorage-persisted),
and renames the LLM tool search_insurance_kb → search_legal_kb.
All existing sources migrate to topic_id=1 (ביטוח לאומי); the UI
defaults to that topic so no visible regression for current users.
Refs Task Master #12
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three issues from the v0.2.0 user test:
1. Sides flipped relative to expectation
In our Hebrew/RTL CRM the user reads primary content (the answer or
the hit list) on the visual right and the supporting reference
(the PDF) on the visual left — that's how Hebrew layouts stage
"main + sidebar". v0.1.10's _buildSplitShell forced direction:ltr on
the splitter container (correct, for mouseX math) but laid the
children in the order [primary, handle, reference], which under LTR
put primary on the LEFT and PDF on the RIGHT. Fixed by renaming the
helper params to {primaryHtml, referenceHtml} and ordering the
children as [reference, handle, primary] — visual LEFT is now the
PDF, visual RIGHT is the answer/list. Default split moved from 42%
to 58% so the reference pane (now on the left) keeps the same
absolute width the PDF column had under v0.1.9's bootstrap rows.
2. No way to start a new query
afterRender replays _lastSearch / _lastAsk on every remount, which
means the prior result hangs around forever. Added a "חיפוש חדש" /
"שאלה חדשה" button next to the submit button. clearResults() cancels
any in-flight request (closes the EventSource for ask, drops
_activeSearch reference for search), removes the cached entry from
sessionStorage, empties the results pane, and refocuses the input.
3. Stale results from another session
Cached results now auto-expire after 30 minutes — _loadJson checks
completedAt against STALE_AFTER_MS and silently drops anything
older. So coming back tomorrow the KB tab opens clean instead of
showing yesterday's answer to whatever you asked then.
Refs Task Master #4, #6
User wanted to see what Shira is doing while she thinks — like Claude
does — instead of staring at an incrementing seconds counter. Now the
ask UI shows one line per agent step as it happens:
3s 🧠 קוראת את השאלה
4s 🔍 מחפשת בבסיס הידע: "תקנה 36"
9s ✓ נמצאו 5 קטעים רלוונטיים
10s 🔍 מחפשת בבסיס הידע: "תקנה 36 ביטוח לאומי"
14s ✓ נמצאו 5 קטעים רלוונטיים
15s 🧠 ממשיכה לחקור
22s ✍️ מנסחת תשובה
→ תשובה
Client wiring:
- runAsk now opens an EventSource at ?entryPoint=KnowledgeBaseAskStream
instead of POSTing JSON. Each SSE event of type thinking/tool_start/
tool_done/tool_error/writing appends a row to a stacked progress log.
The final {type:answer,text,sources} event triggers renderAskAnswer
with the existing split view. {type:error} surfaces via showError.
- Module-level state from v0.1.9 unchanged in spirit but now stores the
EventSource + accumulated events. afterRender on view re-mount replays
the entire event list and re-binds onmessage/onerror — the SSE
connection itself doesn't drop because it lives at module scope.
- Elapsed-time pill stays for cadence but is decoration; the real
signal is the per-event log.
EspoCRM proxy:
- New EntryPoint KnowledgeBaseAskStream — required because EventSource
is GET-only and ?message= must come via the URL. PHP buffering is
disabled (zlib + ob + apache_setenv no-gzip), Apache headers are set
for SSE (no-cache, X-Accel-Buffering: no), and we cURL into
shira-hermes /kb/ask/stream with CURLOPT_WRITEFUNCTION echoing each
byte and flushing immediately. The handler ends with exit; to bypass
EspoCRM's Response wrapper which would otherwise emit headers/body
on top of our raw stream.
- KnowledgeBaseService gained two public accessors (getStreamingUrl,
getStreamingApiKey) so the EntryPoint can build the cURL without
going through request() (which buffers the whole body).
- entryPoints.json registers the new class.
Server (shira-hermes 76fd77f):
- AgentRunner.run accepts an on_event sync callback fired before each
LLM call (thinking) and around each tool call (tool_start, tool_done,
tool_error). Hebrew labels in agent_runner._tool_label / _tool_done_label
ship over the wire ready-to-display — keeps the client free of
translation concerns.
- POST /kb/ask/stream wraps the existing agent runner in an asyncio
task feeding an asyncio.Queue, returns StreamingResponse with proper
SSE headers (X-Accel-Buffering, Cache-Control). 15s comment-line
heartbeats keep the connection open through proxy idle-timeouts.
- Original POST /kb/ask still works — additive change.
Refs Task Master #8
User reported the search card "looks broken" when the selected hit is the
law (Wikisource source 5) — the right preview pane just duplicates the
chunk text already visible in the left list card. For PDF sources the
right pane shows the PDF page, but for text-only sources it had nothing
useful to add and fell back to a copy.
Fix:
- /kb/search hits gained chunk_index (shira-hermes b127caf), and
/kb/source/{id}/chunks gained an `around=<chunk_index>&ctx=<k>` window.
- Espo proxy plumbing: getActionChunks reads `around`/`ctx` query
params and forwards them; KnowledgeBaseService::sourceChunks accepts
optional $around + $ctx and appends them to the upstream URL.
- showSearchPreview for non-PDF hits now fetches the matched chunk plus
2 sections before and 2 after, renders them as stacked panels: the
matched section gets a yellow #fff3cd highlight, neighbors render as
muted context blocks. The panel header carries a prominent
"פתח ב-Wikisource" button so users can jump to the public source page.
After render the matched section auto-scrolls into view.
- Race protection via _previewToken: a click on a different hit while a
fetch is in flight cancels the stale render so the user always sees
the section that matches the currently-selected card.
PDF path unchanged.
Refs Task Master #7
Two requests on top of v0.1.9:
1. Drag-to-resize splitter
The previous fixed Bootstrap col-md-5/col-md-7 split locked the
user into 41.66% answer / 58.33% PDF. Replaced with a flex-row
layout containing a 6px drag handle. mousedown spawns a fixed
transparent overlay so the PDF.js iframe doesn't swallow mouse
events while dragging; mouseup persists the chosen ratio (clamped
15-85%) to localStorage as kb-split-pct, so the next page-load and
the next session both come back to the same layout. Both the
ask split view and the search split view share _buildSplitShell +
_wireSplitter — single source of truth for the geometry.
2. Search survives a view switch
Search had the same view-bound promise problem ask had before
v0.1.9. Hoisted the in-flight request to a module-level
_activeSearch and the most recent {query, kind, hits, selectedIdx}
to _lastSearch + sessionStorage. afterRender restores the input,
the kind filter, the hit list (with the same hit highlighted), and
the right-pane preview. Click handler updates selectedIdx so a
round-trip lands on the user's last-clicked hit, not always on the
top-ranked one.
The persistence helpers grew a tiny refactor: _loadJson/_saveJson/_clear
replace the ask-only SS_KEY shim and now back both ask + search keys.
Refs Task Master #5, #6
Two recurring frustrations from the v0.1.8 demo:
1. Searching "מהי תקנה 37" only returned the תקנה 37 חוזר and the law —
ספר הליקויים, the canonical medical reference, was crowded out by the
document whose title literally contained the query terms. Fixed in
shira-hermes 1441a41 by adding an LLM-driven query-expansion path to
/kb/search (default expand=true). Same query now also returns 2 hits
from src 29 (ספר הליקויים).
2. Asking Shira a question and clicking another EspoCRM view dropped the
in-flight request — coming back rendered a blank kb-results pane, even
though the request had completed in the background. Fixed here:
{question, promise, startedAt} are now stored at module scope (closure
shared across remounts of this view). afterRender re-binds handlers
for any active ask, resumes the progress panel with the original start
time so the elapsed counter doesn't reset, and replays the most recent
completed answer from sessionStorage so the user sees their result
even after a hard reload. State is intentionally session-scoped — no
answers persist across browser sessions.
Refs Task Master #3, #4
Search and ask modes now render results as a two-column layout: a left
column with ranked hit cards (search) or Shira's answer + source picker
(ask), and a right column with an iframe showing the source PDF scrolled
to page_number. Clicking a hit (search) or source pill (ask) swaps the
iframe's src, so users can verify a quote against the original PDF
without leaving the KB tab.
- search: renderSearchResults lays out results as panel cards on the
left (with kind + title + section + "עמ׳ N" label); the top hit is
pre-selected and its PDF loads on the right. Clicking any card
re-highlights it and swaps the preview. Law-kind hits (Wikisource
text) gracefully fall back to a chunk-text panel with a Wikisource
link so the right pane never 404s on a text source.
- ask: renderAskAnswer dedups /kb/ask's sources[] by source_id,
collects all cited pages per source, and renders a picker row plus
per-source page-jump buttons. First source's first page loads on
initial render; buttons swap the iframe without re-running the query.
Depends on shira-hermes commits e534709 + 1ca1cc0 (chunker page_number
propagation + null-byte strip) — without them, every PDF collapses to
page 1 in the DB and the jump links are cosmetic.
Refs Task Master #1
The ask mode goes through the full agent loop (hybrid search + rerank +
Claude). Typical runs land at 30-90s; long queries reach 90-120s. With
the previous UX the user saw only "Loading…" on the submit button and
"מחפש…" in the results area — indistinguishable from a stuck request.
- Render a dedicated progress panel on ask with a spinner, a description
of what shira is doing, and a live elapsed-seconds counter.
- After 2 minutes, append a "taking longer than usual" warning so users
know the request is still running.
- Override Espo.Ajax timeout to 240s; jQuery's default (undefined) or
EspoCRM's request-level default would kill the browser request well
before shira finishes on a cold cache.
- Version 0.1.7.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
KnowledgeBase extension for EspoCRM: UI on top of the shira-hermes KB of
Israeli National Insurance law, regulations, and circulars.
Three modes:
- חיפוש — Hybrid (pgvector + tsvector) + Voyage rerank-2.5, returns ranked
chunks with heading_path and citations.
- שאל את שירה — Full agent loop; Shira picks up search_insurance_kb as
needed and returns a summary with citations.
- עיון — Browse all active sources. Click a source:
- PDF source (ספר הליקויים, ספר המבחנים, circulars): renders the
original PDF inline via an iframe proxied through the
KnowledgeBasePdf EntryPoint, so the layout/columns/tables are
preserved and Ctrl+F works natively.
- TXT source (חוק הביטוח הלאומי scraped from Wikisource): falls back
to the hierarchical chunk list with RTL styling.
Architecture:
- Controller: KnowledgeBase.php — thin proxy to shira-hermes /kb/*.
- Service: KnowledgeBaseService.php — shared curl plumbing; derives the
shira-hermes base URL from the SmartAssistant integration record so
there is no second admin config.
- EntryPoint: KnowledgeBasePdf.php — streams the PDF inline, wraps the
body in a php://temp stream for setBody, applies a locked-down CSP.
- JS: views/kb/index.js branches on source.original_path; modes wired
through the SmartAssistant fa_IR i18n convention.
Auth model:
- Browser → EspoCRM: session cookie / X-Api-Key (EspoCRM's existing auth).
- EspoCRM → shira-hermes: X-Api-Key from the SmartAssistant integration
(never exposed to the browser).
- Portal users are blocked at both the Controller and the EntryPoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chaim