Old behavior: clicking "עיבוד מחדש" on a source quietly re-chunked +
re-embedded the file but kept the existing metadata (title, identifier,
summary, labels). Sources whose original AI classification produced weak
metadata had no path to recover short of deleting and re-uploading.
New behavior: re-process now opens the same batch review screen as a
fresh upload — AI suggests fresh metadata (kind, title, identifier,
summary, labels, dates), the user reviews/edits, and committing
overwrites kb_source + replaces chunks. The card is tagged "עיבוד מחדש"
with a 🔄 icon so the user knows what they're committing.
Form precedence on the review card: user edits > AI suggestion > existing
source metadata. Existing labels are pre-filled in metadata_json and
surface as fallback if the AI classifier times out — so the form is
never blank, and existing labels are never silently dropped just because
the AI didn't re-suggest them.
Backend: shira-hermes 4ad569d.
Refs Task Master #22
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- mark task #21 (commit-race + collapsible panels fix) as done — followup to v0.8.1 release commit 95c48c7
- add SECURITY_AUDIT_2026-04-25.md from the deep audit of v0.8.0 (1 critical, 4 high, 5 medium, 4 low findings — kept for reference; remediation tracked separately)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two unrelated UX issues in the Knowledge Base manage tab:
1. Double-commit 400 error. The polling tick at index.js:1217 was
blindly overwriting a locally-set 'committing' status with whatever
the server reported. If the GET /batch poll raced ahead of the POST
commitJob reaching the DB, the card flipped back to 'awaiting_review',
the "אשר הכל" button re-enabled, and a second click triggered a duplicate
commit which the backend rejected with HTTP 400 ("job N is in status
'done', expected 'awaiting_review'"). Fix: never downgrade
'committing' → 'awaiting_review' from the polling response.
2. Admin tab scrolling. The "מקורות בנושא", "תוויות תת-נושא" and
"משימות אחרונות" panels were always expanded on tab entry, pushing
the actual upload form far down the page. Wrapped each in a
collapsible body with a chevron toggle, default collapsed. Data
lazy-loads on first expand.
Refs Task Master #21
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
v0.8.0 ships the four-feature bundle from the v0.8.0 plan plus
mid-flight fixes discovered when batch-testing 5 circulars.
New capabilities:
* Multi-file batch upload. AI classifier (Sonnet via ai-gateway)
reads the first 3 pages of each PDF and proposes kind / title /
identifier / labels / dates / summary. User reviews + edits per
card before committing. Two-phase ingest: classify →
awaiting_review → embed.
* 'tool' kind for academic assessment instruments (GMFCS, MACS).
* Labels (kb_label / kb_source_label) for sub-topic navigation;
classifier proposes shared label slugs so the graph builds
itself.
* 'תוויות' admin sub-section for merge/delete of unused labels.
* NEW 'ממתינים לאישור' panel: lists batches still in
awaiting_review with status counters so a hard-refreshed user
can resume their review — closes the RAM-only _activeBatch gap.
Mid-flight fixes folded in:
* classifier timeout 45s→90s, concurrency 5→2. ai-gateway
serializes through a single Claude OAuth session; with conc=5,
jobs 4-5 of a 5-file batch consistently timed out.
* labels schema array-of-string → comma-separated string. Claude
via ai-gateway returned [{}, {}, {}] for items:{type:"string"}.
_normalize accepts both shapes defensively.
Backfilled ai_classified_at + description + 1-3 labels for the 10
sources ingested in earlier sessions so the filter UX is uniform.
Backend (shira-hermes) in commits 0d678da (Phase 6) + bb23b8a
(this session's fixes).
Refs Task Master #20
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Short Hebrew label shown in the admin "ההרחבות שלנו" panel.
Synced into extensions.description on next sync-registry run
(or on next /publish for this extension).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a topics-management panel above the sources table. Admins can
create new legal domains (דיני עבודה, דין פלילי, נדל"ן) without
running SQL — slug + name + description + system prompt addendum
all live in one inline form. Per-row actions:
✏ Edit — opens inline form with all fields except slug
(slug is locked post-create — the S3 layout
depends on it)
👁 Toggle active — soft-disable: hides the topic from the user-facing
<select> while keeping data intact. Eye icon
flips between "השבת" and "הפעל"
🗑 Delete — hard-delete; only allowed when source_count=0.
Otherwise the UI directs the user to soft-disable
instead
Any topic mutation (create / rename / toggle / delete) invalidates
the cached _topics list so the user-facing dropdown re-fetches
immediately — admins don't have to refresh the page to see their
own changes.
Service.request() now distinguishes 4xx from 5xx upstream errors:
4xx is rethrown as BadRequest (so the user sees "slug already exists"
instead of a generic 500), 5xx stays as Error.
Backend: depends on shira-hermes commit 49503ca (admin topic
endpoints). No new migration — kb_topic schema from migration 001
is sufficient.
Description field in manifest was rewritten to Hebrew to match the
in-CRM scopeNames label ("מאגר ידע").
Refs Task Master #15
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds full CRUD for KB sources from the EspoCRM UI: a sortable table at
the top of the 'ניהול' tab with kind / title-identifier-updated / chunk
count / actions columns, a kind filter (חוק / תקנות / חוזרים / פסיקה),
and three per-row actions:
✏ Edit — inline form with title, identifier, dates, source URL
↻ Reingest — re-parses the original file in place, replacing chunks
while keeping the source row + hand-edited metadata
🗑 Delete — confirm dialog with chunk count; double confirm above
100 chunks; also deletes the MinIO object
Browser polls the standard /KnowledgeBase/action/job endpoint for
re-ingest progress (same banner as upload). On terminal status
(done|failed) the sources table auto-refreshes so chunk_count and
last_ingest reflect the new state. Topic switch invalidates the
per-topic admin-sources cache so a stale list doesn't bleed across
topics.
Backend: depends on shira-hermes commit 17f93b5
(GET/PUT/DELETE /admin/kb/sources + POST /reingest).
Refs Task Master #14
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Court rulings get their own kind so users can filter search results
to "פסיקה" specifically and upload case-law PDFs from the ניהול tab.
PHP validators in Controller + Service accept the new kind; the search
dropdown and upload kind dropdown gain a "פסיקה" option; kindHe in
index.js maps caselaw → "פסיקה" so jobs list, search results, and
browse view all label it consistently.
Backend: depends on shira-hermes commit 16eeeff (kind validators +
chunker fallback) and migration 003_caselaw_kind.sql (already applied
on dev — alters CHECK constraints on kb_source.kind and
kb_ingest_job.kind to include 'caselaw').
Refs Task Master #18
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a "ניהול" tab with multipart file upload, async job tracking, and
a recent-jobs list with live status polling. Browser POSTs to
/KnowledgeBase/action/upload (PHP proxy), which forwards as multipart
to shira-hermes /admin/kb/upload. shira-hermes returns a job_id
immediately and processes parse/chunk/embed in a background task; the
browser polls every 2s until status hits done|failed.
New EspoCRM endpoints:
POST /KnowledgeBase/action/upload (multipart, $_FILES['file'])
GET /KnowledgeBase/action/jobs (list, filtered by topicId/status)
GET /KnowledgeBase/action/job?id (single-job detail)
The X-User-Name header is forwarded so kb_ingest_job records who
uploaded each file. Cross-topic guard in switchTopic stops polling
when the user changes topics mid-upload (the job continues server-side).
Depends on shira-hermes commit 0cb89fb (admin upload endpoints +
migration 002).
Refs Task Master #13
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Generalize the Knowledge Base from insurance-only to multi-domain.
Introduces kb_topic table (shira-hermes), topic-aware endpoints
(/kb/topics, /kb/search, /kb/ask, /kb/ask/stream all accept topic_id),
a topic picker <select> in the EspoCRM UI (localStorage-persisted),
and renames the LLM tool search_insurance_kb → search_legal_kb.
All existing sources migrate to topic_id=1 (ביטוח לאומי); the UI
defaults to that topic so no visible regression for current users.
Refs Task Master #12
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wires the EspoCRM extension to the multi-topic backend introduced in
shira-hermes commit f03721e. The KB page header now shows a topic
<select> ('נושא:'); the user's selection persists in localStorage
('kb-topic') and is sent with every search/ask/sources call so cross-
domain bleed-over is impossible. Single seeded topic ('ביטוח לאומי')
means existing users see no functional change until a second topic is
added in Phase 4 (Task Master #15).
- Resources/routes.json: GET /KnowledgeBase/action/topics.
- Controllers/KnowledgeBase.php: actionTopics + topicId passthrough on
search/sources/ask. Numeric coercion for the request payload.
- Services/KnowledgeBaseService.php: listTopics(); search/listSources/
ask take an optional topicId and forward it as topic_id to upstream.
- EntryPoints/KnowledgeBaseAskStream.php: ?topicId=N becomes topic_id
in the JSON body sent to /kb/ask/stream.
- res/templates/kb/index.tpl: topic <select> in the page header,
with a kb-topic-name-suffix span that JS fills with the active name.
- src/views/kb/index.js: module-level _topics cache + LS_TOPIC. setup
reads localStorage; afterRender gates the picker fill + sources fetch
on topics being loaded so a stale id from a deleted topic doesn't
fire a 400-ing /sources call. switchTopic clears _activeAsk/Search +
_lastAsk/Search + sessionStorage entries + this._sources, then
reRenders. Cross-topic guard in afterRender drops cached state whose
topicId !== this.topicId. _activeAsk / _lastAsk / _activeSearch /
_lastSearch all carry topicId. EventSource URL gains &topicId.
Refs Task Master #12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comprehensive Coolify-prod runbook covering: pgvector PG provisioning,
MinIO bucket, shira-hermes dockerimage app, Infisical /prod secrets,
data migration choice (pg_dump+S3 mirror vs fresh ingest), KB extension
install on prod EspoCRM, SmartAssistant integration config,
optional n8n auto-scan workflow clone, DNS, and end-to-end smoke test
sequence. Currently the entire stack is dev-only — this task makes
KB live on the production CRM at crm.prod.marcus-law.co.il.
Captures the open-but-not-blocking items from the v0.2.1 sessions plus
the comprehensive plan for generalizing the KB beyond ביטוח לאומי.
#9-11: carry-over polish noted at the v0.2.1 close
- #9 SSE rejoin (in-flight stream survives F5 / tab close)
- #10 n8n release-asset auto-attach has been failing for every release
from v0.1.8 through v0.2.1; manual curl uploads as workaround
- #11 ask mode source picker hides text-source citations (Wikisource law)
because /kb/ask filters sources_used to PDFs only
#12-16: multi-topic KB plan (next session)
- #12 Phase 1 — DB schema with kb_topic + kb_source.topic_id, /kb/topics
endpoint, topic-aware system prompts, topic dropdown in UI;
backwards-compat by seeding topic_id=1 ביטוח לאומי and migrating
the 6 existing sources to it
- #13 Phase 2 — file upload from the management panel, async ingestion
via kb_ingest_job table, live job-status polling
- #14 Phase 3 — sources management UI: edit metadata, delete (with
cascade messaging), re-ingest, jobs/failures view
- #15 Phase 4 — topic CRUD UI for admins (add דיני עבודה, דין פלילי,
etc., edit prompts, soft-disable)
- #16 Phase 5 (optional) — per-topic ACL by EspoCRM role with view vs
manage permissions
Phases 1-3 are the must-ship subset for what the user asked. Phase 4 is
admin convenience. Phase 5 is scoped only if firms ask for it.
Three issues from the v0.2.0 user test:
1. Sides flipped relative to expectation
In our Hebrew/RTL CRM the user reads primary content (the answer or
the hit list) on the visual right and the supporting reference
(the PDF) on the visual left — that's how Hebrew layouts stage
"main + sidebar". v0.1.10's _buildSplitShell forced direction:ltr on
the splitter container (correct, for mouseX math) but laid the
children in the order [primary, handle, reference], which under LTR
put primary on the LEFT and PDF on the RIGHT. Fixed by renaming the
helper params to {primaryHtml, referenceHtml} and ordering the
children as [reference, handle, primary] — visual LEFT is now the
PDF, visual RIGHT is the answer/list. Default split moved from 42%
to 58% so the reference pane (now on the left) keeps the same
absolute width the PDF column had under v0.1.9's bootstrap rows.
2. No way to start a new query
afterRender replays _lastSearch / _lastAsk on every remount, which
means the prior result hangs around forever. Added a "חיפוש חדש" /
"שאלה חדשה" button next to the submit button. clearResults() cancels
any in-flight request (closes the EventSource for ask, drops
_activeSearch reference for search), removes the cached entry from
sessionStorage, empties the results pane, and refocuses the input.
3. Stale results from another session
Cached results now auto-expire after 30 minutes — _loadJson checks
completedAt against STALE_AFTER_MS and silently drops anything
older. So coming back tomorrow the KB tab opens clean instead of
showing yesterday's answer to whatever you asked then.
Refs Task Master #4, #6
User wanted to see what Shira is doing while she thinks — like Claude
does — instead of staring at an incrementing seconds counter. Now the
ask UI shows one line per agent step as it happens:
3s 🧠 קוראת את השאלה
4s 🔍 מחפשת בבסיס הידע: "תקנה 36"
9s ✓ נמצאו 5 קטעים רלוונטיים
10s 🔍 מחפשת בבסיס הידע: "תקנה 36 ביטוח לאומי"
14s ✓ נמצאו 5 קטעים רלוונטיים
15s 🧠 ממשיכה לחקור
22s ✍️ מנסחת תשובה
→ תשובה
Client wiring:
- runAsk now opens an EventSource at ?entryPoint=KnowledgeBaseAskStream
instead of POSTing JSON. Each SSE event of type thinking/tool_start/
tool_done/tool_error/writing appends a row to a stacked progress log.
The final {type:answer,text,sources} event triggers renderAskAnswer
with the existing split view. {type:error} surfaces via showError.
- Module-level state from v0.1.9 unchanged in spirit but now stores the
EventSource + accumulated events. afterRender on view re-mount replays
the entire event list and re-binds onmessage/onerror — the SSE
connection itself doesn't drop because it lives at module scope.
- Elapsed-time pill stays for cadence but is decoration; the real
signal is the per-event log.
EspoCRM proxy:
- New EntryPoint KnowledgeBaseAskStream — required because EventSource
is GET-only and ?message= must come via the URL. PHP buffering is
disabled (zlib + ob + apache_setenv no-gzip), Apache headers are set
for SSE (no-cache, X-Accel-Buffering: no), and we cURL into
shira-hermes /kb/ask/stream with CURLOPT_WRITEFUNCTION echoing each
byte and flushing immediately. The handler ends with exit; to bypass
EspoCRM's Response wrapper which would otherwise emit headers/body
on top of our raw stream.
- KnowledgeBaseService gained two public accessors (getStreamingUrl,
getStreamingApiKey) so the EntryPoint can build the cURL without
going through request() (which buffers the whole body).
- entryPoints.json registers the new class.
Server (shira-hermes 76fd77f):
- AgentRunner.run accepts an on_event sync callback fired before each
LLM call (thinking) and around each tool call (tool_start, tool_done,
tool_error). Hebrew labels in agent_runner._tool_label / _tool_done_label
ship over the wire ready-to-display — keeps the client free of
translation concerns.
- POST /kb/ask/stream wraps the existing agent runner in an asyncio
task feeding an asyncio.Queue, returns StreamingResponse with proper
SSE headers (X-Accel-Buffering, Cache-Control). 15s comment-line
heartbeats keep the connection open through proxy idle-timeouts.
- Original POST /kb/ask still works — additive change.
Refs Task Master #8
User reported the search card "looks broken" when the selected hit is the
law (Wikisource source 5) — the right preview pane just duplicates the
chunk text already visible in the left list card. For PDF sources the
right pane shows the PDF page, but for text-only sources it had nothing
useful to add and fell back to a copy.
Fix:
- /kb/search hits gained chunk_index (shira-hermes b127caf), and
/kb/source/{id}/chunks gained an `around=<chunk_index>&ctx=<k>` window.
- Espo proxy plumbing: getActionChunks reads `around`/`ctx` query
params and forwards them; KnowledgeBaseService::sourceChunks accepts
optional $around + $ctx and appends them to the upstream URL.
- showSearchPreview for non-PDF hits now fetches the matched chunk plus
2 sections before and 2 after, renders them as stacked panels: the
matched section gets a yellow #fff3cd highlight, neighbors render as
muted context blocks. The panel header carries a prominent
"פתח ב-Wikisource" button so users can jump to the public source page.
After render the matched section auto-scrolls into view.
- Race protection via _previewToken: a click on a different hit while a
fetch is in flight cancels the stale render so the user always sees
the section that matches the currently-selected card.
PDF path unchanged.
Refs Task Master #7
Two requests on top of v0.1.9:
1. Drag-to-resize splitter
The previous fixed Bootstrap col-md-5/col-md-7 split locked the
user into 41.66% answer / 58.33% PDF. Replaced with a flex-row
layout containing a 6px drag handle. mousedown spawns a fixed
transparent overlay so the PDF.js iframe doesn't swallow mouse
events while dragging; mouseup persists the chosen ratio (clamped
15-85%) to localStorage as kb-split-pct, so the next page-load and
the next session both come back to the same layout. Both the
ask split view and the search split view share _buildSplitShell +
_wireSplitter — single source of truth for the geometry.
2. Search survives a view switch
Search had the same view-bound promise problem ask had before
v0.1.9. Hoisted the in-flight request to a module-level
_activeSearch and the most recent {query, kind, hits, selectedIdx}
to _lastSearch + sessionStorage. afterRender restores the input,
the kind filter, the hit list (with the same hit highlighted), and
the right-pane preview. Click handler updates selectedIdx so a
round-trip lands on the user's last-clicked hit, not always on the
top-ranked one.
The persistence helpers grew a tiny refactor: _loadJson/_saveJson/_clear
replace the ask-only SS_KEY shim and now back both ask + search keys.
Refs Task Master #5, #6
Two recurring frustrations from the v0.1.8 demo:
1. Searching "מהי תקנה 37" only returned the תקנה 37 חוזר and the law —
ספר הליקויים, the canonical medical reference, was crowded out by the
document whose title literally contained the query terms. Fixed in
shira-hermes 1441a41 by adding an LLM-driven query-expansion path to
/kb/search (default expand=true). Same query now also returns 2 hits
from src 29 (ספר הליקויים).
2. Asking Shira a question and clicking another EspoCRM view dropped the
in-flight request — coming back rendered a blank kb-results pane, even
though the request had completed in the background. Fixed here:
{question, promise, startedAt} are now stored at module scope (closure
shared across remounts of this view). afterRender re-binds handlers
for any active ask, resumes the progress panel with the original start
time so the elapsed counter doesn't reset, and replays the most recent
completed answer from sessionStorage so the user sees their result
even after a hard reload. State is intentionally session-scoped — no
answers persist across browser sessions.
Refs Task Master #3, #4
Search and ask modes now render results as a two-column layout: a left
column with ranked hit cards (search) or Shira's answer + source picker
(ask), and a right column with an iframe showing the source PDF scrolled
to page_number. Clicking a hit (search) or source pill (ask) swaps the
iframe's src, so users can verify a quote against the original PDF
without leaving the KB tab.
- search: renderSearchResults lays out results as panel cards on the
left (with kind + title + section + "עמ׳ N" label); the top hit is
pre-selected and its PDF loads on the right. Clicking any card
re-highlights it and swaps the preview. Law-kind hits (Wikisource
text) gracefully fall back to a chunk-text panel with a Wikisource
link so the right pane never 404s on a text source.
- ask: renderAskAnswer dedups /kb/ask's sources[] by source_id,
collects all cited pages per source, and renders a picker row plus
per-source page-jump buttons. First source's first page loads on
initial render; buttons swap the iframe without re-running the query.
Depends on shira-hermes commits e534709 + 1ca1cc0 (chunker page_number
propagation + null-byte strip) — without them, every PDF collapses to
page 1 in the DB and the jump links are cosmetic.
Refs Task Master #1
The ask mode goes through the full agent loop (hybrid search + rerank +
Claude). Typical runs land at 30-90s; long queries reach 90-120s. With
the previous UX the user saw only "Loading…" on the submit button and
"מחפש…" in the results area — indistinguishable from a stuck request.
- Render a dedicated progress panel on ask with a spinner, a description
of what shira is doing, and a live elapsed-seconds counter.
- After 2 minutes, append a "taking longer than usual" warning so users
know the request is still running.
- Override Espo.Ajax timeout to 240s; jQuery's default (undefined) or
EspoCRM's request-level default would kill the browser request well
before shira finishes on a cold cache.
- Version 0.1.7.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
KnowledgeBase extension for EspoCRM: UI on top of the shira-hermes KB of
Israeli National Insurance law, regulations, and circulars.
Three modes:
- חיפוש — Hybrid (pgvector + tsvector) + Voyage rerank-2.5, returns ranked
chunks with heading_path and citations.
- שאל את שירה — Full agent loop; Shira picks up search_insurance_kb as
needed and returns a summary with citations.
- עיון — Browse all active sources. Click a source:
- PDF source (ספר הליקויים, ספר המבחנים, circulars): renders the
original PDF inline via an iframe proxied through the
KnowledgeBasePdf EntryPoint, so the layout/columns/tables are
preserved and Ctrl+F works natively.
- TXT source (חוק הביטוח הלאומי scraped from Wikisource): falls back
to the hierarchical chunk list with RTL styling.
Architecture:
- Controller: KnowledgeBase.php — thin proxy to shira-hermes /kb/*.
- Service: KnowledgeBaseService.php — shared curl plumbing; derives the
shira-hermes base URL from the SmartAssistant integration record so
there is no second admin config.
- EntryPoint: KnowledgeBasePdf.php — streams the PDF inline, wraps the
body in a php://temp stream for setBody, applies a locked-down CSP.
- JS: views/kb/index.js branches on source.original_path; modes wired
through the SmartAssistant fa_IR i18n convention.
Auth model:
- Browser → EspoCRM: session cookie / X-Api-Key (EspoCRM's existing auth).
- EspoCRM → shira-hermes: X-Api-Key from the SmartAssistant integration
(never exposed to the browser).
- Portal users are blocked at both the Controller and the EntryPoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chaim