Labor-law caselaw uploads (4 DOCX, 2026-04-28) timed out at exactly 90s,
landing in awaiting_review with empty ai_suggestions. A live retry of the
same payload through ai-gateway returned valid metadata in 91.8s — one
second over the cap. PDF caselaw can be larger (20K-char cap vs DOCX 6K)
and ai-gateway's serialized OAuth session lengthens the tail under batch
load, so 180s would not be enough headroom either.
240s keeps the call fail-soft (user can still fill metadata manually) but
covers realistic Hebrew legal documents. Override via KB_CLASSIFIER_TIMEOUT_S.
Two production-blocker fixes for the v0.8.0 batch upload UI, plus a
small endpoint to recover review state after a hard refresh.
classifier.py:
- timeout 45s → 90s and concurrency 5 → 2. ai-gateway proxies a
single Claude OAuth session so requests effectively serialize at
the gateway; with concurrency=5, the 4th and 5th files in a batch
of 5 timed out from queue waits alone (observed 2026-04-25 on
user upload of 5 circulars: jobs 10 and 13 returned empty).
- labels schema/prompt: arrays of strings via ai-gateway came back
as [{}, {}, {}] (proxy mangled items.type). Switched the param
to a comma-separated string with explicit format guidance, then
split in _normalize. _normalize is defensive — accepts both
string and list[str|dict] shapes so older callers don't break.
- Sharpened the prompt distinction between subject (free text on
the source) and labels (navigation tags), since identical
examples were causing the model to emit one and skip the other.
ingest_jobs.list_pending_batches + admin_kb GET /admin/kb/batches:
the review screen lives in JS RAM (_activeBatch). A hard refresh
wipes it, leaving uploaded jobs orphaned in awaiting_review with
no UI path back. The new endpoint groups jobs by batch_id with
status counters; the EspoCRM 'ממתינים לאישור' panel uses it to
let users resume any pending review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 6 of the multi-topic KB refactor (backend half). Subsumes the
original Task #19 design discussion (subject vs labels). EspoCRM
extension UI ships separately.
Migrations (4 new, all idempotent, transaction-wrapped):
004_kind_tool — adds 'tool' to kb_source/kb_ingest_job CHECK
005_source_description — kb_source.description + ai_classified_at
006_labels — kb_label + kb_source_label (M:N), GRANTs
007_ingest_classifier — awaiting_review status + processing_stage
+ ai_suggestions JSONB + batch_id
Two-phase ingest (api/services/kb/ingest_jobs.py):
queued → processing(stage=classifying)
→ awaiting_review ← user reviews AI output
→ processing(stage=embedding) ← after user commits
→ done | failed
process_classify_stage() pulls the file, runs parse_first_pages
(3-page text extract), calls classifier.classify (Claude tool-call
via ai-gateway, 45s timeout, 5-concurrent semaphore, fail-soft on
any error → empty suggestions), writes ai_suggestions JSONB,
transitions to awaiting_review.
commit_job() resolves user-confirmed labels (existing by slug,
new ones via slugify+ON CONFLICT DO NOTHING), transitions to
processing(embedding), schedules process_embed_stage.
process_embed_stage() runs the legacy ingest_source path with
user-edited metadata + summary as kb_source.description, then
apply_to_source for labels.
discard_job() removes a file from a batch (status=failed,
S3 deleted).
list_jobs_by_batch() — single-roundtrip review-screen load.
Labels (NEW api/services/kb/labels.py):
Hebrew → ASCII slugify (letter-by-letter map, no external dep);
CRUD; lookup_or_create_batch (race-safe); apply_to_source +
replace_for_source maintain usage_count; merge race-safely
(UPDATE assignments + ON CONFLICT DO NOTHING).
Classifier (NEW api/services/kb/classifier.py):
AsyncOpenAI to ai-gateway, Sonnet, OpenAI-style tool calling for
guaranteed JSON shape, Hebrew system prompt with 5 kind values,
citation format examples, "do not invent" rule, summary length
bounds 80-200 chars.
Routes (api/routes/admin_kb.py — 8 new on top of existing 11):
POST /admin/kb/upload-batch (multi-file, AI classify)
GET /admin/kb/batch/{batch_id} (review-screen load)
POST /admin/kb/jobs/{id}/commit (user confirms metadata)
POST /admin/kb/jobs/{id}/discard (user removes from batch)
GET /admin/kb/labels (typeahead)
POST /admin/kb/labels (explicit create)
POST /admin/kb/labels/{id}/merge (admin cleanup)
DELETE /admin/kb/labels/{id} (only if usage=0)
Public /kb/search gains optional label_id filter.
admin_sources.list_sources LEFT JOINs labels into each row's
output. update_source accepts label_ids to replace the set.
End-to-end smoke test passed: synthetic Hebrew circular →
upload-batch → background classify → awaiting_review with kind,
title, subject, summary, identifier, published_at all populated
correctly.
Refs Task Master #20 (espocrm-extensions/KnowledgeBase v0.8.0)
Subsumes: Task #19 (labels for sub-topics)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
chaim