{ "master": { "tasks": [ { "id": "1", "title": "Discovery tools + medical_case_analysis skill", "description": "Add browse_folder/find_case_folder/set_case_folder_path tools, enrich list_documents with diagnostic, update prompt rules, and add medical_case_analysis skill.", "details": "", "testStrategy": "", "status": "in-progress", "dependencies": [], "priority": "medium", "subtasks": [], "updatedAt": "2026-04-19T06:11:18.162Z" }, { "id": "2", "title": "feat(kb): Phase 1 — Israeli National Insurance KB", "description": "Phase 1 of the Israeli National Insurance knowledge base. Create insurance_kb database on the shared PostgreSQL instance with pgvector and pg_trgm extensions. Schema: kb_source (law/regulation/circular + supersession) and kb_chunk (content, heading_path, section_ref, tsvector, vector(1024) with HNSW cosine index). Create MinIO bucket insurance-kb. Add KB module to shira-hermes: config, asyncpg pool, Voyage client, parse+chunk+embed pipeline, hybrid search with RRF, search_insurance_kb tool (registered in mcp_server), POST /admin/kb/ingest, GET /admin/kb/stats. Store KB_DATABASE_URL in Infisical /espocrm. Validate end-to-end by ingesting 2-3 sample circulars and issuing a semantic query via the tool.", "details": "See plan in conversation on 2026-04-21. Chunking strategy differs per kind: laws/regulations split by section; circulars split header-aware ~600 tokens with 80 overlap. Supersession: soft (superseded_by FK) not delete. Dependencies to add: asyncpg, voyageai (or REST via httpx), python-docx. Voyage model: voyage-multilingual-2, dim=1024, input_type=document for ingest / query for search.", "testStrategy": "Manual: upload 2-3 sample PDF circulars via POST /admin/kb/ingest, verify kb_source + kb_chunk rows, verify embedding column non-null, call search_insurance_kb with a Hebrew query and confirm top-k results cite the right section_ref.", "status": "in-progress", "dependencies": [], "priority": "high", "subtasks": [], "updatedAt": "2026-04-21T00:00:00.000Z" }, { "id": "3", "title": "Add תקנות הביטוח הלאומי (~100 regulations) from Wikisource to KB", "description": "Extend scripts/fetch_national_insurance_law.py to bulk-fetch ~100 regulation pages from Hebrew Wikisource via MediaWiki API. Output text + sidecar JSON pairs to /tmp/regulations/. Upload to MinIO inbox/regulation/ for the existing scan-inbox cron to embed via Voyage. Estimated 30-40MB Postgres growth, ~$0.36 embedding cost. Sample 3 first (small/medium/large), verify chunker output, then bulk.", "details": "", "testStrategy": "", "status": "done", "dependencies": [], "priority": "high", "subtasks": [], "updatedAt": "2026-04-25T20:39:11.905Z" }, { "id": "4", "title": "Add 9 tools: create_document, write/create/rename/move folder ops, mark_document_for_deletion, update_case_fields, search_entities, draft_email", "description": "Expose new EspoCRM SmartAssistant actions through Python tools and update prompt_builder with new behavioral rules.", "details": "Tools added in mcp_server/tools/document_tools.py: create_document, write_document_to_folder, create_subfolder, rename_item, move_item, mark_document_for_deletion. Tools added in mcp_server/tools/crm_tools.py: update_case_fields, search_entities, draft_email. Prompt updates in api/services/prompt_builder.py: FREE DOCUMENT CREATION rule, FOLDER MANAGEMENT rule, FILE DELETION rule (always mark_document_for_deletion). Pairs with SmartAssistant extension release that adds FreeDocumentGenerator + CaseFolderManager + 6 controller actions.", "testStrategy": "End-to-end test in EspoCRM dev: ask Shira to draft a letter, create/rename/move folders, mark for deletion, update case fields, search entities, draft email.", "priority": "medium", "status": "in-progress", "dependencies": [], "subtasks": [], "updatedAt": "2026-05-14T12:39:55.190Z" }, { "id": "5", "title": "fix(ocr): Extract text from word/document.xml before OCR fallback for text-heavy DOCX files", "description": "The ocr_docx_images function in api/services/ocr.py only extracts and OCRs images from DOCX files, ignoring the actual text content in word/document.xml. When DocumentAnalyzer (PHP) fails and we fall back to OCR, text-heavy DOCX files with minimal images result in hallucinated content. Fix by parsing word/document.xml first, stripping XML tags to extract text, then appending OCR'd images.", "details": "## Current Behavior (api/services/ocr.py:87-112)\n\nThe `ocr_docx_images` function:\n1. Opens DOCX as a zip archive\n2. Finds all files in `word/media/` (images only)\n3. Extracts and OCRs each image via Claude Vision\n4. Returns combined image OCR results\n5. Returns error if no images found\n\n**Critical flaw:** DOCX files are zip archives where the actual text lives in `word/document.xml`, not in `word/media/`. A text-heavy document with only a signature image will return just \"[signature]\" as content.\n\n**Production evidence:** Case 46 prod, 'הודעה לבית הדין-כתב ערעור-973.docx' — 50KB appeal brief, only image is a signature → LLM hallucinates the rest.\n\n## Implementation Steps\n\n1. **Add XML parsing to ocr_docx_images:**\n - After opening the zipfile, first check for `word/document.xml`\n - If found, read and parse it using Python's built-in `xml.etree.ElementTree` (no new dependencies needed)\n - Extract all text nodes (typically in `` tags under namespace `http://schemas.openxmlformats.org/wordprocessingml/2006/main`)\n - Strip all XML tags and formatting, preserve only text content and basic structure (paragraph breaks)\n - Store as `document_text`\n\n2. **Extract images (existing logic):**\n - Keep the existing `word/media/` extraction and OCR loop\n - Store results in `image_extracts` list as before\n\n3. **Combine results:**\n - If `document_text` exists and is non-empty, prepend it to output: `=== תוכן המסמך ===\\n{document_text}\\n\\n`\n - Then append all `image_extracts` entries\n - If both are empty, return the existing error message\n - Update the \"no images found\" logic to check if text was extracted\n\n4. **Update logging:**\n - Log both text extraction and image count: `logger.info(\"[ocr] docx text=%d chars, images=%d\", len(document_text), len(media))`\n\n5. **Error handling:**\n - Wrap XML parsing in try/except — if it fails, log warning and proceed with image-only extraction\n - Don't let XML parse errors block the existing image OCR functionality\n\n## Code Structure\n\n```python\nasync def ocr_docx_images(base64_data: str) -> str:\n \"\"\"Extract text from word/document.xml and OCR images from DOCX.\"\"\"\n import zipfile\n import xml.etree.ElementTree as ET\n \n docx_bytes = base64.b64decode(base64_data)\n document_text = \"\"\n image_extracts = []\n \n try:\n with zipfile.ZipFile(io.BytesIO(docx_bytes)) as z:\n # 1. Extract text from word/document.xml\n if \"word/document.xml\" in z.namelist():\n try:\n xml_bytes = z.read(\"word/document.xml\")\n root = ET.fromstring(xml_bytes)\n # Define Word namespace\n ns = {\"w\": \"http://schemas.openxmlformats.org/wordprocessingml/2006/main\"}\n # Extract all text nodes\n text_nodes = root.findall(\".//w:t\", ns)\n document_text = \"\".join(node.text or \"\" for node in text_nodes)\n logger.info(\"[ocr] docx text extracted: %d chars\", len(document_text))\n except Exception as e:\n logger.warning(\"[ocr] failed to parse word/document.xml: %s\", e)\n \n # 2. Extract and OCR images (existing logic)\n media = [n for n in z.namelist() if n.startswith(\"word/media/\")]\n logger.info(\"[ocr] docx images found=%d\", len(media))\n for i, name in enumerate(media[:MAX_PDF_PAGES]):\n # ... existing image OCR logic ...\n \n except zipfile.BadZipFile:\n return \"❌ הקובץ אינו DOCX תקין.\"\n \n # 3. Combine results\n parts = []\n if document_text.strip():\n parts.append(f\"=== תוכן המסמך ===\\n{document_text.strip()}\")\n parts.extend(image_extracts)\n \n if not parts:\n return \"❌ לא נמצא תוכן ב-DOCX — לא טקסט ולא תמונות.\"\n \n return \"\\n\\n\".join(parts)\n```\n\n## Dependencies\n\n- No new dependencies required — `xml.etree.ElementTree` is part of Python stdlib\n- `zipfile` and `io` are already imported\n- `python-docx` is already in pyproject.toml (line 16) but NOT needed for this fix — we're doing raw XML parsing\n\n## Namespace Handling\n\nWord 2007+ DOCX uses OOXML namespace `http://schemas.openxmlformats.org/wordprocessingml/2006/main`. Text lives in `` elements. Paragraph structure is in ``, but for initial fix we can flatten — preserving paragraph breaks is optional enhancement.\n\n## Testing Requirements\n\n1. **Unit test:** Create a minimal DOCX in memory with known text + 1 image, verify both are extracted\n2. **Production case:** Download case 46 prod 'הודעה לבית הדין-כתב ערעור-973.docx', run through fixed `ocr_docx_images`, verify appeal text is extracted (not just signature)\n3. **Edge cases:**\n - DOCX with text but no images (should extract text, no error)\n - DOCX with images but no text (existing behavior preserved)\n - DOCX with neither (error message)\n - Corrupted word/document.xml (should log warning, proceed with images only)", "testStrategy": "## Manual Testing\n\n1. **Fetch the production case:**\n - Access case 46 on prod EspoCRM: https://crm.prod.marcus-law.co.il\n - Download 'הודעה לבית הדין-כתב ערעור-973.docx'\n - Verify file size is ~50KB and contains appeal text + signature image\n\n2. **Test via read_document_ocr tool:**\n - Deploy fixed code to dev instance\n - In EspoCRM dev, open any case and upload the test DOCX\n - Invoke Shira with: \"קרא את המסמך הזה עם OCR\" (forcing OCR path)\n - Verify response includes:\n - `=== תוכן המסמך ===` section with full appeal text (Hebrew)\n - `=== תמונה 1 ===` section with signature OCR\n - No hallucinated content\n\n3. **Regression test — image-only DOCX:**\n - Create DOCX with only embedded images (no text)\n - Verify existing behavior: extracts and OCRs all images\n - No \"לא נמצא תוכן\" error\n\n4. **Edge case — text-only DOCX:**\n - Create DOCX with text but no images\n - Verify: extracts text, no image section, no errors\n\n5. **Check logs:**\n - Verify log line shows both text and image counts: `[ocr] docx text= chars, images=`\n - If XML parsing fails, verify warning is logged but images still OCR\n\n## Automated Testing (optional follow-up)\n\nCreate `tests/test_ocr.py` with pytest fixtures:\n- Mock DOCX files (minimal zip with word/document.xml + word/media/image.png)\n- Assert `ocr_docx_images` returns both text and image sections\n- Mock Claude Vision API calls to avoid external dependencies\n\n## Acceptance Criteria\n\n✅ Case 46 appeal brief extracts full text (not just signature) \n✅ Text-heavy DOCX files no longer cause LLM hallucination \n✅ Image-only DOCX files still work (regression protected) \n✅ Corrupted word/document.xml doesn't break image extraction \n✅ Log output shows text length + image count for debugging", "status": "done", "dependencies": [], "priority": "medium", "subtasks": [], "updatedAt": "2026-05-26T13:33:43.889Z" }, { "id": "6", "title": "fix(documents): empty/minimal OCR result must return explicit failure, not pass through to LLM", "description": "When read_document's OCR fallback returns only \"[signature]\" or <50 chars, the LLM hallucinates content. Add validation to detect 'no real text extracted' and return a clear failure message that the LLM can recognize, preventing hallucination.", "details": "## Problem\n\nIn `mcp_server/tools/document_tools.py:118`, the `_ocr_fallback` function wraps **any** OCR result in `ok()`, including minimal extractions like \"[signature]\" (12 chars) or empty strings. The LLM interprets these as successful extractions and often hallucinates missing content, leading to factually incorrect responses.\n\n## Root Cause\n\n1. `api/services/ocr.py:ocr_docx_images()` returns `\"❌ לא נמצאו תמונות ב-DOCX לחילוץ טקסט.\"` when no images exist (line 111), which is correctly detected as failure.\n2. But when a single signature image exists, it returns `\"=== תמונה 1 (...) ===\\n[signature]\"` as a \"success\" (line 104).\n3. `_ocr_fallback` at line 115-118 calls `extract_text_from_bytes` and wraps the result in `ok()` without validating text quality.\n4. The `ok()` helper (defined in `mcp_server/tools/_helpers.py:46-48`) simply returns the message as-is—there's no success/failure metadata.\n5. The agent sees \"success\" and passes the minimal text to the LLM context, triggering hallucination.\n\n## Solution\n\nAdd a **signal-to-noise validation** step in `_ocr_fallback` after line 117, before wrapping in `ok()`:\n\n```python\n# api/services/ocr.py (lines 115-120)\ntext = await extract_text_from_bytes(\n bytes_result[\"base64\"], mime_type, file_name\n)\n\n# NEW: Validate text quality\ncleaned = text.strip()\n# Remove common OCR noise patterns\nif cleaned.startswith(\"===\"):\n # Strip header like \"=== תמונה 1 (...) ===\"\n cleaned = \"\\n\".join(cleaned.split(\"\\n\")[1:]).strip()\n\n# Check for failure indicators\nif (\n cleaned.startswith(\"❌\") # Explicit error from OCR functions\n or len(cleaned) < 50 # Too short to be real content\n or cleaned.lower() in [\"[signature]\", \"[חתימה]\", \"signature\"] # Known noise\n):\n return fail(f'לא הצלחתי לחלץ טקסט מהמסמך \"{file_name}\" — נסה לפתוח אותו ידנית או ודא שהקובץ מכיל טקסט ברור.')\n\nreturn ok(f'=== {file_name} (OCR via Vision) ===\\n\\n{text}')\n```\n\n## Rationale\n\n- **50-char threshold:** Based on typical Hebrew/English sentences, 50 chars is the minimum for a single meaningful sentence. Anything shorter is likely OCR noise.\n- **Explicit noise patterns:** `[signature]` is the most common false-positive from Claude Vision when OCRing signature images.\n- **Error prefixes:** `❌` is already used by `ocr.py` functions to signal hard errors (lines 61, 108, 111, 131).\n- **fail() wrapper:** Uses the existing `fail()` helper (`_helpers.py:51-53`) to prefix `❌`, ensuring the agent interprets it as a tool failure.\n\n## Edge Cases\n\n1. **Multi-page PDFs with one junk page:** Current code aggregates pages—validation happens on the **final combined text**, not per-page. If page 1 has content but page 2 is just \"[signature]\", the combined text will pass the 50-char threshold. This is acceptable—we only reject when the **entire document** is junk.\n2. **Hebrew-only short content:** A 40-char Hebrew sentence (e.g., \"התביעה נדחתה מחוסר סמכות עניינית\") is legitimate. However, such documents are **extremely rare** in legal contexts—most legal docs exceed 200 chars. The 50-char threshold is a pragmatic compromise.\n3. **DOCX with only images (Task 5 context):** After Task 5 is completed, `ocr_docx_images` will first extract `word/document.xml` text. If that yields <50 chars **and** image OCR also yields <50 chars, this validation will correctly fail both paths.\n\n## File Changes\n\n**File:** `mcp_server/tools/document_tools.py` \n**Function:** `_ocr_fallback` (lines 100-120) \n**Action:** Add validation block after line 117, before the `return ok(...)` at line 118.\n\n## Dependencies\n\n- **Task 5 (optional but complementary):** Task 5 fixes DOCX text extraction by parsing `word/document.xml` first. This task (Task 6) is **independent**—it validates **any** OCR result, regardless of source (PDF, DOCX, image).\n- If Task 5 is completed first, this validation will catch cases where both XML extraction **and** image OCR fail to produce meaningful text.\n- If this task (Task 6) is completed first, it will immediately prevent hallucination for existing problematic DOCX files (like case 46's appeal document), even before Task 5's XML extraction is implemented.", "testStrategy": "## Manual Testing\n\n1. **Reproduce the hallucination (baseline):**\n - Deploy current code to dev environment\n - Open case 46 on dev EspoCRM (or use the prod DOCX file mentioned in Task 5 context)\n - Call `read_document_ocr` on `הודעה לבית הדין-כתב ערעור-973.docx`\n - **Expected (current broken behavior):** Returns `ok()` with just \"[signature]\"\n - Ask Shira to summarize the document\n - **Expected (current broken behavior):** LLM hallucinates content\n\n2. **Test the fix:**\n - Apply the validation code from the \"Solution\" section above\n - Redeploy to dev\n - Call `read_document_ocr` on the same file\n - **Expected:** Returns `fail(\"לא הצלחתי לחלץ טקסט מהמסמך...\")` with ❌ prefix\n - Ask Shira to summarize the document\n - **Expected:** Agent recognizes tool failure and responds \"לא הצלחתי לקרוא את המסמך — נסה לפתוח אותו ידנית\"\n\n3. **Edge case: legitimate short content:**\n - Create a test DOCX with exactly 60 chars of Hebrew text (above threshold)\n - Upload to dev EspoCRM, attach to a test case\n - Call `read_document_ocr` on the file\n - **Expected:** Returns `ok()` with the full 60 chars\n - Verify the text is passed to the LLM without hallucination\n\n4. **Edge case: multi-page PDF with one junk page:**\n - Create a 2-page PDF: page 1 has 200 chars of text, page 2 is just a signature image\n - Call `read_document_ocr` on the PDF\n - **Expected:** Combined text from both pages exceeds 50 chars, returns `ok()`\n - Verify page 1 content is correctly extracted\n\n5. **Integration test with Task 5 (if completed):**\n - Use the same DOCX from case 46 that triggered this bug\n - After Task 5's XML extraction is deployed, verify `ocr_docx_images` returns the appeal text from `word/document.xml`\n - The validation should **pass** (text length >> 50 chars)\n - Verify no hallucination occurs\n\n6. **Error message visibility:**\n - Trigger the failure condition (signature-only DOCX)\n - Check the agent's tool call response in Shira's logs\n - **Expected:** Tool result starts with `❌` (from `fail()` helper)\n - Verify the agent's next message to the user mentions the file name and suggests opening manually\n\n## Automated Testing (optional follow-up)\n\nCreate unit tests in `tests/test_document_tools.py`:\n- Mock `extract_text_from_bytes` to return various short strings\n- Assert `_ocr_fallback` returns `fail()` for inputs <50 chars, `[signature]`, etc.\n- Assert `_ocr_fallback` returns `ok()` for inputs ≥50 chars", "status": "done", "dependencies": [ "5" ], "priority": "high", "subtasks": [], "updatedAt": "2026-05-26T13:35:43.770Z" }, { "id": "7", "title": "fix(prompt): add anti-hallucination rule to TOOL_RULES — never invent content from empty/minimal tool results", "description": "When read_document/read_document_ocr returns minimal text (e.g., only \"[signature]\" or <50 chars), the LLM often hallucinates missing content from memory, inventing quotes, analyses, or summaries that don't exist in the tool result. Add an explicit anti-hallucination rule to TOOL_RULES in prompt_builder.py forbidding the model from inventing document content and requiring it to explicitly state when extraction failed.", "details": "## Problem\n\nEven after Task 6 adds explicit failure detection at the tool layer (`_ocr_fallback`), the LLM can still misinterpret legitimate minimal results or rely on memory from previous conversations. The model needs a clear behavioral guardrail in its system prompt.\n\n## Current Implementation (api/services/prompt_builder.py:40-66)\n\n`TOOL_RULES` currently has 26 rules covering tool usage, document discovery, folder management, billing, etc. None explicitly forbid hallucination from minimal tool results.\n\n## Solution\n\nAdd a new rule to `TOOL_RULES` (insert after line 52, near the existing DOCUMENTS section) with the following content:\n\n```python\n\"- DOCUMENT CONTENT INTEGRITY: If read_document/read_document_ocr returns very short text (<50 chars), only '[signature]', or an error message starting with ❌ — you MUST NOT invent or quote content. Tell the user explicitly: 'לא הצלחתי לחלץ תוכן משמעותי מהמסמך' and do NOT offer analysis, quotes, or summaries. NEVER quote a document from memory or previous conversations — only from a tool result returned in THIS conversation.\\n\"\n```\n\n## Key Principles\n\n1. **Memory isolation**: Prevent the model from mixing current tool results with past conversation memory\n2. **Explicit failure reporting**: Force the model to admit when extraction failed instead of filling gaps\n3. **Clear threshold**: Use objective metrics (<50 chars, '[signature]', ❌ prefix) so the model can self-check\n4. **Hebrew UX**: The failure message should be in Hebrew and professional\n\n## Implementation Location\n\nFile: `api/services/prompt_builder.py`\n\nLine: After line 52 (after the DOCUMENTS rule, before DOCUMENT DISCOVERY)\n\nInsert the new rule as a continuation of the `TOOL_RULES` string concatenation.\n\n## Testing Notes\n\nThis is a **prompt-level safeguard** that complements Task 6's tool-level validation. Even if Task 6 is not yet implemented, this rule will catch cases where:\n- The OCR legitimately extracts only minimal text (e.g., a signature page)\n- The tool returns an error but the model tries to be \"helpful\"\n- The model has seen the document in a previous conversation and attempts to recall it\n\n## Integration with Task 6\n\nIf Task 6 is completed first, it will return clear failure messages like:\n```\n❌ לא נמצא תוכן משמעותי במסמך (רק 12 תווים: [signature])\n```\n\nThis rule ensures the model respects that failure signal and does not attempt to \"work around\" it.", "testStrategy": "## Manual Testing\n\n1. **Test with minimal OCR result (pre-Task 6):**\n - Deploy to dev environment\n - Upload a document with only a signature image to a case\n - Call `read_document_ocr` on the document via Shira\n - **Expected:** Shira should say \"לא הצלחתי לחלץ תוכן משמעותי מהמסמך\" and NOT provide analysis or quotes\n - **Current (without rule):** Shira might hallucinate content\n\n2. **Test with explicit failure (post-Task 6):**\n - Assume Task 6 is deployed (tool returns ❌ message for minimal results)\n - Ask Shira to read a document that triggers the failure\n - **Expected:** Shira should acknowledge the failure: \"לא הצלחתי לקרוא את המסמך\" without inventing content\n - **Failure mode:** Shira tries to provide analysis anyway (\"לפי מה שראיתי...\")\n\n3. **Test memory isolation:**\n - In conversation A: successfully read a multi-page document, ask Shira to summarize\n - In conversation B (new conversation, same case): ask about the same document but DON'T call read_document\n - **Expected:** Shira should say \"אני צריכה לקרוא את המסמך קודם\" or call read_document herself\n - **Failure mode:** Shira quotes from memory (\"כפי שראינו קודם...\")\n\n4. **Verify Hebrew error message:**\n - Trigger any of the above scenarios\n - **Expected:** Error message should be professional Hebrew: \"לא הצלחתי לחלץ תוכן משמעותי מהמסמך\"\n - **Not:** English errors or technical jargon exposed to user\n\n5. **Verify no regression:**\n - Test with a normal document (>1000 chars of real text)\n - **Expected:** Shira should read, summarize, and analyze normally\n - **Failure mode:** Shira becomes overly cautious and refuses to summarize even valid documents\n\n## Automated Testing (future)\n\nAdd a test case to `tests/test_prompt_builder.py` (if it exists) that:\n1. Builds a case prompt with the new rule\n2. Asserts the TOOL_RULES string contains \"DOCUMENT CONTENT INTEGRITY\"\n3. Asserts it contains the Hebrew failure message\n\n## Production Validation\n\nAfter deployment to prod:\n1. Monitor Mattermost #סוכנים-לוגים for any hallucination reports\n2. Check EspoCRM SmartAssistant logs for cases where read_document returned <50 chars\n3. Verify agent responses in those cases included \"לא הצלחתי לחלץ\"", "status": "done", "dependencies": [ "6" ], "priority": "high", "subtasks": [], "updatedAt": "2026-05-26T13:37:20.194Z" }, { "id": "8", "title": "fix(SmartAssistant): DocumentAnalyzer fails to extract text from valid DOCX — investigate and fix extractFullText", "description": "The SmartAssistant extension's DocumentAnalyzer::extractFullText returns null on standard DOCX files (e.g., 50KB Friedman appeal), causing unnecessary OCR fallback. Add detailed logging to identify the failure mode in the PHP extraction logic, then fix the actual extraction to handle word/document.xml properly.", "details": "## Problem Context\n\nThe **SmartAssistant EspoCRM extension** (repo: `espocrm-extensions/SmartAssistant`) contains a PHP service `DocumentAnalyzer` with an `extractFullText` method that should extract text from DOCX files. Currently it returns `null` on valid DOCX files, forcing shira-hermes to fall back to OCR (which only extracts images, not text — Task 5's scope).\n\n**Call chain:**\n1. `mcp_server/tools/document_tools.py:124` → `crm.post(\"SmartAssistant/action/readDocument\", {\"filePath\": ...})`\n2. SmartAssistant PHP extension → `DocumentAnalyzer::extractFullText()`\n3. Returns `{\"success\": false}` or `{\"success\": true, \"text\": null}`\n4. shira-hermes falls back to `_ocr_fallback` → `api/services/ocr.py:ocr_docx_images` → only extracts images (Task 5 proven this works)\n\n**Evidence:**\n- DOCX file `הודעה לבית הדין-כתב ערעור-973.docx` (~50KB) from case 46 on prod EspoCRM\n- The file **is valid** — we successfully extracted `word/document.xml` in 3 lines of Python (see Task 5's context)\n- The PHP `extractFullText` method is returning null, not the Python OCR layer\n\n## Implementation Steps\n\n### 1. Locate the PHP Code\nFile: `espocrm-extensions/SmartAssistant/files/custom/Espo/Modules/SmartAssistant/Services/DocumentAnalyzer.php`\n\nExpected method signature:\n```php\npublic function extractFullText(string $filePath): ?string\n```\n\n### 2. Add Detailed Logging\n\nBefore fixing, instrument the method to identify the failure mode:\n\n```php\nuse Espo\\Core\\Utils\\Log;\n\npublic function extractFullText(string $filePath): ?string\n{\n $GLOBALS['log']->info(\"DocumentAnalyzer: extractFullText called with filePath={$filePath}\");\n \n // Check file exists\n if (!file_exists($filePath)) {\n $GLOBALS['log']->error(\"DocumentAnalyzer: file not found at {$filePath}\");\n return null;\n }\n \n $size = filesize($filePath);\n $mime = mime_content_type($filePath);\n $GLOBALS['log']->info(\"DocumentAnalyzer: file size={$size}, mime={$mime}\");\n \n // Existing DOCX extraction logic...\n try {\n $zip = new \\ZipArchive();\n $opened = $zip->open($filePath);\n \n if ($opened !== true) {\n $GLOBALS['log']->error(\"DocumentAnalyzer: ZipArchive::open failed with code={$opened}\");\n return null;\n }\n \n $xml = $zip->getFromName('word/document.xml');\n if ($xml === false) {\n $GLOBALS['log']->error(\"DocumentAnalyzer: word/document.xml not found in archive\");\n $GLOBALS['log']->debug(\"DocumentAnalyzer: available files=\" . json_encode($this->listZipContents($zip)));\n $zip->close();\n return null;\n }\n \n $GLOBALS['log']->info(\"DocumentAnalyzer: word/document.xml extracted, length=\" . strlen($xml));\n \n // Parse XML and extract text...\n // (Add logging at each stage: XML parsing, text node extraction, final text length)\n \n } catch (\\Exception $e) {\n $GLOBALS['log']->error(\"DocumentAnalyzer: exception during extraction: \" . $e->getMessage());\n return null;\n }\n}\n\nprivate function listZipContents(\\ZipArchive $zip): array\n{\n $files = [];\n for ($i = 0; $i < $zip->numFiles; $i++) {\n $files[] = $zip->getNameIndex($i);\n }\n return $files;\n}\n```\n\n### 3. Test with Production DOCX\n\n- Deploy instrumented version to dev EspoCRM\n- Call `SmartAssistant/action/readDocument` via shira-hermes with the Friedman appeal file\n- Check `data/logs/espo-YYYY-MM-DD.log` for the new log lines\n- Identify the exact failure point (ZipArchive open? XML parsing? Text extraction?)\n\n### 4. Fix the Extraction Logic\n\nCommon failure modes and fixes:\n\n**A. ZipArchive::open fails:**\n- Check PHP `zip` extension is enabled: `php -m | grep zip`\n- Check file permissions: `ls -la `\n- Try opening with `ZIPARCHIVE::RDONLY` flag explicitly\n\n**B. word/document.xml not found:**\n- List archive contents (via `listZipContents` helper) — maybe different structure (e.g., LibreOffice uses different paths)\n- Fallback to alternative paths: `word/document2.xml`, `content.xml`\n\n**C. XML parsing fails:**\n- Check for malformed XML (rare for standard DOCX)\n- Use `libxml_use_internal_errors(true)` to capture errors\n- Log `libxml_get_errors()`\n\n**D. Text extraction returns empty:**\n- DOCX XML uses namespaces (`` tags inside `` paragraphs)\n- Ensure namespace handling:\n```php\n$doc = new \\DOMDocument();\n$doc->loadXML($xml);\n$xpath = new \\DOMXPath($doc);\n$xpath->registerNamespace('w', 'http://schemas.openxmlformats.org/wordprocessingml/2006/main');\n$textNodes = $xpath->query('//w:t');\n$paragraphs = [];\nforeach ($textNodes as $node) {\n $paragraphs[] = $node->nodeValue;\n}\nreturn implode(\"\\n\", $paragraphs);\n```\n\n### 5. Reference Implementation (Python — from Task 5)\n\nThe Python version that works (for comparison):\n```python\nimport zipfile\nimport re\n\nwith zipfile.ZipFile(docx_path) as z:\n xml = z.read('word/document.xml').decode('utf-8')\n text = re.sub('<[^>]+>', '', xml) # Strip all XML tags\n text = re.sub(r'\\s+', ' ', text).strip() # Normalize whitespace\n return text\n```\n\nThe PHP version should achieve the same: extract `word/document.xml`, strip tags, normalize whitespace.\n\n## Dependencies\n\nDepends on **Task 5** (fix ocr.py to extract text from word/document.xml) because:\n- Task 5 validates the DOCX structure and proves the file is valid\n- Task 5's Python extraction provides a working reference implementation\n- Fixing PHP first doesn't unblock anything (the Python OCR fallback still needs Task 5 to work)\n- The holistic fix requires both: PHP extraction (this task) + Python OCR fallback (Task 5)", "testStrategy": "## Testing Strategy\n\n### 1. Manual Testing on Dev Environment\n\n**Prerequisites:**\n- Access to dev EspoCRM: https://espocrm.dev.marcus-law.co.il\n- The production DOCX file from case 46: `הודעה לבית הדין-כתב ערעור-973.docx` (~50KB)\n- SSH access to dev server to tail logs\n\n**Test Steps:**\n\n1. **Deploy instrumented version:**\n - Add logging to `DocumentAnalyzer.php` (step 2 above)\n - Release SmartAssistant extension to dev EspoCRM\n - Verify extension installed: Administration → Extensions\n\n2. **Upload test DOCX:**\n - Open any case on dev EspoCRM\n - Upload `הודעה לבית הדין-כתב ערעור-973.docx` via Documents entity\n - Note the file path (e.g., `/data/upload/documents/abc123.docx`)\n\n3. **Call via shira-hermes:**\n - POST to shira-hermes dev: https://shira.dev.marcus-law.co.il/api/smart-assistant/chat\n - Message: \"תקרא את המסמך הודעה לבית הדין-כתב ערעור-973.docx\"\n - This triggers `read_document` tool → `SmartAssistant/action/readDocument` → `DocumentAnalyzer::extractFullText`\n\n4. **Check logs:**\n ```bash\n ssh chaim@192.168.10.206\n tail -f /data/coolify/applications/ew98wn7sbn1oh3mayk6x9ur9-232618564624/data/logs/espo-$(date +%Y-%m-%d).log | grep DocumentAnalyzer\n ```\n - Look for the new log lines: file exists? ZipArchive opened? XML extracted? Text extracted?\n - Identify the exact failure point\n\n5. **Fix and re-test:**\n - Implement fix based on logs (step 4 above)\n - Release updated extension\n - Repeat step 3\n - **Expected:** `read_document` returns the actual document text (not falls back to OCR)\n - **Expected log:** `DocumentAnalyzer: word/document.xml extracted, length=`, no errors\n\n### 2. Automated Test (Optional)\n\nCreate a PHPUnit test in SmartAssistant extension:\n\n```php\n// tests/unit/Espo/Modules/SmartAssistant/Services/DocumentAnalyzerTest.php\n\npublic function testExtractFullTextFromValidDocx(): void\n{\n $analyzer = $this->getContainer()->get('SmartAssistant\\\\Services\\\\DocumentAnalyzer');\n \n $testFile = 'tests/fixtures/sample-appeal.docx'; // Copy the Friedman file here\n $text = $analyzer->extractFullText($testFile);\n \n $this->assertNotNull($text, 'extractFullText should not return null for valid DOCX');\n $this->assertStringContainsString('ערעור', $text, 'Should extract Hebrew text');\n $this->assertGreaterThan(1000, strlen($text), 'Should extract substantial text (>1000 chars)');\n}\n```\n\n### 3. Integration Test via shira-hermes\n\nAfter PHP fix is deployed, verify the end-to-end flow works:\n\n1. **Call read_document tool** (not read_document_ocr) on the test DOCX\n2. **Expected result:** Returns document text directly, no OCR fallback\n3. **Verify in logs:** No `[ocr]` log lines from `api/services/ocr.py`\n4. **Verify response time:** Should be <1 second (PHP extraction is fast, OCR takes 5-10s)\n\n### 4. Regression Test\n\nTest with multiple DOCX types:\n- Small DOCX (1 page, <10KB)\n- Large DOCX (20+ pages, >1MB)\n- DOCX with images + text\n- DOCX with only text (no images)\n- DOCX created by different tools (MS Word, LibreOffice, Google Docs exported)\n\n**Expected:** All should extract text successfully, no null returns.\n\n### 5. Success Criteria\n\n- ✅ `DocumentAnalyzer::extractFullText` returns non-null text for the Friedman appeal DOCX\n- ✅ No errors in `espo-YYYY-MM-DD.log` during extraction\n- ✅ shira-hermes `read_document` returns text without falling back to OCR\n- ✅ Response time <1 second (vs 5-10s for OCR fallback)\n- ✅ Extracted text length >1000 characters (the appeal is substantial)\n- ✅ Extracted text contains Hebrew content (verify with grep or manual inspection)", "status": "done", "dependencies": [ "5" ], "priority": "medium", "subtasks": [], "updatedAt": "2026-05-26T13:41:18.508Z" }, { "id": "9", "title": "chore(prod-cleanup): remove fabricated data from case 46 Friedman in prod EspoCRM", "description": "Remove Shira-hallucinated data from production case 6a02d01e3c3ab6333 (case 46, Friedman): delete fabricated save_memory Note entries about knee injury/surgery/committee dates, delete or correct the Call dated 2026-05-12, delete or correct the Meeting dated 2026-05-17, delete the Task 'הגשת ערעור' with wrong dateEnd 2026-07-06, and verify/correct any auto-set case fields.", "details": "## Problem Context\n\nShira hallucinated case details for case 46 (Friedman) and wrote the fabricated data to production EspoCRM via the `save_memory`, `create_call`, `create_meeting`, and `create_task` tools. This poisoned production data and must be cleaned up.\n\n**Affected case:**\n- **Case ID:** `6a02d01e3c3ab6333`\n- **Case Number:** 46\n- **Name:** Friedman\n- **EspoCRM instance:** Production (`https://crm.prod.marcus-law.co.il`)\n\n## Data Cleanup Required\n\n### 1. Delete fabricated `save_memory` Note entries\n\n**Tool:** `save_memory` (crm_tools.py:180-192) creates a Note entity with `type=\"Post\"`, `post=\"🧠 {category}: {content}\"`, linked to the case.\n\n**Search strategy:**\n- Use EspoCRM API: `GET /api/v1/Note?where[0][type]=equals&where[0][attribute]=parentId&where[0][value]=6a02d01e3c3ab6333&maxSize=100`\n- Filter Notes with `post` starting with \"🧠\" (the memory prefix)\n- Identify entries mentioning:\n - Knee injury / פגיעה בברך\n - Surgery / ניתוח\n - 0% disability rating / 0% נכות\n - Committee date 31.3.2026 / ועדה 31.3.2026\n \n**Delete:** Use the `delete_note` tool or direct API call `DELETE /api/v1/Note/{noteId}` for each fabricated entry.\n\n### 2. Delete or correct Call dated 2026-05-12\n\n**Tool:** `create_call` (crm_tools.py:98-123) creates a Call entity with `dateStart`, `name`, `description`, linked to the case.\n\n**Search strategy:**\n- Use EspoCRM API: `GET /api/v1/Call?where[0][type]=equals&where[0][attribute]=parentId&where[0][value]=6a02d01e3c3ab6333&maxSize=50`\n- Find Call with `dateStart` = \"2026-05-12\" (or similar normalized format)\n\n**Action:**\n- If the Call is entirely fabricated (no real phone conversation happened on that date), **delete** via `DELETE /api/v1/Call/{callId}`\n- If the Call date is wrong but a conversation did happen, **update** via `PUT /api/v1/Call/{callId}` with correct `dateStart` and `description`\n\n### 3. Delete or correct Meeting dated 2026-05-17\n\n**Tool:** `create_meeting` (crm_tools.py:79-96) creates a Meeting entity with `dateStart`, `name`, `description`, linked to the case.\n\n**Search strategy:**\n- Use EspoCRM API: `GET /api/v1/Meeting?where[0][type]=equals&where[0][attribute]=parentId&where[0][value]=6a02d01e3c3ab6333&maxSize=50`\n- Find Meeting with `dateStart` = \"2026-05-17\"\n\n**Action:**\n- If the Meeting is entirely fabricated, **delete** via `DELETE /api/v1/Meeting/{meetingId}`\n- If the Meeting date is wrong but a meeting did happen, **update** via `PUT /api/v1/Meeting/{meetingId}`\n\n### 4. Delete Task 'הגשת ערעור' with wrong dateEnd 2026-07-06\n\n**Tool:** `create_task` (crm_tools.py:40-56) creates a Task entity with `name`, `dateEnd`, `priority`, linked to the case.\n\n**Search strategy:**\n- Use EspoCRM API: `GET /api/v1/Task?where[0][type]=equals&where[0][attribute]=parentId&where[0][value]=6a02d01e3c3ab6333&maxSize=50`\n- Find Task with `name` containing \"הגשת ערעור\" AND `dateEnd` = \"2026-07-06\"\n\n**Delete:** `DELETE /api/v1/Task/{taskId}`\n\n### 5. Verify and correct Case entity fields\n\n**Risk:** `generate_initial_report` or other tools may have auto-set case fields to wrong values (status, priority, nextHearingAt, urgency, etc.).\n\n**Verification:**\n- Fetch the case: `GET /api/v1/Case/6a02d01e3c3ab6333`\n- Check these fields:\n - `status` — should not be incorrectly set to PendingHearing, PendingDecision, etc.\n - `cNextHearing` / `nextHearingAt` — should not contain fabricated hearing date (e.g., 2026-05-17)\n - `priority` / `urgency` — should not be incorrectly escalated\n - `description` / case notes — should not contain hallucinated facts\n\n**Correction:**\n- If any fields are wrong, use `PUT /api/v1/Case/6a02d01e3c3ab6333` with correct values\n- If uncertain about correct values, consult the user (Chaim) before modifying\n\n## Implementation Steps\n\n1. **Set up EspoCRM client:**\n - Use `mcp_server/espocrm_client.py:EspoCrmClient`\n - Credentials from Infisical `/espocrm` (prod environment): `ESPOCRM_URL`, `ESPOCRM_API_KEY`\n - Instantiate: `crm = EspoCrmClient(base_url=prod_url, api_key=prod_key)`\n\n2. **Create cleanup script:**\n - New file: `scripts/cleanup_case_46_friedman.py`\n - Import `EspoCrmClient`, `asyncio`, `json`, `logging`\n - Load credentials from environment or Infisical MCP\n - Implement async functions for each cleanup task (1-5 above)\n - Add dry-run mode (`--dry-run` flag) that prints what would be deleted/updated without making changes\n\n3. **Execute cleanup:**\n - Run script in dry-run mode first to review changes: `python scripts/cleanup_case_46_friedman.py --dry-run`\n - Review output with user (Chaim)\n - Run script in live mode: `python scripts/cleanup_case_46_friedman.py`\n - Log all deleted/updated entity IDs for audit trail\n\n4. **Verify cleanup:**\n - Re-fetch case and related entities\n - Confirm no fabricated data remains\n - Document cleanup in `.taskmaster/tasks/tasks.json` notes\n\n## API Reference\n\n**EspoCRM REST API endpoints used:**\n- `GET /api/v1/{EntityType}?where[...]` — Search entities with filters\n- `GET /api/v1/{EntityType}/{id}` — Fetch single entity\n- `DELETE /api/v1/{EntityType}/{id}` — Delete entity\n- `PUT /api/v1/{EntityType}/{id}` — Update entity fields\n\n**Authentication:** `X-Api-Key: ` header\n\n**Existing tools reference:**\n- `mcp_server/espocrm_client.py` — HTTP client wrapper\n- `mcp_server/tools/crm_tools.py` — Tool implementations (reference for entity structure)\n- `mcp_server/tools/_helpers.py` — `normalize_date()` helper (if needed for date comparisons)\n\n## Dependencies\n\nThis task depends on Task 4 because:\n- Task 4 added the CRM tools (`save_memory`, `create_call`, `create_meeting`, `create_task`) that were used to create the fabricated data\n- Understanding the tool schemas (parameters, entity structure) from Task 4's implementation is necessary to locate and delete the fabricated entities\n- The cleanup script will use the same `EspoCrmClient` and API patterns established in Task 4's `crm_tools.py`", "testStrategy": "## Manual Testing Strategy\n\n### Prerequisites\n- Access to production EspoCRM: `https://crm.prod.marcus-law.co.il`\n- EspoCRM API key from Infisical `/espocrm` (prod environment)\n- SSH access to prod server (if testing via deployed shira-hermes container) OR local environment with Infisical MCP\n\n### Test Plan\n\n#### 1. Pre-cleanup Verification (Baseline)\n**Objective:** Document the current fabricated data state before cleanup.\n\n**Steps:**\n1. Log in to prod EspoCRM web UI\n2. Navigate to case 46 (Friedman, ID `6a02d01e3c3ab6333`)\n3. Document all Notes, Calls, Meetings, Tasks linked to the case:\n - Take screenshots\n - Note entity IDs\n - Export as CSV if possible\n4. Record current case field values:\n - Status\n - cNextHearing / nextHearingAt\n - Priority / urgency\n - Description\n\n**Expected fabricated data to find:**\n- Note(s) with \"🧠\" prefix mentioning knee injury, surgery, 0%, 31.3.2026\n- Call dated 2026-05-12\n- Meeting dated 2026-05-17\n- Task \"הגשת ערעור\" with dateEnd 2026-07-06\n\n#### 2. Dry-Run Execution\n**Objective:** Verify the script identifies correct entities without making changes.\n\n**Steps:**\n1. Run cleanup script in dry-run mode:\n ```bash\n cd /home/chaim/espocrm-extensions/shira-hermes\n python scripts/cleanup_case_46_friedman.py --dry-run\n ```\n2. Review console output — script should list:\n - Note IDs to delete (with `post` content preview)\n - Call ID(s) to delete/update (with date and name)\n - Meeting ID(s) to delete/update (with date and name)\n - Task ID(s) to delete (with name and dateEnd)\n - Case fields to update (if any)\n3. Compare output against baseline (step 1) — confirm IDs match fabricated data\n4. **STOP if output includes legitimate data** — refine script filters\n\n#### 3. Live Cleanup Execution\n**Objective:** Delete fabricated data from production.\n\n**Steps:**\n1. Get user approval (Chaim) for dry-run output\n2. Run cleanup script in live mode:\n ```bash\n python scripts/cleanup_case_46_friedman.py\n ```\n3. Review console output — script should log each deleted/updated entity ID\n4. Confirm no errors (HTTP 404, 403, 500, etc.)\n\n#### 4. Post-cleanup Verification\n**Objective:** Confirm fabricated data is removed and legitimate data remains intact.\n\n**Steps:**\n1. Log in to prod EspoCRM web UI\n2. Navigate to case 46 (Friedman, ID `6a02d01e3c3ab6333`)\n3. Verify:\n - **Notes:** No \"🧠\" entries mentioning knee injury, surgery, 0%, 31.3.2026\n - **Calls:** No Call dated 2026-05-12 (unless it was a legitimate call that was corrected)\n - **Meetings:** No Meeting dated 2026-05-17 (unless legitimate and corrected)\n - **Tasks:** No Task \"הגשת ערעור\" with dateEnd 2026-07-06\n - **Case fields:** Status, nextHearing, priority, description are correct (not fabricated)\n4. Verify legitimate data was NOT deleted:\n - Check case history/stream for any recent deletions\n - Confirm expected Notes, Calls, Meetings, Tasks are still present\n5. Take screenshots of cleaned-up case\n\n#### 5. API-Level Verification\n**Objective:** Confirm cleanup via direct API queries.\n\n**Steps:**\n1. Use `curl` or Postman with prod API key to query:\n ```bash\n # Notes\n curl -H \"X-Api-Key: $ESPOCRM_API_KEY\" \\\n \"https://crm.prod.marcus-law.co.il/api/v1/Note?where[0][type]=equals&where[0][attribute]=parentId&where[0][value]=6a02d01e3c3ab6333&maxSize=100\"\n \n # Calls\n curl -H \"X-Api-Key: $ESPOCRM_API_KEY\" \\\n \"https://crm.prod.marcus-law.co.il/api/v1/Call?where[0][type]=equals&where[0][attribute]=parentId&where[0][value]=6a02d01e3c3ab6333&maxSize=50\"\n \n # Meetings\n curl -H \"X-Api-Key: $ESPOCRM_API_KEY\" \\\n \"https://crm.prod.marcus-law.co.il/api/v1/Meeting?where[0][type]=equals&where[0][attribute]=parentId&where[0][value]=6a02d01e3c3ab6333&maxSize=50\"\n \n # Tasks\n curl -H \"X-Api-Key: $ESPOCRM_API_KEY\" \\\n \"https://crm.prod.marcus-law.co.il/api/v1/Task?where[0][type]=equals&where[0][attribute]=parentId&where[0][value]=6a02d01e3c3ab6333&maxSize=50\"\n ```\n2. Confirm API responses do NOT contain fabricated data\n3. Optionally: diff pre-cleanup and post-cleanup API responses\n\n#### 6. Audit Trail\n**Objective:** Document cleanup for future reference.\n\n**Steps:**\n1. Save cleanup script console output to file: `cleanup_case_46_friedman_YYYY-MM-DD.log`\n2. Update Task Master notes with:\n - Deleted entity IDs\n - Updated entity IDs (if any)\n - Verification screenshots\n3. Commit log file to git (optional, if repo is private)\n\n## Acceptance Criteria\n- [ ] All fabricated \"🧠\" memory Notes deleted\n- [ ] Fabricated Call (2026-05-12) deleted or corrected\n- [ ] Fabricated Meeting (2026-05-17) deleted or corrected\n- [ ] Fabricated Task \"הגשת ערעור\" (2026-07-06) deleted\n- [ ] Case entity fields verified correct (no fabricated status/dates)\n- [ ] No legitimate data was deleted (confirmed via case history)\n- [ ] Cleanup logged and auditable\n- [ ] Script can be reused for future cleanup tasks (generic where possible)", "status": "done", "dependencies": [ "4" ], "priority": "high", "subtasks": [], "updatedAt": "2026-05-26T13:50:11.559Z" } ], "metadata": { "version": "1.0.0", "lastModified": "2026-05-26T13:50:11.559Z", "taskCount": 9, "completedCount": 6, "tags": [ "master" ] } } }