Each file in the inbox can now be accompanied by a `<filename>.meta.json`
sidecar that supplies identifier, published_at, effective_at, and
source_url. The scanner applies filename-derived defaults first and lets
the sidecar override individual fields.
- list_inbox / list_failed skip `.meta.json` so sidecars aren't ingested
as standalone documents.
- mark_processed, mark_failed, and requeue-failed move the sidecar
alongside its parent (best-effort).
- mark_failed now sanitizes the error string to ASCII before writing it
to S3 object metadata (HTTP header encoding).
- search SQL selects published_at/effective_at; the formatter shows the
full heading_path plus publication date so citations are unambiguous.
- scripts/kb_sidecar_template.json documents the expected shape.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-shot script that fetches the consolidated National Insurance Law from
Hebrew Wikisource and emits plain text with heading lines the chunker can
parse (פרק / סימן / סעיף).
Key quirks of the source markup:
- {{ח:קטע2}} carries the level name inside the title parameter ("פרק א׳:
…"), not in the template name. Emit the title verbatim and let the
chunker's existing prefix regex handle it.
- Named args (e.g. 'אחר=[פרק א׳]') must be filtered — the original
positional parser was treating them as positional titles and producing
gibberish "חלק אחר=[פרק א׳]" headings.
- No חלק level exists in this law — only פרק > סימן > סעיף.
Run as: python3 scripts/fetch_national_insurance_law.py <out_path>
Then upload the output .txt to s3://insurance-kb/inbox/law/ and invoke
POST /admin/kb/scan-inbox.
Refs Task Master #2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chaim