feat(kb): bulk-fetch תקנות הביטוח הלאומי from Wikisource

Extends scripts/fetch_national_insurance_law.py with a --regulations mode
that pulls all canonical (non-redirect) תקנות הביטוח הלאומי pages from the
MediaWiki API and writes <slug>.txt + <slug>.txt.meta.json pairs ready for
the existing scan-inbox cron.

Used to seed the live insurance KB with 92 regulations (~1,500 chunks,
~22MB Postgres) on dev. Wikisource's redirect graph dedups spelling
variants (קיצבה/קצבה, etc.) for free.

Bumps version to 0.2.0.

Refs Task Master #3

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-25 20:43:31 +00:00
parent bb23b8a120
commit 83a3f63a37
3 changed files with 134 additions and 8 deletions
+16 -4
View File
@@ -24,16 +24,28 @@
"priority": "high",
"subtasks": [],
"updatedAt": "2026-04-21T00:00:00.000Z"
},
{
"id": "3",
"title": "Add תקנות הביטוח הלאומי (~100 regulations) from Wikisource to KB",
"description": "Extend scripts/fetch_national_insurance_law.py to bulk-fetch ~100 regulation pages from Hebrew Wikisource via MediaWiki API. Output text + sidecar JSON pairs to /tmp/regulations/. Upload to MinIO inbox/regulation/ for the existing scan-inbox cron to embed via Voyage. Estimated 30-40MB Postgres growth, ~$0.36 embedding cost. Sample 3 first (small/medium/large), verify chunker output, then bulk.",
"details": "",
"testStrategy": "",
"status": "done",
"dependencies": [],
"priority": "high",
"subtasks": [],
"updatedAt": "2026-04-25T20:39:11.905Z"
}
],
"metadata": {
"version": "1.0.0",
"lastModified": "2026-04-21T00:00:00.000Z",
"taskCount": 2,
"completedCount": 0,
"lastModified": "2026-04-25T20:39:11.905Z",
"taskCount": 3,
"completedCount": 1,
"tags": [
"master"
]
}
}
}
}