feat(kb): bulk-fetch תקנות הביטוח הלאומי from Wikisource
Extends scripts/fetch_national_insurance_law.py with a --regulations mode that pulls all canonical (non-redirect) תקנות הביטוח הלאומי pages from the MediaWiki API and writes <slug>.txt + <slug>.txt.meta.json pairs ready for the existing scan-inbox cron. Used to seed the live insurance KB with 92 regulations (~1,500 chunks, ~22MB Postgres) on dev. Wikisource's redirect graph dedups spelling variants (קיצבה/קצבה, etc.) for free. Bumps version to 0.2.0. Refs Task Master #3 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -24,16 +24,28 @@
|
||||
"priority": "high",
|
||||
"subtasks": [],
|
||||
"updatedAt": "2026-04-21T00:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "3",
|
||||
"title": "Add תקנות הביטוח הלאומי (~100 regulations) from Wikisource to KB",
|
||||
"description": "Extend scripts/fetch_national_insurance_law.py to bulk-fetch ~100 regulation pages from Hebrew Wikisource via MediaWiki API. Output text + sidecar JSON pairs to /tmp/regulations/. Upload to MinIO inbox/regulation/ for the existing scan-inbox cron to embed via Voyage. Estimated 30-40MB Postgres growth, ~$0.36 embedding cost. Sample 3 first (small/medium/large), verify chunker output, then bulk.",
|
||||
"details": "",
|
||||
"testStrategy": "",
|
||||
"status": "done",
|
||||
"dependencies": [],
|
||||
"priority": "high",
|
||||
"subtasks": [],
|
||||
"updatedAt": "2026-04-25T20:39:11.905Z"
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"version": "1.0.0",
|
||||
"lastModified": "2026-04-21T00:00:00.000Z",
|
||||
"taskCount": 2,
|
||||
"completedCount": 0,
|
||||
"lastModified": "2026-04-25T20:39:11.905Z",
|
||||
"taskCount": 3,
|
||||
"completedCount": 1,
|
||||
"tags": [
|
||||
"master"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user