espocrm-extensions/shira-hermes
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(kb/chunker): drop false section matches from PDF tables of contents

Browse Source 
ספר הליקויים produced 698 chunks, most of them junk. A line like
"3 . . . . . . . . . . . . . . . . . . . . . . . . . . הוראות החוק" from
its table of contents was being treated as section 3. Phone numbers like
"02-6709701" became section 02. Both of these polluted the index and
broke the browse view's readability.

Tighten _RE_SECTION so it requires either the explicit prefix (סעיף /
תקנה / §) OR real text immediately after the terminator — a row of dots
or more digits disqualifies. The scraped חוק הביטוח הלאומי still chunks
correctly because the scraper emits "סעיף N. ..." with the prefix.

Refs Task Master #2

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chaim Marcus
2026-04-24 09:45:09 +00:00
parent 13f2e14050
commit 1739fa1211
1 changed files with 14 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
api/services/kb/chunker.py
+14 -2
View File
@@ -32,9 +32,21 @@ _RE_PART = re.compile(rf"(?m)^\s*חלק\s+({_HEB_ORD})\s*[:\-–—]?\s*(.*?)\s*
_RE_CHAPTER = re.compile(rf"(?m)^\s*פרק\s+({_HEB_ORD})\s*[:\-–—]?\s*(.*?)\s*$") _RE_CHAPTER = re.compile(rf"(?m)^\s*פרק\s+({_HEB_ORD})\s*[:\-–—]?\s*(.*?)\s*$")
_RE_SUBCHAPTER = re.compile(rf"(?m)^\s*סימן\s+({_HEB_ORD})\s*[:\-–—]?\s*(.*?)\s*$") _RE_SUBCHAPTER = re.compile(rf"(?m)^\s*סימן\s+({_HEB_ORD})\s*[:\-–—]?\s*(.*?)\s*$")
# Section header: "סעיף 12.", "12א.", "תקנה 5.", at start of a line, followed by # Section header: "סעיף 12.", "12א.", "תקנה 5.", at start of a line, followed by
# optional sub-letter "(ב)" and then a dot / colon / dash or newline. # optional sub-letter "(ב)" and then a dot / colon / dash, then either EOL or
# real Hebrew/English text (NOT a row of dots / a phone number / more digits).
#
# The key change vs the original: the tail `[\.:\-–—]` must be followed by
# at least one letter within the same short lookahead. This filters out:
# - phone numbers like "02-6709701"
# - table-of-contents rows like "3 . . . . . . . . . . . . . . ."
# - page-number-only lines like "23."
# while still catching real section headers like "סעיף 12. הגדרות" or
# "195. חישוב קצבת נכות".
_RE_SECTION = re.compile( _RE_SECTION = re.compile(
r"(?m)^\s*(?:סעיף\s+|תקנה\s+|§\s*)?(\d+[א-ת]?)\s*(?:\(([א-ת\d]+)\))?\s*[\.:\-–—]" r"(?m)^\s*(?:סעיף\s+|תקנה\s+|§\s*)" # prefix is now REQUIRED (one of סעיף / תקנה / §)
r"(\d+[א-ת]?)\s*(?:\(([א-ת\d]+)\))?"
r"\s*[\.:\-–—]\s*"
r"(?=[^\d\.\s])" # next non-space char must not be digit/dot — i.e. real text follows
) )
_STATUTE_MARKERS = [ _STATUTE_MARKERS = [