fix(kb/chunker): strip stray \x00 bytes from chunk content

_split_oversized slices at fixed char offsets and can cut a
\x00KB_PAGE:N\x00 sentinel in half — the broken prefix/suffix no longer
matches _PAGE_MARKER_RE so _strip_page_markers leaves the \x00 byte in
place. PostgreSQL's UTF-8 check then rejects the insert with
"invalid byte sequence for encoding UTF8: 0x00", and ingest of
ספר הליקויים fails on every scan. Belt-and-suspenders: strip all
remaining \x00 bytes from clean content before returning.

Refs Task Master #2
This commit is contained in:
2026-04-24 15:45:41 +00:00
parent e534709706
commit 1ca1cc0050
+4
View File
@@ -128,6 +128,10 @@ def _as_dicts(chunks: list[Chunk]) -> list[dict]:
result: list[dict] = []
for c in chunks:
clean, page_in_body = _strip_page_markers(c.content)
# _split_oversized may slice a piece in the middle of a marker
# sentinel, leaving a stray \x00 that PostgreSQL's UTF-8 check
# rejects. Drop any surviving null bytes.
clean = clean.replace("\x00", "")
# Prefer the page the caller computed from the chunk's START offset
# (the "where does this quote begin" answer). Only fall back to any
# marker inside the body if the caller didn't set one.