chaim
3948c387b5
The previous batching sent up to 128 inputs per request but didn't cap total tokens. On large Hebrew statutes (ספר הליקויים, ספר המבחנים) a single request exceeded Voyage's 120K-token-per-batch hard cap and failed with 400. - voyage.embed now pre-computes an estimated token count per input (Hebrew is ~0.55 tokens/char in voyage-multilingual-2) and emits multiple requests so each stays under a 90K-token safety margin. - chunker lowers the per-chunk ceiling to 1500 tokens and updates the chars→tokens estimate to match the observed tokenizer behavior. Refs Task Master #2 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>