Hi, thanks for the project!
I’m reproducing the finance example (task_name=finer, mode=offline). The intermediate playbook/bulletpoints grow very quickly — >100K tokens before 300 examples — and then LLM calls start failing with rate limiting (likely due to prompt size with the entire playbook).
I also noticed curator doesn’t seem to support merging/deleting bulletpoints, so the playbook only grows.
Commands
Without Bulletpoint Analyzer (README):
python -m finance.run --task_name finer --mode offline --save_path results
With Bulletpoint Analyzer:
python -m finance.run --task_name finer --mode offline --save_path results --use_bulletpoint_analyzer
Questions
- Is
--use_bulletpoint_analyzer required to reproduce the reported finance results? (README doesn’t include it.)
- Is this rapid playbook growth expected, or a bug/misconfig?
- Curator ignores
--playbook_token_budget. Are there recommended settings/strategies to bound playbook size (max bulletpoints, summarization, truncation, etc.)?
Hi, thanks for the project!
I’m reproducing the
financeexample (task_name=finer,mode=offline). The intermediate playbook/bulletpoints grow very quickly — >100K tokens before 300 examples — and then LLM calls start failing with rate limiting (likely due to prompt size with the entire playbook).I also noticed
curatordoesn’t seem to support merging/deleting bulletpoints, so the playbook only grows.Commands
Without Bulletpoint Analyzer (README):
With Bulletpoint Analyzer:
Questions
--use_bulletpoint_analyzerrequired to reproduce the reportedfinanceresults? (README doesn’t include it.)--playbook_token_budget. Are there recommended settings/strategies to bound playbook size (max bulletpoints, summarization, truncation, etc.)?