Repository navigation
explain: default deployment gpt-6.1-sol - #251
Merged
Merged
Conversation
Blind grading of 24 stored plans: luna 2.8, sol 4.25, astra 4.46 on a 1 to 5 scale for how well the answer explains why the plan looks the way it does. Sol costs a fifth of astra, about 0.02 USD per explanation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #250, which merged before this commit was pushed.
Default deployment
gpt-6.1-solinstead ofgpt-6-luna. The container app sets onlyOPTIMIZER_AZURE_OPENAI_ENDPOINTandOPTIMIZER_AZURE_OPENAI_KEY, so the default applies on deploy.Blind grading of the 24 stored plans with the #250 prompt, 1 to 5 for how well the answer explains why the plan looks the way it does: luna 2.79, sol 4.25, astra 4.46. Luna never scores 5 and keeps inventing default reasons regardless of the prompt; sol and astra had no wrong reasons. Head to head sol vs astra: 1 plan better, 6 worse, 17 tied, every loss by one point.
Cost (Global Standard, USD per 1M tokens, input / cache write / cached read / output): luna 0.12 / 0.15 / 0.012 / 0.60, sol 2 / 2.5 / 0.1 / 10, astra 10 / 12.5 / 1 / 50. With the 24h cached prefix an explanation is about 0.0015 USD on luna, 0.021 on sol and 0.105 on astra, at October's 131 explanations 0.19, 2.8 and 14 USD a month. Sol answered in 6.5 s median with 189 completion tokens and no reasoning tokens, within the 150 words and without dashes. One sol answer in the benchmark hit a 180 s read timeout; the service timeout stays at 45 s, such a request fails with 502 instead of hanging.
🤖 Generated with Claude Code