mirror of
https://github.com/yusufipk/dikte.git
synced 2026-09-11 10:56:10 +00:00
yusufipek
70bc4c16fa
llama.cpp counts the thinking towards max_tokens along with the answer it precedes, and the local ceiling was sized for the answer alone. Turning Thinking up therefore came out of the reply rather than being added to it, and on a short dictation the 512 floor is the whole budget, so the model spent it in the think block and came back with nothing to paste. Each rung of the ladder now carries its own budget, doubling from 256 at "minimal" to 8192 at "maximum", added on top of the answer's share rather than taken out of it. The rungs are small because cleanup is punctuation and locally every one of these tokens is also a second of somebody standing in front of the screen. "Off" keeps the old tight ceiling untouched, and an empty setting is given a middling amount, since a template that can think thinks by default and there is no way to ask which kind of model this is. The ceiling is also held under what the server was started with. Above the context it is not a ceiling at all: the runaway it exists to stop would run to the end of the context instead, which on CPU is minutes of waiting. The prompt keeps its share at two characters to the token, which is under any tokeniser's rate for natural language and so reserves too much rather than promising room that is not there. Separately, a reply cut off at somebody's ceiling was returned as if it were whole. Half a sentence looks like a cleaned-up transcript and is not one, so finish_reason is now read in both cleanup and chat. The callers already keep the transcript they started with, which is the better of the two. This one is not local-only: a hosted provider stopping at its own output limit was silently pasted the same way.