Give the memory back when a local model has been sitting unused

A whisper.cpp or llama.cpp server started for one dictation stayed loaded
until Dikte quit. On this machine that is 1.3 GB of VRAM for large-v3 plus
whatever the cleanup LLM takes, held all day between dictations that last
seconds.

Each Server now carries an idle window. A watcher thread per launch stops the
server once nothing has asked it anything for that long, and the next request
loads it again through serve(), which already starts what is not running.
Settings has one checkbox and one number for both servers, on by default at ten
minutes, and it only appears for a machine that runs a model here. The tray menu
says which models are loaded and offers to unload them now.

Two things the clock alone gets wrong, both held off by a count of requests in
flight:

  * A file or a meeting is one address lookup and then minutes of work, which
    to a clock started at the lookup looks exactly like a model nobody wants.
    api.py and cleanup.py hold the count for the length of the request.

  * The count must survive the start it triggered. cleanup._local takes the
    hold and only then asks for the address, so a cold start happens inside it;
    neither serve() nor _stop_now() resets the count any more.

Unloading by hand runs on the interface's thread, so it asks for the start lock
rather than waiting on it: a model still being read in is refused, the way one
in the middle of a request is, instead of freezing the window for as long as
the load takes.
This commit is contained in:
2026-09-05 12:15:14 +03:00
parent b13b08fc38
commit 825f089fe9
13 changed files with 444 additions and 17 deletions
+22
View File
@@ -83,6 +83,8 @@ CHANGED = {
"local_llm_gpu": False,
"local_llm_preload": True,
"local_llm_reasoning": "low",
"local_idle_unload": False,
"local_idle_minutes": 45,
"cleanup_prompt": "Only fix the punctuation.",
"file_cleanup_prompt": "Keep the stamps where they are.",
"transcribe_prompt": "Paraşüt, OpenFrame",
@@ -1701,6 +1703,26 @@ class LocalModels(DikteTest):
# Its own thinking box, because the two default to opposite things.
self.assertFalse(window.cleanup_form.isRowVisible(window.cleanup_reasoning))
def test_the_idle_unload_is_offered_to_whoever_runs_a_model_here(self):
for transcriber, cleaner in (("local", "openrouter"),
("openai", "local"),
("local", "local")):
with self.subTest(transcriber=transcriber, cleaner=cleaner):
window = self.window(self.config(transcribe_provider=transcriber,
cleanup_provider=cleaner))
self.assertTrue(window.local_box.isVisibleTo(window))
def test_a_machine_that_runs_neither_is_not_asked_about_memory(self):
window = self.window(self.config(transcribe_provider="openai",
cleanup_provider="openrouter"))
self.assertFalse(window.local_box.isVisibleTo(window))
def test_the_minutes_follow_the_checkbox(self):
window = self.window(self.config(local_idle_unload=False))
self.assertFalse(window.local_idle_minutes.isEnabled())
window.local_idle_unload.setChecked(True)
self.assertTrue(window.local_idle_minutes.isEnabled())
def test_each_cleaner_brings_its_own_model_row_and_no_other(self):
window = self.window(cfg.Config())
rows = {"openrouter": window.cleanup_model_row,