mirror of
https://github.com/yusufipk/dikte.git
synced 2026-09-11 19:06:11 +00:00
Give the memory back when a local model has been sitting unused
A whisper.cpp or llama.cpp server started for one dictation stayed loaded
until Dikte quit. On this machine that is 1.3 GB of VRAM for large-v3 plus
whatever the cleanup LLM takes, held all day between dictations that last
seconds.
Each Server now carries an idle window. A watcher thread per launch stops the
server once nothing has asked it anything for that long, and the next request
loads it again through serve(), which already starts what is not running.
Settings has one checkbox and one number for both servers, on by default at ten
minutes, and it only appears for a machine that runs a model here. The tray menu
says which models are loaded and offers to unload them now.
Two things the clock alone gets wrong, both held off by a count of requests in
flight:
* A file or a meeting is one address lookup and then minutes of work, which
to a clock started at the lookup looks exactly like a model nobody wants.
api.py and cleanup.py hold the count for the length of the request.
* The count must survive the start it triggered. cleanup._local takes the
hold and only then asks for the address, so a cold start happens inside it;
neither serve() nor _stop_now() resets the count any more.
Unloading by hand runs on the interface's thread, so it asks for the start lock
rather than waiting on it: a model still being read in is refused, the way one
in the middle of a request is, instead of freezing the window for as long as
the load takes.
This commit is contained in:
@@ -689,3 +689,24 @@ class ReadyToRun(DikteTest):
|
||||
self.assertEqual(ggml.whisper.settings()["threads"], 4)
|
||||
self.assertFalse(ggml.whisper.settings()["gpu"])
|
||||
self.assertEqual(ggml.llm.settings()["context"], 4096)
|
||||
|
||||
def test_the_idle_window_is_in_seconds(self):
|
||||
conf = self.config(local_idle_unload=True, local_idle_minutes=15)
|
||||
self.assertEqual(conf.idle_seconds(), 900)
|
||||
|
||||
def test_an_unchecked_box_keeps_the_model(self):
|
||||
conf = self.config(local_idle_unload=False, local_idle_minutes=15)
|
||||
self.assertEqual(conf.idle_seconds(), 0)
|
||||
|
||||
def test_a_window_of_no_minutes_is_still_a_window(self):
|
||||
"""The spin box will not go below one; a config edited by hand can."""
|
||||
conf = self.config(local_idle_unload=True, local_idle_minutes=0)
|
||||
self.assertEqual(conf.idle_seconds(), 60)
|
||||
|
||||
def test_both_servers_are_told_the_window(self):
|
||||
conf = self.config(local_idle_unload=True, local_idle_minutes=3)
|
||||
self.addCleanup(ggml.llm.set_idle, 0)
|
||||
self.addCleanup(ggml.whisper.set_idle, 0)
|
||||
conf.apply_local()
|
||||
self.assertEqual(ggml.whisper.idle, 180)
|
||||
self.assertEqual(ggml.llm.idle, 180)
|
||||
|
||||
Reference in New Issue
Block a user