yusufipek/dikte
1
0
Fork 0
mirror of https://github.com/yusufipk/dikte.git synced 2026-09-11 10:56:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

Give the memory back when a local model has been sitting unused

Browse Source 
A whisper.cpp or llama.cpp server started for one dictation stayed loaded
until Dikte quit. On this machine that is 1.3 GB of VRAM for large-v3 plus
whatever the cleanup LLM takes, held all day between dictations that last
seconds.

Each Server now carries an idle window. A watcher thread per launch stops the
server once nothing has asked it anything for that long, and the next request
loads it again through serve(), which already starts what is not running.
Settings has one checkbox and one number for both servers, on by default at ten
minutes, and it only appears for a machine that runs a model here. The tray menu
says which models are loaded and offers to unload them now.

Two things the clock alone gets wrong, both held off by a count of requests in
flight:

  * A file or a meeting is one address lookup and then minutes of work, which
    to a clock started at the lookup looks exactly like a model nobody wants.
    api.py and cleanup.py hold the count for the length of the request.

  * The count must survive the start it triggered. cleanup._local takes the
    hold and only then asks for the address, so a cold start happens inside it;
    neither serve() nor _stop_now() resets the count any more.

Unloading by hand runs on the interface's thread, so it asks for the start lock
rather than waiting on it: a model still being read in is refused, the way one
in the middle of a request is, instead of freezing the window for as long as
the load takes.
This commit is contained in:
yusufipek
2026-09-05 12:15:14 +03:00
parent b13b08fc38
commit 825f089fe9
13 changed files with 444 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.tr.md
+2 -1
View File
@@ -158,7 +158,8 @@ olmasını ister.
whisper.cpp, temizleme llama.cpp üzerinde; ikisini de önceden kurman gerekmez:
ayarlar penceresi programı ve modeli indirir, sha256'sını doğrular,
checksum'suz yayınlanmış bir indirmeyi reddeder, sen dikte ettikçe sunucuyu
ayakta tutar. Model listesi dosya boyutuna değil modele göre gruplanır ve bu
ayakta tutar ve on dakika kullanılmayan modelin belleğini geri verir. Model
listesi dosya boyutuna değil modele göre gruplanır ve bu
makinenin belleğine ve ekran kartına uyan satır işaretlenir. Derleme
destekliyorsa ekran kartına CUDA, ROCm ya da Vulkan
üzerinden ulaşılır. Anahtar yok, hesap yok, makineden çıkan bir şey yok.