We should be happy that compute is once again improving and machines are getting outdated rapidly. Which is better - a world where your laptop is competitive for 5+ years but everything stays the same? Or one where entire new realms of advancement open up every 18 months?
It’s a no contest option 2 for me.
Just use llama.cpp with any of the available UIs. It will be usable with 4 but quantization on CPU. You can use any of the “Q4_M” “GGUF” models that TheBloke puts out on Huggingface.
It’s a no contest option 2 for me.
Just use llama.cpp with any of the available UIs. It will be usable with 4 but quantization on CPU. You can use any of the “Q4_M” “GGUF” models that TheBloke puts out on Huggingface.
https://github.com/ggerganov/llama.cpp
UI projects in description.
https://huggingface.co/TheBloke
A closed source option is LMStudio.
https://lmstudio.ai/