textgen

What people said

• “Guanaco is indeed very capable and can replace GPT 3.5 in almost all scenarios, based on my tests. Easy way to self-host it is to use text-generation-webui and 33B 4-bit quantized GGML model from TheBloke.…”

• “Works on all platforms, but runs much better on Linux. Running this in Docker on my 2080Ti, can barely fit 13B-4bit models into 11G of VRAM, but it works fine, produces around 10-15 tokens/second most of the time. It also has an…”