true I was hoping we could use maybe character.ai characters or something like that

the free chat install says server overlaod please try again later lmao :V

Maybe we could create a plugin/mod that lets us use llama.cpp or exllama locally, do you think it would be worth to investigate it further?

Originally posted by insomnyawolf:

Maybe we could create a plugin/mod that lets us use llama.cpp or exllama locally, do you think it would be worth to investigate it further?

idk what that is, but if it works then yes ofc

Originally posted by insomnyawolf:

Maybe we could create a plugin/mod that lets us use llama.cpp or exllama locally, do you think it would be worth to investigate it further?

LLaMa cpp python bindings already have server available through openai's module. Perhaps we can replace openai's with local

Originally posted by den0620:

Originally posted by insomnyawolf:

Maybe we could create a plugin/mod that lets us use llama.cpp or exllama locally, do you think it would be worth to investigate it further?

LLaMa cpp python bindings already have server available through openai's module. Perhaps we can replace openai's with local

I beg of you

It advanced a lot lately we could try nowadays, even ooba's uo has an "open ai" compatible api even with streaming.

And models/libraries have advanced a lot as well.

Exllama2 is nuts for speed, if you can load the whole model in VRAM it can answer in less than 5 seconds (but requieres modern nvidia gpus)

Llamacpp is a bit slower but can work anywhere (even without gpu but it will be slow) and use some system memory if vram is not enough.

Models deppends on what are you searching for, i have found out that nowadays there are models that can have a nice conversation with the user but they still narrate things and break sometimes.

If anyone of you is still really interested, send me a message anywhere and we can try to make it real

Originally posted by insomnyawolf:

It advanced a lot lately we could try nowadays, even ooba's uo has an "open ai" compatible api even with streaming.

And models/libraries have advanced a lot as well.

Exllama2 is nuts for speed, if you can load the whole model in VRAM it can answer in less than 5 seconds (but requieres modern nvidia gpus)

Llamacpp is a bit slower but can work anywhere (even without gpu but it will be slow) and use some system memory if vram is not enough.

Models deppends on what are you searching for, i have found out that nowadays there are models that can have a nice conversation with the user but they still narrate things and break sometimes.

If anyone of you is still really interested, send me a message anywhere and we can try to make it real

if you actually can that would be sick lol

Originally posted by insomnyawolf:

It advanced a lot lately we could try nowadays, even ooba's uo has an "open ai" compatible api even with streaming.

And models/libraries have advanced a lot as well.

Exllama2 is nuts for speed, if you can load the whole model in VRAM it can answer in less than 5 seconds (but requieres modern nvidia gpus)

Llamacpp is a bit slower but can work anywhere (even without gpu but it will be slow) and use some system memory if vram is not enough.

Models deppends on what are you searching for, i have found out that nowadays there are models that can have a nice conversation with the user but they still narrate things and break sometimes.

If anyone of you is still really interested, send me a message anywhere and we can try to make it real

Does ExLLaMa have a server? I tried to move from llama.cpp to exl as soon as got powerful enough gpu but couldnt find it

Connect the chatgpt mod to text-generation-webui:

https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API

ZERO cost. You can run one instance almost everywhere.