the cost would be depending on GPU type/serving system/traffic pattern. check out some throughput comparison from vllm's blog post https://vllm.ai/ if you serve 7B on cost-optimized GPUs (A10G/L4) and keep it busy, it can be a lot cheaper than gpt3.5 turbo. tho it's not a fair comparison as 3.5's quality is still far better.

Great reference!

Just want to add about hosting your own LLM vs using ChatGPT. Cost is definitely a thing to consider, but it also depends on whether it is ok to share the requests to your product with OpenAI.

Also, something you cannot do with ChatGPT is to custom it with your own data, such as internal documents, etc. As shown in the blog, the model trained by ourselves can easily know its identity.

say using A10G ~$1.2/hr and with full utilization on vllm 112 reqs/min => per req ~$0.00018 versus gpt-3.5 turbo $0.002 per 1k token

Quick question: what would you estimate the running cost of Llama 2 70b to be? (On GPU, and assuming maximum utilization)?

yeah, that's the real question here

leaderboard link: https://chat.lmsys.org/?leaderboard

Surprised to learn StableLM is worse than plain LLaMA. link to their leaderboard: leaderboard.lmsys.org

I’ve heard that it’s really bad for it’s size

I asked GPT4All one of Vicuna's benchmark questions:

"What if the Internet had been invented during the Renaissance period?"

Check out their responses: https://imgur.com/a/mPrdZ1W More questions here: https://vicuna.lmsys.org/eval/

Note: not an apple-to-apple comparison but that's the model checkpoint I found on their git repo.

See the original Vicuna post: https://news.ycombinator.com/item?id=35378683

just tried it worked okay for me tho.

> Can you name all of the soccer players whose names contain a Z

> Here are all the soccer players whose names contain a "Z" that I could find:

Zlatan Ibrahimovic Zinedine Zidane Zlatko Zahovic Zeljko Kalac Zoran Banovic Zlatko Arsovic Zoran Savic Zoran Jovanovic Zoran Djurdjic Zoran Radosavljevic Zoran Zekic Zoran Mirkovic Zoran Sretenovic Zoran Sosic Zoran Luburic Zoran Vulin Zlatan Ljubijankic Zoran Lekovic Zoran Culibrk Zoran Bojovic Please note that this list may not be exhaustive, and there may be other soccer players with names containing "Z" that are not included.

The Zoran repetition looked suspicious to me and from some quick googling of some of the ones in the middle of that I can't find "Zoran Vulin football" or soccer or "Luburic Zoran football" or soccer results. So it's less blantant but still seems to have lost the plot.