1
$\begingroup$

A lot pf AI hardware coming out lately has its performance mentioned in TOPS i.e trillion operations per second.

Does anyone have an Idea how to estimate the llm performance on such hardware in tokens per second.

for example I have a hardware of 45 TOPS performance. if I perform inferencing of a 7 billion parameter model what performance would I get in tokens per second.

just to clarify even further there's another term going around called TFLOPS i.e trillion floating point operations per second (used for quite a lot of Nvidia hardware). I'm not asking about that. I'm asking specifically about TOPS i.e trillion operations per second.

$\endgroup$

2 Answers 2

1
$\begingroup$

You can find a lot written on the Internet on the number of tokens/second you can expect from different GPUs. I suggest you pick a specific LLM (e.g., Llama 2) and a specific GPU or hardware, and do some searching. Many people report their experience.

There is no simple formula. So you'll need to rely on the experience/measurements of others.

It is not just a function of operations per second. Rather, in many cases, memory bandwidth is one of the limiting factors. You should not expect any simple "conversion factor" to convert TOPs to tokens per second.

$\endgroup$
1
1
$\begingroup$

seem to have found something

https://www.qualcomm.com/news/onq/2023/10/rethink-whats-possible-with-new-snapdragon-x-elite-platform#:~:text=Snapdragon%20X%20Elite%20can%20efficiently,%2Dlatency%2C%20advanced%20multitasking%20experiences.

integrated Qualcomm Hexagon neural processor unit (NPU) has 45 TOPS

and runs 7 billion parameter models at roughly 30 tokens/sec.

Here's the details on how to calculate it in much more detail https://www.qualcomm.com/news/onq/2024/04/a-guide-to-ai-tops-and-npu-performance-metrics

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.