Given how an inference model stresses a GPU so much more than any game does how when will site like this start using them to benchmark PCs you think?
Im thinking MLPerf Inference, HuggingFace Optimum etc
The thing about benchmarks is that they have to be scripted.
Run the same query on a inference model multiple times and the results will vary and thus the load on the system will not be perfectly predictable.
cyclone3d wrote on Today, 20:48:The thing about benchmarks is that they have to be scripted.
Run the same query on a inference model multiple times and the results will vary and thus the load on the system will not be perfectly predictable.
Thats why you use a Benchmark with a predictable outcome, just like game benchmarks do. Which is why I asked.
How are you going to make it have the same exact outcome every time though?
Inference means it infers things from the data it is fed.
I won't necessarily always come to the same conclusion or go through the same exact steps.
If it does, then it is not inference anymore.
Thats like saying you cant benchmark a game because in the game world you never get the exact same frame twice!
You can benchmark things like tokens per second, batch performance, memory efficiency, parallel requests, context window, VRAM usuage, model load times,
