Given how an inference model stresses a GPU so much more than any game does how when will site like this start using them to benchmark PCs you think?
Im thinking MLPerf Inference, HuggingFace Optimum etc
The thing about benchmarks is that they have to be scripted.
Run the same query on a inference model multiple times and the results will vary and thus the load on the system will not be perfectly predictable.
cyclone3d wrote on 2026-08-05, 20:48:The thing about benchmarks is that they have to be scripted.
Run the same query on a inference model multiple times and the results will vary and thus the load on the system will not be perfectly predictable.
Thats why you use a Benchmark with a predictable outcome, just like game benchmarks do. Which is why I asked.
How are you going to make it have the same exact outcome every time though?
Inference means it infers things from the data it is fed.
I won't necessarily always come to the same conclusion or go through the same exact steps.
If it does, then it is not inference anymore.
Thats like saying you cant benchmark a game because in the game world you never get the exact same frame twice!
You can benchmark things like tokens per second, batch performance, memory efficiency, parallel requests, context window, VRAM usuage, model load times,
The game benchmarks use scripted runs... Aka, the frames should be exactly the same every single run.
Yes, you could test those things, but how is that actually benching, in a repeatable way, the inference model itself? If the model takes different paths to come to a result, the speed at which it completes that exact prompt that was used for the bench, the time to completion may be wildly different from run to run.
And what if you have a mixed CPU and GPU setup where things can spill over to another asset if one or more run out of resources. If the RAM usage is different between runs, the assets used could also be different thus affecting the benchmark scores.
At the very least, you would need to take an average over many runs with th same exact prompt and any acquired "training" resulting from the prompt being run, erased between runs.
Maybe a shorter answer is "when that sort of benchmarking has some relevance to what the site is about".
cyclone3d wrote on 2026-08-06, 04:26:The game benchmarks use scripted runs... Aka, the frames should be exactly the same every single run. […]
The game benchmarks use scripted runs... Aka, the frames should be exactly the same every single run.
Yes, you could test those things, but how is that actually benching, in a repeatable way, the inference model itself? If the model takes different paths to come to a result, the speed at which it completes that exact prompt that was used for the bench, the time to completion may be wildly different from run to run.
And what if you have a mixed CPU and GPU setup where things can spill over to another asset if one or more run out of resources. If the RAM usage is different between runs, the assets used could also be different thus affecting the benchmark scores.
At the very least, you would need to take an average over many runs with th same exact prompt and any acquired "training" resulting from the prompt being run, erased between runs.
I don't know where to start on this one.
An inference model is already trained, that part is finished. When you run it locally, you’re not training anything, you’re just doing a forward pass through fixed weights. Nothing inside the model changes between runs.
If you have a mixed CPU/GPU setup because the GPU can’t hold the whole model, then that’s simply the hardware configuration you’re benchmarking. It doesn’t “test the card differently” depending on the output. The workload per token is identical.
An inference model isn’t like a game engine. Games have to process the same sequence of frames to make a benchmark meaningful, because the amount of work changes depending on what’s on screen. LLMs don’t work that way.
With an inference model, whether it’s text to image, image to image, or an LLM writing a 10 page fantasy in the style of Poe the content of the output might change, but the amount of computation per token does not. A image of a zebra or an ocean doesn’t change the math. A different sentence doesn’t change the math. The model always performs the same operations for each token.
So even if the output varies, the computational workload is the same, which is why tokens per second is a valid and repeatable benchmark.
ratfink wrote on 2026-08-06, 05:02:Maybe a shorter answer is "when that sort of benchmarking has some relevance to what the site is about".
Oh I'm very sorry... Next time I'll be sure to make sure it's about games and nothing but. But just for you!
For now in this thread maybe we can talk about benchmarking for inference models as they stress GPUs much much more than games do to the point where many GPUs that are still considered current for games played today are pitifully underpowered when considering an inference model.
Ok, looking up stuff more, it is possible to have a model get trained by the input prompts, but is not normally done.
After a bit of back and forth, this is what ChatGPT output for a list of best practices for benching inference and I learned some things along the way.
Best practices for benchmarking LLM inference1. Define what you are testingRun separate tests for:Cold start: model load through completed responseWarm inference: model already loaded, fresh request statePrompt-cached inference: repeated prefix caching enabledSustained throughput: continuous requests over a longer periodFor most practical comparisons, warm inference with a fresh KV cache is the most useful test.2. Keep the test conditions fixedUse the same:Model and quantizationInference software and versionPrompt and system promptContext lengthMaximum generated-token countSampling settingsHardware power and performance settingsNumber of simultaneous requestsSpeculative-decoding configurationUse deterministic decoding where possible, such as greedy decoding or temperature 0.3. Control caching correctlyFor a standard warm-inference benchmark:Keep the model loadedPerform several warm-up runs firstStart each measured request with a fresh KV cacheDisable prompt or prefix cachingDisable response cachingDo not flush the operating-system disk cacheDo not unload and reload the model between runs
Do not clear the GPU allocator cache between every run unless that is specifically what you are testingFor a cold-start benchmark, restart the inference process before every measured run.4. Control output lengthOutput length strongly affects total time.Best options:Force the same number of generated tokensUse a prompt unlikely to produce an early stopDisable early stopping or stop strings when possibleRecord the actual generated-token count for every runDo not compare total completion times when the models generate substantially different numbers of tokens.5. Warm up before measuringBefore collecting results:Load the modelRun the exact workload several timesAllow kernels, memory allocations, and GPU clocks to stabilizeExclude warm-up runs from the results6. Measure the right metricsRecord:Time to first tokenPrompt-processing speed, in input tokens per secondGeneration speed, in output tokens per secondEnd-to-end latencyActual input and output token countsPeak memory usagePower consumption, when efficiency mattersFor concurrent testing, also measure total system throughput and latency percentiles.7. Run enough repetitionsUse at least:3–5 warm-up runs10 or more measured runsMore runs when results vary noticeablyReport:MedianMeanMinimum and maximumStandard deviation or percentile spreadThe median is generally more useful than reporting the fastest run.8. Keep the system stableDuring testing:Stop unnecessary workloadsUse a consistent power profileAvoid thermal throttlingMonitor GPU temperature, clocks, power, and utilizationKeep CPU affinity and thread counts consistentKeep the same NUMA configurationAvoid changing GPU offload or batch settings between comparisons9. Test realistic prompt sizesBenchmark several workloads rather than one tiny prompt:Short prompt, short responseLong prompt, short responseShort prompt, long responseLong prompt, long responseThis separates prompt-processing performance from token-generation performance.Recommended default procedure1. Load the model once.2. Disable response and prefix caching.3. Use a fresh KV cache for every request.4. Fix the prompt, context, sampling settings, and generated-token count.5. Run five unmeasured warm-up requests.6. Run at least ten measured requests.7. Record time to first token, prompt speed, generation speed, and total latency.8. Report the median and variation.9. Test cold-start and cached-prefix performance separately.[\code]
ElectroSoldier wrote on 2026-08-06, 06:11:ratfink wrote on 2026-08-06, 05:02:Maybe a shorter answer is "when that sort of benchmarking has some relevance to what the site is about".
Oh I'm very sorry... Next time I'll be sure to make sure it's about games and nothing but. But just for you!
For now in this thread maybe we can talk about benchmarking for inference models as they stress GPUs much much more than games do to the point where many GPUs that are still considered current for games played today are pitifully underpowered when considering an inference model.
I answered your question, what IS your problem?
ratfink wrote on 2026-08-06, 08:43:ElectroSoldier wrote on 2026-08-06, 06:11:ratfink wrote on 2026-08-06, 05:02:Maybe a shorter answer is "when that sort of benchmarking has some relevance to what the site is about".
Oh I'm very sorry... Next time I'll be sure to make sure it's about games and nothing but. But just for you!
For now in this thread maybe we can talk about benchmarking for inference models as they stress GPUs much much more than games do to the point where many GPUs that are still considered current for games played today are pitifully underpowered when considering an inference model.I answered your question, what IS your problem?
But we talk about a lot more than just Very Old Games On New Systems on this site.
Your answer makes it sound like any talk other than that isnt wanted.
