So here I am getting excited about what all these frontend inference engines can do when all the fancy stuff happens courtesy of llama.cpp

"Well duh, that's a given Bee Dubbya." But I am meaning the flexibility of hardware target, it's all in the llama.cpp flags and switches.

Firstly the tensor pairing, that's where your GPUs are working in kind of like a RAID 0 stripe and the work is split between, but dependent on the other for completeness. They need high bandwidth to swap data, so this is not the config for PCIe 1x. This is more ideal for your fast slots, enables large models in the pooled VRAM.

Then there's the layered mode, where it just dispenses an amount of layers to work on that the GPU can hold, kind of your JBOD version. This doesn't need fast inter GPU communications, for all I know it will work on a PCI to PCI 1x riser or USB3 EGPU setup even. Provided your run time per job is sufficiently long that the time it takes to get 8GB there and back isn't too ridiculous.

Asymmetry can be handled by specifying the loading proportion per GPU, so you can have a slow 2GB doing it's share along with a fast 12GB. Mixed precision is handled well also, most commonly seen with single GPU plus CPU in a "regular desktop workstation". But it also means you can have very mixed targets in same system Vulkan, RDNA, CUDA and OpenCL (Presuming you get the drivers to play nice, that can always be a thing with mixed generations)

Then you don't even have to have all the GPU in the same machine, you can run one instance across multiple machines with an RPC configuration. Woohoo.

Now what I have to dig into is whether tensor mode is a whole instance thing or whether you can designate one pair in tensor and layer mode to extra GPU, or tensor them if you get 2 slowpokes who aren't massively affected by communicating at PCIe 1x.

Heh, guess it's back to "distributed computing" times, like SETI/Folding/Scrypt mining where you have to figure out where to distribute your computing around the house so as not to strain your circuits or get any room/floor too hot. 🤣 Actually, I have to dig out what the Ethereum and Litecoin guys knew about power tuning if I get much running. Basically a combo of undervolt and downclock, optimising GPU/RAM clock ratios until you're at 90% of the speed for 50% of power. Some workloads though, you can actually optimise for power and get higher speed too, think Ether was more like that because of memory intensity.

Well anyway, that's part of why I'm on a push going into winter, wanna get the high load training going in the cold weather so it's only like paying 50% for my heat. (I mean that as a cost offset, will still be more elec than not doing it) Then in spring start tuning, and chron jobbing machine activation times so it's queuing jobs until cheap elec/cool eves. Would be cool if by then I can get voice assistant frontend up and running on just a few dozen watts off solar through day.

edit: the horses mouth https://github.com/ggml-org/llama.cpp/blob/ma … cs/multi-gpu.md

EDITII: Now that's interesting, apparently the R9 390s had "overkill" memory performance when used for mining Ethereum, such that memclocks were often lowered for power save. However core clocks were known to spike power use rapidly over 1050. So definitely have to keep core clocks down. I wonder how low they can go before AI perf impact. However, is overkill for Ether memory speed enough? It is known on the more crippled memory bus of the RX series, "memsift" ("memshift"?) memory timing strap mods were required to get good mem performance. Maybe I will look at how those work and give the R9s something similar, insane VRAM speeds for practically anything else sound like barely enough in AI context. These seem like guides I will need later... https://www.overclock.net/threads/undervoltin … reader.1632351/ https://www.overclock.net/threads/guide-how-t … lt-gcn.1592963/

BitWrangler wrote on 2026-11-07, 15:45:

ElectroSoldier wrote on 2026-11-07, 14:14:

750Ti would be interesting, In that I would be interested to know how that performs.

I will keep you posted if I see any results for one. I don't have one so won't be direct from me. I'll let you know if I spin up the 650ti and what that does.

Many of these cards seem to have a lot of under the hood grunt, but not the actual pixel pushing or DX support to do high FPS on modern games. So they don't rank high in game focused benchmarks any more, but seem to have useable FLOPs and RAM bandwidth in there that can be leveraged.

I have one myself in an old HP SSF desktop PC with an i7-3770 but its running WinXP right now. If I get the chance over the next few weeks ill get the drive out and fire up Win10 and see what it can do if you want, but given the workflow is yours and the software too then I dont know what I can do my end to "help"...

ElectroSoldier wrote on Yesterday, 23:01:

BitWrangler wrote on 2026-11-07, 15:45:

ElectroSoldier wrote on 2026-11-07, 14:14:

750Ti would be interesting, In that I would be interested to know how that performs.

I will keep you posted if I see any results for one. I don't have one so won't be direct from me. I'll let you know if I spin up the 650ti and what that does.

Many of these cards seem to have a lot of under the hood grunt, but not the actual pixel pushing or DX support to do high FPS on modern games. So they don't rank high in game focused benchmarks any more, but seem to have useable FLOPs and RAM bandwidth in there that can be leveraged.

I have one myself in an old HP SSF desktop PC with an i7-3770 but its running WinXP right now. If I get the chance over the next few weeks ill get the drive out and fire up Win10 and see what it can do if you want, but given the workflow is yours and the software too then I dont know what I can do my end to "help"...

I'm lost, thought it was you who wanted to know about the 750ti, 🤣

I found what is supposed to be one of the more comprehensive setup and tuning guides for llama.cpp on the net..
https://blog.steelph0enix.dev/posts/llama-cpp-guide/
If it was actually on the net, I'm getting cloudflare blocked, someone on my IP block has been a naughty boy I guess, or there's an active attack and they are blocking a whole bunch of class 1s. Ohhh, PHP, maybe he hasn't patched it yet.

However, I went round the long way and got it from archive.org so ...
https://web.archive.org/web/20260926183158/ht … lama-cpp-guide/

It is 2 years old though, so there's a bit of new stuff as well not covered.

In that it appears llama.cpp itself would consider AVX2 instructions optional and supported if available, with a base level requirement of AVX (1/classic/OG/whatev) so that bodes well for GPU holes here, adds 7 or 8 potentials.

Some great config tips in there it seems like, handling models should be the same etc.

edit: it's on his github too in here... https://github.com/SteelPh0enix/steelph0enix. … r/content/posts

I am finding this review of the "AI appliance" AMD mini PC fascinating as it takes you through performance factors and explains the whole shebang. https://www.techpowerup.com/review/amd-ryzen- … x-halo-mini-pc/ couple of interesting graphs that probably have the same shape if different scale on lesser hardware, like the brick wall when context gets too large for your available (V)RAM. Then also near end it touches on network performance for distributed use.

Not sure if it was in that or elsewhere it mentioned AI loads aren't actually "full load"-ing most hardware, power and heat wise, but have a maximum point where the peak is. I am guessing this is due to RAM use intensity vs actual math core hammering. Might mean I can use PSU with less "safety margin", or get one more GPU per PSU in some instances. But anyway, looking at a power cost estimate, I'd have to be using a kilowatt 24/7 for four years to justify that AMD mini on power saving vs "pile o' random parts", but cloud AI token savings probably outweigh that quite a bit. This is assuming the price doesn't soar because someone figures out you can give a Slate truck better autopilot than a Tesla with it or something. 120W and it's diminutive size is definitely in the ballpark for adding it to things, cars, robots etc.

Found someone testing AI on an R9 390, Buuuut, didn't seem to use effectively, looks like he's trying to do coding with a Qwen-4B general model, IDK. Looks like it's ripping thru things with that lightweight but obviously "going off into the weeds" with results because of being such a dumbed down model. https://www.youtube.com/watch?v=SmZ2tWmB93o&t=680s jump in at 11:20 if the timestamp doesn't work (is it just me or are timestamp links still really hit and miss??)