Well my minimal setup for llama.cpp was Core2Duo (as a CPU-only setup) so I know a little bit about this 😁
(Yes, it did work, slowly.)

My FX-8320E has AVX1 which is a huge benefit compared to C2D but as you said, packages usually go for AVX2 minimum.
So Haswell is a good choice in that sense.
I usually try to keep somewhat up to date with llama.cpp so I'd compile things myself anyway.

GT730 and 1x riser though...well, let's just say I'll be interested to hear what you can get out of them 😁
(I did dabble a bit with opencl+low VRAM combo myself a few years ago, it didn't go well.)

ElectroSoldier wrote on Yesterday, 10:05:

Im struggling to come up with a constructive answer to this buddy because Im not really sure what it is youre wanting to do...

It sounds like you want to find the slowest PC possible to run an LLM in LMStudio... Right?

Hah, not exactly it's more like how can I use what I already have sitting here to best advantage. Which happens to be about the slowest possible.

I am not sure what I am trying to do, beyond learning about it. I'm about in the state like it's 1995 I've got a 386 with 16MB in it and I want to install Win95 and learn web development, pretty much you can bang out html in MSDOS Edit on a 5150, buuut, I'm trying to be running a browser so I can see what I am doing. The "pros" are using a Pentium 133 right now, but if I dump 2000 into one of those, I'm not using it effectively for a year or so, so could have got better hardware by the time I am able to use it.

Not planning to make a career on this per se, just use it to enable my further interests, but if it were a generic business plan of the "collect underpants, ???, profit" type, I am collecting underpants on the way to ??? stage. After all, how will you know what the underpants can do for you, until you have the underpants.

mr.cat wrote on Yesterday, 12:08:

GT730 and 1x riser though...well, let's just say I'll be interested to hear what you can get out of them 😁
(I did dabble a bit with opencl+low VRAM combo myself a few years ago, it didn't go well.)

Yeah I am not expecting a real lot, but from what I am gathering things have changed in just the last year or two as to how the models can divide experts according to resource type such that less capable hardware can also contribute. I am viewing it like the situation with windows swap files. You've got your fast hard drive, but if both system and swap are using it, they are basically using it at half speed or less each because of dividing the resource and the seeking back and forth. But even if you put in an extra HDD that is half as fast to dedicate to swap, because it gets the full speed of that, and seeks less the whole system speeds up, because now system can use full speed of original hdd and the swap gets the effective speed it has before without so much seeking penalty. That's the minimum example, the best setup would be that your swap is on the faster of course, which might be inconvenient due to that one being huge and your extra being tiny, but I assume the resource scheduling would be more efficient. I'm just saying in worst dumb case of adding slower hardware it can make a difference.

edit:

BitWrangler wrote on Yesterday, 15:42:

Hah, not exactly it's more like how can I use what I already have sitting here to best advantage. Which happens to be about the […]
Show full quote

ElectroSoldier wrote on Yesterday, 10:05:

Im struggling to come up with a constructive answer to this buddy because Im not really sure what it is youre wanting to do...

It sounds like you want to find the slowest PC possible to run an LLM in LMStudio... Right?

Hah, not exactly it's more like how can I use what I already have sitting here to best advantage. Which happens to be about the slowest possible.

I am not sure what I am trying to do, beyond learning about it. I'm about in the state like it's 1995 I've got a 386 with 16MB in it and I want to install Win95 and learn web development, pretty much you can bang out html in MSDOS Edit on a 5150, buuut, I'm trying to be running a browser so I can see what I am doing. The "pros" are using a Pentium 133 right now, but if I dump 2000 into one of those, I'm not using it effectively for a year or so, so could have got better hardware by the time I am able to use it.

Not planning to make a career on this per se, just use it to enable my further interests, but if it were a generic business plan of the "collect underpants, ???, profit" type, I am collecting underpants on the way to ??? stage. After all, how will you know what the underpants can do for you, until you have the underpants.

Yeah I see what you mean now.

I was running LMStudio on an i7-7700 with 32Gb RAM, which I bought before the price went sky high, I also have a 3090 24Gb GPU that was also bought before prices went through the roof... I think if I sold it now I would double my money on it.

Im not sure what I could run it on now, given what I have to run it on now...

To me a 7th gen i7 is old... Still very capable but it is old. Up until a few weeks ago I was browsing the web with a 4th gen Haswell E3-1225 V3 Xeon system with an NVS 510 GPU. But I never considered running LMStudio on that system, and its now been replaced with an i7-12700T based HP Elite Mini G9 system, which probably could run an LLM, though it wouldnt be quick.

an HP EliteDesk 800 G3 with an i7-7700/32Gb/2256Gb will run a 26B Gemma model but it runs it extremely slowly. In the order of several minutes to generate an answer but it does run it.
Im not sure what would happen if I tried to put an agent with that model though.

Give us a clue as to how old you are thinking of going?

Just hearing about a GPU-free method of running an LLM that competes with Claude Opus. Named Colibri, the work of Vincenzo Fornaro.

Basically it's EMM for AIs, 10GB "page frame" in system RAM and streams the rest off HDD, one would assume fast SSD. Needs a bit of a biggie though, the model it runs is GLM-5.2 and it's a chonker. 370MB, so if you've only got 512GB SSD it needs to get downloaded to external or secondary storage I guess.

This is a 744b token monster though, putting the LARGE back in LLM, only some 50B are active apparently. This model is particularly well regarded for cybersecurity, it's not a one trick pony though. It can be run in server mode as endpoint with Colibri also.

Colibri can be cloned and built from github, dev page to find it from is github.com/JustVugg and also has a presence on huggingface.co/Justvugg

Colibri is Italian for hummingbird, which would be an appropriate name for it's "light" memory footprint compared to size of model. But also it may pay tribute to a historic Stanford AI Lab LISP machine named COLIBRI.

Anyway, planning to look further into this one also. May be able to scrape up hardware to put one of these up additional to the other stuff, might not get a great CPU though. Might look for recent CPU, smashed screen etc crappified laptop for cheap and just have it on the local lan as an AI appliance.

Edit: Damn, was getting that info off a talking head and on reading https://github.com/JustVugg/colibri#which-mod … -for-my-machine it doesn't seem entirely accurate, and no-GPU mode doesn't exactly fly at 0.1 token per second on GLM 5.2 . Also dude on there mixed up minimum for other models and size needed for full fat GLM.

ElectroSoldier wrote on Today, 01:55:

Yeah I see what you mean now. […]
Show full quote

BitWrangler wrote on Yesterday, 15:42:

Hah, not exactly it's more like how can I use what I already have sitting here to best advantage. Which happens to be about the […]
Show full quote

ElectroSoldier wrote on Yesterday, 10:05:

Im struggling to come up with a constructive answer to this buddy because Im not really sure what it is youre wanting to do...

It sounds like you want to find the slowest PC possible to run an LLM in LMStudio... Right?

Hah, not exactly it's more like how can I use what I already have sitting here to best advantage. Which happens to be about the slowest possible.

I am not sure what I am trying to do, beyond learning about it. I'm about in the state like it's 1995 I've got a 386 with 16MB in it and I want to install Win95 and learn web development, pretty much you can bang out html in MSDOS Edit on a 5150, buuut, I'm trying to be running a browser so I can see what I am doing. The "pros" are using a Pentium 133 right now, but if I dump 2000 into one of those, I'm not using it effectively for a year or so, so could have got better hardware by the time I am able to use it.

Not planning to make a career on this per se, just use it to enable my further interests, but if it were a generic business plan of the "collect underpants, ???, profit" type, I am collecting underpants on the way to ??? stage. After all, how will you know what the underpants can do for you, until you have the underpants.

Yeah I see what you mean now.

I was running LMStudio on an i7-7700 with 32Gb RAM, which I bought before the price went sky high, I also have a 3090 24Gb GPU that was also bought before prices went through the roof... I think if I sold it now I would double my money on it.

Im not sure what I could run it on now, given what I have to run it on now...

To me a 7th gen i7 is old... Still very capable but it is old. Up until a few weeks ago I was browsing the web with a 4th gen Haswell E3-1225 V3 Xeon system with an NVS 510 GPU. But I never considered running LMStudio on that system, and its now been replaced with an i7-12700T based HP Elite Mini G9 system, which probably could run an LLM, though it wouldnt be quick.

an HP EliteDesk 800 G3 with an i7-7700/32Gb/2256Gb will run a 26B Gemma model but it runs it extremely slowly. In the order of several minutes to generate an answer but it does run it.
Im not sure what would happen if I tried to put an agent with that model though.

Give us a clue as to how old you are thinking of going?

Thanks yeah sounds like I'll be a bit slow unless I keep model sizes down.
I have,
Ryzen B550 mATX board, 32GB DDR4 to go in it, temporary CPU R5 1500x, looking for better. PCIe 16x one of.
Z87 haswell board 32GB DDR3 to go in it, i7-4770k that's meant to do 4.5Ghz, three long slots, not sure what the lane assignment is, 16,8,4 or what.
generic haswell boards 2 of with only one slot, another 4770, a 4570 and a 4690T low power.
Then there's 4 8GB GPUs with reasonable mem bandwidth and high floating point performance, 3x R9 390, 1x R9 295X, one inbound with 4GB a K4200, the might run it for a joke GT730 4GB, and a shoulda bought the 6GB 1060 with 3GB which might not figure much.

Oh man, they're coming out of the woodwork all over, I guess this is one for keeping the Keplers Keppling....

https://github.com/ahtx/kindi

Only seems to run Ternary-Bonsai-2 models at the moment, but speeds on k1200 make me hopeful it might even be useful on the K4200 or evern K620s paired...

So he's mentioning tensor pairing for GPUs there, I wonder if that what was being mentioned when I saw about more than one extra GPU on 1x being slower, because the first one paired up with the first, but the second extra, third card didn't glom on. Hmmm.

Guess that GT730 is getting a different job, replacing a K620 in a machine I was gonna use that in. Though also I have a 650ti and a plain 650 to dick around with if this is any good.

Gotta see where the CPU line is drawn here, as the next 2 or 3 long slot boards down are either AM3+ or C2D.

Edit: I guess "the algorithm" is just firehosing me with stuff now (makes a change to get something I am actually a bit interested in vs the one time I looked at a toilet part and got 6 months of plumbing supply and bathroom reno) but here's another lightweight inference one https://www.makeuseof.com/27b-llm-model-fits-4gb-vram/ but article explains a bit about how bonsai models are trimmed from Qwen which was a little illuminating.

BitWrangler wrote on Today, 04:42:

Oh man, they're coming out of the woodwork all over, I guess this is one for keeping the Keplers Keppling.... […]
Show full quote

Oh man, they're coming out of the woodwork all over, I guess this is one for keeping the Keplers Keppling....

https://github.com/ahtx/kindi

Only seems to run Ternary-Bonsai-2 models at the moment, but speeds on k1200 make me hopeful it might even be useful on the K4200 or evern K620s paired...

So he's mentioning tensor pairing for GPUs there, I wonder if that what was being mentioned when I saw about more than one extra GPU on 1x being slower, because the first one paired up with the first, but the second extra, third card didn't glom on. Hmmm.

Guess that GT730 is getting a different job, replacing a K620 in a machine I was gonna use that in. Though also I have a 650ti and a plain 650 to dick around with if this is any good.

Gotta see where the CPU line is drawn here, as the next 2 or 3 long slot boards down are either AM3+ or C2D.

Just wanted to mention this in case you weren't aware, but the K620, K1200 and K2200 are actually Maxwell-based, not Kepler like all of the rest of the K series (K420, K600, K2000, K4200, etc.). Thanks for the naming consistency Nvidia! 🤦 😀

Ozzuneoj wrote on Today, 04:55:

BitWrangler wrote on Today, 04:42:

Oh man, they're coming out of the woodwork all over, I guess this is one for keeping the Keplers Keppling.... […]
Show full quote

Oh man, they're coming out of the woodwork all over, I guess this is one for keeping the Keplers Keppling....

https://github.com/ahtx/kindi

Only seems to run Ternary-Bonsai-2 models at the moment, but speeds on k1200 make me hopeful it might even be useful on the K4200 or evern K620s paired...

So he's mentioning tensor pairing for GPUs there, I wonder if that what was being mentioned when I saw about more than one extra GPU on 1x being slower, because the first one paired up with the first, but the second extra, third card didn't glom on. Hmmm.

Guess that GT730 is getting a different job, replacing a K620 in a machine I was gonna use that in. Though also I have a 650ti and a plain 650 to dick around with if this is any good.

Gotta see where the CPU line is drawn here, as the next 2 or 3 long slot boards down are either AM3+ or C2D.

Just wanted to mention this in case you weren't aware, but the K620, K1200 and K2200 are actually Maxwell-based, not Kepler like all of the rest of the K series (K420, K600, K2000, K4200, etc.). Thanks for the naming consistency Nvidia! 🤦 😀

Right, TechPowerup is weird with it too, lists them under quadrokepler tag but not under kepler, and gives gen as kepler core as maxwell, ah well, nvidiocy I guess.

BitWrangler wrote on Today, 05:03:

Ozzuneoj wrote on Today, 04:55:

BitWrangler wrote on Today, 04:42:

Oh man, they're coming out of the woodwork all over, I guess this is one for keeping the Keplers Keppling.... […]
Show full quote

Oh man, they're coming out of the woodwork all over, I guess this is one for keeping the Keplers Keppling....

https://github.com/ahtx/kindi

Only seems to run Ternary-Bonsai-2 models at the moment, but speeds on k1200 make me hopeful it might even be useful on the K4200 or evern K620s paired...

So he's mentioning tensor pairing for GPUs there, I wonder if that what was being mentioned when I saw about more than one extra GPU on 1x being slower, because the first one paired up with the first, but the second extra, third card didn't glom on. Hmmm.

Guess that GT730 is getting a different job, replacing a K620 in a machine I was gonna use that in. Though also I have a 650ti and a plain 650 to dick around with if this is any good.

Gotta see where the CPU line is drawn here, as the next 2 or 3 long slot boards down are either AM3+ or C2D.

Just wanted to mention this in case you weren't aware, but the K620, K1200 and K2200 are actually Maxwell-based, not Kepler like all of the rest of the K series (K420, K600, K2000, K4200, etc.). Thanks for the naming consistency Nvidia! 🤦 😀

Right, TechPowerup is weird with it too, lists them under quadrokepler tag but not under kepler, and gives gen as kepler core as maxwell, ah well, nvidiocy I guess.

Pretty sure the K2200 is based on the the Maxwell 750Ti so that explains that card. The K1200 and K620 are all cut down professional variants of the Maxwell 750Ti core too, K1200 is a GTX750 and the K620 is the GTX745 ...a POS display adapter.

The only real differences are the slower memory and core clocks and the larger memory on the Quadro cards for professional use, K2200 is a nice card though.

I think the reason Techpower UP lists these under Kepler is because they were released under that gen of cards, these three along with the GTX745, 750 and 750Ti are Maxwell 1.0 not Maxwell 2.0 that Quadro Maxwell and the GTX900 series used. I'm not entirely sure what changed between 1 and 2 but 2.0 cards are significantly more powerful.

750Ti would be interesting, In that I would be interested to know how that performs.

With an expansion of what GPU may be useful, it's easy to find you don't have enough seats for GPUs in AVX2 class machines, but are they just basically providing disk and i/o for GPUs? Many inference engines seem to be compiled for AVX2, probably expecting inference to be run on CPU, which of course is dog slow at the low end of AVX2 support. But if all the real work is pushed to GPU, maybe it's not so important.

Claude users among which are probably "from my cold dead hands" less current thinkpad fanatics etc, have found this problem running tools to use Claude remotely, so have come up with some AVX2 workarounds which appear to be useful in general cases.

Firstly got a QEMU wrapper: https://github.com/AdversaLLC/claude-science-no-avx2

But also a library: https://github.com/Mavergreen/avxemu (For Mac, ported to Linux)

I don't think you wanna be using these on your i7-2600 to run that GLM-5.2 on Colibri in CPU mode unless you want to ask a couple of dozen tokens worth at suppertime and get the answer at breakfast. But if you're like me, your idea of multitasking is to have an additional machine to use while the other n machines are busy.

I was on the "find holes for GPUs" mission though, meaning I have multi x16 length slots on non-AVX2 platforms. The next problem is putting enough RAM on them, so DDR2 boards gonna come up short, unless you have server class boards that take registered. Might get one of the lightweights running a 4GB GPU on a C2D though. In the main though for my personal use I was thinking of an AM3+ board that will take 32GB and the fastest FX. But I am seeing a lot of GTX970 4GBs around cheap in the local market so if a configuration on 4GB old CUDA seems like a keeper, I could grab one of those, stick it on sandybridge to preserve that and free up the better platform for other testing/GPU or something. (Though idea of running 27b model on 2x 970 got appealing but I gotta prove utility of any of this shit to me first)

Another use might be dedicated head ends for all this crap in server mode in your basement or wherever, given I've got several non-AVX2 laptops but the front ends might need it.

ElectroSoldier wrote on Today, 14:14:

750Ti would be interesting, In that I would be interested to know how that performs.

I will keep you posted if I see any results for one. I don't have one so won't be direct from me. I'll let you know if I spin up the 650ti and what that does.

Many of these cards seem to have a lot of under the hood grunt, but not the actual pixel pushing or DX support to do high FPS on modern games. So they don't rank high in game focused benchmarks any more, but seem to have useable FLOPs and RAM bandwidth in there that can be leveraged.

BitWrangler wrote on 33 minutes ago:

With an expansion of what GPU may be useful, it's easy to find you don't have enough seats for GPUs in AVX2 class machines, but […]
Show full quote

With an expansion of what GPU may be useful, it's easy to find you don't have enough seats for GPUs in AVX2 class machines, but are they just basically providing disk and i/o for GPUs? Many inference engines seem to be compiled for AVX2, probably expecting inference to be run on CPU, which of course is dog slow at the low end of AVX2 support. But if all the real work is pushed to GPU, maybe it's not so important.

Claude users among which are probably "from my cold dead hands" less current thinkpad fanatics etc, have found this problem running tools to use Claude remotely, so have come up with some AVX2 workarounds which appear to be useful in general cases.

Firstly got a QEMU wrapper: https://github.com/AdversaLLC/claude-science-no-avx2

But also a library: https://github.com/Mavergreen/avxemu (For Mac, ported to Linux)

I don't think you wanna be using these on your i7-2600 to run that GLM-5.2 on Colibri in CPU mode unless you want to ask a couple of dozen tokens worth at suppertime and get the answer at breakfast. But if you're like me, your idea of multitasking is to have an additional machine to use while the other n machines are busy.

I was on the "find holes for GPUs" mission though, meaning I have multi x16 length slots on non-AVX2 platforms. The next problem is putting enough RAM on them, so DDR2 boards gonna come up short, unless you have server class boards that take registered. Might get one of the lightweights running a 4GB GPU on a C2D though. In the main though for my personal use I was thinking of an AM3+ board that will take 32GB and the fastest FX. But I am seeing a lot of GTX970 4GBs around cheap in the local market so if a configuration on 4GB old CUDA seems like a keeper, I could grab one of those, stick it on sandybridge to preserve that and free up the better platform for other testing/GPU or something. (Though idea of running 27b model on 2x 970 got appealing but I gotta prove utility of any of this shit to me first)

Another use might be dedicated head ends for all this crap in server mode in your basement or wherever, given I've got several non-AVX2 laptops but the front ends might need it.

ElectroSoldier wrote on Today, 14:14:

750Ti would be interesting, In that I would be interested to know how that performs.

I will keep you posted if I see any results for one. I don't have one so won't be direct from me. I'll let you know if I spin up the 650ti and what that does.

Many of these cards seem to have a lot of under the hood grunt, but not the actual pixel pushing or DX support to do high FPS on modern games. So they don't rank high in game focused benchmarks any more, but seem to have useable FLOPs and RAM bandwidth in there that can be leveraged.

That K2200 is as close to a 750ti as you can get and its 4Gb over the 2Gb the 750Ti has ...results from the K2200 should be close to or identical to the 750Ti or good enough to get a general idea.