First post, by BitWrangler
- Rank
- l33t++
Howdy folks,
So Big AI may or may not be dying/problematic/lame/evil/smelly etc etc, if you've got opinions on that, take it to one of the numerous other threads on the topic. This here is about crafting tools. Not the most expensive up to date tools that soak up 10s of thousands of dollars that you can jack off to, more like beating old bits and pieces from the workshop into a different shape. Remember the dickheads you meet everywhere else on teh interwebs that insist you can't possibly even think of playing a game until you have a wraparound 8k monitor a 5090, 48 core CPU and enough lights for Vegas? Don't be like that with this.
How marginal is the hardware? I will initially be working with GPU that have 8GB, OpenCL, Vulkan 1.2, that's it, no matrix acceleration. Small matrices are supposed to accelerate better on general GPU compute than they do on custom HW support anyway. Later, MUCH later, do not be pushing for this, it's about experience and comfort zone, I may have a model on my more modern laptop with more advanced but very middling GPU.
I am more in the realm of "If the printer makes a noise you don't recognize, shoot it" class of wayward tech paranoia, I won't have smart speaker Echo/Alexa type devices here, to pay for big tech to bug my own house, and will not buy the current form of "Smart TV" as it's smarts are not for your benefit. Therefore first step is figuring out what kind of containerisation or virtualisation I can use, keep it in it's own padded cell with limited access. Another consideration for that is so I can move the whole thing wholesale and also mess with different models on the same machine and switch them up.
So the only game in town for this for virtualisation with any sliver of efficiency seems to be full GPU passthrough, which appears to mean I will need to use chipset graphics to administer the machine. I am not very familiar with such setups. I get the "exclusive use of GPU" thing, but not clear on the mechanics. Can I freeze the VM and use the GPU for something else? Or would I have to quit out of VM entirely? Or does it mean you can't have the host OS install drivers or use GPU at all before you launch VM even?
Then also I am trying to figure the bare minimum RAM I could get away with for that, figuring something like 8GB for the host OS, then minimal guest OS needs 4GB plus the software overhead 4g? Plus the size of the model you want to run. Let's say bare minimum one model would be 8GB, so 24GB RAM I am thinking to support single 8GB GPU.... though it would seem you have to have system RAM to match per GPU for total model footprint with multi GPU, i.e. add another 8GB GPU, need another 8GB system. Two years ago, you'd have been thinking "just throw ALL the RAM at it" but yeah, it's hard to scrape up for cheap now so trying to work to minimums.
Thus would containerisation be the more RAM efficient approach or not? Not sure what kind of overheads and restrictions that brings in, and again don't know enough about it.
I haven't even mentioned what I want to get into really. I am thinking of something like Qwen for indexing all the tech docs and books I have pdfs of, with natural language query. Then something like Phi for actually putting tech knowledge to use, and maybe LLama or GPT4All for general purpose "rubber duck" noodling around ideas... That's what I am thinking I should be able to run from reading around sites like https://everylocalai.com/ with selecting the low RAM requirement.
Stretch goal, when two are three are trained independently would be to have speech to text query frontend and be able to use them in their own areas of (in)competence like a StarTrek computer or JARVIS etc. Maybe also co-operate with independent home automation and security systems.
Anyway, there's a lot to know and figure out, so just trying to get basics down for beginning to mess with this stuff.
Unicorn herding operations are proceeding, but all the totes of hens teeth and barrels of rocking horse poop give them plenty of hiding spots.
