rasz_pl wrote on 2024-11-20, 22:59:
LLMs predict next word (token) dependent on previously fed words (tokens), nothing more nothing less.
It doesnt contain knowledge in traditional sense, just word sequence order probability weights. Its a glorified One-Word Story Game.
It doesnt retain any new data from its previous interactions and cant learn new things. There are tricks like very larger context windows, but its all smoke and mirrors.
yes, its important not to confuse it with what used to be called "expert systems" which were attempts to capture, in something like a matrix-workflow, the accurate expertise in a domain, like medicine
an LLM just repeats what "comes next" as aggregated from a truly huge library of reference material. It does appear to do a lot of weighted assessment of input and likely of output though, maintaining context and even consistency over long passages of output text. It can be unnerving
Yet it has no model of the world, demonstrable in some of its hallucinations, but does often appear to simply because all the reference texts were largely written by humans who do have a model of the world
because it's algorithmically driven my understanding is no one really knows what it is doing in any one case, there are no lines of code to follow for specific cases - it would be necessary to somehow trace every weighting and calculation that lead up to the output and i don't think anyone is keeping logs. That makes it a higher risk in my view, the detailed process is unknowable and the output essentially unpredictable
CharlieFoxtrot wrote on 2024-11-20, 06:39:
The viability of this technology lies in the belief that it can still improve, costs don’t continue ballooning up and there are profitable business cases for it cover burned money and create actual returns for investment. So far this is very uncertain and IMO the hype stems from rotten big tech corporate world, where they desperately need ”the next big thing” to maintain their growth status as there is nothing else: big data hype died (you can argue that current AI hype is direct continuation of that) and it is difficult to get two digit growth from established services, such as cloud, anymore.
LLMs won’t go away, but I’m skeptical if they ever actually deliver tangible benefits especially compared to the cost. Because they just make statistical predictions based on existing information, they can’t actually create anything new, they will remain error prone and it is ultimately also the inherent flaw of the technology.
a good point, the impression given is that we are at the beginning of a revolution and that the tech will improve hugely - but its also possible that the tech is already most of the way there and what is left now is many years of impressive but essentially refining improvements - each ever more minor improvement coming at the fairly linear increase in cost, i.e. with lower and lower returns.
that's for LLMs anyway, i'd expect the same for all generative AI - i'm sure there will be AI novels, music and movies flooding our choices but also marked by mediocrity and and oddness
actual artificial general intelligence my be well over the horizon, and if it can be said to arrive what will it be - self aware? possibly not, and perhaps not as revolutionary as predicted either. human general intelligence exists as organic response to organic inputs (senses, hormones etc) and all built on an inherent "expert system" the complexity of which (in terms of the interactions that give rise to the specifics action of intelligence) is still not fully comprehended. I suspect making AGI equivalent to a human just means making a biological human and that machine AGI will be different in most aspects, quite "alien" to us
in the meantime some more people will lose jobs, ushered into jobs serving coffee to the smaller pools of people still able to escape "ai automation", to put a downer on it