With the pace of AI, and with AI helping to pave the way for faster/better AI, I keep wondering if hardware like this will become obsolete well before it has a meaningful ROI. Huge AI models can be run with less resources already through quantization and offloading, but that's just the beginning.
One day, maybe not far from now, a breakthrough will allow huge LLMs (say 200B in size) to run well on an old 5 year old Dell desktop. Think that's crazy? Look at the size of the first hard drives. The IBM 350 was a disk with 50 platters, 24 inches in diameter, that held 3.5Mb, and was leased for today's equivalent of $35K.
Compare that to a multi-terabyte ssd. Now apply that improvement to how an LLM is architected and run now. With AI assisting, it won't be long before a leap occurs and these data centers with all their current ultra-cutting edge Nvidia cards are nearly obsolete overnight.
They do. Mythos kicked ass while it lasted. And what we know of the scaling law curves promises us even more gains in the future.
"The future" being "whenever training and inference at increased scale becomes economical". Which is probably bounded by new generations of hardware, but might also be pushed forward by algorithmic advances.
The likes of Mythos show that the scaling laws are real, and you can x5/x2 the total/active params and get meaningful gains. If "inference per param" gets cheaper? Up the params and get more intelligence for the same price.
Very true, and all I am basing my comment on is the improvement in speed AI has demonstrated when applied to software development, and inferring it might enable a similar 10X or 100X improvement in both hardware architecture as well LLM structure and/or interface methods. If that speed improvement applies to performance of AI, that could mean the 70 years it took for people to improve storage technology might be able to be compressed to achieve a step change in AI performance in a drastically shorter timeframe.
I think Jevons Paradox and scaling laws will make this not the case. If bigger models are always better (which seems they are), then will always need high-end hardware.
> One day, maybe not far from now, a breakthrough will allow huge LLMs (say 200B in size) to run well on an old 5 year old Dell desktop.
I think there will be specialized hardware (beside GPUs) that would be custom made for LLMs. Yes TPUs exist, but mainly for datacenter. GPUs exist, but they are adapted from mainly graphic application. Once all the demand from data center dries up, innovation will kick in.
True but as someone else pointed out; at that time we'd be interested in running 200T parameter model rather than 200B. Why, you might ask? Law of human laziness - a human will become as lazy as the technology allows it to. With the 200T or 20,000 T model - I'd be heavily incentivized to ask it to make the bread for me that I enjoy making now or create a movie for me (featuring myself) which will maximize the dopamine production in my brain.
I know a lot of level-headed engineers here may not side with me, but I say let the companies who abandoned their people at the drop of a hat, with CEOs who waved their flag around on social media, proudly declaring how they'd now run their companies with 75% fewer employees wither and die. If I had been let go, there's no way I'd go back to a company like that, and there should be a black list of CEOs who acted this way established and kept public. These CEOs are not holistic thinkers, and are too susceptible to mass hysteria and too irresponsible to real people and their lives to be trusted with the vision for any company ever again.
GM just did this in the last 30 days [1], and their sales are likely going to be just fine. In fact the auto industry has repeatedly automated jobs over the last 100 years, and they still make decent sales numbers.
If you decided to boycott every company that replaced staff with automation, you would be forced to exit the economy. Every company does this to some degree and the customers who vote with their wallet do not seem to care about a reduction in force.
Robots that replace auto industry factory workers exist; the CEO of GM didn't imagine them as part of some sort of business media induced psychotic episode.
The same is not true for the software industry execs.
This is true, and I'm sure AI cuts will continue, but it's obvious that the ones who went "all in" at AI's mass introduction were drinking a special kind of Kool-Aid reserved for the truly sycophantic Wall Street lap dogs, not the CEOs who think about risk and are cautious about betting the farm on a relatively new and mostly untested technology. GM is over 100 years old, and no doubt released improvements that were well-tested and predictable, because you don't take massive chances with a company that well established. It was a couple years into the mass AI deployment that studies on the minimal overall productivity gains of AI even started to come out(!) This was "get on the bandwagon" thinking at a massive scale, which shows you how many CEOs are not independent thinkers at all, but are really just followers. Yes, use AI, but do it responsibly, never forgetting that your investors aren't your only stakeholders - so are your people.
> GM just did this in the last 30 days [1], and their sales are likely going to be just fine. In fact the auto industry has repeatedly automated jobs over the last 100 years, and they still make decent sales numbers.
I worked at Verizon during their layoffs last year. Biggest layoffs in the USA.
As someone who’s been laid off before, I knew that it generally boosts the stock price.
I bought VZ because of that. It’s up 15% since the layoffs.
Microsoft, an AI stock, is down 30% in the same timeframe.
What was really behind the push to get everything browser based in the first place? Is this all to make everything cloud based, software as a service, or did some exec see a demo of Windows 8 and think "web is the future" and over-rotate?
Turns out the browser is an incredibly sophisticated and highly performant layout engine that works on almost every platform out there. Native UI frameworks are always going to be more efficient and let you access more of the hardware, but it's more expensive to maintain 2 or 3 separate codebases.
The push is that web developers are easy to find, and native software developers barely event exist anymore (and if they do can get paid to work on things like trading software, though web is even picking up there!).
Microsoft don't even _have_ a reasonable desktop UI stack, having been through at least 4-5 which gained minimal traction before being abandoned. The last successful one was Windows Forms, which is what I'd pick up today if I ever had to touch Windows again.
The least cynical argument is internationalization. Rendering engines already implement robust text layout when multiple languages are involved, so it makes sense to leverage it. The more cynical argument is that it's easier to find web designers than desktop UI designers. The more-cynical-still argument is the one you said.
I am finding this exceptionally difficult, to be honest. It has been nearly impossible (so far) to get traditional media to take the book seriously (there have been a few exceptions). And the perversity of the modern social media algorithms means that it is extremely difficult to get discussions of the book to break through that barrier. I've had people who follow me say their feed is absolutely saturated with content about the book, and others who have no idea the book has come out. The algorithm decides all, having hundreds of thousands of followers is next to useless anymore.
So, to the extent the book is getting attention at all, it's being driven primarily by word of mouth. People who read the book are inspired to become evangelists for it, far more than my prior books. I think this one is more emotionally resonant for folks, and has applicability way beyond founders or even product people. So I'm extremely grateful for all that support.
But if you look at the view counts on YouTube, you'll see that they are all also suffering from the algorithm's fickle attention. Most have very few views (with a few notable exceptions).
How is having local AI going to produce a result that's any better than using OpenAI or Anthropic? Isn't what we really need programmers who rely on themselves more than AI so they avoid technical debt accumulation?
This is what I have been thinking. Business will always try to do more with less because their only true goal is figuring out how to make more money. They will sacrifice giving those juniors time to learn from their mistakes for the sake of making more widgets (code). From the wider generational view, they will rob today's juniors from the chance to learn and thereby keep the talent pipeline full so they can profit today, the future (and the developers who will arrive there) be damned. The economic game is flawed because it only ever comes down to a single output that is optimized for: money. One solution? I think software people might consider forming unions. I know that's antithetical to the lone coder ethos, but if what this comment reflects is true, the industry needs a check and balance to prevent it from destroying its foundation from the inside.
Why would a business train anyone when they can lean on the govt to provide unbankruptable loans to the student to go to university to learn themselves
https://www.computerhistory.org/storageengine/first-commerci...
Compare that to a multi-terabyte ssd. Now apply that improvement to how an LLM is architected and run now. With AI assisting, it won't be long before a leap occurs and these data centers with all their current ultra-cutting edge Nvidia cards are nearly obsolete overnight.
reply