>Isn't it a bit exaggerating to say that users cannot use Snapdragon laptops except for running LLMs?
I think maybe what OP meant was that the memory occupied by the model meant you couldn't do anything alongside inferecing, e.g. have a compile job or whatever running (unless you unload the model once you've done asking it questions.)
to be honest, we could really do with RAM abundance. Imagine if 128GB ram became like 8GB ram is today - now that would normalize local LLM inferencing (or atleast, make a decent attempt.)
i cant believe we're still putting NPUs into new designs.
silicon taken up that couldve been used for a few more compute units on the GPU, which is often faster at inference anyway and way more useful/flexible/programmable/documented.
You can thank Microsoft for that. Intel architects in fact did not want to waste area on an NPU. That caused Microsoft to launch their AI-whatever branded PCs with Qualcomm who were happy to throw in whatever Microsoft wanted to get to be the launch partner. After than Intel had to follow suit to make Microsoft happy.
That doesn’t explain why Apple “wastes” die area on their NPU.
The thing is, when you get an Apple product and you take a picture, those devices are performing ML tasks while sipping battery life.
Microsoft maybe shouldn’t be chasing Apple especially since they don’t actually have any marketshare in tablets or phones, but I see where they’re getting at: they are probably tired of their OS living on devices that get half the battery life of their main competition.
And here’s the thing, Qualcomm’s solution blows Intel out of the water. The only reason not to use it is because Microsoft can’t provide the level of architecture transition that Apple does. Apple can get 100% of their users to switch architecture in about 7 years whenever they want.
Bingo. Maybe Microsoft shouldn’t even be chasing them but I think they have a point to try and stay competitive. They can’t just have their OS getting half the battery life of their main competitor.
When you use an Apple device, it’s performing ML tasks while barely using any battery life. That’s the whole point of the NPU. It’s not there to outperform the GPU.
Every modern chip needs some percentage dedicated to dark silicon. There is no cheating the thermal reality. You could add more compute units in the GPU, but you then have to make up for it somewhere else. It’s a balancing act.
The Core Ultra lineup is supposed to be low-power, low-heat, right? If you want more compute power, pick something from a different product series.
> Every modern chip needs some percentage dedicated to dark silicon. There is no cheating the thermal reality. You could add more compute units in the GPU, but you then have to make up for it somewhere else. It’s a balancing act.
I think that "dark silicon" mentality is mostly lingering trauma from when the industry first hit a wall with the end of Dennard scaling. These days, it's quite clear that you can have a chip that's more or less fully utilized, certainly with no "dark" blocks that are as large as a NPU. You just need to have the ability to run the chip at lower clock speeds to stay within power and thermal constraints—something that was not well-developed in 2005's processors. For the kind of parallel compute that GPUs and NPUs tackle, adding more cores but running them at lower clock speeds and lower voltages usually does result in better efficiency in practice.
The real answer to the GPU vs NPU question isn't that the GPU couldn't grow, but that the NPU has a drastically different architecture making very different power vs performance tradeoffs that theoretically give it a niche of use cases where the NPU is a better choice than the GPU for some inference tasks.
i have an even stupider question, which is what if we scrapped ip addresses and just used ethernet addresses to route everything? just make the entire internet network be one big switch.
i think the guy who created tailscale wrote about something like this...
theres a fun "K-shaped" optionality with LLMs: on one hand, its possible to deal with otherwise large API surfaces.
but on the other hand, you can 'go oldschool' but with the hot new tools: install ubuntu, launch claude with yolo mode, and just tell it what you want as if it were a sysadmin from the early 2000s/late 90s.
both roads very reasonable, but that the old way of doing things is new again is interesting.
I've been having a great time prompting "vanilla JavaScript, no react" and building neat things that use browser APIs exclusively (including modern stuff like web audio APIs and Web Components and WASM) because I don't need to learn a bunch of boilerplate stuff first anymore.
Feels like coding in the 200xs and I'm enjoying every minute of it.
And it's not an either-or. For example, I found that a quick way to get a web frontend for a console app is to prompt it to turn that into a CGI app. But said CGI app can still serve HTML with fancy JS and what not, and use modern frameworks for that if desired.
embedded dev with a billion toolchains, GPU development with each vendors bespoke API, ffmpeg with its billion parameters - if anything, you could say LLMs bailed us out of the impending ultra-specialization. without LLMs, we might be facing a world where 30% of the workforce is in software dev.
i am keeping my eyes peeled on vibe-coding PCB layouts and schematics. a lot of eyes in that direction already but its still early.
Not if you sit at home wearing boots and looking at photos of mountains.
If you want to have boots, that's cool. But is replacing walking with ordering boots and photos making hiking fun again? Or were you only interested in the photos anyway?
What part of the process of hiking do you enjoy? And why is it so hard to hear what part of the process of programming people enjoy?
This is just obtuse. Some folks have fun building their own pizza oven, curing & slicing their own meat, and mixing their own dough. Some folks like to buy mostly pre-made stuff and just play with a few special ingredients. Some folks want to make 5 different pizzas with different flavors. Some folks just order a pizza.
Some folks walk out of their house and start hiking. Some folks drive somewhere and then start walking. Some folks take photos from the car. Some folks take a roadtrip.
All of these things ask for different effort & commitment with different experiences & results as the payoff. At least be honest about that.
Like, fine, here's a personal example: I wanted to build a system that posts web links I share to a bot account on the fediverse. That seemed like a fun result to me.
I wanted to self-host the links, so I installed Linkding. (I didn't write Linkding.) For the fediverse bot, I installed gotosocial as the service host (I didn't write gotosocial.)
From there, a cronjob running a small program using Linkding and gotosocial APIs could do the trick. Decided to do it in golang, because the standalone binaries are easy to deploy.
Writing that small program didn't seem like fun - I've already played with those APIs and golang. What I wanted, for my enjoyment, was the completed system.
So, I took 10 minutes to write out a quick spec for the program and what I wanted it to do. I loaded that up as context for Claude Code along with some pointers for building CLI apps in golang. I let it rip and, in about 20 minutes, Claude produced a functional tool. It also wrote a decent README based on my original prose.
I reviewed the code, did some testing, made some tweaks, called it done. My bookmarks are now regularly posted to a bot account on the fediverse. This is an enjoyable outcome for me - and I didn't have to type every line of code myself.
For bonus points, I also had Claude Code gin up some GitHub Actions workflows to lint, test, build, and release multi-platform binaries for this tool. I've done these things before, but they're tedious. More enjoyable to have the resulting automations than to build them. And now I have them: I can make tweaks to this tool and get builds just through the GitHub web UI.
I've since repeated this pattern with a handful of other small personal tools. In each case, I wanted the tool and the utility it offered. I didn't care about the process of writing the code. It's working pretty well for me.
Seeing the output I want when I describe it, and making changes to get to the vision in my mind. I don't have aphantasia so maybe it's different for those who do, but I can literally see the UI of the app I want to build and of course I can build it by writing code manually too, but I can make it exist much faster with an LLM than without.
100% the opposite. LLMs lack high level creativity, wisdom and taste. Being a generalist is how you build these.
For example, there's a common core to music, art, food, writing, etc that you don't see until you've gotten good at 3+ aesthetic fields. There are common patterns in different academic disciplines and activities that can supercharge your priors and help you make better decisions.
LLMs can "see" these these connections if explicitly prompted with domains and details, but they don't seem to reason with them in mind or lean on them by default. On the other hand, LLMs are being aggressively RL'd by the top 10% of various fields, so single field expertise by some of the best in the world is 100% baked in and the default.
I agree with this. I've been able to tackle projects I've been wanting to for ages with LLMs because they let me focus on abstractions first and get over the friction of starting the project.
Once I get my footing, I can use them to generate more and more specialized code and ultimately get to a place where the code is good.
its fun seeing HN articles with huge upvotes but no comments, similar to when some super esoteric maths gets posted: everyone upvotes out of a common understanding of its genius, but indeed by virtue of its genius most of us are not sufficiently cognitively gifted to provide any meaningful commentary.
the karpathy vids are very cool but having watched it, for me the takeaway was "i had better leave this for the clever guys". thankfully digital carpentry and plumbing is still in demand, for now!
actually it's quite the opposite: lectures are as approachable as one could possible make them, no fancy math and a walkthrough over attention is all you need
I think maybe what OP meant was that the memory occupied by the model meant you couldn't do anything alongside inferecing, e.g. have a compile job or whatever running (unless you unload the model once you've done asking it questions.)
to be honest, we could really do with RAM abundance. Imagine if 128GB ram became like 8GB ram is today - now that would normalize local LLM inferencing (or atleast, make a decent attempt.)
ofcourse youd need the bandwidth too...
reply