I think this says more about how much of our tasks and demonstrations of ability as developers revolve around boilerplate and design patterns than it does about the Cognitive abilities of modern LLMs.

I say this fully aware that a kitted out tech company will be using LLMs to write code more conformant to style and higher volume with greater test coverage than I am able to individually.

I'd counsel you to work with LLMs daily and agree that we're no where close to LLMs that work properly consistently outside of toy use cases, where examples can be scraped from the internet. If we can agree on that we can agree that General Intelligence is not the same thing as a, sometimes, seemingly random guess at the next word...

I think "we" have accidentally cracked language from a computational perspective. The embedding of knowledge is incidental and we're far away from anything that "Generally Intelligent", let alone Advanced in that. LLMs do tend to make documented knowledge very searchable which is nice. But if you use these models everyday to do work of some kind that becomes pretty obvious that they aren't nearly as intelligent as they seem.

They're about as smart as a person who's kind of decent at every field. If you're a pro, it's pretty clear when it's BSing. But if you're not, the answers are often close enough.

And just like humans, they can be very confidently wrong. When any person tells us something, we assume there's some degree of imperfection in their statements. If a nurse at a hospital tells you the doctor's office is 3 doors down on the right, most people will still look at the first and second doors to make sure those are wrong, then look at the nameplate on the third door to verify that it's right. If the doctor's name is Smith but the door says Stein, most people will pause and consider that maybe the nurse made a mistake. We might also consider that she's right, but the nameplate is wrong for whatever reason. So we verify that info by asking someone else, or going in and asking the doctor themselves.

As a programmer, I'll ask other devs for some guidance on topics. Some people can be absolute geniuses but still dispense completely wrong advice from time to time. But oftentimes they'll lead me generally in the right way, but I still need to use my own head to analyze whether it's correct and implement the final solution myself.

The way AI dispenses its advice is quite human. The big problem is it's harder to validate much of its info, and that's because we're using it alone in a room and not comparing it against anyone else's info.

> They're about as smart as a person who's kind of decent at every field. If you're a pro, it's pretty clear when it's BSing. But if you're not, the answers are often close enough.

No they are not smart at all. Not even a little. They cannot reason about anything except that their training data overwhelmingly agrees or disagrees with their output nor can they learn and adept. They are just text compression and rearrangement machines. Brilliant and extremely useful tooling but if you use them enough it becomes painfully obvious.

Something about an LLM response has a major impact on some people. Last weekend I was in in Ft. Lauderdale FL with a friend who's pretty sharp ( licensed architect, decades long successful career etc) and went to the horse track. I've never been to a horse race and didn't understand the betting so I took a snapshot of the race program, gave it to chatGPT and asked it to devise a low risk set of bets using $100. It came back with what you'd expect, a detailed, very confident answer. My friend was completely taken with it and insisted on following it to the letter. After the race he turned his $100 into $28 and was dumbfounded. I told him "it can't tell the future, what were you expecting?". Something about getting the answer from a computer or the level of detail had him convinced it was a sure thing. I donm't understand it but LLMs have a profound effect on some people.

edit: i'm very thankful my friend didn't end up winning more than he bet. idk what he would have done if his feelings towards the LLM was confirmed by adding money to his pocket..

If anything, the main thing LLMs are showing is that the humans need to be pushed to up their game. And that desire to be better, I think, will yield an increase in supply of high-quality labour than what exists today. Ive personally witnessed so many 'so-so' people within firms who dont bring anything new to the table and focus on rent seeking expenditures (optics) who frankly deserve to be replaced by a machine.

E.g. I read all the time about gains from SWEs. But nobody questions how good of a SWE they even are. What proportion of SWEs can be deemed high quality?

Yes, exactly. LLMs are lossy compressors of human language in much the same way JPEG is a lossy compressor of images. The difference is that the bits that JPEG throws away were manually designed by our understanding of the human visual cortex, while LLMs figured out the lossy bits automatically because we don't know enough about the human language processing chain to design that manually.

LLMs are useful but that doesn't make them intelligent.

Completely agree (https://news.ycombinator.com/item?id=45627451) - LLMs are like the human-understood output of a hypothetical AGI, 'we' haven't cracked the knowledge & reasoning 'general intelligence' piece yet, imo, the bit that would hypothetically come before the LLM, feeding the information to it to convey to the human. I think that's going to turn out to be a different piece of the puzzle.

So, where is my sandwich? I am hungry

You have to keep moving the goalposts if you keep putting them in the wrong place.