This seems roughly equivalent to liability insurance.

It’s much easier to accept fatalities caused by other humans because there is someone to hold responsible. Will autonomous vehicle companies be held responsible when they cause fatalities?

It also goes beyond just the total number of fatalities. Just like we don’t accept DUIs, we shouldn’t accept negligence or laziness from autonomous vehicle developers even if their product is safer than human drivers.

On the other hand we have very lenient punishments for damage, injury and deaths caused by drivers, and are often reluctant to actually apply them. As long as no DUI is involved we are willing to accept a lot of negligence from human drivers

I think the more interesting question is who will be on the panel?

A group of ex frontier lab employees? You could declare AGI today. A more diverse group across academia and industry might actually have some backbone and be able to stand up to OpenAI.

How is this not a terrible deal for Microsoft? I’m not confident that an “expert panel” will prevent OpenAI from prematurely declaring AGI.

This was really a bad release for OpenAI, if benchmarks are even somewhat indicative of how the model will perform in practice.

I actually don't agree. Tool use is the key to successful enterprise product integration and they have done some very good work here. This is much more important to commercialization than, for example, creative writing quality (which it reportedly is not good at).

In what ways?

OpenAI’s systems haven’t been pure language models since the o models though, right? Their RL approach may very well still generalize, but it’s not just a big pre-trained model that is one-shotting these problems.

The key difference is that they claim to have not used any verifiers.

What do you mean by “pure language model”? The reasoning step is still just the LLM spitting out tokens and this was confirmed by Deepseek replicating the o models. There’s not also a proof verifier or something similar running alongside it according to the openai researchers.

If you mean pure as in there’s not additional training beyond the pretraining, I don’t think any model has been pure since gpt-3.5.

Local models you can get just the pretrained versions of, no RLHF. IIRC both Llama and Gemma make them available.

I’m not trying to take away from the difficulty of the competition. But I went to a relatively well regarded high school and never even heard of IMO until I met competitors during undergrad.

I think that the number of students who are even aware of the competition is way lower than the total number of students.

I mean, I don’t think I’d have been a great competitor even if I tried. But I’m pretty sure there are a lot of students that could do well if given the opportunity.

Are you in the US? Have you heard of the AMC (used to be AHMSE) and the AIME? Those are the feeders to the IMO.

If your school had a math team and you were on it, would be surprised if you didn't hear of it

You may not have heard of the IMO because no one in school district, possibly even state got in. It is extremely selective (like 20 students in the entire country)

I’m in the US but this was a while back, in the south. It was a highly ranked school and ended up producing lots of PhDs, but many of the families were blue collar and so there just wasn’t any awareness of things like this.

I'm curious on your response to GP's question. Have you heard of AHSME, AMC, or AIME?

Nobody mentioned them in high school (1997) until I heard of them online and got my school to participate. 30 kids took the AHSME. Only one qualified for the AIME. And nobody qualified for IMO (though I tell myself I was close).

I believe the 1 in a million number.

Never heard of those either. And I took the highest level math courses offered. The only competition I can remember is participating in is Academic Decathlon.

Has anyone independently reviewed these solutions?

My proving skills are extremely rusty so I can’t look at these and validate them. They certainly are not traditional proofs though.

I read through P1, and it seemed to be correct. Though you could explain the central idea of the proof into about 3 sentences and a few drawings.

It reads like someone who found the correct answer but seemingly had no understanding of what they did and just handed in the draft paper.

Which seems odd, shouldn't an LLM be better at prose?

One would think. I suppose OpenAI threw the majority of their compute budget at producing and verifying solutions. It would certainly be interesting to see whether or not this new model can distill its responses to just those steps necessary to convey its result to a given audience.

Access to cheap goods may lead to short term satisfaction, but will never lead to long term fulfillment.

Where are the expectations coming from? The major labs continually claim that these models are now PhD level, whatever that even means.