What is the piano keyboard test? Your link requires granting AI Studio access to Google Drive, which I do not want to do.

Just ask it to generate a correct piano keyboard. It's something the current gen of image generator AIs fail at.

Do most humans pass?

Most humans fail at 4 digits multiplication, or drawing a cube in perspective.

Presumably most humans with a camera do

2-2-1-2-2-2-1

I still feel like most humans would fail, haha.

Maybe, but anyone who knows what a chromatic scale is should be able to reason it out. E# == F, B# == C, so no black keys between those.

The selling point of this model really seems to be it's consistency between generations rather than it's raw generating ability.

for instance:

https://aistudio.google.com/app/prompts/1gTG-D92MyzSKaKUeBu2...

I can’t see it. You probably need to set permissions to “anyone with the link can access.”

Are their models that have vector space that includes ideas, not just words/media but not entirely corporeal aspects?

So when generating a video of someone playing a keyboard the model would incorporate the idea of repeating groups of 8 tones, which is a fixed ideational aspect which might not be strongly represented in words adjacent to "piano".

It seems like models need help with knowing what should be static, or homomorphic, across or within images associated with the same word vectors and that words alone don't provide a strong enough basis [*1] for this.

*1 - it's so hard to find non-conflicting words, obviously I don't mean basis as in basis vectors, though there is some weak analogy.

How would you encode those ideas?

I don't know, in part that's why I asked ... I wonder if there's a way to provide a loosely-defined space.

Perhaps it's a second word-vector space that allows context defined associations? Maybe it just needs tighter association of piano_keyboard with 8-step_repetition??

Interesting! I feel like that's maybe similar to the business of being able to correctly generate images of text— it looks like the idea of a keyboard to a non-musician, but is immediately wrong to someone who is actually familiar with it at all.

I wonder if the bot is forced to generate something new— certainly for a prompt like that it would be acceptable to just pick the first result off a google image search and be like "there, there's your picture of a piano keyboard".

Anything that is heavily periodic can definitely trip up image gen - that being I just used Flux Kontext T2I and got a got pretty close (disregard the hammers though since thats a right mess). Only towards the upper register did it start to make mistakes.

https://imgur.com/a/fyX42my

Doesn't pass the analog clock test either.

Like most image models, except GPT-4o, it also didn't pass the wooden Penrose triangle test. (It creates normal triangles.)

or my "hands with palms facing down" test.... no matter how hard I try it just can't get open hands, palms down.

It's probably just a matter of rerolling a few times. I was able to get it around 25% of the time.

https://imgur.com/a/H9gH3Zy

that's pretty good. I was using a cartoon girl as an example of a dance move for kids.

https://g.co/gemini/share/0e0de0d42029

I guess the vast majority of images have the palms the other way, that this biases the output. It's like how we misinterpret images to generate optical illusions, because we're expecting valid 3D structures (Escher's staircases, say).

Yes - it's the same reason generating a 5-leaf clover fails - massive amounts of training data that predisposes the model against it.

Failed my horizontal text test as well.