Honestly... it's complicated. As I understand it, African-American is one of the more recent ways of saying (with the intent of being polite) that someone is a black skinned and ethnically African and is American.
The term was not meant to extend to corner cases like mine - I'm white, but one of my ancestors grew up on the African continent in an island in the Atlantic and spoke Portuguese.
I suspect that this term will fall out of favor soon and be replaced by another.
By that logic, does American Asian only refer to people with certain facial characteristics? And are American Europeans only those with a white skin color?
I mean here in Europe we have plenty of Europeans that are not white. So, if a black skinned German went to the US, people would suddenly label him "African" American, just based on his skin color?
Americans of European descent tend to refer to themselves by nation of immigrant origin (eg: Italian American, Irish American, etc.)
However, because African slaves brought to the New World were denied the ability to culturally identify with, or even know (after generations) which African nation or tribe they descended from, the only common cultural identity they could form, apart from that which slavery had forced upon them, was as a diaspora from "Africa" as a whole.
"African American" is a term with a specific cultural and historical context behind it linked to the identity of the descendants of slaves and to black immigrants from Africa, and there is controversy about its use even within the black community, but it isn't going to correctly refer to the descendant of white Dutch colonizers in South Africa even if it does technically fit. Or at least not without coming across as racist itself.
Consider that racist if you like, but race and culture are messy and complicated everywhere, not just in the US.
Racism will exist as long as people look at the skin color or other generic markers to judge others. If you think it's okay to do so, that's sad.
Fighting racism with even more racism, will world three opposite result. It's not the first time in human history, that people mean well and create hell. It's just a different topic this time.
Edit: why the downvote? Is he not?