GPT-2 has had these consistency issues from the very first demo. The things it gets right are incredibly impressive (orc who can't count -> use math to confuse orc -> orc starts fighting with other orcs).