We asked the LLM after seeing complaints about GPT-5 hallucinating on social media. “generate a map of the USA with each state named.” The LLM responded by giving us an illustration that has the shapes and sizes of the states correct but many of the names are misspelled or made-up.
Click to enlarge the map of the US drawn in GPT-5.
As you can tell, Oregon is “Onegon,” Oklahoma is “Gelahbrin,” while Minnesota is “Ternia.” All of the state names except for Montana and Kansas are incorrect. Some of the letters can’t be read.
We asked GPT-5 to “generate a map of South America” all countries listed in order to see if it only had a problem with US. It got more names right this time, but there were still some notable mistakes.
It shows Suriname and Uruguay as “Guriname,” but Ecuador is “Felizio,” . We also wanted to see if this fact-drawing issue would affect a drawing which is not a geographical map. We asked GPT-5 to “draw a timeline of the US presidency with the names of all presidents.”
Of all the graphics that we requested, the timeline graphic GPT-5 provided was the least accurate. The timeline graphic GPT-5 provided was the least accurate of all the graphics we requested. It only listed 26 presidents and the years weren’t in the correct order. Many of the names were also made up.
While Jefferson’s name is misspelled, and the third president didn’t serve in 1931, the first three lines are mostly correct. Our fourth president is “Willian H. Brusen,” and he lived in the White House in 1991. Benlohin Barrion served in 1879, and Henbert Bowen in 1934.
Oddly, when we asked GPT-5 “make an infographic showing all the actors who played James Bond in order,” to give us perfect text. The first time we asked it this question, it gave us the answers in text only. It was quite good when we followed it up with “you didn’t draw an image,” . We’ll forgive the fact that Connery’s role from “Diamonds Are Forever” (1971) was not included.
GPT-5 can provide the correct answers for queries in text form, even when it fails to draw accurate images. It was accurate when we asked for a complete list of US states and South American countries. It was accurate in its list of US presidents, except for the fact that Joe Biden is shown as “2021-present.” . This could be because GPT-5 has not been trained to keep up with the latest news. OpenAI hasn’t disclosed the dates of training for this model. However, it is likely that the period predates the second term of President Trump.
It is not clear why GPT-5 has problems with names of places and individuals when drawing infographics. OpenAI’s spokesperson told us GPT-5 uses Image Gen 4o. While they said that it does a much better job of rendering text on images than previous models, they acknowledged that distortions do still occur. It did not explain why. OpenAI removes ChatGPT’s self-doxing feature
But we have some theories. Image generators use a diffusion process to train on other images. They take the training pictures, turn them into random noise, and then reconstruct the images. All image generators face a challenge when it comes to accurately generating text for image outputs. Text generated by diffusion models used to look more like hieroglyphics, or alien languages than anything that looked like English.
We got similar results when we asked Bing Image Creator for a map of the US that included state names.
Click to enlarge the US map drawn by Bing Image creator
Even worse, Bing called the country the “United States Ameriicca.” And it failed the James Bond Test. Who do you think the men with white hair belong to?
The output looks more like a list of boxes than a map.
Interestingly enough, after encouraging GPT-5, it produced a map that was accurate. It’s probably because it was using code creation, which is a different process.
Google Gemini has done a poorer job with state names than GPT-5. On the map below, there is not a single state that is correct.
(Click to enlarge) Map of the US drawn by Gemini
On the plus side, Gemini has created a wonderful James Bond Infographic. It deserves extra credit for not only including the main actors in the films but also more than two dozen recurring characters on a time line.
It is clear that drawing text within an image is difficult and neither GPT-5 or its competitors have yet mastered it. . . If you ask about James Bond, they will not answer. (r)
Updated on Aug 8 at 2300 GMT to include OpenAI’s comments.
