They traveled in an easterly direction towards Bethlehem. The Wise Men came from the East.

They went east from a westerly direction.  So how could they come from the east? How could they be the Wise Men from the east?

(That is the riddle.)

The answer: They lived in a place that was called the East. Short for the east side of something. Probably a lagoon. Fishing grounds they shared with the Hebrews. Most likely in northern Africa, on the coast.