Artificial intelligence systems may be good at generating text, recognizing images, and even solving basic math problems—but when it comes to advanced mathematical reasoning, they are hitting a wall.

In the third century BCE, Apollonius of Perga asked how many circles one could draw that would touch three given circles at exactly one point each. It would take 1,800 years to prove the answer: eight ...