On a night when former European champions Porto and Celtic were both knocked out, Ajax faced elimination when its former youth player Diego Biseswar scored for PAOK in the 23rd minute to put the Greek team 3-2 ahead on aggregate.
Ajax captain Dusan Tadic saw one penalty saved, but scored a second to draw level. A Nicolas Tagliafico header finally put the Dutch team ahead in the 79th, before PAOK's rough, physical style handed Ajax a third penalty converted by Tadic.
Biseswar scored again in stoppage time for PAOK to again put Ajax in danger, but the Dutch champion held on to win. PAOK's players and staff surrounded the referee to protest after the final whistle.
"It was a tough evening. I'm glad we got there in the end, but don't ask me how," Ajax midfielder Donny van de Beek said. "It's a very tough team to play against."
In the playoff round, the last stage of qualifying, Ajax faces a likely easier task against APOEL of Cyprus, which beat Azerbaijan's Qarabag 3-2 on aggregate.
The fact that Ajax even had to play qualifying prompted debate over UEFA's rules, which tend to favor less-storied teams from big countries over high achievers from smaller nations.
Ajax's place in the qualifying rounds was down to the Dutch league's low ranking, despite its heroics last season. By contrast, Italy's strong record means Atalanta gets a Champions League group stage spot despite never having played in the competition before.
Elsewhere, Porto was beaten on away goals by Russia's FC Krasnodar after conceding three in the first half. At 3-3 on aggregate, the Portuguese team threw players forward in search of a winning goal, but dogged defending kept the Russians ahead. They will face Olympiacos for a place in the group stage.
Romania's CFR Cluj scored two late goals against Celtic to win 4-3 on the night and 5-4 on aggregate following after a tense game in Glasgow, while former champion Red Star Belgrade came through a marathon penalty shootout after a 2-2 aggregate draw.
Red Star and FC Copenhagen took 11 penalties apiece before Jonas Wind's miss sent the Serbian team through. Wind and Red Star's Radovan Pankov took two penalties apiece in the shootout because Red Star had finished the game with 10 men.
Former group-stage regular Dynamo Kiev lost in qualifying for the third straight year, losing 4-3 on aggregate to Brugge.
Dinamo Zagreb, LASK Linz and Rosenborg also went through.