Arsenal was moments away from advancing in regulation time on Wednesday until Divock Origi's equalizer in the fourth minute of stoppage time made it 5-5.
Then, only Arsenal midfielder Dani Ceballos missed his penalty, allowing teenager Curtis Jones to seal Liverpool's progress with 5-4 shootout victory on Wednesday.
There was less drama at Stamford Bridge but still a stunning winning goal. Marcus Rashford struck a free kick from 30 yards that dipped high into the net to clinch a 2-1 win for Manchester United against Chelsea after the striker's first-half penalty had been canceled out by Michy Batshuayi.
Aston Villa also advanced on Wednesday after Anwar El Ghazi and Ahmed Elmohamady scored in a 2-1 victory over Wolverhampton.
Holder Manchester City, Everton and Leicester joined lower-league sides Colchester and Oxford in reaching the quarterfinals on Tuesday.
ANFIELD FRENZY
On only his third appearance of the season, Mesut Özil provided a reminder of his abilities as he had a hand in three of Arsenal's goals.
However, he was taken off just past the hour with Liverpool in the ascendancy having twice fought back from two-goal deficits to level at 4-4.
The first goal of the night came after six minutes when Arsenal defender Shkodran Mustafi diverted the ball into his own net after coming under pressure from Rhian Brewster after Alex Oxlade-Chamberlain's cross came into the penalty area.
Arsenal drew level in the 19th minute. Bukayo Saka's shot from Ozil's cutback was parried by goalkeeper Caoimhin Kelleher and Lucas Torreira netted while offside.
But with no VAR used in the competition, the goal stood.
Liverpool fell behind when Kelleher parried Sepp Van Den Berg's deflection to Gabriel Martinelli who netted in the 26th. Martinelli, the Brazilian teenager, then made it 3-1 from another rebound 10 minutes later — taking his tally for the season to seven goals.
The absence of video replays cost Liverpool again when Naby Keita's shot hit the outstretched hand of Hector Bellerin. But from the resulting corner, Harvey Elliott was fouled by Martinelli and Milner netted from the penalty spot.
Origi should have powered home a header before the break but the biggest surprise of the night was Milner leaving a back-pass short 10 minutes into the second half.
Maitland-Niles overran the ball but Ozil's drag-back retrieved the ball close to the byline presenting his teammate with a simple goal to put Arsenal 4-2 in front.
But after former Arsenal forward Alex Oxlade-Chamberlain's strike from distance, Origi produced the first of Liverpool's equalizers.
He was required again deep into stoppage time with a volley, just when it seemed Joe Willock's 70th-minute strike would be enough to send Arsenal through.
But after scoring five in regulation time, Arsenal couldn't repeat the feat. Liverpool did, the fifth netted by the 18-year-old Jones.
"Being a local lad it's always been my dream to get an appearance at Anfield," he told the BBC. "I was calm, I picked my spot and I was fortunate enough that the winning penalty went in. The team played great, they showed character and belief.
With the shootout win, Liverpool remains in contention for every competition this season while sitting six points clear at the top of the Premier League.