Ali played 53 ODIs, scoring 1,845 runs at an average of 36.90. He scored three centuries and 12 half-centuries, including a 59 against India in the triumphant Champions Trophy final last year in London.
Ali lost the confidence of the selectors in ODIs this year and was dropped after Pakistan was swept in New Zealand in January. He was replaced as opener by Imam-ul-Haq, the nephew of chief selector Inzamam-ul-Haq. Imam has cemented his place with a prolific run in ODIs.
"I didn't decide to retire after any hard feelings, it's my personal decision," Ali said.
"As a former ODI captain, I wish the team the very best for what is an incredibly important season ahead of the World Cup next June."
Pakistan slipped to an all-time low of No. 9 in the ODI rankings during Ali's tenure as skipper from 2015 to early 2017. Pakistan won four and lost five bilateral series.
Pakistan Cricket Board chairman Ehsan Mani said Ali will remain an "integral part" of Pakistan's test team.
Ali, who turns 34 in February, has played 67 tests for an average of 44. He featured last month in the home series win over Australia. He has never played Twenty20s for Pakistan.