On Wednesday, a motion of no confidence was submitted as MPs blamed Prime Minister Soumeylou Boubeye Maiga for failing to handle the unrest.
Last month, scores of herders were killed by a rival ethnic group.
President Ibrahim Boubacar Keita said in a statement that he accepted Mr Maiga and his ministers' resignation.
"A prime minister will be named very soon and a new government will be put in place after consultations with all political forces," the statement said.
Mali has been struggling to control violence since Al-Qaeda-linked Islamist extremists gripped the desert north of the country in 2012.
Despite an ongoing military drive and a 2015 peace agreement, jihadists still dominate areas huge areas of the country, having migrated from the north to the more heavily populated centre of the country.
The government has come under increasing pressure over its inability to restore stability, particularly after the massacre of 160 Fulani herders in the Mopti region.
Armed with guns and machetes, the attackers appeared to be members of the Dogon ethnic group, which has a long history of tension with the nomadic Fulani people.
The country was shocked by the killings and tens of thousands of people protested on the streets of the capital Bamako on April 5.
The president said in a televised address on Tuesday that he had "heard the anger".