Rouhani proposed an Islamic financial payment with Muslim countries trading in local currencies and the creation of a Muslim cryptocurrency to cut reliance on the U.S. dollar and weather the effects of market fluctuation.
He repeated his assertion that the U.S. has used economic sanctions as the "main tools of domineering hegemony and bullying" of other nations.
Iran has been gripped by violent protests since last month amid anger over a sharp hike in gasoline prices following U.S. sanctions after Washington withdrew last year from Iran's 2015 nuclear deal with world powers. There has been a wave of arrests in Iran and rights group Amnesty International has said at least 304 people were killed in the anti-government protests.
Rouhani said the U.S. sought to cripple Iran with the "heaviest sanctions" but the country's economy is on the mend and moving away from reliance on oil.
"The Muslim world should be designing measures to save themselves from the domination of the United States dollar and the American financial regime," he said at the opening ceremony of a three-day conference that includes leaders from Turkey, Qatar and host Malaysia.
Rouhani suggested the creation of a special banking and financial system among Muslim nations, using local currencies for trade and giving each other trade privileges to deepen linkages.
He said rising extremism as well as challenges such as weak governance, poverty and corruption are endangering sovereignty and paving the way for Western interference in Syria, Yemen, Afghanistan and other Muslim nations. But if Muslim countries harness their collective strength, they could confront such problems, he said.
Rouhani proposed that the conference in Kuala Lumpur establish a joint fund to finance technological cooperation among Muslim nations and set up a a joint research center in artificial intelligence and cyberspace.
Notably absent from the conference is Saudi Arabia, which snubbed the meeting because it was not held under the Saudi-based Organization of Islamic Cooperation. Saudi Arabia and Iran are bitter rivals. The world's two biggest Islamic nations, Indonesia and Pakistan, also didn't join the conference.
Turkish President Recep Tayyip Erdogan launched a veiled criticism of the OIC, telling the conference that the platform that brings the Muslim world together was besieged by a "lack of implementation."
"We still haven't made any progress regarding the Palestinian cause, we still can't stop the exploitation of our resources, we still can't say 'stop' to the fragmentation of the Muslim world over sectarianism, that's why," said Erdogan.
Muslims account for 94% of worldwide conflict with one out of three weapons sold globally finding its way to the Middle East, he said. "The Muslims are using their resources for armament and for conflict and while doing so they are enriching the Western arm dealers," he said.
Host Malaysian Prime Minister Mahathir Mohamad downplayed divisions over the conference, saying it was merely a small step to address Muslim issues and wasn't aimed at isolating anyone.
Mahathir, the world's oldest leader at 94, said Muslims have damaged their own religion by their acts of terror that led to an "unjustified fear of Islam" worldwide. Their claims of jihad led to more oppression of Muslims and killing of innocent victims worldwide, he said.
Echoing Rouhani and Erdogan, Mahathir said the Muslim bloc must refocus on education and step up its scientific and technological capabilities to catch up with developed nations.
"While we may not have been able to fully dissect all that had caused our pain and anguish, we are mostly in agreement that it is our inability to keep up with the progress and development of the non-Muslims that have left us in the lurch," he added.
Rouhani leaves Friday for a two-day visit to Japan.