The visit is the first by an Iranian president in 19 years and comes just before Japan is expected to announce a plan to deploy its troops to the region to ensure the safety of Japanese vessels transporting oil.
Japan is keen to restore stability in the Middle East, the source of more than 80% of Japan's oil.
A U.S. ally that has traditionally had friendly relations with Iran, Japan also seeks to serve as a mediator between Tehran and Washington. Abe has said such a role is also expected by the international community.
Rouhani, currently in Malaysia to attend an Islamic conference, is making an overnight trip just to meet Abe and have dinner with him, Japanese Foreign Ministry officials said.
Tensions have escalated between Tehran and Washington since President Donald Trump's decision last year to withdraw from Iran's 2015 nuclear deal with world powers.
Abe, who has developed friendly ties with Trump, traveled to Iran in June in an unsuccessful effort to encourage Iran and the U.S. to hold talks to reduce tensions. U.S. sanctions on Iran, which block it from selling crude oil abroad, have crippled its economy, and Tehran has gradually reduced its commitment to the nuclear deal.
Abe's government is to announce a contentious plan to send its military to the Middle East to ensure Japanese oil tankers' safety. Japan's troop dispatch won't be part of a U.S.-led coalition protecting Middle East waterways, apparently an attempt to maintain neutrality in a show of consideration to Iran.
Abe plans to thoroughly explain this to Rouhani to gain his understanding, officials said.
Sending warships to areas of military tension is a highly sensitive issue in Japan because its pacifist post-World War II constitution strictly limits the use of force by the military. Abe, however, has gradually expanded Japan's military role in recent years.
In June, a Japanese-operated tanker was attacked in the Gulf of Oman. Washington said Iran was responsible and urged Japan to join the U.S.-led military initiative.
Rouhani may ask Japan to resume purchase of Iranian oil, but it may be difficult because of international sanctions still in place. Japanese companies have stopped importing Iranian oil since May.