"The expression I used ran counter to the spirit of the Olympics and Paralympics. It was inappropriate," Yoshiro Mori told reporters. "I would like to retract the remarks and apologize to anyone I have offended."
Mori had complained on Wednesday about the Japanese Olympic Committee's push for gender diversity on its board, saying women have "a strong sense of rivalry" and that when one female member raises her hand to speak, "everyone ends up saying something."
Also read:Speculation over Tokyo Olympics: 2021, 2032 or not at all?
He made the comments in a JOC meeting held online and open to the media.
The JOC has set a goal of increasing the number of women on its board to 40 percent. Women currently comprise only 20 percent of members.
Mori's comments caused a stir in Japan and abroad, sparking calls for him to resign.
When reached for comment by Kyodo News on Thursday before he spoke to the reporters, the 83-year-old former prime minister said he is aware of the public outcry and that he would step down if the controversy, or any other issue he is involved in, has a negative impact on the games.
Also read:Tokyo Olympics postponed
Japan's top government spokesman, chief Cabinet secretary Katsunobu Kato, told a press conference he was not aware of the specifics of Mori's comments, but the government is committed to gender equality and will continue to push for the promotion of women to leadership roles in sports organizations.
Tokyo Gov. Yuriko Koike said it "goes without saying" that women should be included in the decision-making process, adding she believes someone "talking too much" is an issue with an individual, rather than their gender.
Also read:Japan and IOC deny Tokyo Olympic cancellation report
Public opposition to the Olympics, set to begin in July after being postponed for one year, is growing amid the COVID-19 pandemic. Around 80 percent of respondents in a Kyodo News survey last month were in favor of cancelling or pushing them back again.