President Donald Trump announced Shanahan's departure in a tweet, and said Army Secretary Mark Esper would be the new acting Pentagon chief.
"I believe my continuing in the confirmation process would force my three children to relive a traumatic chapter in our family's life and reopen wounds we have worked years to heal," Shanahan said in a statement. "Ultimately, their safety and well-being is my highest priority."
His withdrawal from one of the most critical positions in the government comes at a time of escalating tensions in the Middle East, a day after the U.S. authorized sending additional troops to the region, and after months of unexplained delays in the confirmation process.
The acting defense secretary did not provide specifics, but court records show a volatile family history around the time of his 2011 divorce. The couple had been married since 1986.
His ex-wife, Kimberley, was arrested several times on charges that included burglary, property damage and assault. The assault charge was a misdemeanor for domestic violence in August 2010 when, according to police records, she hit Shanahan a number of times, giving him a bloody nose and black eye. The police report said she was not injured, and he was not charged.
There was also a separate November 2011 incident in which the couple's son, who was 17 at the time, struck his mother with a baseball bat in the home where he lived with her in Sarasota, Florida, according to court records. He pleaded guilty to battery and was sentenced to four years of probation.
In an interview with The Washington Post shortly before Trump announced that Shanahan was withdrawing his nomination, Shanahan spoke about the circumstances surrounding his 2011 divorce and said he didn't want to drag his children through the experience again.
"Bad things can happen to good families ... and this is a tragedy, really," Shanahan told the Post.
In his statement, Shanahan said he asked to be withdrawn from the nomination process and would work on an "appropriate transition."
The Pentagon, in a statement, said Esper will take over the job at midnight Sunday. Esper and Shanahan met at length Tuesday to begin transition planning.
In his tweet, Trump simply said Shanahan had done "a wonderful job" but would step aside to "devote more time to his family." Later, Trump told reporters at the White House that he heard about the problems for the first time Monday.
"I didn't ask him to withdraw, but he walked in this morning," said Trump. "He said it's going to be a rough time for him because of obviously what happened."
In noting Esper's move, Trump added, "I know Mark, and have no doubt he will do a fantastic job!" He said it's "most likely" he will nominate Esper for the job "pretty soon."
The post atop the Pentagon has not been filled permanently since retired Gen. James Mattis abruptly stepped down in December after delivering a blunt letter to Trump outlining a list of foreign policy differences and a warning that the administration should not allow relations with allies to fray.
Shanahan was put in place as acting secretary, but it wasn't until May that Trump announced he would nominate Shanahan. That formal nomination has never come, inexplicably delaying the Senate process.
On Capitol Hill, the Shanahan news was met with mixed reactions.
Top Democrats said his sudden withdrawal underscores the shortcomings of White House vetting for key Trump administration jobs.
Senate Minority Leader Chuck Schumer said Tuesday that "this Shanahan fiasco" shows that the administration's national security policy is "a shambles."
Senators said they were largely unaware of allegations involving Shanahan's family situation when he was confirmed as deputy defense secretary in 2017.
Democratic Sen. Richard Blumenthal raised the possibility that Shanahan deliberately concealed the domestic problems, and he called for an investigation by the Defense Department's inspector general. Shanahan, he said, "had an obligation to reveal it himself. This is potentially a violation of law."
Trump defended the vetting process, calling it "great," and said the Shanahan issues were "very unfortunate," and they "came up a little bit over the last short period of time."
Sen. James Inhofe, the Oklahoma Republican who is chairman of the Senate Armed Services Committee, said Trump called him about Shanahan on Tuesday. The president didn't offer any specifics, Inhofe said, but mentioned "allegations that would be very uncomfortable and really not worth making sacrifices for."
Shanahan, a former Boeing executive, has been leading the Pentagon as acting secretary since Jan. 1, a highly unusual arrangement for arguably the most sensitive Cabinet position.
His prospects for confirmation have been spotty due in large part to questions about his lengthy work as former Boeing executive and persistent questions about possible conflicts of interest.
The Defense Department's Inspector General cleared Shanahan of any wrongdoing in connection with accusations he had shown favoritism toward Boeing during his time as deputy defense secretary, while disparaging Boeing competitors.
In Shanahan's tenure at the department he's had to deal with a wide array of international hotspots, ranging from missile launches by North Korea to the sudden shift of military ships and aircraft to the Middle East to deal with potential threats from Iran.
Shanahan, 56, had extensive of experience in the defense industry but little in government. In more than six months as the acting secretary, he emphasized a shift from the resources and tactics required to fight small wars against extremist groups to what Shanahan called "great power" competition with China and Russia.