Annegret Kramp-Karrenbauer succeeded Ursula von der Leyen, who was elected Tuesday as the new head of the European Commission.
Kramp-Karrenbauer narrowly won the leadership of Merkel's center-right Christian Democratic Union in December. She has concentrated so far on trying to renew the party, among other things seeking to heal divisions with conservatives who were irked by Merkel's welcoming approach to migrants in 2015 and other moves.
However, she has seen her poll ratings drop sharply following several gaffes, and has struggled to raise her profile given that she lacks a seat in parliament and also wasn't in the Cabinet. Kramp-Karrenbauer had insisted in recent days that she would continue to concentrate on the party, but has now changed tack abruptly.
Merkel says she won't seek a fifth term as chancellor. It isn't yet clear who will run as the CDU's candidate for chancellor in Germany's next election, which is due in 2021 but could come earlier if her fractious coalition with the struggling center-left Social Democrats collapses.
The defense ministry offers Kramp-Karrenbauer, 56, who has served as governor of Saarland state but not in the federal government, a chance to gain foreign and security policy experience.
But it also is notoriously difficult to run, with concerns including the poor state of the German military's equipment and managing a transition from conscription to a professional force.
While von der Leyen's unexpected elevation to the European Union's top job after 5½ years as defense minister suggests that the post can be a springboard to promotion, it has left most recent occupants as diminished figures in domestic politics. Von der Leyen herself was long considered a potential successor to Merkel but faded out of contention while at the defense ministry.
An ambitious younger conservative who also ran in last year's CDU leadership election, Health Minister Jens Spahn, had been considered favorite to become defense minister. Kramp-Karrenbauer's move to take the job avoids giving a potential rival more exposure.
Kramp-Karrenbauer didn't address the reasons for her choice in brief remarks to reporters at the defense ministry, and took no questions.
She said she was taking on the job "with great respect, whole-heartedly and full of conviction," and is "very much looking forward to this task."