Seeking a ninth title at Basel — where he has once a ball boy and reached the final in his last 11 appearances — Federer last dropped a set in his opening match 10 years ago.
The top-seeded Swiss seemed in control after winning seven straight games to take the first set and lead by a break in the second.
The 35th-ranked Serb then broke Federer's serve three times, including back-to-back to clinch the set and level the match.
Federer got the key break early in the deciding set and reeled off five straight points when facing three break chances in the sixth game. He clinched the win when Krajinovic sent a backhand long.
While Federer hit 10 aces, he also had six double faults and landed fewer than half of his first serves.
Federer next plays Jan-Lennard Struff on Thursday. The 52nd-ranked German beat Australia's John Millman, who eliminated Federer at the U.S. Open last month, 7-6 (3), 6-2.
Seventh-seeded Daniil Medvedev, who is also in Federer's half of the draw, beat Maximilian Marterer of Germany 6-3, 7-5.
Medvedev next faces Andreas Seppi, the 34-year-old Italian, who eased past qualifier Taro Daniel 6-0, 6-4.