The National Fire Agency on Wednesday identified two of the victims as Indonesian and another as Filipino. The fourth body had not been identified.
Taiwan's military deployed a floating platform to help workers remove debris and try to extract two of the boats from under collapsed sections of the bridge. The third boat was recovered the previous day. All the victims are believed to be fishing boat workers from Indonesia and the Philippines.
The 140-meter (460-foot) -long arched bridge collapsed Tuesday into a bay on Taiwan's east coast, about 60 kilometers (40 miles) southeast of Taipei, the capital. A typhoon swept by the island earlier, but the weather was sunny when the bridge collapsed, and it wasn't clear if the storm was a factor.
Ten people were taken to hospitals with injuries, including the Taiwanese driver of an oil tanker truck that fell off the bridge. The other nine are Filipino and Indonesian fishermen, with one, an Indonesian, in serious condition, Taiwan's official Central News Agency said, citing an employment agency that recruited the workers.
Many Filipinos and Indonesians work on fishing boats registered in Taiwan, where the pay is better than in their home countries.
The 18-meter-high (nearly 60 feet) high bridge in Nanfangao, a Pacific coast fishing port, opened in 1998 and replaced a lower span that prevented large fishing vessels from passing underneath.
The company that designed the bridge, MAA Consultants, says it's the only single-span arch bridge in Taiwan supported by cables and the second single arch-cable steel bridge in the world.