Firefighters had to fight the blaze from 100 meters (yards) away because it broke out in one of the area's many alleyways, tangled in electrical wire and too narrow for vehicles to access, authorities said.
A resident of the area, Mohammed Naushad, said he was woken by people wailing at around 4:30 a.m. He went outside to find smoke and flames shooting out of a building near Sadar Bazaar, New Delhi's largest wholesale market for household goods. Inside, he found the fourth floor engulfed in flames. One floor below, he saw "20 to 25 people lying on the floor."
"I don't know if they were dead or unconscious, but they were not moving," Naushad said.
He said he carried at least 10 people out of the flames on his shoulders and into the arms of emergency responders.
Outside the mortuary, which was guarded by dozens of police officers, some of the workers' relatives said they received phone calls from the men trapped inside, who begged them to call the fire brigade.
Many of the men were migrant workers from the impoverished border state of Bihar in eastern India, relatives said. They earned as little as 150 rupees ($2.10) per day making handbags, caps and other garments, sleeping at the factory between lengthy shifts.
Many of the victims were asleep when the blaze began, according to Yogesh, a police spokesman who uses one name.
The cause of the fire was not immediately clear.
A case of culpable homicide not amounting to murder was registered against the building's owner, but no arrests were immediately made, said assistant New Delhi police commissioner Anil Kumar Mittal, adding that authorities were investigating whether the factory was operating legally.
Dr. Kishore Singh said rescuers brought victims to his government-run hospital and two others in the city. Another 16 people were being treated for burns or smoke inhalation and were in stable condition, Singh said.
Police barred relatives from entering Lok Nayak hospital, where some of the victims were taken. Relatives of the workers cried, consoled one another and jostled for information.
"I was told by someone my nephew is inside, but I haven't seen him," said Mohammad Moti, who was searching for his 22-year-old nephew, Mohammad Chedi.
Fire Services chief Atul Garg said it took 25 fire trucks to put out the blaze. About 60 people, including some of the dead, were taken out of the building, Mittal said.
Distraught relatives also appeared at the site of the blaze.
The Press Trust of India news agency quoted Manoj, who uses one name, as saying that his 18-year-old brother, Naveen, was working in a handbag factory in the building.
"I got a call from his friend informing that he has been injured in the incident. I have no clue which hospital he has been taken to," he said.
New Delhi's chief minister, Arvind Kejriwal, also appeared at the scene of the fire, promising victims' families compensation.
Prime Minister Narendra Modi described the fire as "extremely horrific."
``My thoughts are with those who lost their loved ones. Wishing the injured a quick recovery," Modi tweeted.
Fires are common in India, where building laws and safety norms are often flouted by builders and residents.
In 1997, a fire in a movie theater in New Delhi killed 59 people. In February this year, 17 people were killed by a fire in a six-story hotel in the capital that started in an illegal rooftop kitchen.