Also, Syrian activists said airstrikes in the country's northern, rebel-held region killed at least nine civilians on Monday.
The exchange between Ankara and Damascus came hours after a large Turkish military convoy entered the northwestern province of Idlib, the last rebel stronghold in Syria. It is likely to further increase tensions between the two neighboring countries as such direct clashes have been rare and could also cause friction between Moscow and Ankara, which have sought to coordinate their actions in Syria.
Earlier, Turkey's National Defense Ministry said the Turkish forces were sent to Idlib as reinforcement and were attacked there despite prior notification of their coordinates to the local authorities. It said Turkish forces responded to the attack, destroying targets. Four Turkish soldiers died at the scene while another soldier and a Turkish civilian member of the military personnel died later in hospital. Seven Turkish troops were wounded.
Speaking to reporters before departing for a visit to Ukraine, President Recep Tayyip Erdogan said Turkish artillery hit some 46 targets in Syria. Erdogan said Turkish warplanes were also involved and claimed that there were between 30 and 35 casualties on the Syrian side but offered no evidence.
"Those who test Turkey's determination with such vile attacks will understand their mistake," Erdogan said. He said Russia was told that Ankara would not stand for any "situation where we are prevented" from responding to Syrian assaults.
"It is not possible for us to remain silent when our soldiers are being martyred," Erdogan said.
The deaths were one of the highest single-day tolls for Turkish troops in Syria — Ankara has lost scores of military personnel in the Syrian war.
Turkey's Defence Minister Hulusi Akar immediately traveled to the Turkey-Syrian border to assure the troops. "Our people should know that the necessary has been done and will continue to be done," he said upon arrival.
The exchange occurred near the Syrian flashpoint town of Saraqeb, according to the the Britain-based Syrian Observatory for Human Rights, a war monitoring group. It added that Turkish troops shelled Syrian army positions in three provinces, killing eight soldiers in Idlib, three in Latakia province and two in the Hama region.
However, Syria's state news agency SANA said government forces captured two new villages on the way to Saraqeb. It added that as Syrian troops were chasing insurgents, four Turkish soldiers were killed and nine wounded triggering a Turkish retaliation — but it claimed there were no casualties among Syrian troops.
The Russian Defense Ministry said Turkey had failed to notify the Russian military about troop movements overnight in Idlib and that the Turkish troops got hit by Syrian fire that was directed at "terrorists" — a reference to al-Qaida-linked militants — west of Saraqeb.
The Russian military, which controls the airspace over Idlib province, said the Turkish aircraft never entered Syria's airspace during Monday's attack. Kremlin spokesman Dmitry Peskov said the Russian military remains in "constant contact" with Turkish counterparts in Syria.
The escalation comes amid a Syrian government offensive into the country's last rebel stronghold, located in Idlib and parts of the nearby Aleppo region. Turkish troops are deployed in some of those rebel-held areas to monitor an earlier cease-fire that was agreed to but that has since collapsed.
Relations between Turkey and Syria have deteriorated sharply since Syria's civil war began in 2011. Syria accuses Turkey of undermining its security by allowing thousands of foreign fighters to come battle the Syrian army. Idlib province is currently dominated by al-Qaida-linked militants.
With Russian backing, the Syria government has been on the offensive since December to capture and reopen a strategic highway held by the rebels since 2012. The offensive ignored a cease-fire deal brokered late last year between Russia and Turkey. The deal has since collapsed.
Syrian government forces captured the key Idlib town of Maaret al-Numan from the rebels last Wednesday, and have now set their sights on Saraqeb. The strategic highway passes through both towns.
Opposition activists did not say who was behind the airstrikes that killed nine civilians in rebel-held parts of northern Syria. The nine were on a minibus carrying people fleeing the violence near the village of Kfar Naha in Aleppo province, according to the Observatory and Baladi news, an activist collective. Four children, three women and two men were killed, according to paramedics.
The province of Idlib is home to some 3 million people, many of them displaced from other parts of Syria in earlier bouts of violence. The United Nations has estimated that about 390,000 Syrians have been displaced there over the past two months — 315,000 in December and 75,000 in January.
Turkey already hosts 3.5 million Syrian refugees, and the current wave of violence in Idlib has raised concerns of a new surge in displaced civilians fleeing toward the Turkish border.