Chavez, who had been battling pneumonia, died in his sleep in the San Diego suburb of Poway, his daughter, Kathleen Chavez, told The Associated Press.
As recently as last May he had traveled to Washington, D.C., where he was honored on Memorial Day by President Donald Trump. The White House Tweeted a statement Wednesday saying it was saddened to hear of his passing.
"We were honored to host him at the White House earlier this year," the statement said. "Thank you for your service to our great nation, Ray!"
Daniel Martinez, chief historian for the National Park Service at Pearl Harbor, Hawaii, confirmed Wednesday that Chavez was the oldest survivor of the attack that killed 2,335 U.S. military personnel and 68 civilians.
"I still feel a loss," Chavez said during 2016 ceremonies marking the attack's 75th anniversary. "We were all together. We were friends and brothers. I feel close to all of them."
Hours before the attack, he was aboard the minesweeper USS Condor as it patrolled the harbor's east entrance when he and others saw the periscope of a Japanese submarine. They notified a destroyer that sunk it shortly before Japanese bombers arrived to strafe the harbor.
By then Chavez, who had worked through the early morning hours, had gone to his nearby home to sleep, ordering his wife not to wake him because he had been up all night.
"It seemed like I only slept about 10 minutes when she called me and said, 'We're being attacked,' " he recalled in 2016. "And I said, 'Who is going to attack us?' "
"She said, 'The Japanese are here, and they're attacking everything.' "
He ran back to the harbor to find it in flames.
Chavez would spend the next week there, working around the clock sifting through the destruction that had crippled the U.S. Navy's Pacific fleet.
Later he was assigned to the transport ship USS La Salle, ferrying troops, tanks and other equipment to war-torn islands across the Pacific, from Guadalcanal to Okinawa.
Although never wounded, he left the military in 1945 suffering from post-traumatic stress disorder that left him anxious and shaking.
Returning to San Diego, where he had grown up, he took a job as a landscaper and groundskeeper, attributing the outdoors, a healthy diet and a strict workout program that he continued into his early 100s with restoring his health.
"He loved trees and he dearly loved plants and he knew everything about a plant or tree that you could possibly want to know," his daughter said Wednesday with a chuckle. "And he finally retired when he was 95."
Still, he would not talk about Pearl Harbor for decades. Then, on a last-minute whim, he decided to return to Hawaii in 1991 for ceremonies marking the attack's 50th anniversary.
"Then we did the 55th, the 60th, the 65th and the 70th, and from then on we went to every one," his daughter recalled, adding that until Chavez's health began to fail he had planned to attend this year's gathering next month.
Born March 12, 1912, in San Bernardino, California, to Mexican immigrant parents, Chavez moved to San Diego as a child, where his family ran a wholesale flower business. He joined the Navy in 1938.
In his later years, as he became well known as the attack's oldest military survivor, he'd be approached at memorial services and other events and asked for his autograph or to pose for pictures. He always maintained that those events were not about him, however, but about those who gave their lives.
"He'd just shrug his shoulders and shake his head and say, 'I was just doing my job,' " said his daughter. "He was just a very nice, quiet man. He never hollered about anything, and he was always pleasant to everybody."
Chavez was preceded in death by his wife, Margaret. His daughter is his only survivor.
Funeral services are pending.