A famous comedy team and a 61-year love affair all began with some stolen silverware.
Stiller's mother, comedy great Anne Meara, who died at 85 in 2015, told Willie in a previous interview that Jerry first impressed her when they went for coffee at a cafeteria in New York City. She asked him if he would steal her some silverware because she needed a second set in her apartment, and once he did it, she knew he was the one.
"And he stole silverware for her the rest of his life," Stiller said, laughing. "He was devoted to her."
The two starred together as a comedic team that appeared on "The Ed Sullivan Show" and "The Tonight Show" with Johnny Carson in the 1960s, and also worked together again in their later years on "The King of Queens."
Watching them perform was a regular part of Stiller's life growing up.
"I think it was always around us,'' the said. "They'd take us to the nightclubs. They'd take us out to California, and I remember walking around the Paramount lot, and I was a huge 'Star Trek' fan, and my mom introducing me to William Shatner. It just was like the end of the world for me."
Stiller even recalled being told that the Pips, of Gladys Knight and the Pips fame, taught him how to swim.
"It's family lore,'' Stiller said. "I have to take it as true."
Stiller followed them into the family business by becoming a comedy star of his own as a writer, director and actor. He got a chance to direct his father in the 2001 hit "Zoolander," in which he also starred. Stiller is also an ambassador for the United Nations High Commissioner for Refugees.
"(Jerry) took everything very, very seriously,'' Stiller said. "He really approached it methodically. So I kinda would try to stay out of his way when he was working."
His father was born and raised in New York City in what Stiller said "wasn't the happiest household." He lied about his age at 16 so he could join the U.S. Army, and he later used the G.I. Bill to attend the theater program at Syracuse University.
The fire that he would one day famously show as Frank Costanza on "Seinfeld" once came out when he made a rough tackle on a military policeman who was saying anti-Semitic things to him during an Army intramural football game in Italy in his military days.
"He had so much inside of him,'' Stiller said. "He took this guy down and he broke his ankle, and the entire stadium cheered. And I just think about that because in real life, he was so quiet and sweet, but he was kind of like this volcano underneath that had a lot of emotion.
"And that would come out on 'Seinfeld,' in these characters. But it never really came out in real life."
Stiller spoke about his father's sweet nature with Savannah Guthrie on Thursday.
"First of all he never, ever yelled at me once in my whole life," Stiller said. "He kind of kept it all under the surface. He was very overprotective and loving."
He was that way with relative strangers as well. Willie recalled seeing Jerry Stiller on the street in the Upper West Side of New York City years ago, and Stiller looked at him and said, "Hey, you're doing a great job, kiddo." Willie didn't think that Stiller even knew who he was.
"He loved connecting with people," his son said. "That generosity, it's kind of what I feel now with his passing, feeling so much love for him, and that karmically coming back. He touched so many people that way, both personally and through his work."