Yes, but for most programs those files are generally under 5 MB and they’re totally inert. They don’t DO anything. On a Windows machine you’d have the program leave marks on the registry which would slow down the computer sometimes, but that’s not the case on the Mac. All your losing is the hard drive space, which is pretty small. The files left behind are nver again used by the computer. So just leave them alone.
If you REALLY want to get rid of them, erase the program and then empty the trash. Then use Spotlight to search for the program. Any files or folders that you still find at that point…erase them.
That’s all you have to do. It should take about a minute. I’d do it for large programs like Final Cut Pro since it may leave large files behind. But something like “Fetch” or “Handbrake?” I wouldn’t even worry about it after I erased the program from the Application folder. At that point the program itself is gone and that’s all I care about.