Arnold & Mary, the first thing I would do is to make sure your sound card drivers are up to date.
If that doesn’t fix things, then consider a re-install of Vegas as this is not normal behaviour.
I just tried it on my system and mute worked perfectly.
I dropped a song on the timeline, boosted the audio control in the track header to maximum as well as raising the master fader to maximum and pressed mute.
Even with my speaker volume at full, I couldn’t hear anything.
If a re-install doesn’t cure it, file a trouble report with Sony.