Would it work as a switch? Probably. Is it the right way to do it? Nope. While it might work for short distances, the cables you are probably using aren’t designed to carry a high definition signal and the two signals will not be in time with each other when you throw the switch. It will cause a big glitch when you switch between your sources. If you are feeding a monitor or a projection system, it will look bad but do no harm. If you are feeding a recorder or digitizing it into a computer, you can cause yourself a whole world of hurt with a bad recording or a corrupt file. Since it’s just a mechanical switch, it will pass the signal, but it’s not designed to carry the kind of signal you are sending through it. Proceed at your own risk.
E=MC2+/-2db