[Hoover Kwan]

yeah I had the same problem. Even when I created a new empty comp with absolutely nothing in it, the selection tool was still locked as a rectangular masking tool. This is the only way I was able to get around this:

Create a new project
Create a new comp
Import your old project in the library.
Note that the imported project comps will still have the selection tool locked problem
however, if you copy&paste your layers into the new comp that you created earlier, the selection tool should go back to normal.

it’s a hassle, i know, but hopefully someone will figure out an easier way when this glitch happens again