Just double-click the element you want to remove from a palette while in the customize palette mode.
It’s not intuitive and leads users to search a lot. I know this post is from 7 years back, but I ended up here searching for the same thing in 2022. Poor UX design that didn’t evolve.
The answer from Jack Cook is still working, but there’s a quicker way to do this.