Yep, I keep everything in the Library. I edit TV shows that have several seasons in one very massive Library. Never any issues. I keep my drives clean, and I’m running a MBP M1 Pro Max. No issues. System drives of any OS must keep minimum 15% free space. But yeah, just another folder.
As a retired IT engineer, I can assure you, it doesn’t matter where you store your media, it’s just files being read off of a drive. Being inside the Library has NOTHING to do with anything. Drive speed and system hygiene is all that matters.