I’m going to have to see your scene file. You have something going on that is not right in your example clip. Spheres are moving backward after the broom has raised above them – it might be the spheres are piling up and simply rolling off the pile but I’d need to see the actual setup.
As to spheres – they look like marbles or ball bearings on a perfectly smooth surface would in reality with a broom pushing them around, so the dynamics system is doing a good job and you are trying to push the settings around until they don’t act like the spheres they are and you might not be able to do that. It might be possible to get the settings right (I’d have to see the file) but I’d try cubes (or something close), they would move like you want with just friction I think. Or if you have to have spheres try using cubes as the dynamic element but place a sphere inside that as a child