Since you’ve isolated this to a single server, have you considered running that server on a port of its own for a short while? You should also get the logs out of this server and the TCP stats. When the port goes away, if there’s enough time, you should try and get a tcpdump output. (In fact, it would be good to do that even while it’s working. It might shed some light on what’s on the wire.)
Older linux kernel TCP stacks had some serious TCP ordering issues. I think those were corrected around 2.6.20.
Another test, when this thing goes away, can you still ping?
Steve