I don’t have time right now to test it with your file, but I would try enabling the “compositing background for HDR maps” in the compositing tag.
Per the help file: Enable this option if you want to integrate rendered objects in a Project that was illuminated using GI and HDRI textures (must be assigned to a Sky object). The Compositing tag will then usually be assigned to a Floor object. Light and shadows from the GI illumination will then be combined with the background image (in our example with an HDRI texture applied to the sky).