Oh yeah, without audio in enabled it always goes faster, but I want audio though of course.
Actually I was using Adobe Media Encoder to do the exporting. But if I export out of Premiere Pro directly, then it goes much faster, it turns out, with the audio as well.