It’s not too confusing if you know where to look.
You’re using non-dropframe timecode separators (colons) in your subtitle script (1320 01:19:46:10 01:19:47:18 Let’s go?).
NTSC DVDs are dropframe timecode. In dropframe timecode, 2 frames are dropped every minute except at the 10 minute mark, and this will cause the drift you’re describing.
Instead, use semi-colons as separators for your script (e.g. 1320 01;19;46;10 01;19;47;18 Let’s go?) and all should be well.