First delete all the keyframes you currently have in that layer.
Fast-forward your video to 5 seconds.
Add a keyframe. (make sure the size is at 40px)
Then fast-forward to another position where you want the transition from 40px to 20px to end and add another keyframe.
Now don’t move the slider anymore, or it’ll screw it up!
Go back to your Transform settings, and set the scaling to 20px.
Then it should work. (you can move the slider after you do that)
If you want these to be “smoother” select the desired keyframe, and hit “F9.”
Hope this helps! =)