You need to set your composition to be duration of the amount of frames you need. In AE an effect cannot increase the duration of a sequence so you have to pre-comp and set duration appropriately or you can also as per above set the top layer as a solid (make the solid after you changed the comp duration) and apply Twixtor on the solid where color source is layer 2 and ALT source is layer 3.
I am not sure I understand what layer 2 looks like. Is it a black and white image of the matte? The scenario I am describing here is to retime a matte without the RGB. As Barend said maybe what you need is just “set channel” (ae included effect) the roto matte into the alpha of the color source. When you do such thing you need to make that into a separate comp as AE does not honor effects applied to a layer when you ask for frames from a layer (unless it’s subcomped).
Again Twixtor Pro only supports embedded controls for multiple-layers (defined by mattes)
Pierre