Seems that the current method for mask generation with polystar is similar to Adobe Illustrator method, that is, you can change the settings while you draw the mask (before you release the mouse button). Ctrl for adjusting the outer radius, arrow keys for adding points and changing the star points roundness.
With shape layers, the settings are adjustable afterwards by twirling down to the contents>polystar settings.