Thank you Dave for your response! I was able to create a shape layer and first group “group1” and added a rectangle, and what I am trying to do now is to add a new group “group2” to the same shape layer “Contents” that I created and add a path container/element to this group “group2” and add three vertex mask to this “path”.
This is how I created first group and added rectangle to it:
var shapeLayer = newComp.layers.addShape(); // adding shape layer
shapeLayer.name = "bannerLayer"; // name the shape layer
var shapeGroup1 = shapeLayer.property("Contents").addProperty("ADBE Vector Group"); / creating a group1
shapeGroup1.name = "Banner"; //name the group1
myRect= shapeGroup1.property("Contents").addProperty("ADBE Vector Shape - Rect"); // adding rectangle to the group1
thanks you for your help