Use an online section calculator, such as:

http://www.engineersedge.com/section_properties_menu.shtml

Input the numbers for your rectangular tube (ignore the radius corners), then jot down I value (area moment of inertia).

Next choose the round tube and input your proposed numbers, adjusting as needed to match the I value for the square one.

Basically, you want the area moment of inertia to match, assuming you are using equivalent materials to what is specified for the square tube when you build your round mast. 

   Thanks, I'll try that...

���I built a hollow mast using 1�1/2 inch strip of sitka spruce, make two 45 degree cuts on one one side, make 8 pieces for an octagon and plane it round from there. You will find info by searching for the Birds mouth hollow mast design. After comparing sectional properties I made the main section 2.5 inches across the flats. You can plane down the top 6 feet of the 1" wide strip to about 3/4" wide at the top for a tapered mast.

   Bobby, did you build for the original rig or the larger sail from RSS, that you posted about? And your spars, round or per plans?

I thought about a birdsmouth mast too, but the jury is still debating if it's worth the extra work.

���Thor, I plan to use the 89 sf RSS sail, still need to order it. I have not launched yet, I still need to paint the outside, varnish inside etc. But I am delayed by selling my house and moving right now. The mast, boom and yard are all glued up with 1 coat of epoxy inside and out, they still need varnish. The strip are ray to make on a table saw. I laid out the strips and put a coat of epoxy inside first and then laid them edge to edge on 3 strip of duct tape, slathered thickened epoxy in the birds mouth and just rolled it up into a cylinder. I then added hose clamps about every 12 to 18 inches. The finished mast,boom and yard are all tapered like in the manual but slightly longer for the RSS sail dimensions. I am really happy with how light these feel. I cut it all from a piece of sitka spruce, 2x9"x14.5 ft. This made 4 oars, the mast, boom and yard.

   What diameters and wall thickness did you use for mast, boom, and yard? Was wall thickness constant through out? Are your oar looms hollow too? 

���The mast is 2.5" across the flats, 1/2" wall, boom/yard are 1.5"across the flats, 3/8" wall. Mast strips are about 1.03" wide on the outside, boom & yard are about 0.70" wide on the outside. Tapering is done by planing down the square side of strips with a hand plane. I made the mast 14.5 feet but will cut it to fit the RSS sail once I get it. I chose the 2.5 inch across rhe flats based on a review of sectional properties compared to 2.25 inch square and solid section properties. Plus I was going slightly longer when using the RSS sail.

���My oars are solid, per John DeLapp info, spoon blades with 4mm oakume blades at 24" long. Search John DeLapp oars, Mystic Revelations pdf file. I made one set 8 ft 9" and am making a secondition set 8 ft 0". I can cut the long ones down if they prove too long.