Hi, 

just looking at the pictures, the issue is not the tape, it's the surface that you varnished and its preperation.   there is something going on there that is preventing the varnish from adhering to the deck. 

prior to refinishiing, the deck should have been sanded back to the epoxy/glass and cleaned with the proper solvent to ensure that there is no wax or other surface contaminant that would prevent the varnish from bonding.

can you describe how you prepped the deck before varnishing it?  was the boat waxed in the past?    the answer is somewhere in the deck prep....resolve that and you can revarnish and no blue tape or fine-line tape will come anywhere close to being able to pull it off.

h  

   I agree with all tha hspira said about surface prep being the issue.  Even so, I always find it helpful in almost all cases, whether painting baseboards in the house (in most for house painting I prefer not to mask anyway, just use a steady hand and a good brush), lines on the basketball backboard, or anything to do with the boat, to pull masking tape as soon as possible after the painting/varnishing is completed.  And that means right away, just as soon as it has served its purpose and the paint/varnish has skim-dried enough not to drip or run.  For one thing, this prevents the "lifting" you are experiencing (though again, it shouldn't be an issue on a properly prepared surface).  For a second thing, this helps make sure you can pull the tape easily - it doesn't get stuck down by the finish. 

And for the most important third thing: pulling the tape while the new finish is still quite wet will often allow the line to naturally "blend" itself; you don't end up with such a sharp little step-edge at the boundary between the masked and unmasked surface.  When pulling the tape, be carefull of where the tape "tail" gets itself - gather it in a ball as you go - to prevent inadvertenetly spreading wet finish where you don't want it.  Sometimes after pulling the tape I even gently run my thumbnail down the edge while the finish is still soft to make the masked line "seamless."

And don't even ask me about the time as a youngster when I varnished some teak handrails on a fiberglass boat using common tan-colored masking tape, left it on overnight in the dew, and didn't come back the next day until the sun had been on it for a few hours.  A terrible mistake that even harmed the gelcoat by the time removal was complete.

   I agree with hispra it was surface preparation.