Sounds like your engine and/or prop need tuning, as well as your transom wedge.  For example, a lot of those old Fisherman outboards were used for slow speed trolling in shallow waters where they sucked up silt and dirt.  That muck clogged up their exhaust passages somewhat, preventing the engines from runnig at top speed.  They'll run all day, just not at high rpm.  Older props typically have nicks and dings that can adversly impact speeds and help the prop ventilate sooner and more excessively in turns, especially sharp turns.  The 35 pounds of ballast you are carrying suggests you need to tuck your outboard's lower unit in closer to the transom.  Try putting thin sticks (e.g., tongue depressors) between the top of the engine's clamps and the wedge to see if you can get rid of some or all of that ballast.

At your weight I wouldn't recommend a larger outboard.  A newer 6-hp Evinrude/Johnson (newer than 1980 but still a 2-stroke) would increase your speed, potentially into the 20s.

DO NOT, I repeat, DO NOT try a long-shaft outboard!  If you do, you won't be happy.  Good luck.