This sounds identical to my last problem, which turned out to be a salt-rotten ground cable. Your issue is likely a bad connection or failing cable(s) at this point. If you bench test the starter and test continuity across the solenoid (before you open up the ebox and have to go through that hassle) you'll know where that gremlin has set up shop. More on pulling the motor - I feel its a good thing to do occasionally anyhow, clean up that bilge, check mounts and hoses/clamps, overall condition of the motor, etc.
Seriously though, while your at this stage of disassembly do yourself a favor and consider hooking your stator up with some sort of proven quick disconnect and also consider the solenoid post relocate (both I got earlier form JSS) that makes the engine removal a whole lot less of a pain if the ebox can stay behind. I think it took around 10 minutes to pull mine last time and Johns stuff is top notch.