When using standard 32P (or mod0.8) pinions, you have to use specific sizes, and it's not necessarily the same as what came with the vehicle.
The "15-16-17" gearboxes used in Dune Racers, Hurricanes, F-150s, Escalades, etc. -- take 16T pinions in the standard pitch. For a 775-class motor the key words are "5mm pinion 32dp" or "5mm pinion 32p". You can keep it simple and get something like this cheap e-bay pinion:https://www.ebay.com/itm/HobbyStar-32DP-Pinion-Gear-16T-Hardened-Steel-5-0mm-Bore-5mm-32-Pitch-16-32P/171955600929?epid=525168309&hash=item28095a3e21:g:-0cAAOSw8GhZhNDf
For more information about why 15/17T aftermarket pinions don't work well, see this thread
. It is possible to use 15/17T but you have to hog out the screw holes in the gearbox and build an offset sleeve for the motor.
Some more reading:http://forum.modifiedpowerwheels.com/viewtopic.php?f=9&t=21628&p=148387#p148400
And yes, you'll need to file a flat on the motor shaft so that the set screw can work correctly. If you just tighten it up on a round shaft it will slip.
As for the last question: The 775 motors are heavier and larger. This means more thermal mass (slower to heat up) and more area to dissipate heat. MHO, heat sinks are of little use since the windings don't touch the case at all. Cooling down the case will have little effect on the winding temperature. (they make a ton of sense on brushless motors, but not brushed ones).
As for longevity options that don't involve scooter motors and crazy gearbox mods, here are a few other options:
(A) Go down to 18V but move up to a 19T gearbox (eg. from a Mustang) and 19T gear. You'll have to use a step-drill bit or similar method to expand the axle hole. Combining this with the Nichibo (cheapo) 775 motor that I pointed you to should yield a 90% speed improvement over the stock 12V / 15tooth combination, and about 50% more torque than stock motor/gearbox @ 12V.
-> Note that you can also get a Corvette gearbox and use a 22T pinion for more speed at the expense of some torque. This would give you a theoretical 120% increase in speed and 35% torque improvement over the stock Dune Racer @ 12V
(B) Stay at 24V, go to a Banebots 18V motor and add relays for the throttle. This motor doesn't spin as fast as the Nichibo and will probably last longer at 24V. However, it has a much lower wiring resistance and the stall current rating is ~180A each at 24V, vs 133 for the Nichibo (at 24V; it's 66A @ 12V, nearly the same as the stock motors). Running with the stock foot pedal is likely to not last very long, but the 80A SPDT car relays should hold up ok.
Note that 24V the Banebots is running 26000 RPM, and it's going to have a huge amount of torque (>3x stock). I predict immediate stripped gearboxes with this setup and 16T gears, though if you run with a 22T gearbox at least that will reduce the torque to something "more reasonable" (2x stock).
An ESC combined with either of these will reduce the starting torque and peak heat dissipation. It will improve longevity, but if you really enjoy the additional torque you have now, an ESC will almost certainly be a step backward. The standard "500/1000W" ESCs on ebay have current limits of 30-40A. The stock motors @ 12V pull >80A (as a pair of motors) at startup...
Good luck, whatever you decide. Cost-wise it's fairly economical to buy two empty Mustang or Corvette gearboxes and you just swap the guts over and purchase the appropriate (19 / 22T) 32DP 5mm pinion gear. Going to 18V is a bit of a bummer because you have to mess with 12V and 6V mixed batteries... which is part of why I came up with my 4x 775 motor solution.
F-150 - 24-36V with homebrew 100A variable-speed motor controller, 4x 775 motors (no, it's NOT all-wheel-drive--look HERE
for more details!)
Pink Rocket Princess Mustang - 15mph, 36V, 4x 775 motors, homebrew PWM controller rev 2....
Hurricane - 24V w/50A step-down controller, 775 motors