[jhv48]

Definitely sounds like heat soak, or a failing starter solenoid.

Is your solenoid separate from the starter? Replaceable?

Granted a new starter/solenoid might fix the problem, but I would try to insulate the unit first. If that doesn't work, replace it.

Back in the seventies, when this happened, we would run a heavier wire from the battery to the solenoid.