Make sure that the cooler is not plugged. If it has been sitting in your garage for a while without caps it could have some critter homes in it.
Running your engine for a few (very few) minutes without
oil pressure PROBABLY won't cause a problem - after all it was not under load - right?
The fact that you can hear pressure release means to me that the cooler is plugged with something.
Next time shut it off sooner when you don't see any pressure. You should see movement of the needle within 2 or 3 seconds of startup.
Good luck!