You don't use both at the same time and, generally, you choose the gasket to match the top of your intake manifold. Back in the old days we used to believe that the four holer was better at low to mid throttle and the open center was better on the top end, but that was probably just "youthful crap" and it was just the IM that made the difference. You're kind of making a "carb spacer" by using them both though. If nothing leaks, and you like the way the car runs, I'd leave it alone even with both of them on there.
EDIT -- Maybe a four holer on an open manifold will start to sag down and get sucked in after time. Surely someone on here has done that before and can give some first hand experience.