[wkooiman]

I don't know if a damper will help, but you are describing bump steer.

The ball joint needs to be as close to the middle of the tire as possible. If it isn't, when you hit a bump, it yanks that tire to the outside - assuming the ball joint center is to the inside of center.

I had the same problem on my Pantera. The stock rims are 15x7 in the front. I switched to 15x8 with the extra 1" to the inside. It transformed the car.

Naturally, I don't know your car or if you can change something to correct the offset. But that's where I'd look if it were my car.