I had a 66 GTO a got a backfire after start up one day. I thought I had bent a push rod but I found the same problem you have. The car was only 4 years old at the time. Recruited my dad who had me place a heavy magnet on the pan and then drain the
oil and pull the pan. I found no sizeable filings or metal pieces at the point of the magnet. We took the heads in for inspection and was told that the rocker arm apparently had a manufacturing defect and had fatigued under use. Naturally, my dad was sure I had been hot rodding (I had) and it was my fault.
Long story short, we replaced the lifters, push rods, and rocker arms and the old GTO lived a nice long life for a GM even with my occassional hot rodding. By the way I used the Crane Roller Rockers on my 302 and loved them. When I sold my engine last year I had about 4000 miles on them with no signs of wear. If you want to go to a roller rocker call Tommy Vaughn Ford in Houston. They have excellent prices and lots of Hot Rod parts.
Good luck.