Rear brake caliper pin holes worn

[Ghoti]

Just had my tyres replaced and was advised that the holes that the pins that hold the brake pads in place go through in my rear caliper have worn badly. They are oval!
I guess this is just because of vibration slowly wearing the metal out, but....
Is this common?
Am I doing something wrong to cause this?
Apart from a new rear brake, is there anything I can do?
Thanks in advance