Take it off Nath
Have a look where the free play is. I think it's probably a rivet there so put it on the vice and hammer the rivet tighter. or drill it out and put a nut and bolt in.
The mirrors should fold in without going loose!
If it's the OEM mirror, then robin78842 is right, it's a rivet and there's no easy way to tighten it up.
I had a similar problem on my right mirror until (after a drop) I had to replace it, but although the first replacement I got had a bolt with a screw head, it appeared to have been secured with Red Threadlock which will only release after being heated to about 300 degrees centigrade(!), so it was impossible to fold in or adjust the tension.
I then got another replacement which I could change the tension on which is good because I like being able to pull in the right mirror when I'm filtering through narrow gaps to give a bit of extra space.
The only thing to watch for when changing over the mirrors is there's a metal tube which holds the front fairing on and it's possible for one of the nuts that holds the mirror securing to fall into it and then it's a real bugger of a job to find something long, thin and flexible enough to push into the tube freom the other side to get the nut out again (voice of experience!!!)