Had to do it a couple of times when stopping at a friends a while back and couldn't remember having any problems with them, so I've just nipped into the garage to check them and whilst they need a 'firm' move, they certainly weren't hard or difficult to move (and my bikes a couple of years old now)
Just to clarify though, that was moving the whole stem as one unit, not twisting the actual mirror, don't think they're designed to move much?
There's a couple of pictures of Grandma doing track days with her mirrors in the upright position.
