If the calipers are working properly then the wheel should spin freely regardless.
But when you remove the wheel you upset the position of the caliper ever so slightly, so it will take a very short time for the pads to bed in again in a different position than they were before.
It’s only minimal but can cause a little friction, similar to when you put new pads on, or if you’ve put the pads back on different sides when refitting them.