The only downside to starting it up occasionally in very cold weather is condensation in the oil, it can cause mayo to form on your oil cap which is worrying if you've never seen it before because you think you have a gasket leak.
Basically the bike doesn't get hot enough to burn it off properly.
Other than that it's good to start it to keep the battery in shape, unless you've taken it off the bike.
The only time i won't ride is snow and ice, 'cos i can't get out of my own road as the sun doesn't hit the road until late afternoon then it's dark within 2hrs, and i would be very focced off if i stacked it next to home so i just won't risk it, it's not worth the aggro.