Firstly, check the clutch is correctly adjusted.
Second gear is the Fazer’s Achilles’ heel.
Not only is the selector for 2nd is prone to wear which makes engagement a hit-or-miss affair, but the mechanism that rotates the selector drum is poorly designed: part of the gear shift shaft can jam against the pins of the drum.
From your description it sounds as though the latter may be what’s affecting your bike.
There’s a longwinded account
hereunfortunately it looks as though my image hosting service is down at the moment. If it doesn’t come back in a few days I’ll have to find another and re-link. The GIF works ok, that and the description may help.
Looking on the bright side, if this does turn out to be the issue, it's an easier fix than changing the drum and selector forks.