Yes, the flap only comes out if you've got the whole assembly is out, which kind of defeats the object of removing it at that point.
I can't remember the exact details from when I took mine off, but there were four bolts that hold the grab rail on, then 2 or four more. It was all pretty easy to work out.