Simply pull on it really hard and it will pop out. If it's been there a long time, you might need to squirt some WD40 down around the handle to allow it to loosen it's grip on the rubber. Be sure to spray it clean before putting it back in afterwards, or it might slip out when pulling up to go to reverse.
I eventually drilled and tapped mine to add bolts to hold it in. I put a Hurst shifter handle and **** on mine later, as I liked it better. But the stock linkage works well for me, so I kept that part.