Thanks for all your help, guys, but I got it!
In order to get the swipe down/up to trigger panorama you have to change these:
browser.gesture.swipe.down > Browser:ShowTabView Browser:ToggleTabView
browser.gesture.swipe.up > Browser:HideTabView Browser:ToggleTabView
Settings in about:config to map three-finger-swipe down to Panorama (and up for back).
Update: Seems like ShowTabView and HideTabView were replaced by ToggleTabView when FF4 went out of beta.