transcoding for psp on linux

Now that my 32GB memory stick has arrived, it's time to fill it up. My previous was 4GB and was full with game data and no room left for video. Mpeg4(h264)/aac is the required video/audio codec for psp, so some transcoding has to be done.
So I dug up my mencoder options from my documentation database (a directory with flat txt files on which I use grep) and fire up mencoder. First error: no faac support, so no aac sound. Fixed this with an updated mencoder rpm (see http://haerench.blogspot.com/2010/07/mencoder-from-rpmfusion-with-aac.html).
Next attempt seems to be good. Transcoded a sample of 30 seconds (experience as a sysadmin has taught me to hope for the best, but prepare for the worst; so I test first).

$ mencoder dvd://1 -aid 128 -sws 9 -ss 120 -endpos 30 \
-vf pullup,softskip,dsize=480:272,scale=0:0,harddup,unsharp=l3×3:0.7 \
-ofps 24000/1001 \
-oac faac -faacopts br=128:mpeg=4:object=2:raw -channels 2 -srate 48000 \
-ovc x264 -x264encopts bitrate=500:global_header:partitions=all:trellis=1:level_idc=30 \
-of lavf -lavfopts format=psp -info name="some_movie" -o "psp_some_movie.mp4"

Copied sample to psp and try to play. Not good. Unsupported data says the PSP. The information screen tells me audio codec is ok, video resolution is ok, but video codec isn't recognized. Hmmm... why did a script that worked before produces now bad files? Last time I used it was on my PSP-1000, now I'm using a PSP-3000 after the analog stick broke on the PSP-1000. Is it a PSP-3000 thing? It shouldn't be, but you never know. Or is it a firmware thing that changed the video player behaviour (change is always good, isn't it?). Or is x264 broken in mencoder?
Ok I try other tools like h264enc which has presets for psp, searches the web for newer scripts (I find also some with -ovc lavc, which is also able to do mpge4/h264). But all fail, some with the same unsupported data, others give broken file and again other produce corrupt data.

Just when I was about to give up, I stumbled upon this blog http://blog.yogarine.com/2007/12/converting-video-for-psp-on-linux.html (which has a nice howto on transcoding for the PSP via avidemux (which you can also install with yum if you have rpmfusion enabled)). It's from 2007 but was updated in 2008 with changes to the video format after a PSP firmware update. Although some things are now less strict (like the framerate), some limitations were added: No reference b-frames and no 8x8 transform.
I first gave avidemux a try and after disabling 8x8 and b-franes in the video config, it produced a sample that was recognized by the PSP. Unfortunately avidemux lacked AAC support. I could do an avidemux recompile, but I'm more of a CLI guy anyway and with the found info I should get mencoder working. After some man paging I added x264encopts like bframes=0:partitions=none:no8x8dct. Transcoded a sample which played nice on the PSP. So the final config is now

$ mencoder dvd://1 -aid 128  -o video.mp4 \
-of lavf -lavfopts format=psp -info name="video_title" \
-ovc x264 \
-x264encopts bitrate=500:global_header:trellis=1:level_idc=30:bframes=0:partitions=none:no8x8dct \
-vf pullup,softskip,dsize=480:272,scale=0:0,harddup,unsharp=l3×3:0.7 \
-oac faac -faacopts br=128:mpeg=4:object=2:raw -channels 2 -srate 48000 

If you prefer lavc to do the h264 encoding, you can use

$ mencoder dvd://1 -aid 128  -o video.mp4 \
-of lavf -lavfopts format=psp -info name="video_title" \
-ovc lavc -lavcopts vbitrate=500:aglobal=1:vglobal=1 \
-vf dsize=320:240:0:16,scale=0:0 \
-oac faac -faacopts br=128:mpeg=4:object=2:raw -channels 2 -srate 48000

In the second example there is also a different video filter option (-vf). You can also use the -vf options from the first example. Based on the resolution and options you like. Both examples are also 1-pass encodes. For me this is good enough to be viewed on the psp. If you intend to use the TV-out option of the PSP, then I suggest you encode to a larger resolution (720x480), use a higher bitrate (eg. 1000) and do 2 or 3-pass encoding.