I'm currently working through my DVD collection, and some of my DVDs are exhibiting frame stuttering at various points. So far, I've identified the following characteristics:
1) Pre-ripping the DVD to hard disk does not seem to affect this result.
2) Whenever I see this problem, the activity log also shows a lot of "MP4Mux: illegal render offset ..." messages. The converse is not true: occurrence of these messages does not guarantee that visible/obvious frame stutter occurs.
3) I can prevent this from happening (and, incidentally, get rid of the "illegal render offset" messages) by turning off B-frames. Disabling detelecine also seems to get rid of the video stutter, but does NOT get rid of the "illegal render offset" messages.
I've pastebinned a dump here for one of the problematic DVDs:http://pastebin.com/pastebin.php?dl=m7e4312a4
Note that this is a multi-megabyte pastebin, primarily because of all the "illegal render offset" messages. I can produce more pastebins and try different settings upon request.
This is reproducible using the built-in High Profile preset with both 0.9.4 GUI and SVN r3036 GUI, using the x86-64 deb on Ubuntu 9.10. For now I'm working around it by watching for the "illegal render offset" messages, and re-doing the affected encodes with B-frames off whenever I see it crop up.