Motion blur in re-encoded AVC sources

[bergerado1]

I would like to re-encode some AVC encoded videos -- mostly 1280x720p TV recordings using 'PCTV's satellite receiver USB stick software -- in order to reduce their file sizes, e.g. from 10 GB to about 2-3GB.
First attempts encoding to H264 yielded the desired file size and were 'watchable', but output videos showed significant image blur even for slow movements.

Question: How can I set up the H264 encoder in a way to reduce or get rid of motion blurs?

The sample code is abridged, the original log file is 8 MB large, due to a constantly repeated line such as "sync: video time didn't advance - dropped 2 frames (delta 60 ms, current 23004, next 30204, dur 7200)"). Encoding time appr. 5 hrs:

Code: Select all

HandBrake 0.9.9.5470 - 32bit Version
OS: Microsoft Windows NT 6.0.6002 Service Pack 2
CPU: AMD Turion(tm) 64 X2 Mobile Technology TL-56
Ram: 2813 MB, Screen: 1280x800
Temp Dir: C:\Users\Privat\AppData\Local\Temp\
Install Dir: C:\Program Files\Handbrake
Data Dir: C:\Users\Privat\AppData\Roaming\HandBrake\HandBrake\0.9.9.5470

-------------------------------------------

CLI Query:  -i "F:\TV Recs\[sw] Die rote Herberge (FR 1951).mpg" -t 1 --angle 1 -c 1 -o "C:\- DVDs\[sw] Die rote Herberge (FR 1951), hDTV.mkv"  -f mkv  --decomb -w 960 -l 640 --crop 40:40:164:164 --modulus 16 -e x264 -q 21 -r 25 --pfr -a 1,3 -E copy,copy -6 auto,auto -R Auto,Auto -B 0,0 -D 0,0 --gain 0,0 --audio-copy-mask none --audio-fallback ffac3 -x ref=1:weightp=1:subq=2:rc-lookahead=10:trellis=0:8x8dct=0 --verbose=1
User Query: False

[12:51:21] hb_init: starting libhb thread
HandBrake 0.9.9 (2013051800) - MinGW i686 - http://handbrake.fr
2 CPUs detected
Opening F:\TV Recs\[sw] Die rote Herberge (FR 1951).mpg...
[12:51:21] hb_scan: path=F:\TV Recs\[sw] Die rote Herberge (FR 1951).mpg, title_index=1
libbluray/bdnav/index_parse.c:162: indx_parse(): error opening F:\TV Recs\[sw] Die rote Herberge (FR 1951).mpg/BDMV/index.bdmv
libbluray/bdnav/index_parse.c:162: indx_parse(): error opening F:\TV Recs\[sw] Die rote Herberge (FR 1951).mpg/BDMV/BACKUP/index.bdmv
libbluray/bluray.c:1725: nav_get_title_list(F:\TV Recs\[sw] Die rote Herberge (FR 1951).mpg) failed (002C3180)
[12:51:21] bd: not a bd - trying as a stream/file instead
libdvdnav: Using dvdnav version 4.1.3
libdvdread: Encrypted DVD support unavailable.
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
libdvdnav: vm: failed to read VIDEO_TS.IFO
[12:51:21] dvd: not a dvd - trying as a stream/file instead
[12:51:21] file is MPEG Transport Stream with 188 byte packets offset 0 bytes
[12:51:21] Found the following PIDS
[12:51:21]     Video PIDS : 
[12:51:21]       0x100 type H.264 (0x1b) (PCR)
[12:51:21]     Audio PIDS : 
[12:51:21]       0x101 type MPEG1 (0x3)
[12:51:21]       0x102 type MPEG1 (0x3)
[12:51:21]       0x103 type AC3 (0x81)
[12:51:21]       0x104 type MPEG1 (0x3)
[12:51:21]     Subtitle PIDS : 
[12:51:21]     Other PIDS : 
[12:51:21] stream id 0x101 (type 0x3 substream 0x0) audio 0x101
[12:51:21] stream id 0x102 (type 0x3 substream 0x0) audio 0x102
[12:51:21] stream id 0x103 (type 0x81 substream 0x0) audio 0x103
[12:51:21] stream id 0x104 (type 0x3 substream 0x0) audio 0x104
[12:51:23] scan: decoding previews for title 1
Scanning title 1 of 1, preview 1, 10.00 %[12:51:23] scan: audio 0x103: AC-3, rate=48000Hz, bitrate=448000 Unknown (AC3) (5.1 ch)
[12:51:23] scan: audio 0x101: mp2, rate=48000Hz, bitrate=200000 Deutsch (MPEG1) (2.0 ch)
[12:51:23] scan: audio 0x102: mp2, rate=48000Hz, bitrate=200000 Francais (MPEG1) (2.0 ch)
[12:51:23] scan: audio 0x104: mp2, rate=48000Hz, bitrate=200000 Unknown (MPEG1) (2.0 ch)
Scanning title 1 of 1, preview 2, 20.00 %
Scanning title 1 of 1, preview 4, 40.00 %
Scanning title 1 of 1, preview 6, 60.00 %
Scanning title 1 of 1, preview 8, 80.00 %[12:51:24] scan: 10 previews, 1280x720, 50.000 fps, autocrop = 0/0/66/164, aspect 16:9, PAR 1:1
[12:51:24] stream: 17 good frames, 0 errors (0%)
Scanning title 1 of 1, preview 10, 100.00 %[12:51:24] libhb: scan thread found 1 valid title(s)
+ title 1:
  + stream: F:\TV Recs\[sw] Die rote Herberge (FR 1951).mpg
  + duration: 01:40:48
  + size: 1280x720, pixel aspect: 1/1, display aspect: 1.78, 50.000 fps
  + autocrop: 0/0/66/164
  + chapters:
    + 1: cells 0->0, 0 blocks, duration 01:40:48
  + audio tracks:
    + 1, Deutsch (MPEG1) (2.0 ch) (iso639-2: deu)
    + 2, Francais (MPEG1) (2.0 ch) (iso639-2: fra)
    + 3, Unknown (AC3) (5.1 ch) (iso639-2: und), 48000Hz, 448000bps
    + 4, Unknown (MPEG1) (2.0 ch) (iso639-2: und)
  + subtitle tracks:
[12:51:24] 1 job(s) to process
[12:51:24] starting job
[12:51:24] yadif thread started for segment 0
[12:51:24] yadif thread started for segment 1
[12:51:24] decomb filter thread started for segment 0
[12:51:24] decomb check thread started for segment 0
[12:51:24] mask filter thread started for segment 0
[12:51:24] mask erode thread started for segment 0
[12:51:24] mask dilate thread started for segment 0
[12:51:24] Auto Passthru: no codecs allowed
[12:51:24] Auto Passthru: fallback is AC3 (ffmpeg)
[12:51:24] Auto Passthru: passthru not possible for track 1, using fallback
[12:51:24] Auto Passthru: passthru not possible for track 2, using fallback
[12:51:24] sync: expecting 302430 video frames
[12:51:24] decomb filter thread started for segment 1
[12:51:24] job configuration:
[12:51:24]  * source
[12:51:24]    + F:\TV Recs\[sw] Die rote Herberge (FR 1951).mpg
[12:51:24]    + title 1, chapter(s) 1 to 1
[12:51:24]  * destination
[12:51:24]    + C:\Users\Public\- DVDs\[sw] Die rote Herberge (FR 1951), hDTV.mkv
[12:51:24]    + container: Matroska (.mkv)
[12:51:24]  * video track
[12:51:24]    + decoder: h264
[12:51:24]      + bitrate 200 kbps
[12:51:24]    + frame rate: 50.000 fps -> peak rate limited to 25.000 fps
[12:51:24]    + filters
[12:51:24]      + Decomb (default settings)
[12:51:24]      + Framerate Shaper (2:27000000:1080000)
[12:51:24]        + frame rate: 25.000 fps -> peak rate limited to 25.000 fps
[12:51:24]      + Crop and Scale (960:640:40:40:164:164)
[12:51:24]        + source: 1280 * 720, crop (40/40/164/164): 952 * 640, scale: 960 * 640
[12:51:24]    + dimensions: 960 * 640, mod 16
[12:51:24]    + encoder: H.264 (x264)
[12:51:24]      + options: ref=1:weightp=1:subq=2:rc-lookahead=10:trellis=0:8x8dct=0
[12:51:24]      + quality: 21.00 (RF)
[12:51:24]  * audio track 1
[12:51:24]    + decoder: Deutsch (MPEG1) (2.0 ch) (track 1, id 0x101)
[12:51:24]      + bitrate: 200 kbps, samplerate: 48000 Hz
[12:51:24]    + mixdown: Stereo
[12:51:24]    + encoder: AC3 (ffmpeg)
[12:51:24]      + bitrate: 224 kbps, samplerate: 48000 Hz
[12:51:24]  * audio track 2
[12:51:24]    + decoder: Unknown (AC3) (5.1 ch) (track 3, id 0x103)
[12:51:24]      + bitrate: 448 kbps, samplerate: 48000 Hz
[12:51:24]    + mixdown: 5.1 Channels
[12:51:24]    + encoder: AC3 (ffmpeg)
[12:51:24]      + bitrate: 640 kbps, samplerate: 48000 Hz
[12:51:24] file is MPEG Transport Stream with 188 byte packets offset 0 bytes
[12:51:24] decomb check thread started for segment 1
[12:51:24] reader: first SCR 180 id 0x100 DTS 360
[12:51:24] encx264: Encoding at constant RF 21.000000
x264 [warning]: --psnr used with psy on: results will be invalid!
x264 [warning]: --tune psnr should be used if attempting to benchmark psnr!
[12:51:24] mask filter thread started for segment 1
[12:51:24] mask erode thread started for segment 1
[12:51:24] mask dilate thread started for segment 1
x264 [info]: using cpu capabilities: MMX2 SSE2Slow SlowCTZ
x264 [info]: profile Main, level 3.1
[12:51:24] h264: "Chapter 1" (1) at frame 0 time 16200
[12:51:24] sync: first pts is 16200
[12:51:24] sync: video time didn't advance - dropped 2 frames (delta 60 ms, current 23004, next 30204, dur 7200)

<<<<<< This repeats for about 14000 lines !!!>>>>>>>

[15:31:45] sync: video time didn't advance - dropped 2 frames (delta 60 ms, current 541257804, next 541265004, dur 7200)
[15:31:45] hb_ts_stream_decode - eof
[15:31:45] reader: done. 1 scr changes
[15:31:45] sync: video time didn't advance - dropped 4 frames (delta 100 ms, current 541265004, next 541272204, dur 7200)
[15:31:45] sync: video time didn't advance - dropped 2 frames (delta 60 ms, current 541272204, next 541279404, dur 7200)
[15:31:45] sync: video time didn't advance - dropped 4 frames (delta 100 ms, current 541279404, next 541286604, dur 7200)
[15:31:45] sync: video time didn't advance - dropped 2 frames (delta 60 ms, current 541286604, next 541293804, dur 7200)
[15:31:46] sync: video time didn't advance - dropped 4 frames (delta 100 ms, current 541293804, next 541301004, dur 7200)
[15:31:46] sync: video time didn't advance - dropped 2 frames (delta 60 ms, current 541301004, next 541308204, dur 7200)
[h264 @ 002C8D40] missing picture in access unit
[h264 @ 02965FE0] no frame!
[15:31:46] work: average encoding speed for job is 7.427197 fps
[15:31:47] sync: got 71464 frames, 302430 expected
[15:31:47] decomb: deinterlaced 0 | blended 0 | unfiltered 71463 | total 71463
[15:31:47] render: 71463 frames output, 0 dropped and 0 duped for CFR/PFR
[15:31:47] render: lost time: 0 (0 frames)
[15:31:47] render: gained time: 0 (0 frames) (0 not accounted for)
[15:31:47] h264-decoder done: 300719 frames, 0 decoder errors, 0 drops
x264 [info]: frame I:863   Avg QP:17.73  size: 64875  PSNR Mean Y:45.71 U:74.89 V:69.11 Avg:47.45 Global:47.17
x264 [info]: frame P:41191 Avg QP:20.97  size: 34151  PSNR Mean Y:41.58 U:64.61 V:61.37 Avg:43.32 Global:43.13
x264 [info]: frame B:29409 Avg QP:22.70  size: 15334  PSNR Mean Y:40.52 U:62.97 V:60.30 Avg:42.25 Global:41.93
x264 [info]: consecutive B-frames: 20.5% 71.7%  6.8%  1.0%
x264 [info]: mb I  I16..4: 19.4%  0.0% 80.6%
x264 [info]: mb P  I16..4: 17.6%  0.0%  6.4%  P16..4: 34.2% 22.3% 17.6%  0.0%  0.0%    skip: 2.0%
x264 [info]: mb B  I16..4:  5.7%  0.0%  0.7%  B16..8: 27.3% 14.2%  3.1%  direct:29.1%  skip:19.9%  L0:27.0% L1:36.9% BI:36.0%
x264 [info]: coded y,uvDC,uvAC intra: 90.0% 0.9% 0.0% inter: 58.0% 0.6% 0.0%
x264 [info]: i16 v,h,dc,p: 19% 11% 57% 13%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 19% 16% 29%  4%  8%  6%  6%  6%  7%
x264 [info]: i8c dc,h,v,p: 98%  1%  1%  0%
x264 [info]: Weighted P-Frames: Y:26.6% UV:0.3%
x264 [info]: SSIM Mean Y:0.9599421 (13.973db)
x264 [info]: PSNR Mean Y:41.213 U:64.090 V:61.042 Avg:42.943 Global:42.645 kb/s:2545.39
[15:31:47] mp2-decoder done: 0 frames, 0 decoder errors, 0 drops
[15:31:47] mux: track 0, 71463 frames, 1913641871 bytes, 2545.36 kbps, fifo 512
[15:31:47] mux: track 1, 187953 frames, 168405888 bytes, 224.00 kbps, fifo 2048
[15:31:47] mux: track 2, 187954 frames, 481162240 bytes, 640.00 kbps, fifo 2048
[15:31:47] stream: 300726 good frames, 0 errors (0%)
[15:31:47] libhb: work result = 0
Encode done!
HandBrake has exited.

[Deleted User 13735]

Likely is dropping all those frames due to transport stream errors.

Run your video through quickstream fix in Videoredo first. There is a free trial.

[bergerado1]

Thanks but I had rather try a solution in Handbrake first... Here are some more details I didn't mention about the recorded files:
The PCTV device software (called 'TVCenter') can record digital satellite TV, both in SD and HD, without re-encoding by directly writing the stream to a file with the extension *.ts (= transport stream). So for SDTV I get a MPEG2 ts file (720x576), and for HDTV I get a AVC ts file (1280x720). Audio formats are either MPEG or AC3, depending on the broadcaster...

Important NOTE: The motion blur problem ONLY occurs if Handbrake's input file was HD (= AVC, encoded to H.264 -- just to use distinguishable terms, I know they're quite the same). It does NOT occur with MGEG (SD) sources.
Obviously, AVC sources will take longer to process as HB needs to decompress them first. And as high compression means fewer key frames, there is less information to build on when re-compressing all frames.
My assumption is therefore that HB's x264 encoder requires certain parameters to process AVC inputs in a better way -- in my case without dropping too many frames which seems to cause the blurring effect.

I don't want to work my way all through the possible H.264 parameters in HB's wiki or elsewhere (I will likely not understand them ). That's why I was hoping some of the experts here might know a 'rule of thumb' for re-encoding from/to the AVC/H.264 compressed video.

Any other ideas please, anyone?

[Smithcraft]

What musicvid said was that the stream files are damaged and need to be fixed first. Handbrake does not correct stream errors, but tries to work around them, so using a faulty source will not result in a good file.

SC

[bergerado1]

Thank you, I did understand him, but wonder what should be wrong with the recordings... (and how you can tell)? Does it say so in the code I posted, or is 'a faulty source' the only possible cause for dropped frames? Many more questions, I'm sorry...
You may be right however, for I now remembered that MKVmerge failed to load the recorded TS files (message: "B-frame without second reference image detected. Please repair MPEG2-video stream")).
Also, after cutting away adverts and re-muxing them in Avidemux (copy mode for both A+V), resulting AVI or MPG files wouldn't play on my DLNA TV.
Strange, I used to believe the recordings were 'written directly to file' 1:1 without modification -- and assuming the TV broadcast stream was properly formed, A/V timing correct, playing fine on TV, I would think it gets recorded as such.

But perhaps not ... Have now checked the Videoredo website too, thank you, but wonder if it works for AVC? They claim it handles "High Definition Transport Streams" though... but these are not MPEG2 are they? (But perhaps HD broadcasters stream their AVC material 'wrapped' inside a MPEG2 transpoted stream, which would explain why recordings are 10 GB large...)
Before the guessing gets any wilder, I better have you experts comment... Thanks in advance!

[Deleted User 13735]

Thank you, I did understand him, but wonder what should be wrong with the recordings... (and how you can tell)?

The PCTV device software (called 'TVCenter') can record digital satellite TV, both in SD and HD, without re-encoding by directly writing the stream to a file with the extension *.ts (= transport stream).

There, you've answered your own question in its entirety!

---> BECAUSE THEY ARE TRANSPORT STREAMS <---
1. Transport streams contain errors. http://www.atsc.org/cms/standards/a_78a.pdf
2. Broadcast and satellite transport streams can contain many thousands of errors.
3. I know of no device or pvr that actually corrects TS errors, although some of them purposely lie and misflag the output as PS.
4. A transcoder, unlike a "player," does not fudge through transport stream errors on the fly. It generally drops frames it cannot understand.
5. Some third-party utilities, including VideoRedo and TSDoctor, correct transport stream and associated indexing errors in non-realtime.
6. When the transport stream errors are too severe, these utilities too, drop frames. But not in nearly the numbers that "doing nothing" does.

I acknowledge your artful speculation, but there is actual experience behind these observations.
VideoRedo has TWO versions, the TVSuite handles your AVC, the Plus version is for MPEG-2. As I mentioned, there is a free trial.

It may be that your satellite streams are simply too error-ridden to inflate suitably for encoding. OTOH, the "fix" may work wonders.
Best of luck; I bet you drove your teachers to dementia!

[bergerado1]

Hm, some useful information indeed. It's true I'm speculating on the basis of incomplete knowledge -- which is why I seek more knowledge in an expert forum... and find it, thanks to yourself!
(Please forbear a non English native speaker his awkward wordings... Apologies implied for causing dementia! I haven't heard of my teachers for too long to tell how sane they still are, but promise to do some reading on TRANSPORT STREAMS to avoid more damage! )

Anyway, bad news for my AVC recordings as I don't fancy buying yet another piece of software!
Fortunately, the SD 720x576 streams seem error free enough to be transcoded into enjoyable widescreen format, at lossless image quality thanks to H.264... Plus, unlike DVDs, these recordings are AR corrected, so the perfect source for Handbrake! (Otherwise I would have to scale down to 720x400... which is a pain HB should get rid of in future versions, such as "Vidcoder" has!)

So sorry to get back to my initial question (please don't bother to reply if you don't feel like):
Is there really no way to make HB's encoder handle transport stream errors?
For example, how about a parameter to "force keyframe every XX seconds" to prevent too many frames getting dropped?

[Deleted User 13735]

No. It has nothing to do with Handbrake encoding. The errors are already in the source. Your broadcast stream might have one keyframe for every 300-500 b/p frames. That's a lot of damage if even one b-frame gets messed up, and a missing b-pyramid would seem to be essentially unrecoverable.

Handbrake can't put the toothpaste back in the tube.
You can force all the i-frames you want with keyint and it won't change a thing in your source.

Your SD recordings have fewer errors because they occupy but a small magnitude of bandwidth compared with your HD.
Satellite is notorious for introducing noise errors in the stream. I have found VideoRedo a worthy investment because it removes commercials, splits and joins episodes, batch renders, resyncs audio, and corrects a wide range of errors.

[bergerado1]

You can force all the i-frames you want with keyint and it won't change a thing in your source.

I see, thanks for this addition. Should the "keyint" string you suggest be a H.264 parameter... ignorant as I am, how to use that in HB? Just put the command in the Video tab's "Extra Options" field, perhaps, add a number etc?

Just for interest - although I agree with you in that TSs are error prone, most likely so via satellite - there may be differences between your ATSC standards (US, as your link above suggests) and European "DVB" streams...

Another solution might be to get another recording software that can fix TS errors, in real time or not, but I guess if such exists it will cost me dearly... "Freevo" looks promising (has commercial detection/removal) but isn't quite ready for Windows...
So you see, even having transcoded hundreds of DVDs using HB, I'm rather new to PVR and its surprises.

[Smithcraft]

TSDoctor mentioned above is about 30 Euros. I'm not sure if that's something that might cost you dearly or not...

SC

[bergerado1]

TSDoctor mentioned above is about 30 Euros. I'm not sure if that's something that might cost you dearly or not...

Yep, I had seen that too, and would prefer that to VideoRedo, price-wise...
But by saying 'cost me' I was also thinking of the extra time spent running yet another process in order to get 1 recording proper...
Plus, having a sizeable video base already, many in HD/HQ (and regarding the lot of rubbish that's on TV), I might not need it very often... Will keep in mind though, thanks!