CC's missing in SD transport stream

Archive of historical bug reports.
Please use the GitHub link above to report issues.
Forum rules
*******************************
Please be aware we are now using GitHub for issue tracking and feature requests.
- This section of the forum is now closed to new topics.

*******************************
Post Reply
PaulF
Novice
Posts: 58
Joined: Thu Nov 21, 2013 7:15 am

CC's missing in SD transport stream

Post by PaulF »

I am a new user here. I am using HandBrake-svn5893_x86_64-Win_GUI because I have a new computer that supports QSV. I record Comcast shows and HD works great. The SD shows scan OK, but the CCs are not found in the video stream user data(ccextractor finds them). I wrote a little bit of the ccextractor code so I still have the MPEG2 TS Packet analyser. I ran the TS analyzer and the only difference I see is the SD files have extra user data(from my reading it may be aspect ratio user data).

TS Packet snippet:
00 00 01 B2 03 81 10 AC 04 06 4B 01 01 80 00 00 01 B2 47 41 39 34 03 C2....long caption snipped

The first user data (00 00 01 B2 start code) is unknown and the second has the GA94 ascii id for captions.

I have no idea if this is the problem, but it looks suspicious to me.

I looked at some of the source code and the test for captions would pass on the second user data, but have
no clue where to look in the code to see if the first user data causes the scan to stop looking.

The activity log is the same as the HD files that work.

Code: Select all

[07:59:40] hb_init: starting libhb thread
[07:59:40] CPU: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
[07:59:40]  - Intel microarchitecture Haswell
[07:59:40]  - logical processor count: 8
[07:59:40] OpenCL device #1: Intel(R) Corporation Intel(R) HD Graphics 4600
[07:59:40]  - OpenCL version: 1.2 
[07:59:40]  - driver version: 9.18.10.3186
[07:59:40]  - device type:    GPU
[07:59:40]  - supported:      YES
[07:59:40] Intel Quick Sync Video support: yes
[07:59:40]  - Intel Media SDK hardware: API 1.6 (minimum: 1.3)
[07:59:40]  - Intel Media SDK software: API 1.7 (minimum: 1.3)
[07:59:40]  - Preferred implementation: hardware (any)
[07:59:40] hb_scan: path=C:\Users\Public\Documents\SnapStream\Beyond TV\shows\Comet Encounter-2013-11-20-0.tp, title_index=0
libbluray/bdnav/index_parse.c:162: indx_parse(): error opening C:\Users\Public\Documents\SnapStream\Beyond TV\shows\Comet Encounter-2013-11-20-0.tp/BDMV/index.bdmv
libbluray/bdnav/index_parse.c:162: indx_parse(): error opening C:\Users\Public\Documents\SnapStream\Beyond TV\shows\Comet Encounter-2013-11-20-0.tp/BDMV/BACKUP/index.bdmv
libbluray/bluray.c:1725: nav_get_title_list(C:\Users\Public\Documents\SnapStream\Beyond TV\shows\Comet Encounter-2013-11-20-0.tp) failed (0000000000a72270)
[07:59:40] bd: not a bd - trying as a stream/file instead
libdvdnav: Using dvdnav version 4.1.3
libdvdread: Encrypted DVD support unavailable.
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
libdvdnav: vm: failed to read VIDEO_TS.IFO
[07:59:40] dvd: not a dvd - trying as a stream/file instead
[07:59:40] file is MPEG Transport Stream with 188 byte packets offset 40 bytes
[07:59:40] Found the following PIDS
[07:59:40]     Video PIDS : 
[07:59:40]       0x7c0 type MPEG2 (0x2) (PCR)
[07:59:40]     Audio PIDS : 
[07:59:40]       0x7c1 type AC3 (0x81)
[07:59:40]       0x7c2 type AC3 (0x81)
[07:59:40]     Subtitle PIDS : 
[07:59:40]     Other PIDS : 
[07:59:40] stream id 0x7c1 (type 0x81 substream 0x0) audio 0x7c1
[07:59:40] stream id 0x7c2 (type 0x81 substream 0x0) audio 0x7c2
[07:59:40] scan: decoding previews for title 1
[07:59:40] scan: audio 0x7c2: AC-3, rate=48000Hz, bitrate=128000 Espanol (AC3) (1.0 ch)
[07:59:40] scan: audio 0x7c1: AC-3, rate=48000Hz, bitrate=384000 English (AC3) (2.0 ch)
[07:59:40] scan: 10 previews, 704x480, 29.970 fps, autocrop = 74/60/0/10, aspect 4:3, PAR 10:11
[07:59:40] stream: 22 good frames, 0 errors (0%)
[07:59:40] libhb: scan thread found 1 valid title(s)
PaulF
Novice
Posts: 58
Joined: Thu Nov 21, 2013 7:15 am

Re: CC's missing in SD transport stream

Post by PaulF »

I was curious so I looked at some more code. I hit a dead end when a single user_data packet was coming from a "struct info" returned by the mpeg 2 decoder. I was thinking of trying IRC when I noticed in the IRC archive that the mpeg 2 decoder is changing to libav. I hope it returns multiple user data packets.
PaulF
Novice
Posts: 58
Joined: Thu Nov 21, 2013 7:15 am

Re: CC's missing in SD transport stream

Post by PaulF »

I waited for all the changes and then forgot about this until I tried an old sd movie that has a lot of French names (I need CC to figure them out :D smiley because of your fr domain). VLC and ccextract work as before. I made small clip this time, now that I have a handbrake drop box folder. The file is chocolat.zip And my dropbox url is:
https://www.dropbox.com/sh/kfsn5s3dpbjkelx/mCS0vMQlTs

Code: Select all

HandBrake svn6164 (Nightly Build) - 64bit Version
OS: Microsoft Windows NT 6.1.7601 Service Pack 1
CPU: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
Ram: 16061 MB, 
GPU Information:
  Intel(R) HD Graphics 4600 - 9.18.10.3257
Screen: 1920x1080
Temp Dir: C:\Users\PAUL\AppData\Local\Temp\
Install Dir: C:\Program Files\Handbrake
Data Dir: C:\Users\PAUL\AppData\Roaming\HandBrake\HandBrake\0.0.0.6164

-------------------------------------------
CLI Query:  -i "C:\cygwin64\home\PAUL\handbrake\chocolat.ts" -t 1 --angle 1 -c 1 -o "C:\cygwin64\home\PAUL\handbrake\chocolat.mp4"  -f mp4  -w 528 --crop 32:36:0:0 --loose-anamorphic  --modulus 2 -e qsv_h264 -q 20 --cfr -a 1 -E av_aac -6 dpl2 -R Auto -B 160 -D 0 --gain 0 --audio-fallback ac3 --markers="C:\Users\PAUL\AppData\Local\Temp\chocolat-1-chapters.csv" --encoder-preset=quality --encoder-level="4.0"  --encoder-profile=main  --verbose=1

[21:00:39] hb_init: starting libhb thread
HandBrake svn6164 (2014041501) - MinGW x86_64 - http://handbrake.fr
8 CPUs detected
Opening C:\cygwin64\home\PAUL\handbrake\chocolat.ts...
[21:00:39] CPU: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
[21:00:39]  - Intel microarchitecture Haswell
[21:00:39]  - logical processor count: 8
[21:00:39] OpenCL device #1: Intel(R) Corporation Intel(R) HD Graphics 4600
[21:00:39]  - OpenCL version: 1.2 
[21:00:39]  - driver version: 9.18.10.3257
[21:00:39]  - device type:    GPU
[21:00:39]  - supported:      YES
[21:00:39] Intel Quick Sync Video support: yes
[21:00:39]  - Intel Media SDK hardware: API 1.7 (minimum: 1.3)
[21:00:39]  - Intel Media SDK software: API 1.8 (minimum: 1.3)
[21:00:39]  - H.264 encoder: yes
[21:00:39]     - preferred implementation: hardware (any)
[21:00:39]  - H.265 encoder: no
[21:00:39] hb_scan: path=C:\cygwin64\home\PAUL\handbrake\chocolat.ts, title_index=1
libbluray/bdnav/index_parse.c:162: indx_parse(): error opening C:\cygwin64\home\PAUL\handbrake\chocolat.ts/BDMV/index.bdmv
libbluray/bdnav/index_parse.c:162: indx_parse(): error opening C:\cygwin64\home\PAUL\handbrake\chocolat.ts/BDMV/BACKUP/index.bdmv
libbluray/bluray.c:2182: nav_get_title_list(C:\cygwin64\home\PAUL\handbrake\chocolat.ts) failed
[21:00:39] bd: not a bd - trying as a stream/file instead
libdvdnav: Using dvdnav version 5.0.0
libdvdread: Encrypted DVD support unavailable.
libdvdread:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdread:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
libdvdnav: vm: failed to read VIDEO_TS.IFO
[21:00:39] dvd: not a dvd - trying as a stream/file instead
[21:00:39] file is MPEG Transport Stream with 188 byte packets offset 0 bytes
[21:00:39] Found the following PIDS
[21:00:39]     Video PIDS : 
[21:00:39]       0x880 type MPEG2 (0x2) (PCR)
[21:00:39]     Audio PIDS : 
[21:00:39]       0x881 type AC3 (0x81)
[21:00:39]       0x882 type AC3 (0x81)
[21:00:39]     Subtitle PIDS : 
[21:00:39]     Other PIDS : 
[21:00:39] stream id 0x881 (type 0x81 substream 0x0) audio 0x881
[21:00:39] stream id 0x882 (type 0x81 substream 0x0) audio 0x882
[21:00:39] scan: decoding previews for title 1
[ac3 @ 000000000214b6a0] frame sync error
[ac3 @ 0000000002168fa0] frame sync error
[ac3 @ 00000000021693a0] frame sync error
[21:00:39] scan: audio 0x882: ac3, rate=48000Hz, bitrate=64000 Espanol (AC3) (1.0 ch)
[ac3 @ 000000000218e980] frame sync error
[21:00:39] scan: audio 0x881: ac3, rate=48000Hz, bitrate=128000 English (AC3) (2.0 ch)
[21:00:39] scan: 10 previews, 528x480, 23.976 fps, autocrop = 32/36/0/0, aspect 4:3, PAR 40:33
[21:00:39] stream: 21 good frames, 0 errors (0%)
[21:00:39] libhb: scan thread found 1 valid title(s)
+ title 1:
  + stream: C:\cygwin64\home\PAUL\handbrake\chocolat.ts
  + duration: 00:05:28
  + size: 528x480, pixel aspect: 40/33, display aspect: 1.33, 23.976 fps
  + autocrop: 32/36/0/0
  + support opencl: yes
  + support hwd: yes
  + chapters:
    + 1: cells 0->0, 0 blocks, duration 00:05:28
  + audio tracks:
    + 1, English (AC3) (2.0 ch) (iso639-2: eng), 48000Hz, 128000bps
    + 2, Espanol (AC3) (1.0 ch) (iso639-2: spa), 48000Hz, 64000bps
  + subtitle tracks:
Reading chapter markers from file C:\Users\PAUL\AppData\Local\Temp\chocolat-1-chapters.csv
[21:00:40] 1 job(s) to process
[21:00:40] starting job
[21:00:40] work: sanitizing track 1 mixdown Dolby Pro Logic II to Stereo
[21:00:40] sync: expecting 7868 video frames
[21:00:40] work: only 1 chapter, disabling chapter markers
[21:00:40] job configuration:
[21:00:40]  * source
[21:00:40]    + C:\cygwin64\home\PAUL\handbrake\chocolat.ts
[21:00:40]    + title 1, chapter(s) 1 to 1
[21:00:40]  * destination
[21:00:40]    + C:\cygwin64\home\PAUL\handbrake\chocolat.mp4
[21:00:40]    + container: MPEG-4 (libavformat)
[21:00:40]  * video track
[21:00:40]    + decoder: mpeg2video
[21:00:40]      + bitrate 15000 kbps
[21:00:40]    + filters
[21:00:40]      + Framerate Shaper (1:27000000:1126125)
[21:00:40]        + frame rate: 23.976 fps -> constant 23.976 fps
[21:00:40]      + Crop and Scale (528:412:32:36:0:0)
[21:00:40]        + source: 528 * 480, crop (32/36/0/0): 528 * 412, scale: 528 * 412
[21:00:40]    + loose anamorphic
[21:00:40]      + storage dimensions: 528 * 412, mod 2
[21:00:40]      + pixel aspect ratio: 40 / 33
[21:00:40]      + display dimensions: 640 * 412
[21:00:40]    + encoder: H.264 (Intel Media SDK)
[21:00:40]      + preset:  quality
[21:00:40]      + profile: main
[21:00:40]      + level:   4.0
[21:00:40]      + quality: 20.00 (QP)
[21:00:40]  * audio track 1
[21:00:40]    + decoder: English (AC3) (2.0 ch) (track 1, id 0x881)
[21:00:40]      + bitrate: 128 kbps, samplerate: 48000 Hz
[21:00:40]    + mixdown: Stereo
[21:00:40]    + encoder: AAC (libavcodec)
[21:00:40]      + bitrate: 160 kbps, samplerate: 48000 Hz
[21:00:40] file is MPEG Transport Stream with 188 byte packets offset 0 bytes
[21:00:40] reader: first SCR 5672486781 id 0x880 DTS 5672486781
[21:00:40] mpeg2video: "Chapter 1" (1) at frame 0 time 81094
[21:00:40] encqsvInit: using encode-only path
[21:00:40] encqsvInit: TargetUsage 2 AsyncDepth 4
[21:00:40] encqsvInit: GopRefDist 4 GopPicSize 32 NumRefFrame 3
[21:00:40] encqsvInit: BFrames on BPyramid on
[21:00:40] encqsvInit: RateControlMethod CQP with QPI 20 QPP 22 QPB 24
[21:00:40] encqsvInit: CAVLC off
[21:00:40] encqsvInit: Trellis off
[21:00:40] encqsvInit: H.264 profile Main @ level 4.0
[ac3 @ 0000000007ba2160] frame sync error
[21:00:40] sync: first pts is 81094
[21:00:40] qsv_enc_init: using 'hardware (1)' implementation, API: 1.7
[21:00:41] sync: audio 0x881 time went backwards 33 ms, dropped 1 frames (start 3208154, next 3211200)
[21:00:41] sync: audio 0x881 time went backwards 33 ms, dropped 1 frames (start 3654549, next 3657600)
[21:00:41] sync: audio 0x881 time went backwards 33 ms, dropped 1 frames (start 3916626, next 3919680)
[21:00:41] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 4530059, next 4533120)
[21:00:41] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 5417092, next 5420160)
[21:00:42] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 6151486, next 6154560)
[21:00:42] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 6531640, next 6534720)
[21:00:42] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 7194036, next 7197120)
[21:00:42] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 7856428, next 7859520)
[21:00:42] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 7928427, next 7931520)
[21:00:42] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 8066665, next 8069760)
[21:00:43] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 9567132, next 9570240)
[21:00:43] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 9889689, next 9892800)
[21:00:43] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 10393688, next 10396800)
[21:00:43] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 10465687, next 10468800)
[21:00:43] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 11168401, next 11171520)
[21:00:44] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 11997835, next 12000960)
[21:00:44] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 12285833, next 12288960)
[21:00:44] sync: audio 0x881 time went backwards 34 ms, dropped 1 frames (start 12640070, next 12643200)
[21:00:45] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 15065010, next 15068160)
[21:00:45] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 15137008, next 15140160)
[21:00:45] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 15724524, next 15727680)
[21:00:45] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 16294760, next 16297920)
[21:00:45] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 17406430, next 17409600)
[21:00:46] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 19264020, next 19267200)
[21:00:47] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 20807689, next 20810880)
[21:00:47] 248.430313s: Film -> Video
[21:00:47] 248.563782s: Video -> Film
[21:00:47] 249.931824s: Film -> Video
[21:00:47] 250.065292s: Video -> Film
[21:00:47] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 22933118, next 22936320)
[21:00:48] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 24427825, next 24431040)
[21:00:48] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 24848302, next 24851520)
[21:00:48] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 25012460, next 25015680)
[21:00:48] 279.333069s: Film -> Video
[21:00:48] 279.633331s: Video -> Film
[21:00:49] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 26728925, next 26732160)
[21:00:49] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 26775004, next 26778240)
[21:00:49] sync: audio 0x881 time went backwards 35 ms, dropped 1 frames (start 26939163, next 26942400)
[21:00:49] sync: audio 0x881 time went backwards 36 ms, dropped 1 frames (start 28174673, next 28177920)
[21:00:49] sync: audio 0x881 time went backwards 36 ms, dropped 1 frames (start 28330191, next 28333440)
[21:00:50] hb_ts_stream_decode - eof
[21:00:50] reader: done. 243 scr changes
[21:00:50] work: average encoding speed for job is 789.624512 fps
[21:00:50] sync: got 8158 frames, 7868 expected
[21:00:50] render: 8017 frames output, 160 dropped and 19 duped for CFR/PFR
[21:00:50] render: lost time: 0 (0 frames)
[21:00:50] render: gained time: 0 (0 frames) (0 not accounted for)
[21:00:50] mpeg2video-decoder done: 8158 frames, 5 decoder errors, 0 drops
[21:00:50] ac3-decoder done: 0 frames, 0 decoder errors, 0 drops
[21:00:50] mux: track 0, 8017 frames, 27902074 bytes, 667.56 kbps, fifo 4096
[21:00:50] mux: track 1, 15636 frames, 6664528 bytes, 159.45 kbps, fifo 8192
[21:00:50] stream: 8166 good frames, 0 errors (0%)
[21:00:50] libhb: work result = 0
Encode done!
HandBrake has exited.
User avatar
JohnAStebbins
HandBrake Team
Posts: 5712
Joined: Sat Feb 09, 2008 7:21 pm

Re: CC's missing in SD transport stream

Post by JohnAStebbins »

Thanks for the sample. I had a look and the CCs are not ATSC format. They are SCTE format. Libavcodec and handbrake only supports ATSC CCs (a.k.a A53 part 4). But now that I have a sample of STCE CCs, I may add support for this eventually. VLC has good example code and the spec is easily found via google, so it shouldn't be all that difficult.
PaulF
Novice
Posts: 58
Joined: Thu Nov 21, 2013 7:15 am

Re: CC's missing in SD transport stream

Post by PaulF »

Oops. Like I said I had forgotten this problem. It turns out that clip I just uploaded was recorded a long time ago and you point out it fails for a different reason than the first post. I tried a more recent SD like the one I first reported. It uses both ATSC CC and Aspect ratio user_data packets and the original problem is fixed. Thanks.

Maybe Comcast has switched to ATSC or ATSC usage depends on the channel.
Deleted User 13735

Re: CC's missing in SD transport stream

Post by Deleted User 13735 »

ATSC was phased in 2009-2011 in N America iirc.
Smithcraft
Veteran User
Posts: 2697
Joined: Thu Jan 22, 2009 8:04 pm

Re: CC's missing in SD transport stream

Post by Smithcraft »

Well, Comcast will most likely implement it in a few years.

SC
Post Reply