Subtitle characters with diacritics are stripped

Support for HandBrake on Linux, Solaris, and other Unix-like platforms
Forum rules
An Activity Log is required for support requests. Please read How-to get an activity log? for details on how and why this should be provided.
Post Reply
John_Jason_Jordan
Posts: 9
Joined: Tue Feb 07, 2017 11:06 pm

Subtitle characters with diacritics are stripped

Post by John_Jason_Jordan » Thu Feb 15, 2018 9:09 pm

I was re-encoding a .mkv file that did not have subtitles. I found subtitles on opensubtitles.org for English, Italian and Spanish and I encoded them as optional subtitles into the file that HandBrake created (not hard coded). After the encoding was finished I discovered that any character that had a diacritic was deleted. This was not a problem for the English subtitles, but it made the Italian and Spanish subtitles really bad, hard to read.

I opened the subtitle files in Subtitle Editor and in Aegisub, and they looked fine. I opened the .mkv file that HandBrake created and added the subtitles from the files instead of selecting the internal subtitles, and then they appeared correct.

This is HandBrake 1.0.4 (x86_64) on Xubuntu 14.04.5 LTS.

I closed HandBrake, renamed the Activity Log, then restarted HandBrake so it would create a clean Activity Log for just this encoding task. I have added it below:

Code: Select all

[09:59:16] gtkgui: HandBrake 1.0.4 (2017040900) - Linux x86_64 - https://handbrake.fr
[09:59:16] hb_init: starting libhb thread
[09:59:16] hb_init: starting libhb thread
[09:59:16] hb_init: starting libhb thread
[10:00:29] CPU: Intel(R) Core(TM) i7-4800MQ CPU @ 2.70GHz
[10:00:29]  - Intel microarchitecture Haswell
[10:00:29]  - logical processor count: 8
[10:00:29] hb_scan: path=/media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv, title_index=0
udfread ERROR: ECMA 167 Volume Recognition failed
disc.c:274: failed opening UDF image /media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv
disc.c:352: error opening file BDMV/index.bdmv
disc.c:352: error opening file BDMV/BACKUP/index.bdmv
[10:00:29] bd: not a bd - trying as a stream/file instead
libdvdnav: Using dvdnav version 5.0.1
libdvdread:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdread:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
libdvdnav: vm: failed to read VIDEO_TS.IFO
[10:00:29] dvd: not a dvd - trying as a stream/file instead
Input #0, matroska,webm, from '/media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv':
  Metadata:
    title           : EVO
  Duration: 01:47:19.55, start: 0.000000, bitrate: N/A
    Chapter #0:0: start 0.000000, end 808.333000
    Metadata:
      title           : 00:00:00.000
    Chapter #0:1: start 808.416000, end 1555.958000
    Metadata:
      title           : 00:13:28.333
    Chapter #0:2: start 1556.041000, end 2345.583000
    Metadata:
      title           : 00:25:55.917
    Chapter #0:3: start 2345.666000, end 3171.833000
    Metadata:
      title           : 00:39:05.542
    Chapter #0:4: start 3171.916000, end 3984.833000
    Metadata:
      title           : 00:52:51.792
    Chapter #0:5: start 3984.916000, end 4764.958000
    Metadata:
      title           : 01:06:24.833
    Chapter #0:6: start 4765.041000, end 5554.208000
    Metadata:
      title           : 01:19:24.917
    Chapter #0:7: start 5554.291000, end 6379.125000
    Metadata:
      title           : 01:32:34.167
    Stream #0:0: Video: h264 (High)
      yuv420p, tv, bt709/bt709/bt709
      1920x800, PAR 1:1 DAR 12:5
      24 fps, 1k tbn (default)
    Stream #0:1(eng): Audio: ac3
      48000 Hz, 5.1, fltp, 640 kb/s (default)
    Metadata:
      title           : Surround
[10:00:29] scan: decoding previews for title 1
[10:00:29] scan: audio 0x1: ac3, rate=48000Hz, bitrate=640000 English (AC3) (5.1 ch)
[10:00:30] scan: 10 previews, 1920x800, 24.000 fps, autocrop = 0/0/0/0, aspect 2.40:1, PAR 1:1
[10:00:30] libhb: scan thread found 1 valid title(s)
[10:03:48] gtkgui: Modified Custom Preset: /H.264 MKV 1080p30-jjj
[10:03:48] 1 job(s) to process
[10:03:48] json job:
{
    "Audio": {
        "AudioList": [
            {
                "Bitrate": 448,
                "CompressionLevel": -1.0,
                "DRC": 0.0,
                "DitherMethod": "auto",
                "Encoder": "ac3",
                "Gain": 0.0,
                "Mixdown": "5point1",
                "NormalizeMixLevel": false,
                "PresetEncoder": "ac3",
                "Quality": -3.0,
                "Samplerate": "auto",
                "Track": 0
            }
        ],
        "CopyMask": [
            "copy:ac3"
        ],
        "FallbackEncoder": "ac3"
    },
    "Destination": {
        "ChapterList": [
            {
                "Name": "00:00:00.000"
            },
            {
                "Name": "00:13:28.333"
            },
            {
                "Name": "00:25:55.917"
            },
            {
                "Name": "00:39:05.542"
            },
            {
                "Name": "00:52:51.792"
            },
            {
                "Name": "01:06:24.833"
            },
            {
                "Name": "01:19:24.917"
            },
            {
                "Name": "01:32:34.167"
            }
        ],
        "ChapterMarkers": true,
        "File": "/media/jjj/Movies/Cold Skin (La piel fría) 2017.mkv",
        "Mp4Options": {
            "IpodAtom": false,
            "Mp4Optimize": false
        },
        "Mux": "mkv"
    },
    "Filters": {
        "FilterList": [
            {
                "ID": 3,
                "Settings": {
                    "block-height": "16",
                    "block-thresh": "40",
                    "block-width": "16",
                    "filter-mode": "2",
                    "mode": "3",
                    "motion-thresh": "1",
                    "spatial-metric": "2",
                    "spatial-thresh": "1"
                }
            },
            {
                "ID": 4,
                "Settings": {
                    "mode": "7"
                }
            },
            {
                "ID": 6,
                "Settings": {
                    "mode": 2,
                    "rate": "27000000/900000"
                }
            },
            {
                "ID": 11,
                "Settings": {
                    "crop-bottom": 0,
                    "crop-left": 0,
                    "crop-right": 0,
                    "crop-top": 0,
                    "height": 800,
                    "width": 1920
                }
            }
        ]
    },
    "Metadata": {
        "Name": "Cold Skin (La piel fría) 2017"
    },
    "PAR": {
        "Den": 1,
        "Num": 1
    },
    "SequenceID": 0,
    "Source": {
        "Angle": 0,
        "Path": "/media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv",
        "Range": {
            "End": 574020000,
            "Start": 0,
            "Type": "time"
        },
        "Title": 1
    },
    "Subtitle": {
        "Search": {
            "Burn": true,
            "Default": false,
            "Enable": false,
            "Forced": false
        },
        "SubtitleList": [
            {
                "Burn": false,
                "Default": true,
                "Forced": false,
                "Offset": 0,
                "SRT": {
                    "Codeset": "UTF-8",
                    "Filename": "/media/jjj/Movies/Cold.Skin.2017.English.srt",
                    "Language": "eng"
                },
                "Track": 0
            },
            {
                "Burn": false,
                "Default": false,
                "Forced": false,
                "Offset": 0,
                "SRT": {
                    "Codeset": "UTF-8",
                    "Filename": "/media/jjj/Movies/Cold.Skin.2017.Italian.srt",
                    "Language": "ita"
                },
                "Track": 0
            },
            {
                "Burn": false,
                "Default": false,
                "Forced": false,
                "Offset": 0,
                "SRT": {
                    "Codeset": "UTF-8",
                    "Filename": "/media/jjj/Movies/Cold.Skin.2017.Spanish.srt",
                    "Language": "spa"
                },
                "Track": 0
            }
        ]
    },
    "Video": {
        "ColorMatrixCode": 0,
        "Encoder": "x264",
        "Level": "4.0",
        "OpenCL": false,
        "Options": "",
        "Preset": "medium",
        "Profile": "main",
        "QSV": {
            "AsyncDepth": 4,
            "Decode": false
        },
        "Quality": 20.0,
        "Tune": "",
        "Turbo": false,
        "TwoPass": false
    }
}
[10:03:48] CPU: Intel(R) Core(TM) i7-4800MQ CPU @ 2.70GHz
[10:03:48]  - Intel microarchitecture Haswell
[10:03:48]  - logical processor count: 8
[10:03:48] hb_scan: path=/media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv, title_index=1
udfread ERROR: ECMA 167 Volume Recognition failed
disc.c:274: failed opening UDF image /media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv
disc.c:352: error opening file BDMV/index.bdmv
disc.c:352: error opening file BDMV/BACKUP/index.bdmv
[10:03:48] bd: not a bd - trying as a stream/file instead
libdvdnav: Using dvdnav version 5.0.1
libdvdread:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdread:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
libdvdnav: vm: failed to read VIDEO_TS.IFO
[10:03:48] dvd: not a dvd - trying as a stream/file instead
Input #0, matroska,webm, from '/media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv':
  Metadata:
    title           : EVO
  Duration: 01:47:19.55, start: 0.000000, bitrate: N/A
    Chapter #0:0: start 0.000000, end 808.333000
    Metadata:
      title           : 00:00:00.000
    Chapter #0:1: start 808.416000, end 1555.958000
    Metadata:
      title           : 00:13:28.333
    Chapter #0:2: start 1556.041000, end 2345.583000
    Metadata:
      title           : 00:25:55.917
    Chapter #0:3: start 2345.666000, end 3171.833000
    Metadata:
      title           : 00:39:05.542
    Chapter #0:4: start 3171.916000, end 3984.833000
    Metadata:
      title           : 00:52:51.792
    Chapter #0:5: start 3984.916000, end 4764.958000
    Metadata:
      title           : 01:06:24.833
    Chapter #0:6: start 4765.041000, end 5554.208000
    Metadata:
      title           : 01:19:24.917
    Chapter #0:7: start 5554.291000, end 6379.125000
    Metadata:
      title           : 01:32:34.167
    Stream #0:0: Video: h264 (High)
      yuv420p, tv, bt709/bt709/bt709
      1920x800, PAR 1:1 DAR 12:5
      24 fps, 1k tbn (default)
    Stream #0:1(eng): Audio: ac3
      48000 Hz, 5.1, fltp, 640 kb/s (default)
    Metadata:
      title           : Surround
[10:03:48] scan: decoding previews for title 1
[10:03:48] scan: audio 0x1: ac3, rate=48000Hz, bitrate=640000 English (AC3) (5.1 ch)
[10:03:48] scan: 10 previews, 1920x800, 24.000 fps, autocrop = 0/0/0/0, aspect 2.40:1, PAR 1:1
[10:03:48] libhb: scan thread found 1 valid title(s)
[10:03:48] starting job
[10:03:48] decomb filter thread started for segment 0
[10:03:48] decomb filter thread started for segment 1
[10:03:48] decomb filter thread started for segment 2
[10:03:48] decomb filter thread started for segment 3
[10:03:48] decomb filter thread started for segment 4
[10:03:48] decomb filter thread started for segment 5
[10:03:48] decomb filter thread started for segment 6
[10:03:48] decomb filter thread started for segment 7
[10:03:48] decomb check thread started for segment 1
[10:03:48] decomb check thread started for segment 0
[10:03:48] decomb check thread started for segment 2
[10:03:48] decomb check thread started for segment 3
[10:03:48] decomb check thread started for segment 4
[10:03:48] decomb check thread started for segment 5
[10:03:48] mask filter thread started for segment 3
[10:03:48] mask filter thread started for segment 4
[10:03:48] mask filter thread started for segment 5
[10:03:48] mask filter thread started for segment 1
[10:03:48] mask filter thread started for segment 2
[10:03:48] decomb check thread started for segment 6
[10:03:48] decomb check thread started for segment 7
[10:03:48] mask filter thread started for segment 0
[10:03:48] mask filter thread started for segment 6
[10:03:48] mask filter thread started for segment 7
[10:03:48] mask erode thread started for segment 0
[10:03:48] mask dilate thread started for segment 1
[10:03:48] mask erode thread started for segment 2
[10:03:48] mask erode thread started for segment 3
[10:03:48] mask erode thread started for segment 4
[10:03:48] mask erode thread started for segment 5
[10:03:48] mask erode thread started for segment 6
[10:03:48] yadif thread started for segment 2
[10:03:48] mask dilate thread started for segment 0
[10:03:48] mask dilate thread started for segment 2
[10:03:48] yadif thread started for segment 7
[10:03:48] mask dilate thread started for segment 3
[10:03:48] mask dilate thread started for segment 5
[10:03:48] job configuration:
[10:03:48] mask dilate thread started for segment 6
[10:03:48] mask dilate thread started for segment 7
[10:03:48] yadif thread started for segment 0
[10:03:48] yadif thread started for segment 1
[10:03:48] mask erode thread started for segment 7
[10:03:48] yadif thread started for segment 3
[10:03:48] yadif thread started for segment 4
[10:03:48] yadif thread started for segment 5
[10:03:48] yadif thread started for segment 6
[10:03:48] mask erode thread started for segment 1
[10:03:48] mask dilate thread started for segment 4
[10:03:48]  * source
[10:03:48]    + /media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv
[10:03:48]    + title 1, start 00:00:0.00 stop 01:46:18.00
[10:03:48]    + container: matroska,webm
[10:03:48]  * destination
[10:03:48]    + /media/jjj/Movies/Cold Skin (La piel fría) 2017.mkv
[10:03:48]    + container: Matroska (libavformat)
[10:03:48]      + chapter markers
[10:03:48]  * video track
[10:03:48]    + decoder: h264
[10:03:48]    + filters
[10:03:48]      + Comb Detect (mode=3:spatial-metric=2:motion-thresh=1:spatial-thresh=1:filter-mode=2:block-thresh=40:block-width=16:block-height=16)
[10:03:48]      + Decomb (mode=39)
[10:03:48]      + Framerate Shaper (mode=2:rate=27000000/900000)
[10:03:48]        + frame rate: 24.000 fps -> peak rate limited to 30.000 fps
[10:03:48]      + Crop and Scale (width=1920:height=800:crop-top=0:crop-bottom=0:crop-left=0:crop-right=0)
[10:03:48]        + source: 1920 * 800, crop (0/0/0/0): 1920 * 800, scale: 1920 * 800
[10:03:48]    + Output geometry
[10:03:48]      + storage dimensions: 1920 x 800
[10:03:48]      + pixel aspect ratio: 1 : 1
[10:03:48]      + display dimensions: 1920 x 800
[10:03:48]    + encoder: H.264 (libx264)
[10:03:48]      + preset:  medium
[10:03:48]      + profile: main
[10:03:48]      + level:   4.0
[10:03:48]      + quality: 20.00 (RF)
[10:03:48]  * subtitle track 1, English (track 0, id 0xff) Text [SRT] -> Passthrough, Default, offset: 0, charset: UTF-8
[10:03:48]  * subtitle track 2, Italian (track 0, id 0x1ff) Text [SRT] -> Passthrough, offset: 0, charset: UTF-8
[10:03:48]  * subtitle track 3, Spanish (track 0, id 0x2ff) Text [SRT] -> Passthrough, offset: 0, charset: UTF-8
[10:03:48]  * audio track 1
[10:03:48]    + decoder: English (AC3) (5.1 ch) (track 1, id 0x1)
[10:03:48]      + bitrate: 640 kbps, samplerate: 48000 Hz
[10:03:48]    + mixdown: 5.1 Channels
[10:03:48]    + encoder: AC3 (libavcodec)
[10:03:48]      + bitrate: 448 kbps, samplerate: 48000 Hz
[10:03:48] sync: expecting 153096 video frames
[10:03:48] encx264: min-keyint: 24, keyint: 240
[10:03:48] encx264: encoding at constant RF 20.000000
[10:03:48] encx264: unparsed options: level=4.0:8x8dct=0:vbv-maxrate=20000:vbv-bufsize=25000
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
x264 [info]: profile Main, level 4.0
[10:03:49] Invalid byte for codeset in input, discard byte
[10:03:49] sync: first pts audio 0x1 is 0
[10:03:49] sync: first pts video is 450
[10:03:49] sync: "00:00:00.000" (1) at frame 1 time 450
[10:04:18] sync: first pts subtitle 0xff is 5043690
[10:04:18] sync: first pts subtitle 0x1ff is 5043690
[10:04:18] sync: first pts subtitle 0x2ff is 5043690
[10:20:33] sync: "00:13:28.333" (2) at frame 19399 time 72742950
[10:38:16] sync: "00:25:55.917" (3) at frame 37342 time 140029200
[10:56:33] sync: "00:39:05.542" (4) at frame 56293 time 211095450
[11:15:25] sync: "00:52:51.792" (5) at frame 76123 time 285457950
[11:34:18] sync: "01:06:24.833" (6) at frame 95636 time 358631700
[11:52:17] sync: "01:19:24.917" (7) at frame 114358 time 428839200
[12:11:08] sync: "01:32:34.167" (8) at frame 133301 time 499875450
[12:23:56] sync: reached audio 0x1 pts 574021440, exiting early
[12:23:57] reader: done. 1 scr changes
[12:23:59] work: average encoding speed for job is 18.201473 fps
[12:23:59] comb detect: heavy 182 | light 2454 | uncombed 150464 | total 153100
[12:23:59] decomb: deinterlaced 182 | blended 2454 | unfiltered 150464 | total 153100
[12:23:59] vfr: 153100 frames output, 0 dropped and 0 duped for CFR/PFR
[12:23:59] vfr: lost time: 0 (0 frames)
[12:23:59] vfr: gained time: 0 (0 frames) (0 not accounted for)
[12:23:59] ac3-decoder done: 201236 frames, 0 decoder errors
[12:23:59] h264-decoder done: 153100 frames, 0 decoder errors
[12:23:59] sync: got 153100 frames, 153096 expected
[12:23:59] sync: framerate min 24.000 fps, max 24.000 fps, avg 24.000 fps
x264 [info]: frame I:2370  Avg QP:15.76  size: 98156
x264 [info]: frame P:91924 Avg QP:19.24  size: 27932
x264 [info]: frame B:58806 Avg QP:21.06  size:  9475
x264 [info]: consecutive B-frames: 38.4% 28.3%  8.4% 24.9%
x264 [info]: mb I  I16..4: 42.1%  0.0% 57.9%
x264 [info]: mb P  I16..4: 12.2%  0.0%  4.1%  P16..4: 43.8% 13.1%  7.1%  0.0%  0.0%    skip:19.6%
x264 [info]: mb B  I16..4:  0.8%  0.0%  0.3%  B16..8: 44.1%  3.6%  0.6%  direct: 4.3%  skip:46.2%  L0:39.0% L1:56.1% BI: 5.0%
x264 [info]: coded y,uvDC,uvAC intra: 44.3% 63.7% 30.6% inter: 17.1% 31.5% 0.9%
x264 [info]: i16 v,h,dc,p: 38% 22% 25% 15%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 20% 23%  6%  7%  6%  6%  5%  4%
x264 [info]: i8c dc,h,v,p: 62% 17% 19%  3%
x264 [info]: Weighted P-Frames: Y:4.7% UV:2.2%
x264 [info]: ref P L0: 67.8% 11.7% 13.6%  6.6%  0.3%
x264 [info]: ref B L0: 89.3%  9.8%  0.9%
x264 [info]: ref B L1: 97.0%  3.0%
x264 [info]: kb/s:4210.51
[12:23:59] mux: track 0, 153100 frames, 3357371379 bytes, 4210.39 kbps, fifo 4096
[12:23:59] mux: track 1, 199312 frames, 357167104 bytes, 447.91 kbps, fifo 8192
[12:23:59] mux: track 2, 483 frames, 25538 bytes, 0.03 kbps, fifo 16
[12:23:59] mux: track 3, 494 frames, 26469 bytes, 0.03 kbps, fifo 16
[12:23:59] mux: track 4, 490 frames, 25981 bytes, 0.03 kbps, fifo 16
[12:23:59] libhb: work result = 0
[code][/i]

Woodstock
Veteran User
Posts: 2663
Joined: Tue Aug 27, 2013 6:39 am

Re: Subtitle characters with diacritics are stripped

Post by Woodstock » Fri Feb 16, 2018 12:16 am

Are you sure the file is encoded with UTF-8?

John_Jason_Jordan
Posts: 9
Joined: Tue Feb 07, 2017 11:06 pm

Re: Subtitle characters with diacritics are stripped

Post by John_Jason_Jordan » Fri Feb 16, 2018 3:44 am

Aha! That was it!

The Spanish subtitles were encoded with Western 8859-15. Using Subtitle Editor I did a Save As where I could change it to UTF-8. I also changed the Newline to Unix.* Then I restarted the encoding using the new .srt file. After 15 minutes I opened the new .mkv file in VLC and selected the Spanish subtitles, but they did not appear at all. So I stopped the encoding, went back to Subtitle Editor and re-saved again, this time with Newline as Windows. Then I started the encoding over again using the new .srt file, and when I checked on it 15 minutes later the Spanish subtitles were appearing properly.

*Regarding Unix newline: All my computers run on Linux, and I often extract subtitles from DVDs and Blu-rays, usually using mkvmerge -i to find the subtitle track, then 'mkvextract tracks' to extract it as sub/idx or .sup, which I then convert to .srt. Every time I do this the Newline is always Unix, and the subtitles work perfectly in HandBrake and, after the encoding is finished, in VLC. I don't know why this time HandBrake couldn't use them.

Oh well. Problem solved and I learned a new trick. In the future I will make a point of checking that the subtitle file is really UTF-8. Thanks heaps for your suggestion!

Post Reply