With upcoming changes to data protection and privacy laws in Europe coming into effect soon, we thought this would be a good time to remind everyone that we do have a privacy policy.
This applies to all users and visitors world-wide.

We have made a few changes to the language to make it clearer in relation to this new regulation but fundamentally, the terms and your rights are unchanged.

If you have any questions about this, please feel free to ask in the General Forum

Subtitle characters with diacritics are stripped

Support for HandBrake on Linux, Solaris, and other Unix-like platforms
Forum rules
An Activity Log is required for support requests. Please read How-to get an activity log? for details on how and why this should be provided.
Post Reply
John_Jason_Jordan
Posts: 9
Joined: Tue Feb 07, 2017 11:06 pm

Subtitle characters with diacritics are stripped

Post by John_Jason_Jordan » Thu Feb 15, 2018 9:09 pm

I was re-encoding a .mkv file that did not have subtitles. I found subtitles on opensubtitles.org for English, Italian and Spanish and I encoded them as optional subtitles into the file that HandBrake created (not hard coded). After the encoding was finished I discovered that any character that had a diacritic was deleted. This was not a problem for the English subtitles, but it made the Italian and Spanish subtitles really bad, hard to read.

I opened the subtitle files in Subtitle Editor and in Aegisub, and they looked fine. I opened the .mkv file that HandBrake created and added the subtitles from the files instead of selecting the internal subtitles, and then they appeared correct.

This is HandBrake 1.0.4 (x86_64) on Xubuntu 14.04.5 LTS.

I closed HandBrake, renamed the Activity Log, then restarted HandBrake so it would create a clean Activity Log for just this encoding task. I have added it below:

Code: Select all

[09:59:16] gtkgui: HandBrake 1.0.4 (2017040900) - Linux x86_64 - https://handbrake.fr
[09:59:16] hb_init: starting libhb thread
[09:59:16] hb_init: starting libhb thread
[09:59:16] hb_init: starting libhb thread
[10:00:29] CPU: Intel(R) Core(TM) i7-4800MQ CPU @ 2.70GHz
[10:00:29]  - Intel microarchitecture Haswell
[10:00:29]  - logical processor count: 8
[10:00:29] hb_scan: path=/media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv, title_index=0
udfread ERROR: ECMA 167 Volume Recognition failed
disc.c:274: failed opening UDF image /media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv
disc.c:352: error opening file BDMV/index.bdmv
disc.c:352: error opening file BDMV/BACKUP/index.bdmv
[10:00:29] bd: not a bd - trying as a stream/file instead
libdvdnav: Using dvdnav version 5.0.1
libdvdread:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdread:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
libdvdnav: vm: failed to read VIDEO_TS.IFO
[10:00:29] dvd: not a dvd - trying as a stream/file instead
Input #0, matroska,webm, from '/media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv':
  Metadata:
    title           : EVO
  Duration: 01:47:19.55, start: 0.000000, bitrate: N/A
    Chapter #0:0: start 0.000000, end 808.333000
    Metadata:
      title           : 00:00:00.000
    Chapter #0:1: start 808.416000, end 1555.958000
    Metadata:
      title           : 00:13:28.333
    Chapter #0:2: start 1556.041000, end 2345.583000
    Metadata:
      title           : 00:25:55.917
    Chapter #0:3: start 2345.666000, end 3171.833000
    Metadata:
      title           : 00:39:05.542
    Chapter #0:4: start 3171.916000, end 3984.833000
    Metadata:
      title           : 00:52:51.792
    Chapter #0:5: start 3984.916000, end 4764.958000
    Metadata:
      title           : 01:06:24.833
    Chapter #0:6: start 4765.041000, end 5554.208000
    Metadata:
      title           : 01:19:24.917
    Chapter #0:7: start 5554.291000, end 6379.125000
    Metadata:
      title           : 01:32:34.167
    Stream #0:0: Video: h264 (High)
      yuv420p, tv, bt709/bt709/bt709
      1920x800, PAR 1:1 DAR 12:5
      24 fps, 1k tbn (default)
    Stream #0:1(eng): Audio: ac3
      48000 Hz, 5.1, fltp, 640 kb/s (default)
    Metadata:
      title           : Surround
[10:00:29] scan: decoding previews for title 1
[10:00:29] scan: audio 0x1: ac3, rate=48000Hz, bitrate=640000 English (AC3) (5.1 ch)
[10:00:30] scan: 10 previews, 1920x800, 24.000 fps, autocrop = 0/0/0/0, aspect 2.40:1, PAR 1:1
[10:00:30] libhb: scan thread found 1 valid title(s)
[10:03:48] gtkgui: Modified Custom Preset: /H.264 MKV 1080p30-jjj
[10:03:48] 1 job(s) to process
[10:03:48] json job:
{
    "Audio": {
        "AudioList": [
            {
                "Bitrate": 448,
                "CompressionLevel": -1.0,
                "DRC": 0.0,
                "DitherMethod": "auto",
                "Encoder": "ac3",
                "Gain": 0.0,
                "Mixdown": "5point1",
                "NormalizeMixLevel": false,
                "PresetEncoder": "ac3",
                "Quality": -3.0,
                "Samplerate": "auto",
                "Track": 0
            }
        ],
        "CopyMask": [
            "copy:ac3"
        ],
        "FallbackEncoder": "ac3"
    },
    "Destination": {
        "ChapterList": [
            {
                "Name": "00:00:00.000"
            },
            {
                "Name": "00:13:28.333"
            },
            {
                "Name": "00:25:55.917"
            },
            {
                "Name": "00:39:05.542"
            },
            {
                "Name": "00:52:51.792"
            },
            {
                "Name": "01:06:24.833"
            },
            {
                "Name": "01:19:24.917"
            },
            {
                "Name": "01:32:34.167"
            }
        ],
        "ChapterMarkers": true,
        "File": "/media/jjj/Movies/Cold Skin (La piel fría) 2017.mkv",
        "Mp4Options": {
            "IpodAtom": false,
            "Mp4Optimize": false
        },
        "Mux": "mkv"
    },
    "Filters": {
        "FilterList": [
            {
                "ID": 3,
                "Settings": {
                    "block-height": "16",
                    "block-thresh": "40",
                    "block-width": "16",
                    "filter-mode": "2",
                    "mode": "3",
                    "motion-thresh": "1",
                    "spatial-metric": "2",
                    "spatial-thresh": "1"
                }
            },
            {
                "ID": 4,
                "Settings": {
                    "mode": "7"
                }
            },
            {
                "ID": 6,
                "Settings": {
                    "mode": 2,
                    "rate": "27000000/900000"
                }
            },
            {
                "ID": 11,
                "Settings": {
                    "crop-bottom": 0,
                    "crop-left": 0,
                    "crop-right": 0,
                    "crop-top": 0,
                    "height": 800,
                    "width": 1920
                }
            }
        ]
    },
    "Metadata": {
        "Name": "Cold Skin (La piel fría) 2017"
    },
    "PAR": {
        "Den": 1,
        "Num": 1
    },
    "SequenceID": 0,
    "Source": {
        "Angle": 0,
        "Path": "/media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv",
        "Range": {
            "End": 574020000,
            "Start": 0,
            "Type": "time"
        },
        "Title": 1
    },
    "Subtitle": {
        "Search": {
            "Burn": true,
            "Default": false,
            "Enable": false,
            "Forced": false
        },
        "SubtitleList": [
            {
                "Burn": false,
                "Default": true,
                "Forced": false,
                "Offset": 0,
                "SRT": {
                    "Codeset": "UTF-8",
                    "Filename": "/media/jjj/Movies/Cold.Skin.2017.English.srt",
                    "Language": "eng"
                },
                "Track": 0
            },
            {
                "Burn": false,
                "Default": false,
                "Forced": false,
                "Offset": 0,
                "SRT": {
                    "Codeset": "UTF-8",
                    "Filename": "/media/jjj/Movies/Cold.Skin.2017.Italian.srt",
                    "Language": "ita"
                },
                "Track": 0
            },
            {
                "Burn": false,
                "Default": false,
                "Forced": false,
                "Offset": 0,
                "SRT": {
                    "Codeset": "UTF-8",
                    "Filename": "/media/jjj/Movies/Cold.Skin.2017.Spanish.srt",
                    "Language": "spa"
                },
                "Track": 0
            }
        ]
    },
    "Video": {
        "ColorMatrixCode": 0,
        "Encoder": "x264",
        "Level": "4.0",
        "OpenCL": false,
        "Options": "",
        "Preset": "medium",
        "Profile": "main",
        "QSV": {
            "AsyncDepth": 4,
            "Decode": false
        },
        "Quality": 20.0,
        "Tune": "",
        "Turbo": false,
        "TwoPass": false
    }
}
[10:03:48] CPU: Intel(R) Core(TM) i7-4800MQ CPU @ 2.70GHz
[10:03:48]  - Intel microarchitecture Haswell
[10:03:48]  - logical processor count: 8
[10:03:48] hb_scan: path=/media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv, title_index=1
udfread ERROR: ECMA 167 Volume Recognition failed
disc.c:274: failed opening UDF image /media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv
disc.c:352: error opening file BDMV/index.bdmv
disc.c:352: error opening file BDMV/BACKUP/index.bdmv
[10:03:48] bd: not a bd - trying as a stream/file instead
libdvdnav: Using dvdnav version 5.0.1
libdvdread:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdread:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
libdvdnav: vm: failed to read VIDEO_TS.IFO
[10:03:48] dvd: not a dvd - trying as a stream/file instead
Input #0, matroska,webm, from '/media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv':
  Metadata:
    title           : EVO
  Duration: 01:47:19.55, start: 0.000000, bitrate: N/A
    Chapter #0:0: start 0.000000, end 808.333000
    Metadata:
      title           : 00:00:00.000
    Chapter #0:1: start 808.416000, end 1555.958000
    Metadata:
      title           : 00:13:28.333
    Chapter #0:2: start 1556.041000, end 2345.583000
    Metadata:
      title           : 00:25:55.917
    Chapter #0:3: start 2345.666000, end 3171.833000
    Metadata:
      title           : 00:39:05.542
    Chapter #0:4: start 3171.916000, end 3984.833000
    Metadata:
      title           : 00:52:51.792
    Chapter #0:5: start 3984.916000, end 4764.958000
    Metadata:
      title           : 01:06:24.833
    Chapter #0:6: start 4765.041000, end 5554.208000
    Metadata:
      title           : 01:19:24.917
    Chapter #0:7: start 5554.291000, end 6379.125000
    Metadata:
      title           : 01:32:34.167
    Stream #0:0: Video: h264 (High)
      yuv420p, tv, bt709/bt709/bt709
      1920x800, PAR 1:1 DAR 12:5
      24 fps, 1k tbn (default)
    Stream #0:1(eng): Audio: ac3
      48000 Hz, 5.1, fltp, 640 kb/s (default)
    Metadata:
      title           : Surround
[10:03:48] scan: decoding previews for title 1
[10:03:48] scan: audio 0x1: ac3, rate=48000Hz, bitrate=640000 English (AC3) (5.1 ch)
[10:03:48] scan: 10 previews, 1920x800, 24.000 fps, autocrop = 0/0/0/0, aspect 2.40:1, PAR 1:1
[10:03:48] libhb: scan thread found 1 valid title(s)
[10:03:48] starting job
[10:03:48] decomb filter thread started for segment 0
[10:03:48] decomb filter thread started for segment 1
[10:03:48] decomb filter thread started for segment 2
[10:03:48] decomb filter thread started for segment 3
[10:03:48] decomb filter thread started for segment 4
[10:03:48] decomb filter thread started for segment 5
[10:03:48] decomb filter thread started for segment 6
[10:03:48] decomb filter thread started for segment 7
[10:03:48] decomb check thread started for segment 1
[10:03:48] decomb check thread started for segment 0
[10:03:48] decomb check thread started for segment 2
[10:03:48] decomb check thread started for segment 3
[10:03:48] decomb check thread started for segment 4
[10:03:48] decomb check thread started for segment 5
[10:03:48] mask filter thread started for segment 3
[10:03:48] mask filter thread started for segment 4
[10:03:48] mask filter thread started for segment 5
[10:03:48] mask filter thread started for segment 1
[10:03:48] mask filter thread started for segment 2
[10:03:48] decomb check thread started for segment 6
[10:03:48] decomb check thread started for segment 7
[10:03:48] mask filter thread started for segment 0
[10:03:48] mask filter thread started for segment 6
[10:03:48] mask filter thread started for segment 7
[10:03:48] mask erode thread started for segment 0
[10:03:48] mask dilate thread started for segment 1
[10:03:48] mask erode thread started for segment 2
[10:03:48] mask erode thread started for segment 3
[10:03:48] mask erode thread started for segment 4
[10:03:48] mask erode thread started for segment 5
[10:03:48] mask erode thread started for segment 6
[10:03:48] yadif thread started for segment 2
[10:03:48] mask dilate thread started for segment 0
[10:03:48] mask dilate thread started for segment 2
[10:03:48] yadif thread started for segment 7
[10:03:48] mask dilate thread started for segment 3
[10:03:48] mask dilate thread started for segment 5
[10:03:48] job configuration:
[10:03:48] mask dilate thread started for segment 6
[10:03:48] mask dilate thread started for segment 7
[10:03:48] yadif thread started for segment 0
[10:03:48] yadif thread started for segment 1
[10:03:48] mask erode thread started for segment 7
[10:03:48] yadif thread started for segment 3
[10:03:48] yadif thread started for segment 4
[10:03:48] yadif thread started for segment 5
[10:03:48] yadif thread started for segment 6
[10:03:48] mask erode thread started for segment 1
[10:03:48] mask dilate thread started for segment 4
[10:03:48]  * source
[10:03:48]    + /media/jjj/Movies/Cold.Skin.2018.1080p.Bluray.X264-EVO.mkv
[10:03:48]    + title 1, start 00:00:0.00 stop 01:46:18.00
[10:03:48]    + container: matroska,webm
[10:03:48]  * destination
[10:03:48]    + /media/jjj/Movies/Cold Skin (La piel fría) 2017.mkv
[10:03:48]    + container: Matroska (libavformat)
[10:03:48]      + chapter markers
[10:03:48]  * video track
[10:03:48]    + decoder: h264
[10:03:48]    + filters
[10:03:48]      + Comb Detect (mode=3:spatial-metric=2:motion-thresh=1:spatial-thresh=1:filter-mode=2:block-thresh=40:block-width=16:block-height=16)
[10:03:48]      + Decomb (mode=39)
[10:03:48]      + Framerate Shaper (mode=2:rate=27000000/900000)
[10:03:48]        + frame rate: 24.000 fps -> peak rate limited to 30.000 fps
[10:03:48]      + Crop and Scale (width=1920:height=800:crop-top=0:crop-bottom=0:crop-left=0:crop-right=0)
[10:03:48]        + source: 1920 * 800, crop (0/0/0/0): 1920 * 800, scale: 1920 * 800
[10:03:48]    + Output geometry
[10:03:48]      + storage dimensions: 1920 x 800
[10:03:48]      + pixel aspect ratio: 1 : 1
[10:03:48]      + display dimensions: 1920 x 800
[10:03:48]    + encoder: H.264 (libx264)
[10:03:48]      + preset:  medium
[10:03:48]      + profile: main
[10:03:48]      + level:   4.0
[10:03:48]      + quality: 20.00 (RF)
[10:03:48]  * subtitle track 1, English (track 0, id 0xff) Text [SRT] -> Passthrough, Default, offset: 0, charset: UTF-8
[10:03:48]  * subtitle track 2, Italian (track 0, id 0x1ff) Text [SRT] -> Passthrough, offset: 0, charset: UTF-8
[10:03:48]  * subtitle track 3, Spanish (track 0, id 0x2ff) Text [SRT] -> Passthrough, offset: 0, charset: UTF-8
[10:03:48]  * audio track 1
[10:03:48]    + decoder: English (AC3) (5.1 ch) (track 1, id 0x1)
[10:03:48]      + bitrate: 640 kbps, samplerate: 48000 Hz
[10:03:48]    + mixdown: 5.1 Channels
[10:03:48]    + encoder: AC3 (libavcodec)
[10:03:48]      + bitrate: 448 kbps, samplerate: 48000 Hz
[10:03:48] sync: expecting 153096 video frames
[10:03:48] encx264: min-keyint: 24, keyint: 240
[10:03:48] encx264: encoding at constant RF 20.000000
[10:03:48] encx264: unparsed options: level=4.0:8x8dct=0:vbv-maxrate=20000:vbv-bufsize=25000
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
x264 [info]: profile Main, level 4.0
[10:03:49] Invalid byte for codeset in input, discard byte
[10:03:49] sync: first pts audio 0x1 is 0
[10:03:49] sync: first pts video is 450
[10:03:49] sync: "00:00:00.000" (1) at frame 1 time 450
[10:04:18] sync: first pts subtitle 0xff is 5043690
[10:04:18] sync: first pts subtitle 0x1ff is 5043690
[10:04:18] sync: first pts subtitle 0x2ff is 5043690
[10:20:33] sync: "00:13:28.333" (2) at frame 19399 time 72742950
[10:38:16] sync: "00:25:55.917" (3) at frame 37342 time 140029200
[10:56:33] sync: "00:39:05.542" (4) at frame 56293 time 211095450
[11:15:25] sync: "00:52:51.792" (5) at frame 76123 time 285457950
[11:34:18] sync: "01:06:24.833" (6) at frame 95636 time 358631700
[11:52:17] sync: "01:19:24.917" (7) at frame 114358 time 428839200
[12:11:08] sync: "01:32:34.167" (8) at frame 133301 time 499875450
[12:23:56] sync: reached audio 0x1 pts 574021440, exiting early
[12:23:57] reader: done. 1 scr changes
[12:23:59] work: average encoding speed for job is 18.201473 fps
[12:23:59] comb detect: heavy 182 | light 2454 | uncombed 150464 | total 153100
[12:23:59] decomb: deinterlaced 182 | blended 2454 | unfiltered 150464 | total 153100
[12:23:59] vfr: 153100 frames output, 0 dropped and 0 duped for CFR/PFR
[12:23:59] vfr: lost time: 0 (0 frames)
[12:23:59] vfr: gained time: 0 (0 frames) (0 not accounted for)
[12:23:59] ac3-decoder done: 201236 frames, 0 decoder errors
[12:23:59] h264-decoder done: 153100 frames, 0 decoder errors
[12:23:59] sync: got 153100 frames, 153096 expected
[12:23:59] sync: framerate min 24.000 fps, max 24.000 fps, avg 24.000 fps
x264 [info]: frame I:2370  Avg QP:15.76  size: 98156
x264 [info]: frame P:91924 Avg QP:19.24  size: 27932
x264 [info]: frame B:58806 Avg QP:21.06  size:  9475
x264 [info]: consecutive B-frames: 38.4% 28.3%  8.4% 24.9%
x264 [info]: mb I  I16..4: 42.1%  0.0% 57.9%
x264 [info]: mb P  I16..4: 12.2%  0.0%  4.1%  P16..4: 43.8% 13.1%  7.1%  0.0%  0.0%    skip:19.6%
x264 [info]: mb B  I16..4:  0.8%  0.0%  0.3%  B16..8: 44.1%  3.6%  0.6%  direct: 4.3%  skip:46.2%  L0:39.0% L1:56.1% BI: 5.0%
x264 [info]: coded y,uvDC,uvAC intra: 44.3% 63.7% 30.6% inter: 17.1% 31.5% 0.9%
x264 [info]: i16 v,h,dc,p: 38% 22% 25% 15%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 20% 23%  6%  7%  6%  6%  5%  4%
x264 [info]: i8c dc,h,v,p: 62% 17% 19%  3%
x264 [info]: Weighted P-Frames: Y:4.7% UV:2.2%
x264 [info]: ref P L0: 67.8% 11.7% 13.6%  6.6%  0.3%
x264 [info]: ref B L0: 89.3%  9.8%  0.9%
x264 [info]: ref B L1: 97.0%  3.0%
x264 [info]: kb/s:4210.51
[12:23:59] mux: track 0, 153100 frames, 3357371379 bytes, 4210.39 kbps, fifo 4096
[12:23:59] mux: track 1, 199312 frames, 357167104 bytes, 447.91 kbps, fifo 8192
[12:23:59] mux: track 2, 483 frames, 25538 bytes, 0.03 kbps, fifo 16
[12:23:59] mux: track 3, 494 frames, 26469 bytes, 0.03 kbps, fifo 16
[12:23:59] mux: track 4, 490 frames, 25981 bytes, 0.03 kbps, fifo 16
[12:23:59] libhb: work result = 0
[code][/i]

Woodstock
Veteran User
Posts: 2540
Joined: Tue Aug 27, 2013 6:39 am

Re: Subtitle characters with diacritics are stripped

Post by Woodstock » Fri Feb 16, 2018 12:16 am

Are you sure the file is encoded with UTF-8?

John_Jason_Jordan
Posts: 9
Joined: Tue Feb 07, 2017 11:06 pm

Re: Subtitle characters with diacritics are stripped

Post by John_Jason_Jordan » Fri Feb 16, 2018 3:44 am

Aha! That was it!

The Spanish subtitles were encoded with Western 8859-15. Using Subtitle Editor I did a Save As where I could change it to UTF-8. I also changed the Newline to Unix.* Then I restarted the encoding using the new .srt file. After 15 minutes I opened the new .mkv file in VLC and selected the Spanish subtitles, but they did not appear at all. So I stopped the encoding, went back to Subtitle Editor and re-saved again, this time with Newline as Windows. Then I started the encoding over again using the new .srt file, and when I checked on it 15 minutes later the Spanish subtitles were appearing properly.

*Regarding Unix newline: All my computers run on Linux, and I often extract subtitles from DVDs and Blu-rays, usually using mkvmerge -i to find the subtitle track, then 'mkvextract tracks' to extract it as sub/idx or .sup, which I then convert to .srt. Every time I do this the Newline is always Unix, and the subtitles work perfectly in HandBrake and, after the encoding is finished, in VLC. I don't know why this time HandBrake couldn't use them.

Oh well. Problem solved and I learned a new trick. In the future I will make a point of checking that the subtitle file is really UTF-8. Thanks heaps for your suggestion!

Post Reply