HandBrakeCLI Queue

Discussion of the HandBrake command line interface (CLI)
Forum rules
An Activity Log is required for support requests. Please read How-to get an activity log? for details on how and why this should be provided.
Post Reply
gorillaninja
Posts: 7
Joined: Tue Apr 05, 2016 6:20 pm

HandBrakeCLI Queue

Post by gorillaninja »

Hello, all.

I felt the need for HandBrakeCLI to have a queue, so I wrote one up in Python. I think it provides a number of useful features:
  • Scan multiple directories for input files.
  • Fully-configurable HandBrakeCLI command-line.
  • Can dynamically modify HandBrakeCLI command-line based on the path or name of the input file.
...among other configuration options.

You can read the full documentation and download it from GitHub: https://github.com/gorillaninja/process-queue

Thank you for your time, and I would love feedback and suggestions.

Doug
Woodstock
Veteran User
Posts: 4614
Joined: Tue Aug 27, 2013 6:39 am

Re: HandBrakeCLI Queue

Post by Woodstock »

You might also want to look at some of the other scripts posted here, to see if there are any other features you want to add.
gorillaninja
Posts: 7
Joined: Tue Apr 05, 2016 6:20 pm

Re: HandBrakeCLI Queue

Post by gorillaninja »

Woodstock wrote:You might also want to look at some of the other scripts posted here, to see if there are any other features you want to add.
Thanks for the reply, Woodstock! That's a great idea; I have been combing through this forum to see what others are up to, and have started compiling a list of ideas for me to think about adding.

You're probably a CLI user, if you're reading this forum; what features would you like to see in a queue system written for HandBrakeCLI?

Thanks!

Doug
Woodstock
Veteran User
Posts: 4614
Joined: Tue Aug 27, 2013 6:39 am

Re: HandBrakeCLI Queue

Post by Woodstock »

I'm actually the worst person to ask about this - While my job involves automating tasks like this, I've found that video has far too many unknowns to rely on scripts. Scripts are great when there is consistency, or you have some way of building a list of characteristics, and reliable rules for making decisions based on those characteristics. But sometimes the same publisher can change their authoring style between titles released in the same month.

But I mainly work with anime titles, and the variations in audio and subtitle tracks, and matching them to the output device, are a major pain. I have examples where the highest quality audio is NOT the track you want to use while watching the show... the main audio is "just" stereo, but there is a 5.1 surround sound commentary. Sometimes subtitle track 1 is the one that matches up with the dubbed audio, sometimes it's track 2, 3, 4, or 5. Sometimes there is no subtitle track for the dubbed audio, just one for use with Japanese audio.

So my "automation" is a text editor with column edit mode, so I can construct the command line once, and copy/paste/edit it for all the episodes.
gorillaninja
Posts: 7
Joined: Tue Apr 05, 2016 6:20 pm

Re: HandBrakeCLI Queue

Post by gorillaninja »

Haha, I haven't transcoded any anime myself, but I have heard many horror stories just like you describe. It sounds like crazy chaos.

My queue processor allows for some decision-making, but nothing that would handle that pain. However, I'll think about what you're doing with anime as a use case. Maybe if there was a way for you to easily construct a HandBrakeCLI command-line manually and then tell the queue "apply this config to everything in this directory" it could at least cut down on the "copy/paste/edit" part of your workflow.

Thanks!

Doug
nhyone
Bright Spark User
Posts: 252
Joined: Fri Jul 24, 2015 4:13 am

Re: HandBrakeCLI Queue

Post by nhyone »

I'll go with a "inheritance"-based config file. If an episode-specific config file exists, use it. Else use the directory-level config file. Else use the parent directory's config file and so on, and it ends with the default config file. By doing this, it is possible to provide sensible defaults for most videos, but is still able to handle problematic ones.

I'll use a simpler higher-level text-based config file too. You can support profiles. For example, is the show a TV series, a movie, MTV, a lecture, and so on. The initial settings for a file will be based on this and the output resolution.

You can also add support for pre- and post-processing scripts for the worst-case scenario. Although they can be used for something as simple as setting/changing the track language names. :)

For TV series, you can add a text file that contains the episode names, so that you can name the output files meaningfully.

For some DVDs and DVD boxsets, I use a config file that looks like this:

Code: Select all

# ep | vol | disc | title | chaps | name
1|1|1|1||Send In The Clones
2|1|1|3||Sphinx for the Memories
3|1|1|4||Where No Duck Has Gone Before
...
10|1|2|1||Duckman of Aquatraz
...
28|2|1|1||Treasure of the Golden Suns Part 1
...
If you have multiple PCs, you can also think about how to allow concurrent encoding using just one queue.
gorillaninja
Posts: 7
Joined: Tue Apr 05, 2016 6:20 pm

Re: HandBrakeCLI Queue

Post by gorillaninja »

Thanks for the reply, nhyone! This is good feedback.
nhyone wrote:I'll go with a "inheritance"-based config file. If an episode-specific config file exists, use it. Else use the directory-level config file. Else use the parent directory's config file and so on, and it ends with the default config file. By doing this, it is possible to provide sensible defaults for most videos, but is still able to handle problematic ones.
I had thought about inheritance in config files, but I'll admit only in the sense that there might be a 'global' config and a machine-specific config that might overwrite some settings, like the paths to the executable or input files. Your idea is much more scalable and, frankly, easier to implement. I'll definitely put some thought into the details and you'll probably see it in my code relatively soon.
nhyone wrote:I'll use a simpler higher-level text-based config file too. You can support profiles. For example, is the show a TV series, a movie, MTV, a lecture, and so on. The initial settings for a file will be based on this and the output resolution.
I actually started with ConfigParser, but so many of the config sections needed to be lists. I was endlessly converting config strings into lists by splitting on commas and such, but then were were commas in some of the text itself, and it got complicated fast. Switching to JSON provides native support for both dictionaries and lists, nested arbitrarily, and shipped as an internal Python module. It's a clear win for this particular use-case.
nhyone wrote:You can also add support for pre- and post-processing scripts for the worst-case scenario. Although they can be used for something as simple as setting/changing the track language names. :)
Good idea!
nhyone wrote:For TV series, you can add a text file that contains the episode names, so that you can name the output files meaningfully.
Interesting; I'd never thought of this. I personally use Plex as my front-end, which only needs the season/episode numbers to look up the titles on its own. Having a text-based lookup to add episodes names if found wouldn't be too hard, but I wonder if I could go the extra mile and actually look up the episode titles if they're not already in the filename or found in a lookup file. I'll check into what APIs provide that service.
nhyone wrote:If you have multiple PCs, you can also think about how to allow concurrent encoding using just one queue.
This one, my queue already does! You can basically mount a filesystem on another machine and run my queue script, and it'll take reasonable precautions to not encode the same file twice. You can also provide machine-specific config files, mentioned above, although without any inheritance yet. I've been using multiple machine on my single queue for weeks now without a hiccup.

Thanks for your feedback! I really appreciate it.

Doug
nhyone
Bright Spark User
Posts: 252
Joined: Fri Jul 24, 2015 4:13 am

Re: HandBrakeCLI Queue

Post by nhyone »

gorillaninja wrote: I was endlessly converting config strings into lists by splitting on commas and such, but then were were commas in some of the text itself, and it got complicated fast.
You can use a very simple format such as one option per line. A simple config file could look something like this:

Code: Select all

-show_type tv
-aspect_ratio 1.85
-audio_track 2
-subtitle 1
-denoise none:light
Use a frontend processor to parse them first instead of passing them directly to HB. :)

gorillaninja wrote: This one, my queue already does! You can basically mount a filesystem on another machine and run my queue script, and it'll take reasonable precautions to not encode the same file twice. You can also provide machine-specific config files, mentioned above, although without any inheritance yet. I've been using multiple machine on my single queue for weeks now without a hiccup.
From what I see, your script will stop running when it runs out of videos to encode, so you got to restart it when you add new videos?

I'm thinking of a setup where the workers are always on standby, and will start encoding automatically when new videos are added to the queue. It might work most of the time, but even if the theory is sound, you need to test it with some "worst-case" scenario to make sure there are no bugs.

Once, I accidentally added an invalid series of 15 episodes to my queue. 6 workers pounding on the queue revealed some unexpected race conditions as they fetched the non-existent work and updated the status within a fraction of a second.
wearesolipsists
Posts: 4
Joined: Mon Jun 20, 2016 5:26 am

Re: HandBrakeCLI Queue

Post by wearesolipsists »

Hi Doug - your script looks like exactly what I'm after. Thanks for your hard work. A few questions:

1) Is there a way to change where the "temp file" is stored as the video is being encoded? Handbrake GUI behaviour as far as I have noticed is to place the temp file in the output directory while it is being filled with data. I prefer your method of moving the video file upon completion to the output directory. However it would be good to have this temp file location off my primary SSD. Is it as simple as storing the script on a different drive?

2) Is there an easy way for your script to bring .srt files across with it to the output directory?

Thanks again.
gorillaninja
Posts: 7
Joined: Tue Apr 05, 2016 6:20 pm

Re: HandBrakeCLI Queue

Post by gorillaninja »

Hey, glad to hear it may be useful for you! I haven't gotten around to any of the enhancements mentioned higher in the thread, but they're still rattling around in my head.

1) There is no way now to set the temp directory used; it calls tempfile.gettempdir() and uses whatever is returned, according to the rules on that page. However, it would be trivially easy to add an optional parameter to the config that is used to set tempfile.tempdir up front, which would then be used throughout the script.

This is a 5min change that I should be able to get done tonight. Look for an update to GitHub this evening or tomorrow.

2) Let me ponder this one. It may be easy to grab a list of all files with the same name as the video file and move them to the same output directory, but is that what you'd expect? For example, if the video file is movie.mkv, will there be one file named movie.srt that you'd want moved next to the transcoded file? Or could there be multiple .srt files? How might they be named?

Thanks for your feedback and for finding this tool useful. I know I do. :)

Doug
wearesolipsists
Posts: 4
Joined: Mon Jun 20, 2016 5:26 am

Re: HandBrakeCLI Queue

Post by wearesolipsists »

Hey Doug. Thanks very much for that tempfile update! I'll keep a lookout.

Regarding point 2) - you hit the nail on the head with your example. For my usage there is sometimes a .srt file with the same basename as the movie file = basename.srt. Occasionally It will be basename.eng.srt and very very occasionally basename.forced.eng.srt. There may also be other types like .smi files or even .txt files.

I don't think there is an agreed standard for naming of .srt files or other sidecar files, so it might be tricky to implement. However from my experience the .srt file (or other sidecar files) will never have a basename that is shorter than the movie file; only ever the movie basename with some suffixes or no suffixes. So it might work to look for any other files that have the same basename, or have a basename that starts with the movie basename.

I'm running your script on the output directory of my filebot/AMC setup, so my input files are already very "well named". However if the input was more basic like "movie.mkv", then picking up any files that start with "movie" in the same folder might be problematic. Could potentially be solved by having the functionality as a switch, so if you have well named input you can bring other files with you, but you can also choose not to.

Thanks again.

I have another question - if I can't bring .srt files with me, an alternative would be for the output to replace the input file with the exact same name (ie. output overwrites the input). Is that something that will work as of now? Or will the deletion phase end up deleting the output file, thinking it was the old input?
wearesolipsists
Posts: 4
Joined: Mon Jun 20, 2016 5:26 am

Re: HandBrakeCLI Queue

Post by wearesolipsists »

Also, taking a look at your code, it looks like input filenames will be checked for the format "Show - sXXeXX..." and then output to /output-dir/Show/Season XX/file.mkv or something similar, if both of your string searches "show_re" or "season_re" are successful. This is perfect for me and saves another step - is it "enabled" by default so long as the input file is named accordingly? My python isn't great - if the file is "Show - s01eXX..." will this output to a "Season 01" folder, or a "Season 1" folder? It doesn't look like the "(\d+)" grouping is cleaned of any leading zeros, but as I said I'm not great with Python. Not a huge deal but I'll have to tweak existing folders if it outputs as "01".

So happy I found your script - I was tearing my hair out trying to automate re-encoding with handbrake before adding to Plex. Thanks again.
gorillaninja
Posts: 7
Joined: Tue Apr 05, 2016 6:20 pm

Re: HandBrakeCLI Queue

Post by gorillaninja »

I was able to write and post the temp_dir addition to the config last night, so you should be able to download the new code and start using that right away.

I hear what you're saying about sidecar files not having a good naming convention, but I think your suggestion is sound. If I do a basename glob that doesn't include a period, I should in most cases pick up only sidecar files. For example, if the input file was "Zoolander 2.mkv" and I do a glob on "Zoolander 2*", I would pick up:
  • Zoolander 2.srt
  • Zoolander 2-french.srt
  • Zoolander 2-poster.jpg
This is totally doable; I'll have a look as soon as I can.

Show/Season subdirectories are enabled by default if the video name contains " - sNNeNN". It will output "Season 01" if it finds "01" in the video name; this is intentional, so shows with more than nine seasons sort properly in the directory list. If your title has " - s1e3", though, it'll just use "Season 1"; no processing is done on the digits between the 's' and 'e', and it'll pick up any number of digits.

Doug
gorillaninja
Posts: 7
Joined: Tue Apr 05, 2016 6:20 pm

Re: HandBrakeCLI Queue

Post by gorillaninja »

"Sidecar" update has been pushed to GitHub. Let me know what you think and if these recent changes are working for you.

Doug
wearesolipsists
Posts: 4
Joined: Mon Jun 20, 2016 5:26 am

Re: HandBrakeCLI Queue

Post by wearesolipsists »

Awesome, thanks for that. I should get a chance to look at it tonight. Thanks!
LaptopFruit
Posts: 1
Joined: Tue Oct 18, 2016 8:40 am

Re: HandBrakeCLI Queue

Post by LaptopFruit »

I'm having trouble running HandBrakeCLI Queue - it doesn't want to read the config file. What am I doing wrong?

Log file

Code: Select all

[10/18/16 08:31:48] Could not read any config file.
[10/18/16 08:31:48]  Machine-specific config: /home/laptopfruit/process-queue-master/pve-121-handbrake-01/queue.json
[10/18/16 08:31:48]  Default config: /home/laptopfruit/process-queue-master/queue.json
Terminal

Code: Select all

laptopfruit@pve-121-handbrake-01 ~/process-queue-master$ ls -l
total 52
-rw-r--r-- 1 laptopfruit users 13807 Jun 21 17:28 README.md
-rwxr-xr-x 1 laptopfruit users   729 Jun 21 17:28 manage_logs.sh
drwxr-xr-x 2 laptopfruit users  4096 Oct 16 08:59 process_queue
drwxr-xr-x 2 laptopfruit users  4096 Oct 18 08:31 pve-121-handbrake-01
-rw-r--r-- 1 laptopfruit users   724 Oct 16 20:30 queue.json
-rw-r--r-- 1 laptopfruit users   271 Jun 21 17:28 queue.json.minimal
-rw-r--r-- 1 laptopfruit users  2597 Jun 21 17:28 queue.json.sample
-rwxr-xr-x 1 laptopfruit users   135 Jun 21 17:28 queue.py
laptopfruit@pve-121-handbrake-01 ~/process-queue-master$ ./queue.py --verbose
laptopfruit@pve-121-handbrake-01 ~/process-queue-master$ 
Post Reply