[Bash]Unable to convert multiple files within a folder.

Discussion of the HandBrake command line interface (CLI)
Forum rules
An Activity Log is required for support requests. Please read How-to get an activity log? for details on how and why this should be provided.
Post Reply
daphatty
Posts: 6
Joined: Sat Apr 02, 2016 4:59 pm

[Bash]Unable to convert multiple files within a folder.

Post by daphatty »

Description of problem or question:

Calling Handbrake from within a Bash script, I am unable to process more than one file from a given folder. Using a While loop, Handbrake simply exits after processing the first file.

To be clear, the encoding process works as expected for the first file. The script simply terminates after that point. The only errors that appear in the Handbrake output are the typical errors associated with scanning for and not finding Bluray or DVD content.

My preference was to leverage HandBrake's Queue functionality but it seems said functionality is either missing from HandBrakeCLI or is present in a separate binary of which I am unaware.

HandBrake version (e.g., 1.0.0):

HandBrake 1.2.1

HandBrake Log Snippet

Code: Select all

username@video-proc:~$ ./convertHand.sh 
Converting /home/username/Converted/Family Guy-Quagmires Dad.mpg to .MP4...
Cannot load libnvidia-encode.so.1
Cannot load libnvidia-encode.so.1
Cannot load libnvidia-encode.so.1
Cannot load libnvidia-encode.so.1
[05:22:13] hb_init: starting libhb thread
[05:22:13] thread 7f40736b1700 started ("libhb")
HandBrake 1.2.1 (2019021900) - Linux x86_64 - https://handbrake.fr
4 CPUs detected
Opening /home/username/Converted...
[05:22:13] CPU: Intel(R) Core(TM) i3-6100U CPU @ 2.30GHz
[05:22:13]  - Intel microarchitecture Skylake
[05:22:13]  - logical processor count: 4
........

x264 [info]: frame I:263   Avg QP:14.70  size: 72240
x264 [info]: frame P:11377 Avg QP:17.63  size: 12984
x264 [info]: frame B:32201 Avg QP:23.46  size:  1758
x264 [info]: consecutive B-frames:  1.2%  1.7%  2.3% 94.8%
x264 [info]: mb I  I16..4:  5.8% 57.1% 37.2%
x264 [info]: mb P  I16..4:  0.7%  3.2%  1.3%  P16..4: 39.7% 16.9% 13.8%  0.0%  0.0%    skip:24.3%
x264 [info]: mb B  I16..4:  0.0%  0.2%  0.1%  B16..8: 26.2%  4.3%  1.4%  direct: 1.7%  skip:66.0%  L0:37.9% L1:53.0% BI: 9.1%
x264 [info]: 8x8 transform intra:59.4% inter:43.5%
x264 [info]: direct mvs  spatial:100.0% temporal:0.0%
x264 [info]: coded y,uvDC,uvAC intra: 77.4% 87.0% 76.0% inter: 10.7% 15.6% 6.4%
x264 [info]: i16 v,h,dc,p: 53% 24%  8% 15%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 12% 13%  7% 10% 11%  9% 10% 11%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 12%  8%  8% 12% 12% 10% 10% 11%
x264 [info]: i8c dc,h,v,p: 41% 23% 23% 13%
x264 [info]: Weighted P-Frames: Y:1.6% UV:1.5%
x264 [info]: ref P L0: 63.8%  8.8% 16.6%  5.5%  5.3%  0.1%  0.0%
x264 [info]: ref B L0: 84.2% 11.2%  3.6%  1.0%
x264 [info]: ref B L1: 93.2%  6.8%
x264 [info]: kb/s:1220.65
[06:01:35] mux: track 0, 43841 frames, 223303107 bytes, 1220.63 kbps, fifo 4096
[06:01:35] mux: track 1, 68582 frames, 29261452 bytes, 159.95 kbps, fifo 4096
[06:01:35] mux: track 2, 45721 frames, 46817714 bytes, 255.92 kbps, fifo 4096
[06:01:35] libhb: work result = 0

Encode done!

HandBrake has exited.
Operating system and version

Ubuntu 18.04 LTS


Code Snippets
This while loop leverages a txt file containing a list of file paths and names for all files to be processed.

Contents of text file

Code: Select all

/home/username/Converted/Family Guy-Quagmires Dad.mpg
/home/username/Converted/Daniel Tigers Neighborhood-The Class Votes; The Neighborhood Votes.mpg
/home/username/Converted/Sesame Street-How to Build a Dinosaur.mpg
/home/username/Converted/Daniel Tigers Neighborhood-Daniels Allergy; Allergies at School.mpg
/home/username/Converted/Last Man Standing-The Wolf Returns.mpg

While Loop

Code: Select all

homeDir="/home/username"
finalDir="/home/username/Complete"
comcutOutputFolder="/home/username/Converted"

touch handBrakeFiles.txt

find "$comcutOutputFolder" -ctime -1 -print | grep .mpg > handBrakeFiles.txt

while read line; do
echo "Converting $line to .MP4..."
handName=$(basename "$line" .mpg)
HandBrakeCLI --preset "HQ 480p30 Surround" -i "$line" -o "$finalDir"/"$handName".mp4

done < handBrakeFiles.txt

musicvid
Veteran User
Posts: 3610
Joined: Sat Jun 27, 2009 1:19 am

Re: [Bash]Unable to convert multiple files within a folder.

Post by musicvid »

Wouldn't you need to tell Handbrake to close every time?

daphatty
Posts: 6
Joined: Sat Apr 02, 2016 4:59 pm

Re: [Bash]Unable to convert multiple files within a folder.

Post by daphatty »

musicvid wrote:
Wed Jul 15, 2020 8:25 am
Wouldn't you need to tell Handbrake to close every time?
Not sure I understand what you mean. Handbrake is closing on its own and breaking the while loop in the process.

mduell
Veteran User
Posts: 7151
Joined: Sat Apr 21, 2007 8:54 pm

Re: [Bash]Unable to convert multiple files within a folder.

Post by mduell »

Your wrapper may not like HB's exit code or that HB outputs to stderr in normal operation. Try redirecting stderr /dev/null or similar.

mbutler
Posts: 2
Joined: Sun Feb 09, 2020 5:51 pm

Re: [Bash]Unable to convert multiple files within a folder.

Post by mbutler »

I presume you've tried the script without the HandBrakeCLI invocation to see if it is acting correctly?
You might also try redirecting stderr to a file to see if an error is occuring. Don't know, does bash exit a while loop if a command returns an error status?

Mark

daphatty
Posts: 6
Joined: Sat Apr 02, 2016 4:59 pm

Re: [Bash]Unable to convert multiple files within a folder.

Post by daphatty »

mbutler wrote:
Wed Jul 15, 2020 5:54 pm
I presume you've tried the script without the HandBrakeCLI invocation to see if it is acting correctly?
You might also try redirecting stderr to a file to see if an error is occuring. Don't know, does bash exit a while loop if a command returns an error status?

Mark
Correct, I am testing this Handbrake workflow separately from the main script to ensure other aspects won't interfere. Once the conversion script works independently, it will be added to the all-in-one script and tested again.

Based on the above suggestions, I modified my Handbrake command to:

Code: Select all

/usr/bin/HandBrakeCLI --preset "HQ 480p30 Surround" -i "$line" -o "$finalDir"/"$handName".mp4 > /dev/null 2>&1
Unfortunately, my script still exited prematurely.

daphatty
Posts: 6
Joined: Sat Apr 02, 2016 4:59 pm

Re: [Bash]Unable to convert multiple files within a folder.

Post by daphatty »

Solved!

I managed to find the answer in this Stack Overflow post. Turns out, HandbrakeCLI somehow leverages the same stdin as my script which in turn causes the behavior I experienced. Adding

Code: Select all

echo "" |
in front of my HandBrakeCLI command prevents the loop from terminating prematurely. The new HandBrakeCLI command looks like this.

Code: Select all

echo "" | /usr/bin/HandBrakeCLI --preset "HQ 480p30 Surround" -i "$line" -o "$finalDir"/"$handName".mp4

User avatar
BradleyS
Moderator
Posts: 1859
Joined: Thu Aug 09, 2007 12:16 pm

Re: [Bash]Unable to convert multiple files within a folder.

Post by BradleyS »

I would think a subshell would be sufficient, plus you could process more than one job simultaneously with some additional modification.

mbutler
Posts: 2
Joined: Sun Feb 09, 2020 5:51 pm

Re: [Bash]Unable to convert multiple files within a folder.

Post by mbutler »

Wow, I didn't expect that! My scripts use ksh, so maybe I'm bypassing some of the bash behavior.

Mark

User avatar
BradleyS
Moderator
Posts: 1859
Joined: Thu Aug 09, 2007 12:16 pm

Re: [Bash]Unable to convert multiple files within a folder.

Post by BradleyS »

Where you do:

Code: Select all

handName=$(basename "$line" .mpg)
The $(...) is a subshell. Instead of running the enclosed command in the same shell instance as your script, it spawns another instance.

I suspect something like the following would work (provided you supply absolute paths):

Code: Select all

result=$(/usr/bin/HandBrakeCLI --preset "HQ 480p30 Surround" -i "$line" -o "$finalDir"/"$handName".mp4 >/dev/null 2>&1; echo $?)
This stores the exit code in $result, which you can use to check and print status while continuing to encode other jobs.

I suspect there's no concern about handBrakeFiles.txt being modified during the encode in your case; however, I would use mktemp or an in-memory array or both to be sure.

Code: Select all

listFile=$(mktemp)
find "$comcutOutputFolder" -ctime -1 -print | grep .mpg > $listFile
Last edited by BradleyS on Thu Jul 16, 2020 6:19 pm, edited 2 times in total.
Reason: Added redirection in subshell so only exit code is echoed.

Post Reply