RAM Runs Low When Bulk Processing

[pinapple99]

Description of problem or question:
When I bulk transcode several files (loading them in the Queue) I can see the available RAM decrease from about 25Gig’s down to 25Meg’s (depending on file size/video time).

I observe this using Cleansweep. When it gets down to about 5Gig’s, I pause HB and ‘Free’ up the memory (using Cleansweep). It works and no problems. But, that sort of defeats / limits Bulk Processing. I've seen it get down to as low as 25 Meg's of RAM

I haven’t bothered to see what happens if I don’t Pause it as I don’t want to invest 6+ hours to let it go through the process with 4 or 5 full length movie files...

Question(s):
1) Should I ignore this or continue to Pause and Free Memory?

2) Is there some feature or setting I’m missing?

3) This can’t be isolated to only my machine so, what are others doing about this and, should I make a Request for adding some management feature to deal with this?

Other useful guidance/comments appreciated


Steps to reproduce the problem (If Applicable):
Happens every time I transcode 3 to 5 or more long movie files


HandBrake version:
1.4.0

Operating system and version:
Mac OSX 10.12.6 Sierra, 5k Retina
32 GB Memory

HandBrake Activity Log
I looked at the Logs but, did not bother to post as there are too many and would not know which to choose.
I can (and will, if needed) delete all the Logs and Transcode some files in Bulk to Reproduce the concern then post it... if requested but, that's at least 5 hours that may not be useful...

[rollin_eng]

Could you please post your HB logs, instructions can be found here:

https://handbrake.fr/docs/en/latest/hel ... y-log.html

[pinapple99]

O.K. pasted below (I grabbed one at random - difficult to know which file would be for a Bulk job as I transcode from 1 to 10 files...)

It would have been cleaner to post it in original Post, sorry...

Here it is:

HandBrake Activity Log for Session: 2021-08-18T02:53:36Z
Handbrake Version: 1.4.0 (2021071800)
Tomorrow_Is_ForeverX_1946.mp4
Preset: my_Roku_720stereo_RF20
[19:53:36] Compile-time hardening features are enabled
[19:53:36] hb_init: starting libhb thread
[19:53:36] macgui: fr.handbrake.HandBrakeXPCService scanning specifically for title: 1
[19:53:36] CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
[19:53:36] - Intel microarchitecture Skylake
[19:53:36] - logical processor count: 4
[19:53:36] hb_scan: path=/Users/bruce/Movies/myMovies/ASIS_Capture/ASIS_Tomorrow_Is_Forever_1946.mov, title_index=1
udfread ERROR: ECMA 167 Volume Recognition failed
disc.c:333: failed opening UDF image /Users/bruce/Movies/myMovies/ASIS_Capture/ASIS_Tomorrow_Is_Forever_1946.mov
disc.c:437: error opening file BDMV/index.bdmv
disc.c:437: error opening file BDMV/BACKUP/index.bdmv
bluray.c nav_get_title_list(/Users/bruce/Movies/myMovies/ASIS_Capture/ASIS_Tomorrow_Is_Forever_1946.mov/) failed
[19:53:36] bd: not a bd - trying as a stream/file instead
libdvdread: DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdnav: vm: vm: failed to read VIDEO_TS.IFO
[19:53:36] dvd: not a dvd - trying as a stream/file instead
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/bruce/Movies/myMovies/ASIS_Capture/ASIS_Tomorrow_Is_Forever_1946.mov':
Metadata:
major_brand : qt
minor_version : 0
compatible_brands: qt
creation_time : 2021-08-17T16:52:53.000000Z
com.apple.quicktime.make: Apple
com.apple.quicktime.model: iMac17,1
com.apple.quicktime.software: Mac OS X 10.12.6 (16G2136)
com.apple.quicktime.creationdate: 2021-08-17T08:08:42-0700
Duration: 01:43:20.66, start: 0.000000, bitrate: 24618 kb/s
Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709), 2168x1572 [SAR 1:1 DAR 542:393], 24291 kb/s, 59.99 fps, 60 tbr, 6k tbn, 50 tbc (default)
Metadata:
creation_time : 2021-08-17T16:52:53.000000Z
handler_name : Core Media Video
vendor_id : [0][0][0][0]
encoder : H.264
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 314 kb/s (default)
Metadata:
creation_time : 2021-08-17T16:52:53.000000Z
handler_name : Core Media Audio
vendor_id : [0][0][0][0]
[19:53:36] scan: decoding previews for title 1
[19:53:36] scan: audio 0x1: aac, rate=44100Hz, bitrate=314496 Unknown (AAC LC) (2.0 ch) (314 kbps)
[19:53:38] scan: 10 previews, 2168x1572, 60.000 fps, autocrop = 0/0/0/0, aspect 4:3, PAR 1:1, color profile: 1-1-1
[19:53:38] libhb: scan thread found 1 valid title(s)
[19:53:38] macgui: fr.handbrake.HandBrakeXPCService scan done
[19:53:38] Starting work at: Tue Aug 17 19:53:38 2021

[19:53:38] macgui: fr.handbrake.HandBrakeXPCService started encoding Tomorrow_Is_ForeverX_1946.mp4
[19:53:38] 1 job(s) to process
[19:53:38] macgui: fr.handbrake.HandBrakeXPCService with preset my_Roku_720stereo_RF20
[19:53:38] Starting Task: Encoding Pass
[19:53:38] work: only 1 chapter, disabling chapter markers
[19:53:38] job configuration:
[19:53:38] * source
[19:53:38] + /Users/bruce/Movies/myMovies/ASIS_Capture/ASIS_Tomorrow_Is_Forever_1946.mov
[19:53:38] + title 1, chapter(s) 1 to 1
[19:53:38] + container: mov,mp4,m4a,3gp,3g2,mj2
[19:53:38] + data rate: 24618 kbps
[19:53:38] * destination
[19:53:38] + /Users/bruce/Movies/myMovies/ASIS_Out/Tomorrow_Is_ForeverX_1946.mp4
[19:53:38] + container: MPEG-4 (libavformat)
[19:53:38] + align initial A/V stream timestamps
[19:53:38] * video track
[19:53:38] + decoder: h264 8-bit
[19:53:38] + bitrate 24291 kbps
[19:53:38] + filters
[19:53:38] + Comb Detect (mode=3:spatial-metric=2:motion-thresh=1:spatial-thresh=1:filter-mode=2:block-thresh=40:block-width=16:block-height=16)
[19:53:38] + Decomb (mode=39)
[19:53:38] + Framerate Shaper (mode=2:rate=27000000/900000)
[19:53:38] + frame rate: 60.000 fps -> peak rate limited to 30.000 fps
[19:53:38] + Crop and Scale (width=1280:height=720:crop-top=0:crop-bottom=0:crop-left=0:crop-right=0)
[19:53:38] + source: 2168 * 1572, crop (0/0/0/0): 2168 * 1572, scale: 1280 * 720
[19:53:38] + Output geometry
[19:53:38] + storage dimensions: 1280 x 720
[19:53:38] + pixel aspect ratio: 813 : 1048
[19:53:38] + display dimensions: 992 x 720
[19:53:38] + encoder: H.264 (libx264)
[19:53:38] + preset: medium
[19:53:38] + profile: main
[19:53:38] + level: 3.1
[19:53:38] + quality: 20.00 (RF)
[19:53:38] + color profile: 1-1-1
[19:53:38] * audio track 1
[19:53:38] + decoder: Unknown (AAC LC) (2.0 ch) (314 kbps) (track 1, id 0x1)
[19:53:38] + bitrate: 314 kbps, samplerate: 44100 Hz
[19:53:38] + mixdown: Stereo
[19:53:38] + gain: 8dB
[19:53:38] + encoder: AAC (Apple AudioToolbox)
[19:53:38] + bitrate: 160 kbps, samplerate: 44100 Hz
[19:53:38] * audio track 2
[19:53:38] + decoder: Unknown (AAC LC) (2.0 ch) (314 kbps) (track 1, id 0x1)
[19:53:38] + bitrate: 314 kbps, samplerate: 44100 Hz
[19:53:38] + AAC Passthru
[19:53:39] sync: expecting 372039 video frames
[19:53:39] encx264: min-keyint: 30, keyint: 300
[19:53:39] encx264: encoding at constant RF 20.000000
[19:53:39] encx264: unparsed options: level=3.1:8x8dct=0:vbv-bufsize=14000:vbv-maxrate=14000
x264 [info]: using SAR=813/1048
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x264 [info]: profile Main, level 3.1, 4:2:0, 8-bit
[19:53:39] sync: first pts video is 0
[19:53:39] sync: "Chapter 1" (1) at frame 1 time 0
[19:53:39] sync: first pts audio 0x1 is 7320
[19:53:39] sync: first pts audio 0x1 is 7320
[20:09:25] sync: audio 0x1 time went backwards 46 ms, dropped 2 frames. PTS 82988969
[20:09:25] sync: audio 0x1 time went backwards 46 ms, dropped 2 frames. PTS 82988969
[20:36:59] sync: video time went backwards 16 ms, dropped 1 frames. PTS 223156110
[21:41:02] reader: done. 1 scr changes
[21:41:05] work: average encoding speed for job is 57.723053 fps
[21:41:05] comb detect: heavy 16 | light 240 | uncombed 371752 | total 372008
[21:41:05] decomb: deinterlaced 16 | blended 240 | unfiltered 371752 | total 372008
[21:41:05] vfr: 186019 frames output, 185989 dropped and 0 duped for CFR/PFR
[21:41:05] vfr: lost time: 0 (0 frames)
[21:41:05] vfr: gained time: 0 (0 frames) (0 not accounted for)
[21:41:05] aac-decoder done: 267039 frames, 0 decoder errors
[21:41:05] aac-decoder done: 267039 frames, 0 decoder errors
[21:41:05] h264-decoder done: 372009 frames, 0 decoder errors
[21:41:05] sync: got 372008 frames, 372039 expected
[21:41:05] sync: framerate min 30.000 fps, max 60.001 fps, avg 59.995 fps
x264 [info]: frame I:914 Avg QP:15.24 size: 57427
x264 [info]: frame P:51586 Avg QP:18.72 size: 13433
x264 [info]: frame B:133519 Avg QP:23.93 size: 2234
x264 [info]: consecutive B-frames: 0.9% 8.2% 6.3% 84.7%
x264 [info]: mb I I16..4: 20.1% 0.0% 79.9%
x264 [info]: mb P I16..4: 4.0% 0.0% 4.1% P16..4: 37.9% 18.0% 6.9% 0.0% 0.0% skip:29.1%
x264 [info]: mb B I16..4: 0.2% 0.0% 0.2% B16..8: 29.0% 2.2% 0.3% direct: 0.8% skip:67.2% L0:46.5% L1:49.1% BI: 4.4%
x264 [info]: coded y,uvDC,uvAC intra: 49.7% 0.0% 0.0% inter: 7.6% 0.0% 0.0%
x264 [info]: i16 v,h,dc,p: 49% 28% 8% 15%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 27% 24% 13% 5% 7% 6% 8% 5% 5%
x264 [info]: i8c dc,h,v,p: 100% 0% 0% 0%
x264 [info]: Weighted P-Frames: Y:3.3% UV:0.0%
x264 [info]: ref P L0: 60.9% 14.7% 19.7% 4.6% 0.1%
x264 [info]: ref B L0: 88.4% 9.8% 1.8%
x264 [info]: ref B L1: 97.5% 2.5%
x264 [info]: kb/s:1346.67
[21:41:05] mux: track 0, 186019 frames, 1043744315 bytes, 1346.61 kbps, fifo 4096
[21:41:05] mux: track 1, 267040 frames, 126888206 bytes, 163.71 kbps, fifo 4096
[21:41:05] mux: track 2, 267037 frames, 243759553 bytes, 314.49 kbps, fifo 4096
[21:41:05] Finished work at: Tue Aug 17 21:41:05 2021

[21:41:05] libhb: work result = 0
[21:41:05] macgui: fr.handbrake.HandBrakeXPCService work done

[s55]

Be careful not to confuse Active vs Inactive vs Free memory.

Likely, Free will be very low, Inactive will be fairly high and Active won't be that high for the HB process. (Probably less than 1GB for a file like the above)

There is no need to manually free inactive memory. macOS handles this.

[pinapple99]

Thank you - I hadn't bothered to look at the Mac's activity panel to see the various Memory useage.

I will do as advised:

There is no need to manually free inactive memory. macOS handles this

[mduell]

That's your OS memory management caching parts of files it has seen used recently, frequently, or expects to be used soon (e.g. based on sequential reads). Unused (free) memory is literally a waste of money.