MacGui mem "leak" w large queues, persists, #1148

[tc60045]

Not sure how best to report this bug, so please respond: (a) if this should be reported as a bug, and (b) where you'd like me to document it properly.

Situation: MacGui, nightly #1148, iMac i7 w/16GB, transcoding .mkv into ipod-friendly (1/50th size) versions, queue of 2 - 8 .mkvs to transcode

Problem: "Free" memory will dwindle to zero as "Inactive" grows to fill all available ram. When "Inactive" (blue) reaches saturation, transcoding slows down to a crawl

Example: n minutes into a transcode:
minute Free Inactive
0 7.31 1.31
1 5.56 3.16
2 5.11 3.76
3 4.58 4.20
4 3.98 4.73
5 3.23 5.45
(you can see where this is headed).

Workaround: Pause HB. switch to terminal and type: purge. Watch inactive turn into free and become zero 0%, whilst free is ~50%. Resume HB. Await next clogging.

Note that I'm using a build that came AFTER the build that "re-instated some clean-up code" -- presuming that clean-up code persisted into this latest release.

Respectfully submitted,

TC

[tc60045]

Update: at minute 16, free memory was 10MB, Inactive was 10.1 GB, and frames / second were down about 40% from average. Hit pause, purge, and will resume again.

Yes, I should be able to just rely on paging in / out to get through this, but I never trust that everything will stay upright and locked when so clearly on the warning track.

[s55]

It's not a bug. This is just how Mac OS X manages memory these days. It prefers paging to disk rather than using the inactive memory.
It's not actually leaking.

There are actually several programs now available to try and "massage" this problem so it's less of a problem. (e.g. https://itunes.apple.com/nz/app/freemem ... 1672?mt=12)

[s55]

Also, this is something that folks intermittently reporting for the last few years. It's not something due to recent code changes. It started happening to people after an OSX upgrade.

[Smithcraft]

Out of curiosity, which version of OS X?

SC

[s55]

Everything I've read, suggests this behavior was introduced in 10.5

[mduell]

This isn't a HB bug. This is an OS decision you should report to your OS vendor.

[Smithcraft]

I don't think I ever had that problem back when I was transcoding on my old HTmini running 10.5.

SC

[tc60045]

Thanks all for the replies.

@all: HandBrakeCLI does not have this problem -- at all. In fact, while running a big encode right now, free memory increased, which is what you'd expect if there is a GC process re-labeling previously-used/now-inactive ram as Free.

The fact that HandbrakeCLI does NOT have the problem and MacGui DOES seems to indicate that there is something different going on between CLI and MacGui in how memory is handled. Yes, this is something that the OS should handle properly, and it does -- via paging. That slows down transcoding a bit, depending on the drive to which you are paging.

@smithcraft: Mountain Lion

@S5: You are right -- HB MacGui has had this problem, on occasion, under every MacOS since I can recall (using it for 4+ years). Seems to come and go, I suspect based on changes in the code. This is the worst I've seen in a long time, but it doesn't break anything, so I hesitate to call it a bug.

@mduell: Please don't bother replying on my thread unless you can add to the discussion -- thanks.

[TedJ]

So you've noticed an issue related to memory consumption when running large queues but it doesn't manifest when using HandBrakeCLI, which doesn't have queue support?

[tc60045]

Ted, the memory is chewed up so quickly by MacGUI that the "queuing" portion of my description is irrelevant: all free ram is consumed, and, by virtue of memory handling, is labeled inactive inside of 20 minutes on the first transcode.

Didn't see that behavior from CLI earlier today, but seeing something now with memory handling on a first pass of CLI on an MKV looking for foreign language subtitles... Have to dig into logs to see what is happening.

[Ritsuka]

By the way Apple recommends to disable the ram cache if the file won't be read again from the disk. OS X still loves to cache the world in ram, and swap out useful things, so disabling file system caching is still a good idea.
http://developer.apple.com/library/ios/ ... eTips.html

[s55]

It was discussed last year. Much of the file handling, with the exception of (TS/PS stuff that we have our own demuxer for) is handled by the libraries we use, so there are numerous places that would need patched.


I don't recall if anyone actually tested this. Would be much preferable for Apple to just fix the dodgy memory management. They seem hell bent on making developers do the job of the operating system in recent times.

[s55]

Based on that article, the one thing that stands out is they zero-write over any spillage onto disk, which is where the major performance hit will occur.

[tc60045]

@s5 -- thanks. Looks like better memory management would require a lot of HB being re-written in Objective C, given my cursory glance at the code. I suspect that is not in the cards for HB. Nevertheless, good link here on how many of the advanced memory management "features" of OS X are now auto-addded at compile: http://developer.apple.com/library/mac/ ... yMgmt.html

Excerpt:

Objective-C provides two methods of application memory management.

In the method described in this guide, referred to as “manual retain-release” or MRR, you explicitly manage memory by keeping track of objects you own. This is implemented using a model, known as reference counting, that the Foundation class NSObject provides in conjunction with the runtime environment.

In Automatic Reference Counting, or ARC, the system uses the same reference counting system as MRR, but it inserts the appropriate memory management method calls for you at compile-time. You are strongly encouraged to use ARC for new projects. If you use ARC, there is typically no need to understand the underlying implementation described in this document, although it may in some situations be helpful. For more about ARC, see Transitioning to ARC Release Notes.

If you plan on writing code for iOS, you must use explicit memory management (the subject of this guide). Further, if you plan on writing library routines, plug-ins, or shared code—code that might be loaded into either a garbage-collection or non-garbage-collection process—you want to write your code using the memory-management techniques described throughout this guide. (Make sure that you then test your code in Xcode, with garbage collection disabled and enabled.)

[Deleted User 11865]

@s5 -- thanks. Looks like better memory management would require a lot of HB being re-written in Objective C, given my cursory glance at the code.

Not necessarily. But the big problem lies elsewhere: HandBrake uses many external libraries. Since we link against them statically (under OS X at least), we could try and patch them, but patches against external libraries are a pain to maintain (depending on their scope, even create) and are usually not considered good practice. So basically, we do it when we have no choice, but prefer to avoid it whenever possible.

[tc60045]

@Rodeo, thanks. Is there anything to be drawn from the difference in HB behavior as observed - CLI and gui just seem to be handling memory so differently. Different calls to same libraries? Different calls to different libraries?

I'll admit my findings are anecdotal thus far, so tell me what I can do to help you understand this fully, even if a future fix is unlikely. I'd just be happy to help advance the state of empirical HB knowledge...