Analyzing Compressed RAM in Mac OS X (and Linux)

The compressed RAM facility in Mac OS X Mavericks places a RAM compressor inline and in front of page swapping in the virtual memory implementation, favoring compressing pages of RAM over swapping them to disk.  In Mac OS X, page compression/decompression are handled by a custom, hand-optimized 64-bit assembler version of WKdm.

My Mavericks-compatible Python port of WKdm is done and I'm busy building the plugins to allow analysis of compressed RAM from Mac OS X memory dumps in Volatility.

+Andrew Case is helping with the Volatility integration, since we want support for decompressing pages to be as transparent as possible across various operating systems that implement compression. He's also tackling the Linux side, which uses a different set of compression algorithms.

I'll be releasing everything and talking about this project at the American Academy of Forensic Sciences (AAFS) meeting in Seattle in Feb 2014.  Hope to see you there.

Update (12/26/2013):

The Volatility plugins are now decompressing compressed RAM pages properly--currently, we're just dumping all the pages, but Andrew and I will integrate the decompression more tightly soon so that, e.g., process memory dumps for Mac OS X will also dump available compressed pages.

Update (12/27/2013):

I was curious if Apple's choice to do a custom assembler version of WKdm compression/decompression was worth it.  It definitely was--the hand-optimized assembler version, which eliminates all function calls and takes the WKdm stuff down to its bare essentials, smokes the original C version.  On a Core i7, with gcc and -O3:

Assembler timing:    268317 compression / decompression pairs per second.

C timing:    142150 compression / decompression pairs per second.

And, of course, there's the Python version we'll be using in the Volatility plugins:

Python timing: 391 compression / decompression pairs per second.

Sigh.

Update (1/2/2014):

Just for fun, I ported the Python version to go--in about an hour.

Go timing: 59880 compression / decompression pairs per second.  Basically, go rocks, and is my new language of choice.  If you haven't checked it out, do.

Update (1/23/2014):

Getting the Volatility plugins to handle decompression of pages for address spaces of individual processes took some effort, but that stuff is now working.  More detail on everything coming soon once I do some cleanup.

Update (4/19/2014):

Our paper describing this work, entitled "In Lieu of Swap: Analyzing Compressed RAM in Mac OS X and Linux" was accepted for publication at DFRWS 2014 in Denver.  See you in August!

Python: Yes, that indentation thing again

No, I'm not going to get over it.

These:   { }  

rock.

And Python clearly rocks, too--because if you accidentally hit tab a few times, or your cat dances on your keyboard, or you fall asleep and your forehead starts typing, why wouldn't you want to have to carefully read the code again to fix it, instead of just re-indenting the code automatically (possible in sane languages, because, surprise!  That's what explicit bracketing allows!)

Please don't suggest that it's always an awesome idea to re-understand code to fix indentation.  I'm porting some complicated compression code from C, and frankly, once it works, I don't want to have to understand it again if I paste it, or forehead it. Or whatever.

The "white space is significant" thing was tried in the past, and failed.  It's a stupid idea.  COBOL had an excuse, but we're not using punchcards anymore.

COBOL whitespace:  fail.

SNOBOL whitespace:  fail.

Python significant whitespace and no bracketing for code blocks:  huge fail, because it's 2013, not 1950.

It's an awesome language in terms of expressiveness, but the lack of bracketing is 100% fail. Fine, it's 90% fail.  That combined with slow and broken threading means it's a huge missed opportunity.

Will I write Python to contribute to projects that use it?  Of course.

Will I voluntarily use Python for other projects?  Maybe.  Not if I can help it.  OK, I'm not sure. Obviously I'm torn.  But are there really great alternatives?  Perl is a broken mess.  Ruby's threading needs help, too.    I love C and in some sense its my "native language", but can you really imagine writing a framework like Volatility in C?  

These:  { }

are awesome.

Proper threading:

is essential.  Modern hardware insists.  And we should listen.

And yes, I know about Whython.  But it's just a toy and it's really too late, right?  Because portability for security tools is essential.  

Maybe I just need a glass of wine.