Teaching Reverse Engineering

I'm presenting a paper at the Cyber Experimenation and Test (CSET 2009) workshop, held in conjunction with USENIX Security 2009, on teaching reverse engineering in academia. I'm also presenting a two day tutorial on reverse enegineering for USENIX Security, which is a condensation to bare essentials of the semester-long class on reverse engineering that I teach at the University of New Orleans.

There are a few very important challenges in teaching reverse engineering in an academic setting:

The first is that students will likely show up with poor assembly skills. This is because assembly language courses, if they exist at all as separate courses in a curriculum, are typically full of things that do not help students become better systems people. While High Level Assembler (which drapes assembler in macros that give it a flavor more like a high level language) might be a good idea for development of large scale applications in assembler, it hides details that students *should* be immersed in when learning assembly language. The pain, the attention to myriad minute details, complex interaction with hardware features, et al are essential. For systems research, the devil really is in the details. The punchline is that students will essentially have to be taught deep assembly skills while learning reverse engineering, all in a single semester, which creates important time constraints. More on this below.

Another challenge is not only teaching students about the potential legal ramifications of reverse engineering, but also avoiding these same legal hurdles while teaching the course. In my case, the class is focused exclusively on the analysis of malware, which relieves many of the legal issues, but adds yet another dimension, that of safety for the academic computing environment. My solution is to carefully screen the malware samples that will be analyzed by students in the lab. Since my approach to teaching this course involves detailed walkthroughs of assembler for each malware sample, I have to do exhaustive analysis of the effects of the malware, anyway. As further protection, the laboratory environment consists of an isolate-able gigabit network connecting workstations running Linux and VMWare. Preconfigured Windows XP guests under VMWare are used for most analysis and the guests typically have networking disabled as a safety precaution. The XP VMWare images contain a licensed version of IDA Pro, ollydbg, WinHex, HBGary's Responder, the sysinternals tools, as well as other tools. In the next iteration of the class, we will also use BinDiff and BinNavi.

In the reverse engineering class, I'm not interested in having students learn what reverse engineering is. I want them to be able to *do* reverse engineering. This rules out the traditional academic format of flipping Powerpoint slides and giving exams. The approach I've used for the class, to deal with the fact that students must gain good assembly skills while learning reverse engineering, all in a single semester, is to immerse them immediately in the analysis of real malware samples. The malware that we analyze in the class and in laboratory assignments increases in difficulty as the semester progresses and each sample is chosen to push the students a little harder and to force them to gain more systems knowledge in order to succeed.
An essential component of the class is reliance on a document camera for in-depth walkthroughs of every malware sample, in class. I drive the discussion, but students are expected to participate and what results is a very deep analysis of each sample, which is then distributed to all of the students.

More details on the class and my approach to teaching it can be found in my CSET paper, which is here.