Good set of slides that provide an overview of chapter 7 in “Assembly Language for Intel based computers” by Kip R. Irvine. These notes are a great resource whether you’re doing crackme’s, reversing larger algorithms, or just dealing with some code that has compiler optimized math functions in it.
Gray Hat Python: Python Programming for Hackers and Reverse Engineers (Paperback)
by Justin Seitz
Publisher Description
Python is fast becoming the programming language of choice for hackers, reverse engineers, and software testers because it’s easy to write quickly, and it has the low-level support and libraries that make hackers happy. But until now, there has been no real manual on how to use Python for a variety of hacking tasks. You had to dig through forum posts and man pages, endlessly tweaking your own code to get everything working. Not anymore.
Gray Hat Python explains the concepts behind hacking tools and techniques like debuggers, trojans, fuzzers, and emulators. But author Justin Seitz goes beyond theory, showing you how to harness existing Python-based security tools – and how to build your own when the pre-built ones won’t cut it.
My Take
I have to start off by saying that I’ve been looking forward to this book being published for months now. Since it initially popped up on the No Starch website I’ve been following it and nagging the publishers about it’s release. I was fortunate enough to get a copy last Friday and I dove into it full force, even my final exam suffered because I was a little too focused on this book. I have mixed feelings about the book overall, there was definitely a few things I gained from it. On the other hand, there were many more things that could have been added. If you are new to Reverse Engineering then you will most likely obtain a leap of knowledge from the book. Packed with code examples on various open source tools and concepts to create your own, you’ll be waste deep in python for a while. If you’ve been around for the last few years and have already checked out many of the open source tools discussed in the book you may only get a few things out of it. I don’t understand CG’s rating system so I’m not going to give you a star formula, or even a numeric rating.
Justin does a great job elaborating through the the code examples used throughout the book. The sheer scope of this book makes it difficult to cover everything but Justin definitely attempts to give you a taste for the more common scenarios you may find yourself in. Below I highlight a few chapters that I found interesting and useful. Overall the entire book is useful but a large portion covers open source tools that have a plethora of documentation and examples in existence.
Chapter 3 is just downright awesome. This chapter walks you through creating your own python based debugger that is similar to pydbg. Between the concepts and actual implementation you obtain a solid understanding of what is going on in a Windows based debugger.
Chapter 6 goes over Hooking in 5 pages. This topic could easily fill a few chapters by itself but Justin sums everything up quite nicely with plenty of examples.
Chapter 7 discusses DLL and Code injection which was a pretty interesting topic. The chapter even gets a little “Evil” by showing you how to hide files on a filesystem using python. It goes one step deeper into evilness by demonstrating how to code a back door into files.
Chapter 8 goes into fuzzing concepts and the demonstration is very practical for anyone doing exploitation. Justin shows us how to create a file format fuzzer which I thought was pretty slick.
The last chapter I found pretty interesting was chapter 10 titled “Fuzzing Windows Drivers.” This chapter outlines methods of identifying Device names and IOCTL codes using Immunity Debugger. It continues to show examples of an IOCTL Fuzzer that can be used in conjunction with the rest of the scripts outlined in the chapter. Definately some cool stuff. Personally, I’m not quite up to a Drive level fuzzing level but I still found the knowledge very useful and was able to translate some of the immunity scripts into IDAPython scripts for static analysis to find the device names and IOCTL Codes.
The Cons
The entire book was based off Dynamic analysis. This is great for those out there doing RE through execution, but it really sucks for people like me who only does dynamic analysis 10% of the time.
The entire book is windows specific. Yes, most RE work is done on windows, for windows binaries. However, there are closed source applications and even embedded stuff that is Linux based. Would have definitely been useful to have some examples for these types of cases.
At times I found the book drifting into the direction of exploitation. Granted, RE and exploitation go hand in hand but I feel those of us who perform RE for the purpose of compatibility, protocol analysis, and other crap were left out a bit.
The chapter on debugger design did not have all that much to say about designing a debugger. In fact, it was probably more inline with a quick introduction to x86 architecture, specifically registers and stacks. It was not until chapter 3 that you really understand the elements involved.
Overall, I equate this book to Hacking the art of exploitation first edition but for Reverse Engineering. The book is packed with useful information for both the novice to the professional but I hope a second edition comes out with more information, that covers both static and dynamic analysis.
When programming it is always nice to be verbose. Be verbose in the commenting of your code and your error messages is something taught in most academic programming courses and even the good programming books. Today however I saw a new level of verboseness which helped cut my reversing time by a large amount.
As with all firmware images the developers stripped the symbols before release. This makes things difficult as I mentioned in my previous post because all you have is assembly and not much to go on. You get a series of sub_xxxxx calls and that’s about it. In regular binary images its not that difficult when symbols are stripped because you can use flirt signatures (amongst other methods) on Elf or PE binaries to fill in a few gaps. You can also occasionally use flirt signatures on firmware but I’ve found it only works on X86 based images, that is of course unless you can develop your own signatures for the architecture you are using.
Often times you will find your print functions fairly quick because they are called the most. You can also utilize string XREFS to follow what’s going on and after a while of doing it you notice patterns and how the arguments are passed. You just pretty much go ok, thats fprintf, or printf, or some homebrew of print_to_term.
Today as I was tracking down some common functions before I really tackled things I stumbled into something interesting. It’s more luck than anything else since it rarely happens, but cool for me none the less. I started seeing function names in strings!
The developers were so verbose in their errors that their error messages for all mallocs and fails stated the function name in it. So as you are scrolling through the strings you suddently find something like:
“Malloc Failed in: parseHttpRqst()\n”
As I mapped the cross reference to the string and followed the code for a minute I realized it was part of the error checking for malloc in the function… parseHttpRqst. I went on to clear out about 12,000 functions out of 67,000 so far.
They may have made my job just a little more difficult but… I got more verbose function names in 5 hours than what I could have gotten on my own within 3 weeks! w00t
Not sure if anyone else is seeing this but it has steadily been vexing me for the past 6 months. One of my clients uses Nessus on a regular basis and about 6 months ago started seeing every host report back “No significant Problems” on almost every single host. Knowing the networks he’s been scanning pretty well this perplexed me. At first I kept wanting to believe the problem was with him forgetting to disable his firewall. Finally he was close enough to town and called me up after having the same problem. I drove out and as I was pilfering through the Nessus logs I started seeing numerous pop ups from AVG.
Once the anti virus is disabled results start populating the hosts almost instantly. I haven’t been able to figure out how AVG is influencing the scans but I know it is. I also googled the shit out of the topic with no results so am I the only person having this problem?
In Late 2007 I took a position doing reverse engineering, mostly on embedded systems. RE was something I wanted to get into on a professional grade for some time but could never find a segway into it. Now being in the thick of it I’ve come to learn quite a bit through my experience. Reverse engineering takes a special breed. It takes a lot of patience to stare at a debugger or disassembler all day long. There are times I walk out of work and my eyes are blood shot from staring at bindiff or IDA all day long. This is the primary reason my blog has fallen off course. By the time I get home as of late my desire to sit being the computer isn’t always there. I mean, I want to do it but my brain tells me no! Here are a few items off the top of my head about reverse engineering embedded systems. Sometimes I’d rather take obfuscated malware then this stuff…
Learning
The ability to learn fast and get spun up on something such as an architecture is essential for doing this. Quite often when it comes to reverse engineering positions the subject matter is dealing with malware, specifically, malware on x86. Although malware can be seen on mac and linux the majority of it is found on intel based windows systems. You need a concrete knowledge of a single operating system and architecture and it will generally serve you well. When it comes to embedded systems however, you are talking about dozens of operating systems over a handful of architectures. You really need to be able to pick up core concepts of operating systems and architectures really fast.
Architectures
For some reasons the developers of the systems I have worked on can’t make up their mind. One device is x86 and then the next version is ARM, then they hopped over to PPC for last years release and this years device is x86 again. WTF! It gets confusing hoping back and forth between languages. What makes it worse is when you have to find differences in certain features such as protocol implementations or the way the device reads in data. Makes DIFFING a little bit trickier.
For embedded reversing your major architectures are: x86, MIPs, ARM, and PPC. Despite what some of my amigos think PowerPC is far from dying. I say this because it is the predominate architecture that I see in the devices I’ve worked on.
Algorithms
Data structures and algorithms help out because you can start to see patterns in disassembly and will be able to know whats going on a lot faster. Aside from that, just being able to identify structures in disassembly will often bring large portions of code together for you and make your life a lot easier.
Symbols
Every so often a vendors development team will screw up and forget to strip symbols from an updated firmware image they push out. When this happens you better be on your game because they will pull it from their site in minutes upon realizing what they did. Most often firmware images do not contain symbols which makes life a lot harder. When you have 40-60,000 unnamed functions, no imports, no exports….nothing it makes life a bitch. Sometimes you can get around pretty well with just string references and figure out whats going on. Any little bit helps but sometimes it would just be so much easier to have symbols ):
Slow Roll Your Analysis
In July of last year Cody Pierce wrote a blog post on DVLABS about cross references. One of the things he bought up was identifying common functions and clearing them out early on. As you are going through a firmware image that has 60,000 functions would you prefer to repeatedly see CALL loc_67499 or would you rather see CALL print_to_term. Instead of going straight to my objective I run an idapython script that loops through and counts the number of xrefs to each function. What I will do is start at the top and work my way down. Usually performing analysis on the first 20 or so functions because they are the most xrefed functions in the image. Later you realize the pay off from this as you are going through code and you see named functions instead of CALL loc_addr/sub_addr names.
Magic Numbers
Get intimately familiar with magic numbers! From ELF to compression they will come in handy if you have them embedded into your brain. Most firmware images are compressed with some algorithm, in some cases you will see numerous compression blocks. Being able to identify these numbers in a hexeditor will save you a lot of time when trying to find what you are really looking for. Many times the first few segments are not compressed and consist of bootloader code and the decompression routine. The meat of what you are looking for is most likely compressed!
HexEditor
This is another thing you need to have a good relationship with. Unlike PE files and ELF files you can’t just drop a binary image into IDA Pro and get magical results. You will need to open the binary in a hex editor first, identify structs and code from the bootloader and decompression routine. Your hex editor will also come in very handy when mapping the general layout of the image. Your next logical place in the hex editor is to remove the data before your compressed code so you can decompress it.
Quick word of wisdom. If you are running the cpio utility in linux and the cpio archive happens to have a directory structure which includes /lib, /usr, /etc/and /sbin or any others similar to a linux file system. It is in your best interest not to run cpio on the archive file as root. Unfortunately the system was not backed up. Data is good to go, but damn you really can’t do much without a /bin directory or the symbolic link to libc.so.6.
Today was a horrible day!
TFTP Request module for Sulley Fuzzing Framework, located here.
If you find something wrong with it, or make it better let me know.
Please Update your RSS Address… I may or may not have messed up the old one!
Not like I update the site anyway, but I might get around to it after I complete some of the studies I’ve been doing. Just been neck deep with family stuff and books lately.
It’s been a month! I’m down to the last handful of classes and I will be finished with my masters degree. For anyone in San Antonio I’ve started a new group.
San Antonio Hackers Association
Sign up for the mailing list! The first meeting is coming up on Oct 1st and I just sent out a call for presentations! If your in town stop in!
I’ll try to get back to posting here! The 20s money magazine I started has also been consuming a lot of my time. The cool part is we had an article featured on MSN which really bought up the traffic levels and revenue! The downside is I’ve been spending all my time on that! Hopefully will have a few posts coming up though since I have about 6-8 penetration tests coming up in the next 2 months.
I was able to pick up a pre-released copy of The IDA Pro book at Defcon in the vendor area, thanks to Adam from No Starch. This book is not an introduction to reverse engineering, its a hard core manual for IDA Pro. IDA Pro is a critical weapon in any reverser’s arsenal, so proficiency in this tool is paramount to your success in reverse engineering. If you are new to IDA Pro you need this book, even if you’ve been working with IDA for a while you will more than likely learn quite a few things after reading it. Unlike the two other books I’ve read on IDA Pro this book has no fluff or filler, its solid information! The funny thing when comparing it to the other two IDA books is its thicker than both combined, and contains an exponentially larger amount of information.
The author takes time to explain things in a very clear manner as you walk through from an introduction to the tool to more advanced usage such as customizing, extending IDA, debugging, and dealing with obfuscated code. The author answered questions I had been spent weeks asking and searching the Internet for.
Likes:
Just about everything. The author walks you through plenty of code and discusses scenarios where you could apply the information he is giving you. The fact that he took his time to elaborate on why, and when you might use a piece of information is unlike many authors whom will give you information and leave the reader wondering “What would I use that for”.
This book does not just talk about Win32 and Portable Executable format, ELF binaries have a continual guest appearance throughout the book, and firmware/binaries are mentioned in numerous chapters.
Side bar elaboration is kept to a minimum, I often find in texts that an author will go on about background information that does not add anything significant to what I am reading. Chris Eagle keeps this to a minimum adding small side bars when necessary but only take up a small amount of real estate.
Dislikes
My only dislike of this book was the use of PE format as the example in chapter 18 – Binary Files and Ida Loader modules. Despite the use of a well known format chosen for this example the concepts were clearly displayed. I think it would have made it more interesting if the author had used a lesser known format, or do as the author of “Reversing, Secrets of Reverse Engineers” did and create his own binary.