What kind of courses to take to learn more about reverse engineering/enabling homebrew

Hi guys,
I'm a freshman in college right now, going to the best university in the world (UC Berkeley ;)) and majoring in EECS. I was just curious about what kind of courses I would want to take if I wanted to be able to do the kinds of things you guys do with hacking and reverse engineering the Wii and other future consoles, or other devices in general. I'm probably going to have an emphasis on the EE side of things, but I haven't committed to either right now (at Berkeley EECS is one major, not two). Obviously homebrew/reverse engineering isn't going to be the focus of my college education, but I wanted to do this kind of thing as a hobby/side thing when I have a job. Next semester I'm taking a class on machine structures: [www.eecs.berkeley.edu]

I'm guessing this might help a bit with that sort of thing. Currently I program a lot, but mostly in Java, which isn't very useful for homebrew on the Wii. I tried learning assembly in middle school for my TI-83 but was frustrated rather quickly.

To be perfectly honest, I'm in still in highschool. Everything I learned came books and the internet.

EDIT: I can reccomend some if you woyld like me to.



Edited 1 time(s). Last edit at 10/19/2008 02:26PM by Arikado.

I recommend looking for some books on CPU architecture. I'm not sure of any specific titles, but some topics to cover would be-

1. CPU Registers -- [en.wikipedia.org]
2. OpCodes -- [en.wikipedia.org]
3. Stacks -- [en.wikipedia.org])
4. IRQ (Interrupt Requests) -- [en.wikipedia.org]
5. Byte Endian -- [en.wikipedia.org]

Actually, I've found this, it's a tutorial for PIC microprocessors, shows a bit of how the hardware relates to the software to.
[www.mstracey.btinternet.co.uk]

These should give you an idea how processors work in general. Then pick your CPU (ARM, 80x86, PowerPC etc.) and get a feel for the assembly language it uses, how the instructions are structured in memory, this will be invaluable in working out where to patch code.

One thing I would recommend (although there may be some frowns or groans), is to get a disassembler/debugger for your PC (like OllyDbg), and download some CrackMe's (Try to find Assembler ones first, over those compiled in C, C++, and Visual BASIC, as they are generally easier (at least IMO)).

CrackMe's are generally just little programs that open a window asking for a password or showing a "Nag" screen with a registration box, and they generally challenge you to find the password, or break it so that it accepts any password etc.

A link I used to read: [www.angelfire.com]
Some of +ORC stuff was interesting too: [www.textfiles.com]

I'd second the suggestion to look at CrackMe's -- they are some of the only places you'll find writeups ("solutions") that describe actual reverse-engineering methods in detail. Then again, I take apart malware that I receive in email for fun, but I'm weird.

It really is all about learning how software works at a low level, and how you can modify it to change the way it works. Books and classes are good, but this is a very deep subject, meaning that you will have to do a lot of learning before you see how it actually applies to, say, homebrew on the Wii. Once you do, you'll have an epiphany of sorts, and you'll suddenly understand why you had to do so much learning.

Yeah, CPU / Computer Architecture courses are what you should seek. I can really relate to the "I found assembly language frustrating" -- I found (and still find) that to be the case when I was trying to learn it on its own, so that I would "know assembly language". It's far, far easier to do these things when you have some specific goal in mind (reverse-engineering some specific program) -- it lets you push through the frustration.

Quote
bushing
I can really relate to the "I found assembly language frustrating" -- I found (and still find) that to be the case when I was trying to learn it on its own

lol, I truely am odd... I found assembler one of the easiest languages to learn. I learnt it before C and C++, which confused the smeg out of me (and still do from time to time).

@iofthestorm (and anybody else, I guess):
Feel free to post here or even email me if there's anything in specific you want to know. I can even make some Windows based (maybe some Wii based ones if I can implement USBGecko debugging support) crack-me's with guides, if that will help. You might find existing ones easier, or it might help speaking to the creator to see how and why some things work they way they do.

The offer is there if anybody wants it.

What the people above me said. One thing I find important aswell, pick a target that seems easy/feasible for you to work on, and pick clear goals. "reverse program X" is a much wider goal then "reverse (part of) program X to see how it does XYZ". Finding critical sections of code in a big lump can be problematic if you _dont know_ what you are exactly looking for. Come up with clever byte search patterns and constants to look for so you can narrow down the amount of routines that you actually want to/should reverse. And most important of all, just have fun in what you're doing and let curiosity be your guide! :-)

Cool, thanks guys. I didn't realize the forum email doesn't work (or maybe i disabled the option or something) so I thought no one was replying to my thread. I've actually got a PIC16F690 with a PICKit 2 for playing around with, but haven't gotten around to doing anything with it besides making some LEDs flash. I think I'll look through that PIC tutorial and maybe start looking at CrackMe's. I really agree that it's important to have some sort of goal, most of the time I find my CS homework boring because it's just asking about some theoretical problem which isn't too difficult, whereas the projects are easier for me because there's more of a goal in sight, and if I'm working on a game or something I can work all day and not feel bored (I literally coded all day this last Tuesday, it was awesome).

Incidentally, I also want to start programming some homebrew applications, (the reverse engineering stuff is more of a long-term goal) and I was trying to figure out whether I can run stuff in an emulator. Would I want to use Dolphin, or something else?

Quote
iofthestorm
Incidentally, I also want to start programming some homebrew applications, (the reverse engineering stuff is more of a long-term goal) and I was trying to figure out whether I can run stuff in an emulator. Would I want to use Dolphin, or something else?

No, there are no emulators available for use. Good luck with your applications ;)

devKitPro has a Cube emulator, I've never used it though. I just do all my testing directly on my Wii...

Quote
whodares
devKitPro has a Cube emulator, I've never used it though. I just do all my testing directly on my Wii...

I was just about to suggest that :). It's a good way to start if you haven't made homebrew applications before.

Cool thanks. The thing is I spend the weekdays at college away from my Wii so I can't always test on real hardware. Of course, I should probably be studying during the week anyway.