Geoff Chappell - Software Analyst
You’re stuck. Your programmers are stuck. You have a problem that’s in the too-hard basket. It has probably been in there for a while and isn’t coming out any time soon. But it must be faced now. If the problem has anything to do with Windows programming, then I’m your guy. I thrill to the Windows programming that others give up on. Give me your too-hard basket!
If you’re a Windows programmer or you’re in the business of managing software development, then you likely have the occasional thought that the process might go easier if you could ask an expert. Perhaps you have an idea but aren’t sure how practical it would be to do or whether it can even be done. Perhaps your programmers tell you they’ll need six months just to have a prototype to show. Perhaps you already got a great idea into production but now you’re learning of problems and you’re wondering if they truly must be so hard to solve. Perhaps you and your programmers wonder every day if there’s not a better (and less exposed) source of advice than asking for free help at this or that blog on the Internet. Yet your need is only ever occasional. It’s never even nearly enough to justify hiring someone full-time, who anyway won’t be the most expert you can get and keep. So, consult!
By the way, though, if you do perceive a full-time need, I’m open to offers of employment. After all, if you think that your software development “pushes the boundaries of operating system integration” (or some such, this being quoted from an ad by Dropbox), then your problems are already what I happily spend all my working hours solving. Who can you hope to find who’s better?
The theoretical work behind the Geoff Chappell website is to develop and refine techniques for studying software without its source code, especially as an economical way of getting detailed and reliable answers to questions that might otherwise seem intractable. The practice comes from directing these techniques at Windows. The commercial application is to solve your problems of Windows programming.
Look around this website: there are few people anywhere who approach the problem of understanding Windows with such breadth and in such detail. Get that working for you!
To get my attention directed to your particular needs, consider a formal consultation. Perhaps look first at my (negotiable) fees and even at my few terms.
Of course, Windows is very big. I don’t know everything about it. But in my specialties, the systematic application of reverse engineering means that I’m an expert’s expert. (Literally: I have had clients who are themselves being consulted.) Few others anywhere can be as well prepared for questions in these areas of Windows programming. But even where I haven’t yet looked, few others anywhere could make up the ground as quickly and reliably. That’s because I don’t play with trial and error, hoping to infer what’s needed. Instead, I reverse-engineer and deduce. That’s much too much work for a simple problem, of course. But if the problem is innovative enough to take you outside what’s covered clearly and cleanly in the documentation, then my methods not only give you a robust solution but they deliver faster.
Inter-operating with a huge, sophisticated, and yet sometimes only vaguely documented, operating system has in its nature that if what your programmers are attempting is at all tricky then there’s a good chance they spend a lot of time on trial and error, hoping to learn by experiment what does and doesn’t work. Mostly, that does succeed. Computers are intrinsically repetitive. You can go a long way by observing and inducing. Until you can’t. But even before you hit that brick wall, the effort that you put into trying to infer what to program is not effort that’s going into programming carefully. You end up with software that is held together by the proverbial rubber bands and that nobody’s confident of, but which still takes ages to develop. If you’re lucky, the bugs don’t bite you back too hard or too soon. Like any handicraft, even on a commercial scale, software by nature cannot be perfect, but your software will be better from the start if you secure the close attention of someone with a record of deducing how the code it interacts with really works.
A lot of Windows code has been analysed here as speculative research. This analysis is a sort of debugging in advance. Before you even imagine a specific problem of programming Windows, let alone before it costs you something in the real world, I may already have done much of the work required for knowing how to solve the problem, if not for its actual solution. Many questions that you, your programmers or your lawyers may have about Windows can be answered quickly and authoritatively—and if the information needed for the answer is not already known, it may be within easy reach. For many programming problems, you can get reliable advice on a design that you expect your own programmers to implement or you can get a solution coded efficiently for you.
Some programming work with Windows just cries out to be done by a specialist who already knows how Windows behaves and what Windows expects or at least has proven ways of finding out quickly and reliably. You perhaps already have experience of a Linux programmer, even your best Linux programmer, feeling confident to do some system-level Windows programming and even of seeming to do it well—and then, over time, it becomes clear that not everything was quite right or even that some things were very wrong. At some low-enough level in the hierarchy from the hardware up through your software and beyond to the generality of other people’s software that will depend on yours, the details are highly specific to the operating system. Get that programming done by a specialist in that operating system!
If you find yourself stepping into Microsoft’s code while debugging your own, you may soon be overwhelmed by the unfamiliarity of it. Consider that I’ve been immersed in this stuff for years. It’s my source material. Much of what you see at this site is alternative documentation of Microsoft’s code. If you’re lost in that code, there’s every chance that I’ve been in there before and I can help you out of it now.
Help from afar is especially easy when the system has itself detected a problem, as when a user-mode program crashes from a fatal exception. Just send me the fullest crash dump you were able to collect and perhaps what symbol files you have for your code in the faulting process. For kernel-mode driver problems that have caused a bug check (blue screen), just send me a mini-dump, your driver and a matching symbol file. Either way, just this information, plus a billing address, is enough nine times out of ten to get an explanation and even a fix.
A long-standing specialty is kernel-mode programming, especially of file-system filter drivers and of device drivers for disk I/O. For years there was not much sign of that specialty here at this site, because this site is for free publication of research into Windows for the public interest, not for showing professional (and thus confidential) work that’s done in your interest. But look now at the Kernel section!
One subject that is much on display at this site is the Windows Shell (including Microsoft’s adaptation of it to support Internet Explorer). Very few Windows programs of any substance do not depend on the Windows Shell. Yet much is undocumented and a lot more is under-documented. For most of the history of Windows, it is often because of something new in the Windows Shell that new versions of Windows programs from Microsoft have a new, distinctive and even useful appearance. If you want that your programs can do some of the interesting things you see in, say, List-View groups and footers in the Windows Explorer from Windows Vista, then the large amount of information I have published here for free may be enough to get you going. But the nature of writing, let alone for a freely accessible website, is that I write up only a tiny proportion of what I find. If you want that the rest of what I find is put to use working for you, then you have to ask.
Although I regard as folly the notion that closed-source software can usefully be studied only by trying to reconstitute the missing source code, I haven’t been studying software for 20-something years without having developed some translation skills.
You are likely familiar with how machine translations of one human language into another often suffice for getting a rough idea of what is meant but do not begin to help with detail and fall far short of being useful for legal proceedings. For any text written in another language, if you want to read it with as close a sense as possible to reading it as intended, then you need a human translator. Indeed, you will need a good one, and will do for many years yet.
So it is with software. The gobbledegook that’s spewed out by automated decompilers is impressive in its way but is just not readable for detail. For a properly crafted translation of x86 or x64 software into source code that actually does look like a human programmer wrote it, you need a human translator. Indeed, you will need a good one, and will do for many years yet.
If you’re in the unfortunate position of having lost the source code to your product, e.g., through contractual dispute, then few people anywhere will be as able to reconstruct readable maintainable source code for you.
If you’re in the better position of knowing that something’s wrong with someone else’s software that you are evaluating or already depend on, and you want your meeting with them to proceed from a position of strength, then just for your own understanding you should want to be armed with reconstructed readable source code in which someone clearly comments the defects. You don’t get that from automated decompilers.
As malware becomes more sophisticated, there is only so much you can sensibly hope to learn from observation, however systematically and carefully you set about it. Even the anti-virus companies see some merit in what they call static analysis as their way to reverse-engineer the malware without running it. If you need to know everything that some malware can possibly do (or have done), then you need to pick apart its code. Even that won’t be enough for malware that works by interacting with Windows in some novel way. For true understanding, you must pick apart both the malware and the bits of Windows that it uses, abuses or exploits. To get that done comprehensively and reliably, ask someone who has made a point of practising it for years.
Much the same applies to the reverse-engineering of proprietary file formats. If you want a competitor’s customers to become your customers, you typically have to overcome the huge obstacle that those customers have stored their work in your competitor’s file format. Your software will need at least to understand that format well enough for reliable conversion to yours.
The apparently traditional way to discover details of an undocumented file format is to create a wide variety of files in more-or-less controlled circumstances and hope to spot what data turns up where. And I must admit that this does get you much of the way, if only after lots of fumbling and who knows how many iterations. But if you need to know everything about the file format, then there is no substitute for picking apart the code of whatever program is the definitive creator or interpreter of files in this format. To get that done comprehensively and reliably, ask someone who has made a point of practising it for years.
For the little while longer that PCs have a ROM BIOS to start executing code from disk in real mode, the exotic niche of real-mode programming just doesn’t want to disappear. Yet efficient coding in real mode is a skill that many people lost years ago, if they ever had it. Even though I, the author of DOS Internals long ago, haven’t written anything for MS-DOS in years, I have kept those real-mode skills alive on commercial software that is written to run from boot sectors before the Windows loader. If you’re in that niche of still needing to run code in real mode, I can help.