With an assembler, your code gets assembled into machine code. All valid assembly instructions should be translated to corresponding machine code bytes. But in reverse, not every possible byte sequence is something that can be produced from assembly source. There a few categories of reasons for this:
Simple redundancies. XOR EAX, EAX can easily be represented as bytes 31c0 or 33c0. Both are equally valid, it only exists because of some redundancy of the ModR/M byte. This isn't bad, it is actually more flexible if anything. However, if you are an assembler, and you are parsing XOR EAX, EAX, and need to make it into machine code, you will pick one of these (and not the other). It doesn't matter which, but you only pick one. So this reason is assembler based, but it still boils down to there being machine code that will not be produced by your assembly code.
Garbage that doesn't execute at all. There's tons in this space; "Instructions" that are invalid and just will not execute. It's safe to say there wouldn't be any assembly that will create this (excluding assembler directives).
Undocumented code. In a debugger, this may look similar to Reason 2. I say 'may' because some debuggers translate these instructions too. This is why the 'SAL' instruction shows up in my "Better Call SAL" post, even though there is no overt reference to its valid machine code in the Intel manual. This post is about Reason 3. I will use an instruction that the debugger I use (edb) does not decode. I have seen other debuggers represent this machine code, however. In the grand scheme of things, this is a very uncomplicated instruction (in the context of both what it does and how it is encoded).
The above screenshot looks like our "Reason 2" case, but when we step, we end up several instructions after this one without an error. This means a couple of things. 1) It is some kind of instruction. 2) Seeing that it skips 4 of the NOPs, it may have a 4 byte operand. Is it skipping the NOPs? Let's try:
Looks like it's a pretty good chance these instructions aren't being executed; otherwise EAX would be 0x00000004 at this point. So we are running some undocumented instruction that I will now refer to as UNDOC. So far the format is something like: UNDOC +4 bytes. If we are familiar with the encoding of other instructions, we would know that f7 shares instructions with TEST, NOT, NEG, MUL, IMUL, DIV, and IDIV. So we already have a basic idea what the first two and last 3 bits of the c8 byte are for (Mod and Register). In this case the Mod would be just registers (not pointers with offsets and such) and the Register would be EAX. The other 3 bytes specify what instruction (of which is not documented). We also know that the next 4 bytes are 32-bit immediate. So: UNDOC EAX, imm32. In the case of the above example seen in edb (the one with the "nops"), we are running: UNDOC EAX, 0x90909090
What did the instruction do? Or what did it change? Not much, Obviously EIP changed. The flags also changed; they changed from cpAzsdo to cPaZsdo. This doesn't really give us the full picture yet though. With instructions, some flags get blindly set to a 1 or 0 no matter what, some can be modified depending on the result, and others have undefined behavior.
Let's try setting all of the flags right before running this instruction, and see what it does. Flags can't be arbitrarily set directly. To do this indirectly, you set a register to the value you want eflags to be, then you push it to the stack, then you use popf (pop from stack to flags). Turning all the flags on and running our mystery instruction turned CPAZSDO into cPazsDo. Doing the same thing with turning off all of the flags; it turns cpazsdo to cPaZsdo. It's hard to determine the difference between hard-coded clearing (0), setting (1), and modification (can go both ways) with these examples. Although, the Zero (Z) flag stands out as a flag we saw go both ways.
I looked at ALL instructions that can both modify the Z flag and has the ability to do imm32 addressing, we are actually only left with 9 instructions. Let's look at the flag setting behaviors of these instructions and see how it compares to our UNDOC instruction. For each flag I use 'M' for 'modify', '0' for 'clearing to 0', and '-' for 'undefined.'
MMMMM-M: ADC
MMMMM-M: ADD
0M-MM-0: AND
MMMMM-M: CMP
0M-MM-0: OR
MMMMM-M: SBB
MMMMM-M: SUB
0M-MM-0: TEST
0M-MM-0: XOR
0M-MM-0: UNDOC
Based on Intel's documentation for flag modification behavior, UNDOC is common to AND, OR, TEST, and XOR. We actually already know TEST and AND to be very similar; TEST is like an AND without storing the result into a register (it just does an AND for the flag side-effects). for UNDOC, we know that if any register would be modified, it would have something to do with EAX, due to the ModR/M byte. Let's set EAX to something and see what happens to it. I'm going to put 0x11223344 into EAX.
In order: I will clear all of the flags to 0, put 0x11223344 into EAX, and then run the 9 instructions seen above with EAX and the immediate of 0x90909090. I will mark an X for flags that got set, and - otherwise (flags being in the debugger order of cpazsdo).
-X--X--: ADC
-X--X--: ADD
-X-----: AND
XX--X-X: CMP
-X--X--: OR
XX--X-X: SBB
XX--X-X: SUB
-X-----: TEST
-X--X--: XOR
-X-----: UNDOC
So based on flags alone, our UNDOC is similar to TEST and AND, but AND does change EAX wheras TEST does not. UNDOC didn't change EAX (AND would have), so UNDOC is closer to TEST than anything else.
We gathered data and made a hypothesis. Let's try testing our hypothesis. If we were to AND 0x11223344 and 0xeeddccbb, the result would be 0x00000000, this would set the Z (zero) flag. If this is a TEST, we would expect the flags for testing EAX of 0x11223344 and the immediate of 0xeeddccbb to set the Z flag. We can test this with UNDOC. Below is code to move the data for cleared flags into eax, push it to the stack, and pop it into our actual eflags (effectively clearing all the flags). We then move 0x11223344 into eax and run UNDOC EAX, 0xeeddccbb. Because this is an undocumented instruction and shows up as 'dw 0xc8f7', the data immediately after this get's interpreted by the debugger as the instruction it would be if it started with that data. We know that this will not actually be a 'mov ebx...' instruction.
Flags after running our 'dw' instruction.
UNDOC faithfully sets the Z flag.
Based on the data and expiriment, it seems as if f7c8 works the same as TEST. I wouldn't rely on the apparent operation of undocumented instructions, but it is sure fun to explore them.
If you want to explore some of these yourself, I recommend starting with all of the blank boxes in the tables of Appendix A of Vol 2 of the Intel manual. I will end with a screenshot of one of the tables, you can see the missing box between the TEST and NOT instructions. You will also notice the missing SAL in the empty box between SHR and SAR (one row above the ghost TEST).
