x86 Instructions and the Stack

The core instruction set you’ll actually see in a disassembler, how the stack is organized, and how arguments and return values get passed between functions. Builds directly on the registers, flags, and addressing modes introduced in x86 Assembly Fundamentals.

Data movement

  • mov dst, src — copy, no flags affected. The workhorse instruction; unlike ARM’s mov it also handles memory↔register and memory↔immediate, since x86 doesn’t force a separate load/store step.
  • lea dst, [expr] — load effective address: computes the address expression and stores the address itself, without dereferencing it. Despite the addressing-mode syntax, lea never touches memory. Compilers abuse it constantly as a free 3-operand adder/shifter with no flag side effects, e.g. lea eax, [rbx+rcx*4+8] computing rbx + rcx*4 + 8 into eax — seeing lea in the middle of otherwise-unrelated arithmetic is a strong tell that this is exactly what’s happening, not an actual address load.
  • push src / pop dst — stack operations, covered below.
  • movzx dst, src / movsx dst, src — move with zero-extend / sign-extend, used when moving a narrower value (byte/word) into a wider register while controlling what fills the upper bits — the explicit x86 equivalent of ARM’s uxtb/sxtb.

Arithmetic and logic

InstructionEffectFlags set
add dst, srcdst += srcZF, SF, CF, OF
sub dst, srcdst -= srcZF, SF, CF, OF
inc dst / dec dstdst += 1 / dst -= 1ZF, SF, OF (not CF)
imul/mulsigned/unsigned multiplyCF, OF only
idiv/divsigned/unsigned divide (rdx:rax / divisor → quotient in rax, remainder in rdx)undefined
and/or/xor dst, srcbitwiseZF, SF (CF/OF cleared)
not dstbitwise complementnone
neg dsttwo’s-complement negateZF, SF, CF, OF
shl/shr dst, nlogical shift left/rightCF, and ZF/SF for the result
sar dst, narithmetic shift right (sign-preserving)same

xor eax, eax is the idiomatic x86 way to zero a register — one byte shorter to encode than mov eax, 0 and the reason you’ll see it constantly instead of an explicit zero move.

Note idiv’s calling shape: unlike ARM’s sdiv, x86 division takes an implicit 128-bit (rdx:rax) or 64-bit (edx:eax) dividend split across two registers — a compiler emitting cdq (sign-extend eax into edx:eax) or cqo (64-bit equivalent) immediately before an idiv is just setting up that implicit operand, not doing something semantically separate.

Comparison and conditional jumps

cmp a, b performs a - b and sets flags without storing the result — purely for the flags. test a, b similarly performs a AND b and discards the result, most often seen as test eax, eax to check if eax is zero/negative (cheaper to encode than cmp eax, 0).

Every conditional jump reads some combination of flags set by the preceding cmp/test/arithmetic instruction — there is no ARM-style explicit condition-code suffix on the instruction that sets the flags, so you always have to look one instruction upward to know what a jcc is actually testing:

MnemonicJumps ifTypical source
je / jzZF=1a == b
jne / jnzZF=0a != b
jg / jnleZF=0 and SF=OFsigned a > b
jge / jnlSF=OFsigned a >= b
jl / jngeSF≠OFsigned a < b
jle / jngZF=1 or SF≠OFsigned a <= b
ja / jnbeCF=0 and ZF=0unsigned a > b
jae / jnbCF=0unsigned a >= b
jb / jnaeCF=1unsigned a < b
jbe / jnaCF=1 or ZF=1unsigned a <= b

The signed/unsigned split (jg vs ja, etc.) exists because x86 has no single “greater than” — the compiler picks based on the operands’ declared C type, and that choice is itself a useful reverse-engineering signal (an ja/jb pair on a value you suspected was int tells you it’s actually unsigned or a pointer/size comparison).

jmp is the unconditional counterpart, and is exactly what a crackme patch (see x86 Reverse Engineering for Games) most often replaces a jcc with, or NOPs out entirely.

The stack

The stack grows downward — push decrements rsp then stores, pop loads then increments rsp — the opposite direction from how the stack is usually drawn, but the same convention ARM’s full-descending stack uses, so this part should feel familiar:

push rbp        ; rsp -= 8;  [rsp] = rbp
pop  rbp        ; rbp = [rsp];  rsp += 8

call target pushes the return address (the instruction after the call) and jumps; ret pops that address back into rip. This is the concrete reason stack corruption (writing past a local buffer) can hijack control flow — the return address lives on the stack right next to local variables, with no separate link register the way ARM’s lr keeps it out of memory unless explicitly spilled.

Stack frames

A classic (frame-pointer-based) function prologue/epilogue:

; prologue
push rbp
mov  rbp, rsp
sub  rsp, 0x20      ; reserve space for locals

; ... function body, locals accessed as [rbp-8], [rbp-0x10], ...
; ... arguments (if spilled) accessed as [rbp+0x10], [rbp+0x18], ...

; epilogue
leave               ; equivalent to: mov rsp, rbp / pop rbp
ret

rbp here is a fixed anchor for the duration of the function, so locals and spilled arguments can be addressed with a constant offset regardless of how rsp moves during the function (e.g. from further nested push/call). Modern optimizing compilers frequently omit this (-fomit-frame-pointer, on by default at -O2+ and in most release game builds) and address everything relative to rsp directly instead, freeing rbp as a 17th general-purpose register — when you see rbp used as an ordinary scratch register in a disassembly, that’s what happened, and the practical effect is that you lean on the disassembler’s stack-frame recovery (Ghidra/IDA both do this well) rather than tracking rsp offsets by hand.

Calling conventions

The convention dictates which registers hold arguments, which hold the return value, and which registers a callee must preserve. Reading disassembly without knowing the active convention means misreading which register is “argument 1” versus a scratch temporary.

System V AMD64 (Linux, macOS — and increasingly what you’ll see if a game ships on Steam/Proton or is built cross-platform):

PurposeRegisters
Integer/pointer args, in orderrdi, rsi, rdx, rcx, r8, r9, then stack
Float/double argsxmm0–xmm7
Return valuerax (rax:rdx for 128-bit)
Caller-saved (clobbered by calls)rax, rcx, rdx, rsi, rdi, r8–r11
Callee-saved (must be preserved)rbx, rbp, r12–r15
Extra128-byte “red zone” below rsp usable without adjusting rsp, for leaf functions

Microsoft x64 (Windows — the convention you’ll spend the most time in for PC games):

PurposeRegisters
Integer/pointer args, in orderrcx, rdx, r8, r9, then stack
Float/double argsxmm0–xmm3 (same slot number as the integer args — arg 2 is either rdx or xmm1, never both)
Return valuerax
Caller-savedrax, rcx, rdx, r8–r11
Callee-savedrbx, rbp, rdi, rsi, r12–r15
ExtraCaller must reserve 32 bytes of “shadow space” on the stack even if all args fit in registers, for the callee to optionally spill them into

The older 32-bit conventions still matter for legacy/32-bit game binaries:

  • cdecl — all args on the stack, pushed right-to-left; caller cleans the stack after the call. Return value in eax.
  • stdcall — all args on the stack, but the callee cleans the stack (ret 0xN popping N bytes) — the standard Win32 API convention.
  • thiscall — MSVC C++ member functions: this pointer in ecx, remaining args on the stack. Recognizing a mov ecx, obj_ptr immediately before a call is one of the fastest ways to spot a C++ method call versus a plain function call in a 32-bit binary — see Reading Compiled Code Patterns for the 64-bit (rcx-as-first-arg, indistinguishable from a normal first integer arg) equivalent.

A quick way to tell Windows x64 code from Linux x64 code purely from a disassembly, without any symbol/string context: if the first two integer-ish arguments before a call land in rcx, rdx, it’s Microsoft x64; if they land in rdi, rsi, it’s System V.

See also