x86 Calling Conventions

I have used the Wikipedia article as my starting point. Then I have added some information that I have found experimentally (by looking at the generated ASM code) and I've written this article in a form that was useful to me.

Introduction

A calling convention determines the way parameters and return values are passed to/from functions. All calling conventions use the stack for parameters; additionally, processor registers can be used.

Call Stack

The call stack is a special memory area, storing:

• Function parameters and return values
• Local variables
• Additional function call information (address of the instruction to return to, original stack pointer etc.)

The processor has special instructions for working with the stack; notably PUSH, POP, CALL and RET.

The x86 call stack grows, perhaps counter-intuitively, from higher memory addresses to lower ones.

Since x86 is a 32-bit architecture, everything on the stack is in 32-bit chunks:

• Values that have less than 32 bits are expanded to 32-bits
• Values that have more than 32 bits are split into 32 bit chunks.
• Structures with sizes that are not a multiple of 32 bits are padded to the nearest multiple of 32 bits.

Parameter Order

The calling convention determines the order in which the parameters are pushed on the stack. It can be either left-to-right or right-to-left.

For example, given the function call:

f(p1, p2, p3, p4, p5)

We can pass the parameters in two ways:

Left-to-Right

Top	Lower address
p1	↑ Stack grows to lower addresses
p2
p3
p4
p5
Bottom	Higher address

Right-to-Left

Top	Lower address
p5	↑ Stack grows to lower addresses
p4
p3
p2
p1
Bottom	Higher address

The left-to-right (a.k.a. Pascal) order is the obvious order. However, the right-to-left order (a.k.a. C) has the advantage that it supports functions with variable number of parameters (of course, for this to work, the caller must clean up the stack). This enables C to have functions like printf.

Stack Clean-up

Either the calling or the called function can restore the stack to its position before the call (i.e. by changing the ESP register).

It is more convenient if the called function restores the stack. However, for functions with variable number of arguments, this doesn’t work anymore, because the called function cannot know how many arguments were passed. Instead, the calling function must clean up the stack.

Function Name Decoration (“Mangling”)

While not part of the calling convention, per se, compilers decorate the function names according to their calling convention. For example, void Fn(int) is decorated to _Fn@4 in the stdcall calling convention (the number after the @ sign is the number of bytes on the stack).

When exporting functions from a DLL, the exported name of the function can be changed (it’s usually exported without any decoration).

stdcall Calling Convention

This is the calling convention used by the Windows API.

Decoration: names are prefixed with an underscore (_) and are followed by an @ sign and the number of bytes required on the stack; for example: void Fn(int) is decorated to _Fn@4.
Note: the return value is not counted, even when it takes up space on the stack.

Arguments are passed right-to-left (C order)¹; the called function cleans up the stack.

¹ However, the 16-bit Windows API used the left-to-right (Pascal) order for arguments.

All arguments are passed on the stack.

• Integers: Arguments smaller than 32 bits are enlarged to 32 bits. 64 bits arguments are passed as 2 32-bit values; first upper half, then lower half. That means that they are in the normal order in the stack memory (i.e. little endian).
• Floats are 32 bits wide; they are passed in one stack position.
• Doubles are 64 bits wide; they are passed in two stack position (similar to 64 bit integers).
• Pointers are treated as 32 bit integers.
• Arrays are passed by reference, i.e. a pointer to their start is passed as a 32 bit value.
• Structures are just copied on the stack, extended to a multiple of 32 bits.

Return values:

• 32 bit integer results (this includes pointers) are returned in the EAX register. Integers shorter than 32 bits are extended to 32 bits.
• 64 bit integer results are returned in EAX (lower half) and EDX (upper half).
• Floating point arguments are returned in ST(0), which is 80 bits wide. From here, they can be copied to float or double variables, with a corresponding precision loss.
• Structures:
  • 1, 2, 4, 8 byte structures are returned in EAX and EDX (upper half, if necessary) but not 3, 5, 6, 7 byte structures!
    NOTE: The IBM Visual Age C++ 3.5 compiler doesn’t do this; it just puts the pointer on the stack! Also the functions don’t take this into account when cleaning up.
  • Everything else is not returned per se, instead, the caller must push a pointer to their start on the top of the stack.

cdecl Calling Convention

This is the default calling convention used by the Microsoft Visual C++ compiler; it’s also supported by most compilers on Windows.
(Differences from the stdcall calling convention are underlined.)

Decoration: names are prefixed with an uderscore (_); for example: void Fn(int) is decorated to _Fn. (But, according to MSDN, they are not decorated when exported from DLLs.)

Arguments are passed right-to-left (C order)¹; the calling function cleans up the stack – this enables calling functions with a variable number of parameters.

All arguments are passed on the stack.

• Integers: Arguments smaller than 32 bits are enlarged to 32 bits. 64 bits arguments are passed as 2 32-bit values; first upper half, then lower half. That means that they are in the normal order in the stack memory (i.e. little endian).
• Floats are 32 bits wide; they are passed in one stack position.
• Doubles are 64 bits wide; they are passed in two stack position (similar to 64 bit integers).
• Pointers are treated as 32 bit integers.
• Arrays are passed by reference, i.e. a pointer to their start is passed as a 32 bit value.
• Structures are just copied on the stack, extended to a multiple of 32 bits.

Return values:

• 32 bit integer results (this includes pointers) are returned in the EAX register. Integers shorter than 32 bits are extended to 32 bits.
• 64 bit integer results are returned in EAX (lower half) and EDX (upper half).
• Floating point arguments are returned in ST(0), which is 80 bits wide. From here, they can be copied to float or double variables, with a corresponding precision loss.
• Structures:
  • 1, 2, 4, 8 byte structures are returned in EAX and EDX (upper half, if necessary) but not 3, 5, 6, 7 byte structures!
  • Everything else is not returned per se, instead, the caller must push a pointer to their start on the top of the stack.

optlink Calling Convention

This is the default calling convention used by the IBM Visual Age compiler.
(Differences from the stdcall calling convention are underlined.)

Decoration: names are prefixed with a question mark (?); for example: void Fn(int) is decorated to ?Fn.

Arguments are passed right-to-left (C order)¹; the calling function cleans up the stack – this enables calling functions with a variable number of parameters.

All arguments are passed on the stack and in certain registers.

• Integers: Arguments smaller than 32 bits are enlarged to 32 bits. 64 bits arguments are passed as 2 32-bit values; first upper half, then lower half. That means that they are in the normal order in the stack memory (i.e. little endian).
• Floats are 32 bits wide; they are passed in one stack position.
• Doubles are 64 bits wide; they are passed in two stack position (similar to 64 bit integers).
• Pointers are treated as 32 bit integers.
• Arrays are passed by reference, i.e. a pointer to their start is passed as a 32 bit value.
• Structures are just copied on the stack, extended to a multiple of 32 bits.

The first 3 integer values are passed in the EAX, EDX and ECX registers (1st argument in EAX, 2nd in EDX, 3rd in ECX). A 64 bit value is treated simply as 2 32-bit values; if the 3rd argument is a 64-bit integer, EDX will contain the lower half and the rest will be on the stack. The first 4 floating point values are passed in four floating point registers: 1st value in ST(0), 2nd in ST(1), 3rd in ST(2) and 4th in ST(3). These registers are 80-bit wide, so all values take up just one register.

If all values fit into registers, nothing is passed onto the stack. However, if they don’t, space is left on the stack, in the regular positions, even for the arguments passed in registers. (The IBM compiler doesn’t initialize this stack space.)

Return values:

• 32 bit integer results (this includes pointers) are returned in the EAX register. Integers shorter than 32 bits are extended to 32 bits.
• 64 bit integer results are returned in EAX (lower half) and EDX (upper half).
• Floating point arguments are returned in ST(0), which is 80 bits wide. From here, they can be copied to float or double variables, with a corresponding precision loss.
• Structures are not returned per se, instead, the caller must push a pointer to their start on the top of the stack, regardless of their size.