Execute/Run code after main() function returns

Recently one of my friends asked me whether it is possible to run/execute code after main() returns; provided the condition that we cannot use atexit() to register a function to run after main() returns.

Here’s my take at it:

#include <iostream>
using namespace std;

void lastfn(void)
{
 cout<<"This is executed after main()"<<endl;
 exit(0); //No more messing with the stack.
}

/*
Lets make main() 'naked' so that there is no prolog or epilog 
appended to the function. So when we enter into the function,
return address is on the top of the stack.
*/
__declspec (naked) void  __cdecl main(int count, char* vector[])
{
 long retaddr;

 __asm{
  pop retaddr  //pop the actual return address
  push lastfn  //push the new address where we 
     //want control after main
 }
 
 cout<<"Main routine"<<endl;

 __asm{
  retn
 }
}

A little explanation:
In this program, I have made the main() function naked. This will instruct the compiler NOT to emit the prolog and the epilog code. And as we know that it is the prolog code that make the stack frame for a new function call by pushing the frame pointer 'ebp' and making the new frame pointer equal to the current stack pointer 'esp'.

A general prolog looks something like this:

  push        ebp                ; Save ebp
  mov         ebp, esp           ; Set stack frame pointer
  sub         esp, localbytes    ; Allocate space for locals
  push                ; Save registers

The epilog code does the reverse of whatever has been done in the prolog code. A typical epilog looks something like below

pop     ; Restore registers
mov  esp, ebp      ; Restore stack pointer
pop  ebp           ; Restore ebp
ret                ; Return from function

Also as we know that when a function is called the stack looks something like this

High Address

|                   |
|-------------------|
|  Input arguments  |
|-------------------|
|  Return Address   |
|-------------------|
|   Saved ebp       |
|-------------------|
|  Local variables  | top of stack
|-------------------|
|                   |

Low Address

But without the prolog code the stack at the function call time looks something like this

High Address

|                   |
|-------------------|
|  Input arguments  |
|-------------------|
|  Return Address   | top of stack
|-------------------|
|                   |

Low Address

This is because there is no code emitted by the compiler that does the task of saving the frame pointer and making room for the local variables. So as we can see, the return address is at the top of the stack, which we can easily remove from the stack using a 'pop' and 'push' our address there.

PS: If someone reading this has a different solution or idea, please leave a comment.

"C" calling convention

So, what the hell is a calling convention ?

Calling convention is a scheme that specifies how the called function receives the parameter from the caller function and how it sends the results back to the caller function.

Calling conventions can differ in the following respects:

1. Setting up and clearing of the function call(stack frame) between the caller and the callee.
2. How the parameters are being passed from the caller to the callee; i.e. whether the parameters are passed in the registers or on the stack or on both.
3. The order in which the parameters are passed; i.e. from left to right or from right to left.

Here I am going to explain the C calling convention. The calling convention divides the rules between the caller and the callee.

Callers' rule:

1. Save the registers: EBX, ECX, EDX etc. so that their contents are not disturbed during the function call.
2. Push the arguments on the stack from right to left.
3. Call the subroutine.
4. Clean the stack.
5. Get the return status from the subroutine in EAX register.
6. Pop the registers back.

Callee's rule:

1. Save the EBP register, to preserve the stack frame.
2. Move the ESP register into EBP register, make a new stack frame.
Note: The caller does not expect the callee to change the EBP register.
3. Make room for the local variables. As with parameters the local variables will be accessed relative to the EBP register.
4. Save the "callee saved" registers,these are EDI and ESI
5. //Code the callee wants to execute.
6. Place the return value in the EAX register.
7. Restore the values of any "Callee saved" registers i.e. ESI and EDI. Registers must be popped in the reverse order of pushing.
8. Free up the local variables by adding some amount to the ESP register. A less error prone way is to "mov %esp %ebp".
9. Immediately before returning, restore the caller’s base pointer by popping the EBP register off the stack.

Remember, this was the first thing that we did, on entering the subroutine.

10. Finally we do a RET, this will pop off the return address and make a jump to it.

Hope this explains the C calling convention......