Dude, where's my main?

So I was writing a debugger. You know the sort of thing: breakpoints, stepping, checking the value of variables. It was going wonderfully until I tried to debug my debugger with my debugger. main was just... gone. It would run, but trying to set a breakpoint on it crashed the program. It took me weeks to figure out why. This is that story.


The program we're going to use to demonstrate the debugger is a classic C "Hello, world!":

// programs/hello.c
#include <stdio.h>

int main(void) {
  printf("Hello, world!\n");
  return 0;

Let's compile this program with debug information and do a few basic operations in the debugger. It's written in Rust, so we invoke it by using Rust's build tool, cargo:

$ gcc -g programs/hello.c -o programs/hello
$ cargo run -- programs/hello
> symbol main
0x401c60 main
> register rip
rip 0x401b30
> break main
> cont
> register rip
rip 0x401c60
> cont
Hello, world!

Our "Hello, world!" program is initially in a stopped state at some address that isn't main, so we set a breakpoint on main, continue, and then verify that we are at the first instruction of main. Then we continue the program and we see it print out "Hello, world!" as expected. Neat!

This got me thinking: maybe I could debug my debugger in my debugger. That would be awesome.

$ cargo run -- target/debug/rust-debugger
> symbol main
0xd2f90 main
> register rip
rip 0x7fb95aa0c100
> break main
libc err: Input/output error

Oh. What happened?

Understanding the problem

The error message we get comes from a piece of code in my debugger that takes any error code we get from a libc function and runs it through strerror. So which libc function did it come from?


The only libc function that gets called when setting a breakpoint is the ptrace function. ptrace is a Linux-specific API that debuggers use to control the processes they're debugging. With it you can read and write memory, read and write registers, advance by 1 instruction, and loads more.

When you set a breakpoint in the debugger, the debugger writes 0xCC to the address we want to pause execution at. 0xCC is an x86 instruction that triggers a software interrupt specific to debuggers. It stops the debuggee and returns control to the debugger.

The error string "Input/output error" corresponds to the error code EIO. Checking the man page for ptrace and looking for EIO tells us ptrace will return EIO when:

Request is invalid, or an attempt was made to read from or write to an invalid area in the tracer's or the tracee's memory, or there was a word-alignment violation, or an invalid signal was specified during a restart request.

We can rule out the request being invalid, as it worked fine in our other program. Same for word-alignment violation. We're not doing anything funky with signals, so that leaves us with a read or write to an invalid area in the debuggee's memory.

Let's look again at the output we got in our debugger:

> symbol main
0xd2f90 main
> register rip
rip 0x7fb95aa0c100

There's a huge difference between the address of the main symbol and the value of the instruction pointer at the start of the program's execution. The difference was much smaller when debugging our "Hello, world!" program, so where does the address for main come from?

Debug info

There are a couple of ways to get the address of a function in a compiled program. The method I went with is to use the debugging info compilers provide you with when you compile with the -g flag.

This debug info is in a format called DWARF. DWARF is a big and complex beast, and with good reason! It has to describe where functions have been inlined, which file each function was defined in, where functions start and begin, where to find every local variable in every function, and a whole lot more.

We're only interested in where functions begin. DWARF debug info is in a tree-like structure, much like code is, and each type of node is identified with a "tag." The tag that represents a function definitoin is DW_TAG_subprogram:

$ readelf --debug-dump programs/hello
 <1><2a>: Abbrev Number: 2 (DW_TAG_subprogram)
    <2b>   DW_AT_low_pc      : 0x401c60
    <33>   DW_AT_high_pc     : 0x11
    <39>   DW_AT_name        : main

The value of DW_AT_low_pc looks familiar. It's the address we found main at when we ran this program in our debugger, and that's because this is exactly where we get it from. Let's check for main in our debugger:

$ readelf --debug-dump target/debug/rust-debugger
 <2><804db>: Abbrev Number: 16 (DW_TAG_subprogram)
    <804dc>   DW_AT_low_pc      : 0xd2f90
    <804e4>   DW_AT_high_pc     : 0x2ef
    <804ee>   DW_AT_name        : main

That address matches up as well, except this time it's wrong. What's different about rust-debugger for this to not be where main ends up?

I was stumped by this for a good while before noticing a difference in the output of readelf -h:

$ readelf -h programs/hello
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 03 00 00 00 00 00 00 00 00
  Class:               ELF64
  Data:                2's complement, little endian
  Version:             1 (current)
  OS/ABI:              UNIX - GNU
  ABI Version:         0
  Type:                EXEC (Executable file)
  Machine:             Advanced Micro Devices X86-64
  Version:             0x1
  Entry point address: 0x401b30
$ readelf -h target/debug/rust-debugger
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:               ELF64
  Data:                2's complement, little endian
  Version:             1 (current)
  OS/ABI:              UNIX - System V
  ABI Version:         0
  Type:                DYN (Shared object file)
  Machine:             Advanced Micro Devices X86-64
  Version:             0x1
  Entry point address: 0x93160

What caught my attention was the Type field. In programs/hello it's a plain old executable, but in target/debug/rust-debugger it's a shared object. Aren't shared objects the type for libraries? Why would an executable be reporting itself as a shared object?

It turns out that this is how "position-independent executables" look. They're executables that have been compiled in such a way that they can live in any part of the address space. This is achieved by making sure there are no absolute memory references. Instead, all memory references are done by making them relative to the current instruction pointer.

This triggered an old memory: is this somehow related to "address space layout randomisation" (ASLR)? I've done some reverse engineering challenges in my time, and know that one of the ways operating systems try to protect programs from attackers is to load them in to a random part of the address space. How could I confirm this?

$ cargo run -- target/debug/rust-debugger
> register rip
rip 0x7f11d4997100
$ cargo run -- target/debug/rust-debugger
> register rip
rip 0x7f810cd16100
$ cargo run -- target/debug/rust-debugger
> register rip
rip 0x7fc08a2b8100

Well, that's good enough for me. It looks like it's only random in a portion of the address space. Still, it changes each time. Can we disable this?

It turns out we can! Linux has the concept of "personalities" for processes. From what I can see in the man page this seems to be some sort of compatibility layer for running programs meant for non-Linux systems. Fortuantely for us, it also lets us disable ASLR. Children inherit their parent's personality, so if we do the equivalent of the following C snippet in our Rust code:

#include <sys/personality.h>

int main() {

Our child should be loaded at a non-random address. Let's try it!

$ cargo run -- target/debug/rust-debugger
> register rip
rip 0x7ffff7fd3100
$ cargo run -- target/debug/rust-debugger
> register rip
rip 0x7ffff7fd3100
$ cargo run -- target/debug/rust-debugger
> register rip
rip 0x7ffff7fd3100

The randomization is gone, but our process is still loaded at an adress that doesn't line up with the addresses in the binary. This isn't the answer to our problem.

Finding the solution

My search eventually led me to this patch to gdb from 2009. It's a patch introducing support for position-independent executables. My thinking was that gdb works on these binaries, so there must be something it does to correct the symbol addresses.

Combing through the patch, I found a function called svr4_relocate_main_executable. It has a huge comment above it that explains some mathematics you can do when the entrypoint of the running process is not the same as the entrypoint given in the binary on disk. Promising.

However, the comment also explains that this mathematics doesn't work if your binary has an "interp section."

$ readelf -S target/debug/rust-debugger | grep interp
  [ 1] .interp           PROGBITS         00000000000002e0  000002e0

Rats. I spent some time trying to get it work despite this, but I got nowhere. I would either get the same error as before, or no error but the breakpoint would never get hit. I assume I was modifying a valid but arbitary part of memory.

A little more searching later and I stumbled on this StackOverflow answer. At the very end of it the user mentions how GDB knows where the binary has been loaded:

This begs the question: how does GDB discover where ld-linux itself has been loaded. The kernel tells it via the auxiliary vector, AT_BASE entry.

Auxillary vector

I'd never heard the term "auxillary vector" before but at this point I was happy to follow any lead.

When the operating system loads a program from disk in to memory and prepares to execute it, it passes in some key pieces of information. Most people are familiar with the arguments passed in from the command line (argv) and environment variables (envp). These values are found on the stack, just above where the stack begins when execution is passed to main.

Just above argv and envp, though, is our newly discovered auxv. We can use the environment variable LD_SHOW_AUXV=1 to print the contents of the auxillary vector for any program:

AT_BASE:              0x7f0790d8f000
AT_FLAGS:             0x0
AT_ENTRY:             0x5641572c6260
AT_PLATFORM:          x86_64
Cargo.lock  programs/  target/
Cargo.toml  src/

I've trimmed the output for brevity. There's a lot of info in there. The things that are most interesting to us are the AT_BASE and AT_ENTRY values. Let's fire up our debugger with this environment variable and see what it says:

$ LD_SHOW_AUXV=1 cargo run -- target/debug/rust-debugger
AT_BASE:              0x7f80cd45f000
AT_ENTRY:             0x55ba24f62160
> cont
AT_BASE:              0x7ffff7fd1000
AT_ENTRY:             0x5555555d9160

The first auxv values are those of the debugger process, and we have to continue the debuggee process in order to see its auxillary vector because it starts in a paused state. Armed with this, and the knowledge it won't move around because we've disabled ASLR, we can try setting some breakpoints:

> register rip
rip 0x7ffff7fd3100
> break 5555555d9160
> cont
> register rip
rip 0x5555555d9160

I won't lie, I lost my shit when the breakpoint worked. We're not there yet, though. Are we actually at main? How would we know?

A function of the debugger I haven't mentioned until now is being able to disassemble things. If you run disas with no arguments, it disassembles from the current instruction pointer:

> disas
0x5555555d9160 f30f1efa       endbr64
0x5555555d9164 31ed           xor ebp,ebp
0x5555555d9166 4989d1         mov r9,rdx
0x5555555d9169 5e             pop rsi
0x5555555d916a 4889e2         mov rdx,rsp
0x5555555d916d 4883e4f0       and rsp,0FFFFFFFFFFFFFFF0h

We can use the objdump command to see if this matches up with our main function:

$ objdump -d target/debug/rust-debugger | less
00000000000b0a60 <main>:
   b0a60:       48 83 ec 18             sub    $0x18,%rsp
   b0a64:       8a 05 14 e5 48 00       mov    0x48e514(%rip),%al
   b0a6a:       48 63 cf                movslq %edi,%rcx
   b0a6d:       48 8d 3d ac f8 ff ff    lea    -0x754(%rip),%rdi
   b0a74:       48 89 74 24 10          mov    %rsi,0x10(%rsp)
   b0a79:       48 89 ce                mov    %rcx,%rsi
   b0a7c:       48 8b 54 24 10          mov    0x10(%rsp),%rdx
   b0a81:       88 44 24 0f             mov    %al,0xf(%rsp)
   b0a85:       e8 d6 6d ff ff          callq  a7860 <_ZN3std2rt10lang_start17h004f6ce3435c8ab1E>
   b0a8a:       48 83 c4 18             add    $0x18,%rsp
   b0a8e:       c3                      retq

What?! They don't match. How can this be? I was so sure.

It took me a few minutes to calm down from the shock before I remembered that programs don't actually start at main. There's a little bit of code that runs before main called _start:

$ objdump -d target/debug/rust-debugger | less
0000000000085160 <_start>:
   85160:       f3 0f 1e fa             endbr64
   85164:       31 ed                   xor    %ebp,%ebp
   85166:       49 89 d1                mov    %rdx,%r9
   85169:       5e                      pop    %rsi
   8516a:       48 89 e2                mov    %rsp,%rdx
   8516d:       48 83 e4 f0             and    $0xfffffffffffffff0,%rsp

It matches! Yes!

Because we know that the whole executable gets shifted by the same amount, because instruction pointer relative addresses wouldn't work otherwise, we can do a little bit of mathematics here to figure out what the address of main should be:

0x5555555d9160 - 0x85160 + 0xb0a60 = 0x555555604a60

Let's try it:

> break 555555604a60
> cont
> register rip
rip 0x555555604a60
> disas
0x555555604a60 4883ec18       sub rsp,18h
0x555555604a64 8a0514e54800   mov al,[rel 555555A92F7Eh]
0x555555604a6a 4863cf         movsxd rcx,edi
0x555555604a6d 488d3dacf8ffff lea rdi,[rel 555555604320h]
0x555555604a74 4889742410     mov [rsp+10h],rsi
0x555555604a79 4889ce         mov rsi,rcx
0x555555604a7c 488b542410     mov rdx,[rsp+10h]
0x555555604a81 8844240f       mov [rsp+0Fh],al
0x555555604a85 e8d66dffff     call 00005555555FB860h
0x555555604a8a 4883c418       add rsp,18h
0x555555604a8e c3             ret

We did it! \o/

Wrapping up

I learned a lot on this journey. I learned that Linux processes have "personalities" that modify their runtime environment. I learned that executables can be position-independent just like libraries, and this is part of how address space layout randomisation works.

Last and certainly not least, I learned that Linux tells a program things about its runtime environment through a thing called an "auxillary vector," which lives on the stack just above the environment variables, and that's how gdb knows where to find main in position-independent executables.

Now I know all of this, I can go ahead and fix my breakpoint code so that it works with position-independent executables.


The code for the debugger is available on GitHub, and if you're interested in the exact state things were in when I wrote this article you can check it out like so:

$ git clone https://github.com/samwho/rust-debugger
$ cd rust-debugger
$ git checkout 00bcdd4772496913d1417feb7f8886d40aed46d5

The commits I made that fixed our debugger for position independent executables are here and here.