IFEO and Managed-debugging

Be careful  of Image File Execution Options (IFEO) with managed debugging - it won't work like you expect.  

IFEO lets you set some registry goo such that when you launch a target app (specified by a registry key name), a debugger (specified by a string named "debugger" under that registry key) is executed instead. The debugger then launches the target app under its control.  (For more about IFEO: see MSDN for details, GreggM talks about debugger details. Junfeng talks about other IFEO tips; MSDN has some tips here; and Raymond Chen has more.)

MSDN warns this only works for native and interop-debugging. It does not work for managed-only debugging. Here's why...

First, look at a step-by-step walkthrough:

1. You setup a registry key for "MyApp.exe" with a string "debugger"="MyDebugger.exe".
2. You attempt to run MyApp.exe via Explorer.
3. Explorer makes a call to CreateProcess("MyApp.exe", flags=NotDebugging)
4. IFEO intercepts that CreateProcess call because flags=NotDebugging. It launches a debugger by concatenating the value in the "debugger" registry string with the parameters from CreateProcess. So it will actually launch: "MyDebugger.exe MyApp.exe" instead of launching MyApp.exe.  
5. It is expected that MyDebugger.exe will then use the command line args to launch a debuggee. Specifically, it will call CreateProcess("MyApp.exe", flags=Debugging).
6. Since that CreateProcess call specifies it is debugging, the call is not intercepted by IFEO and MyApp.exe is created under the debugger as normal.

GreggM discusses a lot of interesting ramifications of this.

So what's the problem for managed-debugging?
Managed-debugging is not built on native-debugging. Managed-debugging has its own debugging channel that built on its own interprocess-communication protocol, which is completely separate from the OS facilities used by native-debugging. That means that launching the debuggee under the managed-debugger will do CreateProcess("MyApp,exe", flags=NotDebugging).  This introduces infinite recursion with IFEO, because that will get intercepted by IFEO and relaunch the debugger. In other words, we'd loop forever between step 4 and step 5.

Interop-debugging is built on OS-facilities, and so looks like a native-debugger to the OS. This is why MSDN tells you to use interop-debugging with IFEO.  Another option may be to disable IFEO after the debugger is launched, but before it lauches the debuggee.

Comments

• Anonymous
  February 16, 2006
  You say to be careful with "managed debugging".  So, if I want to debug managed code using WinDbg/SOS, will it work as expected?
• Anonymous
  February 16, 2006
  Yes, windbg/sos is native-debugging and so everything should work as intended.
  I was focusing more on Visual Studio and ICorDebug-based debugging.
• Anonymous
  February 17, 2006
  So isn´t it possible to launch an exe file and debug it with Visual Studio debugger from the beginning as you can do with others debuggers ?
• Anonymous
  February 17, 2006
  Adrian -
  1. IFEO works great for native + interop-debugging; this is just an issue for managed-only debugging.
  2. This has nothign to do with VS. Any other managed debugger (eg, MDbg) would have the same problem.
  
• Anonymous
  September 08, 2006
  I'm putting this here so that I don't forget where to look for it later when I need it as it has...