This work is just out of curiosity, since there is a MATLAB toolbox called xPC Target Embedded Option that basically doing the same thing.
The xPC Target, a component of the well-known MATLAB, is a product of Mathworks. It is tightly integrated into MATLAB and Simulink, and enables swift real-time system prototyping and deployment. Although xPC is very powerful, a bit of hacking may reveal more surprises. Starting from its floppy disk loading mechanism, a bit reverse engineering explored many aspects of the system, including loading kernel image, switching to 32-bit, setting up C runtime environment and starting service tasks, etc. I first found its originality. The kernel of xPC target is from On-time, GmbH, a company in Germany dedicated to embedded OS solutions. I am so lucky that there is no encryption, obfuscation or compression (they really should have this at least) in the kernel binary. Yet it still gave me a tough time because I have not even written such system software before. IDA and Bochs offer great help during this work. The outcome is listed below:
A patched kernel that loads DLM files from filesystem. Just type "xpcboot xpctgox.rtb <full path of dlm>" to run. After dlm exits, the system will hang and need reset. The command line can be put in autoexec.bat to achieve auto run at start up.
The modified kernel above only works for the executable (dlm files) generated from xPC in MATLAB R2011a.