Kernel Simulator

1. Introduction
2. Implementation
3. Building ksim
4. Using ksim
5. Possible uses for ksim
6. Future directions
7. Download ksim

One of the things that is especially painful about debugging the kernel is the inability to easily run the thing under a debugger. The information on this page describes a means by which you can set up a simple executable that is capable of exercising the bulk of the SCSI code.

This application can be run under gdb just like any other user-space application, and there is no danger of corrupting any data on your disk drives as there could be if you screwed something up in a real kernel.

At the center of this beast is the scsi_debug "host" adapter. It isn't a real host adapter, but it essentially fakes it. It can be programmed to pretend that there are some number of devices present on the bus, and to respond to the various commands that come down from above. With scsi_debug, there is no actual hardware involved.

The scsi code above this doesn't really care that much whether it is running in a user-space executable or an actual kernel. Same goes for ll_rw_blk, the buffer cache, and block_dev.c for that matter. The main trick is to implement a substitute memory manager that is compatible with a user-space executable (this would typically allocate a few MB of memory, and then dole it out as if it were the pages on a real machine. The other tricky bit is that you need to implement a means for handling multiple kernel threads - to do this, I implemented a simple non-preemptive pthreads-like scheduler which can be used to switch from task to task. This itself is trivially implemented on top of setjmp/longjmp. The kernel itself doesn't preempt itself, so this is a perfectly good model.

The actual code outside of the kernel consists of two source files. One of these compiles against kernel headers and data structures, the other one compiles against the normal user space headers. This split is needed as we need to supply stub functions for a number of things in the kernel, and we also need to hook into stdio so that printk essentially dumps the output to stderr. These source files also contain a simple command line parser which then exercises various bits of code in the actual kernel.

The actual user-space program is built using the two source files mentioned above, plus a *handful* of actual kernel object files. This is the bit that gets really ugly, and a lot of this is because we need to provide a memory manager and scheduler surrogate which will allow the thing to run in user space.

To build the simulator, you must first start with a pristine kernel for a version that goes along with the emulator source you downloaded. Choosing a different version is not recommended for beginners - this thing is extremely fragile (mainly the memory management and scheduling hacks), and to get the thing to work with a new kernel version can take a bit of fiddling.

You then apply the kernel patches which come with the simulator. The patches do a couple of things - first turn on debugging when you compile the kernel (otherwise gdb won't be of any use). Secondly, we change out a couple of instructions (cli, sti, etc) into nops, since these would fault if attempted in a user space program. A few other very minor changes, and there is also a hook in scsi_debug so as to allow the top level driver to "load" it as if it were a module.

You must then configure the kernel - for this purpose it works best to turn on SMP (so spinlocks work correctly, and when the thing deadlocks you can see where you screwed up), turn on CONFIG_USER_DEBUG, turn on the scsi_debug host adapter along with the high level SCSI drivers you are interested in, and finally and turn off most of the other crud that are not of interest (i.e. filesystems, networking, etc). Build the kernel - the build will fail while attempting to link vmlinux - don't worry about this, as this is normal.

The next step is to go over to the actual emulator source tree, and configure the Makefile - in particular you need to properly indicate the head of the linux source tree that you just built.

Finally, build it. The program name is "ksim" - it should compile and link cleanly. The makefile will use a couple of selected .o and .a files from the kernel tree - just enough to get the thing to build. Getting the right mix of kernel objects is something else that is kind of a black art, and it can change from one kernel release to the next (i.e. you do whatever it takes to get the damned thing to link).

To use ksim, simply run it, either standalone, or under the debugger. You should see output appear on the screen not unlike what you get at boot time (i.e. it should recognize the scsi_debug host adapter, and scan the bus). You should then end up with a "ksim>" prompt. At this point there are a couple of things you can do - type "help" for a list of commands.

An example command would be "read". This will issue read commands through block_read, the buffer cache, ll_rw_blk.c, sd.c, scsi.c, and finally scsi_debug.c. If you wish, you can step through this with the debugger to follow the control flow.

Let me toss out a few ideas for how to test various things in the kernel.

1. It would be possible to test error handling by configuring scsi_debug to "fail" in certain predictable ways. I have already done this to some extent in testing the new queuing code, but there is always room for more testing. In particular, you can introduce failure modes where the device "locks up" when you attempt to read a specific block from the disk, and then you can step through the code that the error handler thread uses to attempt to clear the condition. It would also be quite easy to introduce a bad block which would return MEDIUM_ERROR if you attempted to read it.
2. It would be possible to do benchmarking, and even profiling to identify hot spots. The only thing that we are not really able to measure is the real latency that would be seen for a real command on a real bus - we don't have much control over that anyways (other than to make sure that we aren't stupid and send commands down the wire that we don't need to send).
3. It is possible to use the thing simply as a learning tool to learn how the SCSI code actually works.
4. I believe it would be possible to use a similar concept for testing filesystems. For something like this, you would probably want to instead hook into a sort of ramdisk-like driver that would write changes back to a normal disk file (or a naked partition) such that the changes would be persistent.

This thing is a real pain in the neck to maintain. Seemingly trivial changes to the kernel can break this quite easily, and the trick is usually to simply hack together enough code such that you get the thing to link. I am going to toss out an idea for how to improve this in a more maintainable fashion.

What we should really do (left as an exercise for the reader) is to define a new processor type which could be used for simulation. This would avoid the nasty hacks in the asm-i386 directory, and the rest of the kernel would essentially compile without any fuss. The arch/simulator directory wouldn't need to do anything other than supply the bare minimum of glue to hold the thing together, and the makefile in the arch directory would build a shared library. The actual driver program at the top level would just call into the shared library at various points to actually do the things that needed to be done. I will lay out sort of a general plan of action here (not unlike porting to a new processor architecture).

1. Start by populating an asm directory. You could start with asm-generic (no point in having any processor-specific instructions). Fill in as required.
2. Fix the configuration to recognize "simulator" as a valid processor architecture.
3. Next, start to populate the arch/simulator, as required. My guess is that this should be pretty lightweight. The main trick here is that some of the guts for memory management simulation and thread scheduling would live in here (copy this stuff from ksim itself). Some of the sources would compile against system headers, others against kernel headers. The ultimate goal would be to form a shared library. Ideally I would like to avoid any host-processor specific stuff in here (no .s files) if possible.
4. Finally, create a driver program that exercises this kernel in ways that are not unlike what ksim is doing. This would link against the shared library, and call into it in various places.

Whether or not this ever happens remains to be seen, and it depends a lot on the degree of interest from the general kernel-hacker community. If nobody else really cares, I would certainly never get around to it, and nobody else would either. If this thing is viewed as the coolest thing since sliced bread, then I would be more motivated to either attempt such a feat or work with someone else who had more time.

Last updated: 9/8/99.