Using QEMU for CS 45 Linux Kernel Projects

Contents

What is QEMU?
Setting up QEMU on the PCs
Running QEMU (logging in, staring qemu, staring an OS, shutting down an OS, exiting qemu)
Compiling, Installing and Booting your Linux kernel changes from with in QEMU
Debugging the kernel
Connecting to a QEMU VM from a remote machine
Troubleshooting QEMU and Linux

QEMU

QEMU is a processor emulator that we are using to boot and run the Linux 2.6.32.44 kernel. You will do your kernel development on the lab machines (modifying the kernel source and rebuilding it), and then you will install, boot, and test your modified kernel on the qemu virtual machine (VM). As a result, if something goes wrong with your modified kernel you do not crash the entire machine but just the qemu VM. The QEMU Homepage

Setting Up QEMU on the PCs

What you need to do to set up qemu on a PC so that you and your project partner can start working.

Logging In

You and your partner will be assigned one of the PCs in the CS Lab on which you will store your virtual qemu machine and kernel source in a subdirectory in /local that only you and your partner can access. Even though you can only run qemu on the machine on which you are assigned (since it is the only one, and should be the only one, with your qemu machine in /local), you can ssh into your CS lab machine from any other lab machine and run qemu remotely.

For example, if the PC's name is cream, then you would type the following to connect (-X enables X forwarding, use -Y if ssh'ing from MAC-OSX): 

    % ssh -X cream
The -X option enables X forwarding in a secure way.

Steps for setting up your QEMU environment

First, get your qemu machine assignment. The qemu image is set up with one user: cs45. The password for both root and cs45 is swatcs.45.

Next, only one of you or your partner should do the following (you and your partner are sharing the same qemu files and kernel source, so it is up to you to coordinate changes that you make to it):

  1. ssh into the PC to which you have been assigned: 
      
    % ssh -X my_pc
  2. Make a working_dir directory in /local that you and your partner will share. For example, if my working_dir is named tia_n_pal: 
      
    % mkdir /local/tia_n_pal 
    % chmod 700 /local/tia_n_pal

  3. Copy over the linux kernel and the Debian image file (located in /scratch/vm into your /local/working_dir subdirectory 
        % cp /scratch/vm/linux-2.6.32.44.tar.bz2  /local/tia_n_pal/. 
    % cp /scratch/vm/qemu/cs45f11_debian.img   /local/tia_n_pal/.

  4. Run easyfacl.py to set acls on your shared directory so that you and your partner can share its contents while keeping them private (easyfacl.py will prompt you to enter information about what and with whom you want to share): 
      % easyfacl.py
    Enter a space separated list of users: newhall pal 
    Enter a pathname (relative or full): /local/tia_n_pal 
    ...

    A note about copying files into directories with acls (vs. creating files in the directory): on directories with acls if you create a new file or directory in the acl directory the default acls apply to the new file or directory (this is great). If, however, you copy an existing file into a directory with acls, the acls do not apply to the copied file (this is unfortunate). There are two things you can do to set acls on a file you copy into the acl directory:

    1. copy the file into the acl directory, then explicitly add acls to the file: 
        cd /local/me_and_pal/.
  cp ~/private/cs45/foo   .
  setfacl -m user:yourname:rwx,user:your_pals_username:rwx foo
    2. use touch to create an empty new file in your acl directory, then overwrite it with the file you are copying: 
        cd /local/me_and_pal/.
  touch foo                   # creates foo with acls set!
  cp ~/private/cs45/foo  .

      See Safe File Sharing for Group Projects for more information on setting, listing, and removing ACLs, and easyfacl.py for more information on easyfacl.py)

  5. Now you are ready to run qemu.

    note: we do not do back-ups of /local, so I suggest you create a bare git repository in one of your home directories. Each of you can then clone a copy of it in a private subdirectory of your home directory, and clone a copy into /local/you_and_pal/. Periodically, add, commit, and push the linux kernel files you change, and other files you create for testing lab project code. Make sure to include in your repository only the linux source and header files that you modify for the current lab assignment, and not the entire linux kernel source tree. These are the files that you will submit for each lab assignment.

Running QEMU

Starting QEMU:

  1. Log into the PC on which your qemu machine is located. For example: 
        % ssh -X cream

  2. Set up tun/tap network interface (you will be asked to enter your password): 
       $ sudo tunctl -b -u your_user_name
   $ sudo /sbin/ifconfig tap0 172.20.0.1

   # I'd do:
   $ sudo tunctl -b -u newhall
   $ sudo /sbin/ifconfig tap0 172.20.0.1

   ## to see results (you should see an entry for tap0):
   $ ifconfig -a
    If you run /sbin/ifconfig -a in another window, you will see that the tap0 entry now has been assigned IP 172.20.0.1

    If you have problems with the tap network you can turn it off and try the above steps again. To turn it off: 

       $ sudo tunctl -d tap0
    The first time you run sudo, you will see warning message and you will be asked to enter your password.

  3. Start qemu on the PC: 
        $ cd /local/you_n_partner/

    # start qemu in graphics mode 
    $  qemu-system-x86_64  -m 256 -net nic -net tap,ifname=tap0,script=no -hda cs45f11_debian.img 

    # -nographic starts qemu without graphics mode
    $  qemu-system-x86_64  -nographic -m 256 -net nic -net tap,ifname=tap0,script=no -hda cs45f11_debian.img
    This starts qemu which will begin the boot process bringing up the grub menu from which you can select a kernel to boot. Initially, the default kernel is 2.6.32.44-cs45start. For most cs45 assignments you will add and boot your own version of the kernel, but always leave this one as an option.

    You can also run qemu in graphics mode (run without the -nographics flag). In graphics mode a window pops up and you can move around in it using:

    CNTRL-ALT   frees your cursor from the QEMU window
    CNTRL-ALT-1   gives you your booted kernel's prompt
    CNTRL-ALT-2   gives you the qemu prompt (you will likely only use this when you quit qemu)
    SHIFT-PageUp   to scroll up in the qemu terminal window
    SHIFT-PageDown   to scroll up in the qemu terminal window

Starting an OS inside of QEMU:

When you boot up Linux inside qemu, you get a grub menu of different kernels to boot. You can use the arrow keys to scroll through the list and then hit the Enter key to choose your kernel. If you don't select one, after a few seconds the kernel at the top of the list is booted.

The qemu image is set up with two accounts, one for root and one for user cs45. The password for both the root and cs45 accounts is swatcs.45. You can change any of your passwords, by running the passwd command. You can also add new users to the system using the adduser command.

Shutting down an OS inside of QEMU:

To shut down your kernel from inside QEMU, do the following:
  1. login as root
  2. type sync; sync; shutdown -h now
  3. or sync; sync; halt
To reboot your kernel from inside QEMU, do the following:
  1. login as root
  2. type sync; sync; reboot

Shutting down QEMU:

  1. first shut down your kernel

  2. then exit qemu
    If you are running in graphics mode, then from the ctrl-alt-2 qemu window, type "quit" after "Power down" to exit the qemu process.

    If you are running in non-graphics mode, then from another window logged into my_pc type "pkill qemu" to send qemu a SIGKILL signal.

  3. After exiting qemu, turn off the tap interface: 
       % sudo tunctl -d tap0
    If you forget to turn off the tap interface, either you or your partner can run this command to turn it off before re-starting it next time you run qemu.

Compiling, Installing and Booting your Kernel changes in QEMU

You will compile and build your changes to the kernel on the CS machines, and then install your changes on your qemu machine.

These steps only need to be done once:

  1. copy the kernel source into your /local directory: 
     $ cp /scratch/vm/linux-2.6.32.44.tar.bz2  /local/you_n_partner/.
  2. unzip and untar the file: 
     $ cd /local/you_n_partner
 $ tar xjf linux-2.6.32.44.tar.bz2
  3. copy the starting point config file into .config in linux-2.6.32.44 and run make menuconfig: 
     $ cd linux-2.6.32.44 
 $ cp /scratch/vm/etc/config-FAST-2.6.32.44 .config
 $ make menuconfig 	# just choose Exit with the arrow keys
  4. follow Option 1 below for building the kernel the first time.

Building the kernel

 There are two ways to build the kernel and which one you use depends on what you have changed when you build:

Option 1: building kernel image (and kernel header) packages

Build this way the first time you build for each lab. Also, if you modify header files, you want to re-build the kernel_headers package. Additionally, if build Option 2 is failing, try this option. Otherwise, use Option 2, which is a much faster build. 
    # NOTE: make-kpkg clean wipes out everything that has been compiled before.
    #       For the very first build you need to do this step, but for subsequent 
    #       builds you may want to not do it, and only if package builds are failing in 
    #       odd ways try doing a make-kpkg clean to re-build 
    #       everything from scratch ("from scratch" takes ~10 minutes)
    #
    $ cd /local/you_n_partner/linux-2.6.32.44
    $ fakeroot make-kpkg clean

    #
    # NOTE: change "whatever" after --append-to-version to something more 
    #       meaningful to you.  for example, I might append -lab2 to my 
    #       kernel version with my lab 2 solution 
    #       (this build will take several minutes)
    #
    $  fakeroot make-kpkg --revision=1.0 --append-to-version -whatever kernel_image

    #
    # if you add new kernel header files, or change existing ones, you need 
    # to build a kernel headers package too that contains your changed files:
    #
    $ fakeroot make-kpkg --revision=1.0 --append-to-version -whatever kernel_image kernel_headers

    make-kpkg creates files named:
    ------------------------------ 
      ../linux-image-2.6.32.44-whatever_1.0_amd64.deb 
      ../linux-headers-2.6.32.44-whatever_1.0_amd64.deb

Option 2: rebuilding the kernel image only

Use this option for subsequent builds that do not involve adding new interface functions or header files (this is much faster than building new packages, removing the old one, and installing the new one each time): 
    % cd /local/you_n_partner/linux-2.6.32.44
    #
    # to re-build the kernel image
    #
    $ make bzImage

    this builds the kernel image in:
    --------------------------------
    /local/you_n_partner/linux-2.6.32.44/arch/x86_64/boot/bzImage
If this fails you can try some of the following and then type 'make bzImage` again...or try build option 1 
    # you don't always need to do this, but if you have changed include files  
    # things don't seem to be getting built correctly do:
    $ make clean

    # do this only if you have added or removed #includes from existing
    # source or header files (you need to rebuild the dependencies in
    # this case):
    $ make dep

Two ways to install and boot your kernel

Which one you choose depends on which option you used to build it.

Option 1: Installing kernel packages:

    # (a) from your CS machine, copy your kernel packages to your qemu machine:
    #
    $ scp linux-image-2.6.32.44-whatever_1.0_i386.deb root@172.20.0.2:.

    (note the first time you scp or ssh you will get RSA key finger print message, just type in yes and continue)

    # copy over kernel headers package:
    #
    $ scp linux-headers-2.6.32.44-whatever_1.0_i386.deb root@172.20.0.2:.


    # (b) Now, as root on your qemu machine install the packages:
    #     from root's home directory (or wherever you scp the files)
    $ ssh root@172.20.0.2
    $ dpkg -i linux-image-2.6.32.44-whatever_1.0_i386.deb

         there will by some "Hmmm..," warnings about symbolic links  
         and missing /lib/modules/version/source it is deleting
         this is okay

    $ dpkg -i linux-headers-2.6.32.44-whatever_1.0_i386.deb

    # (c) reboot your qemu machine and choose your -whatever kernel
    #     from the grub menu
    #
    %  sync; sync; reboot


    # (d) verify that the new version of your kernel rebooted
    #
    $ uname -a
    Linux debian-amd64 2.6.32.44-whatever #1 Tue Sep 13 13:29:14 EDT 2011 x86_64 GNU/Linux


    # note: If this is a re-install of a kernel package that you 
    # have already installed (i.e. the same -whatever flag as an 
    # installed kernel package), you need to first remove the 
    # old package(s), before you do the dpkg -i of the new ones:

    % dpkg -r linux-image-2.6.32.44-whatever

    # You can use the -I option to dpkg to list info about the package file, 
    # including its name (used in the -r option)

Option 2: Installing just a new kernel image file (bzImage):

Copy the new bzImage file to your qemu machine and replace it with the appropriate vmlinuz file in /boot: 
    # (a) from your CS machine copy bzImage to your qemu machine 
    # from /local/your_user_name/linux-2.6.32.44-lab2/arch/x86_64/boot/
    #
    % scp bzImage root@172.20.0.2:.

    # (b) on your qemu machine replace your old image with the new one:
    #
    %  mv /root/bzImage /boot/vmlinuz-2.6.32.44-whatever

    # (c) then update the grub loader 
    #   it should find your version of the kernel /boot/vmlinuz-2.6.32.44-whatever 
    #
    %  update-grub 

    # (c) reboot your qemu machine and choose your -whatever kernel
    #     from the grub menu
    #
    %  sync; sync; reboot
Important Note about Modifying Kernel Source Files: As you make changes to kernel source and header files, you should copy over these changed files somewhere into your home directory. You may want to use a git repository to manage revisions to these files. We do periodic backups of /home directories but do not backup /local on the CS Lab machines (i.e. your changes to files in /local/me_n_pal/linux-2.6.32.44/ are not backed up). By copying over your kernel files that you modify somewhere in your /home, you can take advantage of our file backup system to recover lost changes you make to these files (see Retrieving Lost Files). Only copy kernel source and header files that you actually modify into your home directory (don't copy all of the kernel source into your home directory, as you will run out of disk space if you do, and you can always grab a new copy of the kernel source from the CS45 starting point).

Debugging the Kernel

You can add debugging printk calls to kernel code. printk output goes to the console and to the files: /var/log/kern.log and /var/log/syslog. If your kernel crashes, it is likely that some of the printk output has not yet been written to the log files. As a result, you may want to jot down the last printk output on the console before rebooting. Also, be very careful about where you put printk calls; you don't want them on a code path that is executed many times per second.

Additionally, you can attach gdb to the kernel executable if you run qemu with the -s option (-s is shorthand for -gdb tcp::1234).

Using gdb:

To use gdb, use the '-s' command line option to qemu: 

% qemu-system-x86_64 -s <the rest of the qemu command depending on how you run it>
qemu will wait for gdb to attach to the vmlinux process: 
 (qemu) Waiting gdb connection on port 1234
In another window, start gdb on the vmlinux executable (from the linux-2.6.32.44 subdirectory): 
%  gdb vmlinux
set the architecture to match qemu's: 
%  set architecture i386:x86-64
then, connect to qemu: 
 (gdb) target remote localhost:1234
to let qemu continue execution: 
 (gdb) cont
You can use CNTL-C to interrupt the kernel and get back the gdb prompt

you can set breakpoints in your system calls: 

 (gdb) break sys_getcurrenttime
see the GDB Guide for more information on using gdb.

If there is a mismatch between the target architecture and the architecture on which you are running gdb, you can set specify the target's architecture by doing this: 

(gdb) set architecture i386:x86-64

Connecting to the QEMU VM using ssh or scp from the host PC

We have set up networking on QEMU so that you can only connect to/from the virtual machine to/from the host PC from which you are running qemu. When qemu starts, a private network connection is created between the PC and the qemu VM. This can be used to ssh and scp to/from the PC and qemu.

Because we do not have X installed on the qemu VM, you will likely want to ssh into your qemu VM from the machine on which you are running qemu. This is a useful way to work as it allows you to have multiple windows logged into your qemu machine, and you can leave the main qemu console for seeing printk output while working in other windows.

From the PC you can remotely log into your qemu machine using ssh (you can log in as root or as some other user account you created on your qemu machine): 

% ssh some_qemu_usr@172.20.0.2

You can copy files to/from your PC from/to the qemu machine using scp: 

# copy file foo.c from the PC to qemu machine in some_qemu_usr's proj2 subdirectory: 
% scp foo.c some_qemu_usr@172.20.0.2:/home/qemu_usr/proj2/.

# copy foo.c from qemu machine to your private cs45 subdirectory on CS machines
% scp foo.c your_usr_name@172.20.0.1:/home/your_usr_name/cs45/.

Details

Below is a detailed description of how to find IP addresses for connecting to/from the qemu VM and the lab machine on which you are running qemu.

After booting Linux on qemu, you can run /sbin/ifconfig from within the qemu window to get your VM's IP addresses. For example: 

% /sbin/ifconfig

eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56  
          inet addr:172.20.0.2  Bcast:172.20.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3765 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1980 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:5556513 (5.2 MiB)  TX bytes:155055 (151.4 KiB)
          Interrupt:9 Base address:0xc100 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:12 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
The eth0 entry contains the IP address of your qemu machine: inet addr:172.20.0.2 (you use this IP to connect from the PC to your qemu machine).

Similarly, on the PC running qemu, you can run /sbin/ifconfig. For example: 

% /sbin/ifconfig -a

eth0      Link encap:Ethernet  HWaddr 00:07:E9:62:68:A5  
          inet addr:130.58.68.47  Bcast:130.58.68.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1869144 errors:0 dropped:0 overruns:0 frame:0
          TX packets:802064 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2469282803 (2.2 GiB)  TX bytes:171113790 (163.1 MiB)
          Base address:0xdf40 Memory:fcfe0000-fd000000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:52765 errors:0 dropped:0 overruns:0 frame:0
          TX packets:52765 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:34671242 (33.0 MiB)  TX bytes:34671242 (33.0 MiB)

tun0      Link encap:Ethernet  HWaddr 00:FF:D8:62:ED:EB  
          inet addr:172.20.0.1  Bcast:172.20.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2004 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3794 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:159434 (155.6 KiB)  TX bytes:5560520 (5.3 MiB)

On the PC, the entry tun0 gives the private qemu network IP for the PC (172.20.0.1 in this example). The qemu machine can ssh or scp to the PC using this IP.

Remember: you can only connect to your qemu machine from the PC on which you are running qemu.

Troubleshooting QEMU and Linux

I'll post answers to qemu and Linux questions here.