-
Notifications
You must be signed in to change notification settings - Fork 0
Kernel debugging env setup for Archlinux users
A few scripts are appended at the end of this page for reference.
- Create a disk image (qemu-img: arch.qcow2)
- (Optional, recommended) create a COW image (qemu-img: cow.qcow2 over arch.qcow2).
- Access the disk image as a partition (qemu-nbd: ?.qcow2 -> /dev/nbd0)
- Format filesystem (mkfs.xfs /dev/nbd0) Don't need a partition table.
- Mount the filesystem (mount /dev/nbd0 /mnt)
- Install Archlinux as usual, add all necessary packages.
We don't need a bootloader. A regular linux kernel package is recommended. The rootfs can boot using the host linux kernel (/boot/vmlinuz-linux and initramfs-linux-fallback.img).
-
Boot the VM, setup network (a simple DHCP suffices) so you can easily transfer files or install new packages. The following steps should be done inside the VM.
-
Download a recent linux src package. Git-clone is too slow so an xz package at https://mirrors.edge.kernel.org/pub/linux/kernel/ is the preferred. This kernel version don't have to be the one you're about to use or debug. We only use it to generate a good config file for use in this VM.
-
Unpack and enter the kernel code directory. Now, download the Archlinux linux config from https://github.com/archlinux/svntogit-packages/tree/packages/linux/trunk, save the config file as ".config".
-
make olddefconfig
This refreshes the config file and add missing entries if any. -
make localyesconfig
This removes the currently unused modules to save time and space, and set to build all loaded modules into the kernel, so we won't need to load modules with the debugging kernel. -
Transfer the .config file to the host for future use. This config file should be good even if you change kernel versions in the future.
-
On the host, copy the .config file to the kernel source code directory that you're about to build.
-
make olddefconfig
This can be omitted. But it's recommended to use it when you switch to different kernel versions. -
make all
This will create vmlinux and bzImage. We will need both of them for booting the VM and debugging. -
During the build process, you may find a few modules are still created (e.g., XEN). It's ok to ignore them. You can also clean them by manually disabling those modules with
make menuconfig
We will use the bzImage to boot the VM, and use the vmlinux for debugging with GDB. This kernel do not depend modules so we don't need a initrd.
Make a few changes to the qemu commands:
- -s -S for waiting GDB at localhost:1234
- -kernel PATH_TO_bzImage
- -append nokaslr (and other existing arguments)
Now, when the qemu starts, you can attach to it from another shell (better use a .gdbinit)
- gdb> target remote localhost:1234
- sym PATH_TO_vmlinux
- b vfs_read
- c
- ...
sudo modprobe nbd
sudo qemu-nbd -c /dev/nbd0 cow.qcow2
sudo mount /dev/nbd0 /mnt
sudo arch-chroot /mnt
sudo sync
sudo sync
sudo umount /mnt
sudo qemu-nbd -d /dev/nbd0
qemu-img commit cow.qcow2
rm cow.qcow2
qemu-img create -f qcow2 -b arch.qcow2 -F qcow2 cow.qcow2
Note: the linux
package in the vm must be up-to-date with that on the host.
If not, mount the rootfs and upgrade manually.
nohup \
qemu-system-x86_64 \
-cpu host -accel kvm -smp 2 -m 8G \
-drive file=cow.qcow2,cache=none,if=virtio \
-nic user,hostfwd=tcp::9122-:22 \
-serial telnet:localhost:9111,server,nowait \
-monitor telnet:localhost:9112,server,nowait \
-kernel /boot/vmlinuz-linux -initrd /boot/initramfs-linux-fallback.img \
-append 'root=/dev/vda console=ttyS0 quiet loglevel=3 audit=0' \
-nographic &>/dev/null &
#
- To connect to the console:
telnet localhost 9111
- To connect to the qemu monitor:
telnet localhost 9112
- To login over ssh if network is already setup:
ssh localhost -p 9122
nohup \
qemu-system-x86_64 \
-s -S \
-cpu host -accel kvm -smp 1 -m 1G \
-drive file=cow.qcow2,cache=none,if=virtio \
-nic user,hostfwd=tcp::9122-:22 \
-serial telnet:localhost:9111,server,nowait \
-monitor telnet:localhost:9112,server,nowait \
-kernel ${PATH_TO_bzImage} \
-append 'root=/dev/vda console=ttyS0 quiet nokaslr loglevel=3 audit=0' \
-nographic &>/dev/null &
#
Enable PMEM and DAX (And also select file systems that you want to debug)
Device Drivers --->
{*} NVDIMM (Non-Volatile Memory Device) Support --->
<*> PMEM: Persistent memory block device support
<*> BLK: Block data window (aperture) device support
Processor type and features --->
[*] Support non-standard NVDIMMs and ADR protected memory
File systems --->
[*] Direct Access (DAX) support
Build the kernel and boot with qemu: -m 20G
... -append 'memmap=4G!16G'
.
the memmap
can be specified multiple times. The example marks the memory from 16G to 16+4G as persistent.
Enable PM support and emulate pmem https://pmem.io/2016/02/22/pm-emulation.html