Issues when implementing memory encryption

[moritz-x64]

Hello everyone,
I am currently trying to implement memory encryption in gem5 at the DRAM level. Each access to the DRAM should be encrypted and decrypted transparently. Currently the encryption is done in the AbstractMemory class. Specifically in the access() and functionalAccess() functions. Full blocks of 64 bytes are encrypted and decrypted and the actual data is read/written at an offset. For testing purposes, the encryption consists of a simple XOR operation with a key like $0xfefefe...fe$ with a length of 64 bytes again. The code is shown below.

void xor_crypt(unsigned char *din, unsigned char *key, int len, unsigned char *dout) {
    for (int i = 0; i < len; i++) {
        dout[i] = din[i] ^ key[i];
    }
}

void
AbstractMemory::access(PacketPtr pkt)
{
    ...
    uint8_t *host_addr = toHostAddr(pkt->getAddr());
    uint64_t addr_offset = ((uint64_t)host_addr) & 63;
    uint8_t* addr_without_offset = ((uint8_t*)(((uint64_t)host_addr) & ~63));
    ...
    } else if (pkt->isRead()) {
        ....
        if (pmemAddr) {
            xor_crypt(addr_without_offset, xor_key, 64, scratchpad);
            pkt->setData(scratchpad + addr_offset);
        }
        ...
    } else if (pkt->isWrite()) {
        if (writeOK(pkt)) {
            if (pmemAddr) {
                xor_crypt(addr_without_offset, xor_key, 64, scratchpad);
                pkt->writeData(scratchpad + addr_offset);
                xor_crypt(scratchpad, xor_key, 64, addr_without_offset);
                ...
            }
            ...
        }
        ...
}

AbstractMemory::functionalAccess(PacketPtr pkt)
{
    ...
    uint8_t *host_addr = toHostAddr(pkt->getAddr());
    uint64_t addr_offset = ((uint64_t)host_addr) & 63;
    uint8_t* addr_without_offset = ((uint8_t*)(((uint64_t)host_addr) & ~63));

    if (pkt->isRead()) {
        if (pmemAddr) {
            xor_crypt(addr_without_offset, xor_key, 64, scratchpad);
            pkt->setData(scratchpad + addr_offset);
        }
        ...
    } else if (pkt->isWrite()) {
        if (pmemAddr) {
            xor_crypt(addr_without_offset, xor_key, 64, scratchpad);
            pkt->writeData(scratchpad + addr_offset);
            xor_crypt(scratchpad, xor_key, 64, addr_without_offset);
        }
        TRACE_PACKET("Write");
        pkt->makeResponse();
    }
    ...
}

When I try to run a FS simulation with the provided x86-ubuntu-run.py, it crashes early with a BadAddressError. From my observations, encryption and offset reads and writes are working fine because when the key is set to all zeros the kernel boots as expected. On closer inspection of a trace, I noticed that the kernel function copy_bootdata is already doing some memory accesses, reading data that should all be zero, but instead is 0x00000000fefefefe. This leads me to the conclusion that at some point the memory is set to zero without using timing or functional accesses which would encrypt the 0x00 to 0xfe. Because when read, the zeroed data will be xored with 0xfe, resulting in 0xfe.

gem5 Simulator System.  https://www.gem5.org
gem5 is copyrighted software; use the --copyright option for details.

gem5 version 24.0.0.1
gem5 compiled Oct 11 2024 09:35:20
gem5 started Oct 11 2024 09:49:13
gem5 executing on gem5cruncher, pid 699258
command line: ./build/X86/gem5.opt --debug-flags=Exec,TLB,DRAM --debug-file=exec_mem_dump.out configs/example/gem5_library/x86-ubuntu-run.py

warn: The X86DemoBoard is solely for demonstration purposes. This board is not known to be be representative of any real-world system. Use with caution.
info: Using default config
Computing md5sum on /home/gem5cruncher/.cache/gem5/x86-ubuntu-18.04-img: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.00G/2.00G [00:04<00:00, 489MB/s]
Global frequency set at 1000000000000 ticks per second
warn: No dot file generated. Please install pydot to generate the dot file and pdf.
src/mem/dram_interface.cc:690: warn: DRAM device capacity (8192 Mbytes) does not match the address range assigned (2048 Mbytes)
src/sim/kernel_workload.cc:46: info: kernel located at: /home/gem5cruncher/.cache/gem5/x86-linux-kernel-5.4.49
src/base/statistics.hh:279: warn: One of the stats is a legacy stat. Legacy stat is a stat that does not belong to any statistics::Group. Legacy stat is deprecated.
board.pc.com_1.device: Listening for connections on port 3456
src/base/statistics.hh:279: warn: One of the stats is a legacy stat. Legacy stat is a stat that does not belong to any statistics::Group. Legacy stat is deprecated.
src/dev/intel_8254_timer.cc:128: warn: Reading current count from inactive timer.
board.remote_gdb: Listening for connections on port 7000
src/sim/simulate.cc:199: info: Entering event queue @ 0.  Starting simulation...
src/mem/ruby/system/Sequencer.cc:680: warn: Replacement policy updates recently became the responsibility of SLICC state machines. Make sure to setMRU() near callbacks in .sm files!
build/X86/arch/x86/generated/exec-ns.cc.inc:27: warn: instruction 'fninit' unimplemented
src/cpu/simple/timing.cc:953: panic: panic condition pkt->isError() occurred: Data access ([0xefefefefeff0a:0xefefefefeff0e]) failed: BadAddressError [efefefefeff0a:efefefefeff0d]
Memory Usage: 2691800 KBytes
Program aborted at tick 3140056800
--- BEGIN LIBC BACKTRACE ---
./build/X86/gem5.opt(_ZN4gem515print_backtraceEv+0x30)[0x559d18429010]
./build/X86/gem5.opt(_ZN4gem512abortHandlerEi+0x4c)[0x559d1845266c]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f9e0e5bb520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f9e0e60f9fc]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f9e0e5bb476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f9e0e5a17f3]
./build/X86/gem5.opt(+0x129af75)[0x559d16c05f75]
./build/X86/gem5.opt(_ZN4gem515TimingSimpleCPU18completeDataAccessEPNS_6PacketE+0x5f1)[0x559d1795dc11]
./build/X86/gem5.opt(_ZN4gem510EventQueue10serviceOneEv+0x16d)[0x559d18440ded]
./build/X86/gem5.opt(_ZN4gem59doSimLoopEPNS_10EventQueueE+0x68)[0x559d1846f0c8]
./build/X86/gem5.opt(_ZN4gem58simulateEm+0x283)[0x559d1846f6c3]
./build/X86/gem5.opt(+0x12cf9c0)[0x559d16c3a9c0]
./build/X86/gem5.opt(+0x11775d4)[0x559d16ae25d4]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x128023)[0x7f9e0f0b9023]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyObject_Call+0x5c)[0x7f9e0f072fec]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x4b16)[0x7f9e0f007776]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7f9e0f1533af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x9d68)[0x7f9e0f00c9c8]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7f9e0f1533af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x829e)[0x7f9e0f00aefe]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7f9e0f1533af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(PyEval_EvalCode+0xbe)[0x7f9e0f14e3de]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1bd96d)[0x7f9e0f14e96d]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1287b3)[0x7f9e0f0b97b3]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69de)[0x7f9e0f00963e]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7f9e0f1533af]
./build/X86/gem5.opt(+0x12a8ff7)[0x559d16c13ff7]
./build/X86/gem5.opt(main+0x213)[0x559d18455623]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f9e0e5a2d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f9e0e5a2e40]
./build/X86/gem5.opt(_start+0x25)[0x559d16ad2485]
--- END LIBC BACKTRACE ---
For more info on how to address this issue, please visit https://www.gem5.org/documentation/general_docs/common-errors/

3022493814: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18    : movs	ES:[rdi], DS:[rsi]   
3022493814: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18. 0 :   MOVS_E_M_M : and   t0, rcx, rcx : IntAlu :  D=0x0000000000000000
3022493814: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18. 1 :   MOVS_E_M_M : br   0xc    : IntAlu : 
3022493814: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18. 2 :   MOVS_E_M_M : ruflag   t0d, 0xa : IntAlu :  D=0x0000000000000020
3022493814: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18. 3 :   MOVS_E_M_M : movi   t3, t3, 0x4 : IntAlu :  D=0x0000000000000004
3022493814: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18. 4 :   MOVS_E_M_M : subi   t4, t0, 0x4 : IntAlu :  D=0xfffffffffffffffc
3022493814: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18. 5 :   MOVS_E_M_M : mov   t3, t3, t4 : IntAlu :  D=0x0000000000000004
3022493814: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18. 6 :   MOVS_E_M_M : ld   t1d, DS:[t0 + rsi] : MemRead :  D=0x00000000fefefefe A=0xffff888000090200
3022559748: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18. 7 :   MOVS_E_M_M : st   t1d, ES:[t0 + rdi] : MemWrite :  D=0x00000000fefefefe A=0xffffffff8222eea0
3022623684: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18. 8 :   MOVS_E_M_M : subi   rcx, rcx, 0x1 : IntAlu :  D=0x0000000000000000
3022623684: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18. 9 :   MOVS_E_M_M : add   rdi, rdi, t3 : IntAlu :  D=0xffffffff8222eea4
3022623684: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18.10 :   MOVS_E_M_M : add   rsi, rsi, t3 : IntAlu :  D=0xffff888000090204
3022623684: board.processor.cores0.core: T0 : 0xffffffff8212317e @copy_bootdata+18.11 :   MOVS_E_M_M : br   0x6    : IntAlu : 

The actual crash at the end is caused by a corrupted address translation, which also looks also very suspicious with all those 0xfe bytes:

3139948908: board.processor.cores0.core.mmu.itb: Translating vaddr 0xffffffff8212a268.
3139948908: board.processor.cores0.core.mmu.itb: In protected mode.
3139948908: board.processor.cores0.core.mmu.itb: Paging enabled.
3139948908: board.processor.cores0.core.mmu.itb: Entry found with paddr 0x2000000, doing protection checks.
3139948908: board.processor.cores0.core.mmu.itb: Translated 0xffffffff8212a268 -> 0x212a268.
3139949574: board.processor.cores0.core.mmu.dtb: Translating vaddr 0xffffffffff200f0a.
3139949574: board.processor.cores0.core.mmu.dtb: In protected mode.
3139949574: board.processor.cores0.core.mmu.dtb: Paging enabled.
3139949574: board.processor.cores0.core.mmu.dtb: Handling a TLB miss for address 0xffffffffff200f0a at pc 0xffffffff8212a26d.
3139951906: board.processor.cores0.core.mmu.dtb: Translating vaddr 0xffffffffff200f0a.
3139951906: board.processor.cores0.core.mmu.dtb: In protected mode.
3139951906: board.processor.cores0.core.mmu.dtb: Paging enabled.
3139951906: board.processor.cores0.core.mmu.dtb: Entry found with paddr 0xefefefefef000, doing protection checks.
3139951906: board.processor.cores0.core.mmu.dtb: Translated 0xffffffffff200f0a -> 0xefefefefeff0a.

Any help in identifying the potential write byassing access() and functionalAccess() or other things I may have overlooked is greatly appreciated.