Instant double fault on syscall instruction - need help

[ChocolateLoverRaj]

I followed https://nfil.dev/kernel/rust/coding/rust-kernel-to-userspace-and-back/#final-code-to-return-to-usermode and /~https://github.com/nikofil/rust-os. Entering ring 3 works. But as soon as ring 3 does syscall, a double fault happens.

I set the following idt entries (so none of these interrupts get called):

• double fault
• breakpoint
• general protection
• page fault
• invalid tss
• segment not present
• invalid opcode
• stack segment fault
• timer interrupt (and i disabled the interrupt)
• spurious
• local apic error

The full code is here: /~https://github.com/ChocolateLoverRaj/code-runner/tree/3c2e6cf730ee7d4616e007a51558b946936e4971. I can remove unused parts needed for easier debugging.

Here are the main parts of the code:

pub unsafe fn jmp_to_usermode(gdt: &Gdt, code: VirtAddr, stack_end: VirtAddr) {
    let cs_idx = {
        let mut code_selector = gdt.user_code_selector.clone();
        code_selector.set_rpl(PrivilegeLevel::Ring3);
        code_selector.0
    };
    let ds_idx = {
        let mut data_selector = gdt.user_data_selector.clone();
        data_selector.set_rpl(PrivilegeLevel::Ring3);
        DS::set_reg(data_selector);
        data_selector.0
    };
    flush_all();
    asm!("\
        push rax   // stack segment
        push rsi   // rsp
        push 0x200 // rflags (only interrupt bit set)
        push rdx   // code segment
        push rdi   // ret to virtual addr
        iretq",
        in("rdi") code.as_u64(), in("rsi") stack_end.as_u64(), in("dx") cs_idx, in("ax") ds_idx);
}

unsafe fn userspace_prog_1() {
    // asm!("nop");
    // asm!("nop");
    asm!("syscall", options(nostack));
    // asm!("nop");
    // asm!("nop");
}

pub struct Gdt {
    gdt: GlobalDescriptorTable,
    kernel_code_selector: SegmentSelector,
    kernel_data_selector: SegmentSelector,
    pub user_code_selector: SegmentSelector,
    pub user_data_selector: SegmentSelector,
    tss_selector: SegmentSelector,
}

impl Gdt {
    pub fn new(tss: &'static TaskStateSegment) -> Self {
        let mut gdt = GlobalDescriptorTable::new();
        let kernel_code_selector = gdt.append(Descriptor::kernel_code_segment());
        let kernel_data_selector = gdt.append(Descriptor::kernel_data_segment());
        let tss_selector = gdt.append(Descriptor::tss_segment(&tss));
        let user_code_selector = gdt.append(Descriptor::user_code_segment());
        let user_data_selector = gdt.append(Descriptor::user_data_segment());

        info!(
            "kernel code: {:?}\nkernel data: {:?}\nuser code: {:?}\nuser data: {:?}\nTSS: {:?}",
            kernel_code_selector,
            kernel_data_selector,
            user_code_selector,
            user_data_selector,
            tss_selector
        );

        Self {
            gdt,
            kernel_code_selector,
            kernel_data_selector,
            user_code_selector,
            user_data_selector,
            tss_selector,
        }
    }

    pub fn init(&self) {
        unsafe { self.gdt.load_unsafe() };
        unsafe {
            // /~https://github.com/rust-osdev/bootloader/blob/5d318bfc8afa4fb116a2c7923d5411febbe7266c/docs/migration/v0.9.md#kernel
            SS::set_reg(self.kernel_data_selector);
            ES::set_reg(SegmentSelector::NULL);
            FS::set_reg(SegmentSelector::NULL);
            GS::set_reg(SegmentSelector::NULL);

            CS::set_reg(self.kernel_code_selector);
            DS::set_reg(self.kernel_data_selector);
            load_tss(self.tss_selector);
        }
    }
}

Serial output:

EXCEPTION: DOUBLE FAULT
InterruptStackFrame {
    instruction_pointer: VirtAddr(
        0x1dfc4,
    ),
    code_segment: SegmentSelector {
        index: 5,
        rpl: Ring3,
    },
    cpu_flags: RFlags(
        INTERRUPT_FLAG | 0x2,
    ),
    stack_pointer: VirtAddr(
        0x1f3e0,
    ),
    stack_segment: SegmentSelector {
        index: 6,
        rpl: Ring3,
    },
}

[tsatke]

Start QEMU with -d int and check v, which is the vector number. Check which interrupt caused the double fault with this table.

I would assume that it is 0x0E (#PF), because your kernel code is probably not mapped as user accessible, which will cause the CPU to fetch instructions from kernel memory (since userspace_prog_1 is kernel code (and your page fault handler is also kernel code, which is also inaccessible).

You can check with info mem in the QEMU monitor what is mapped with which flags.

Ideally, you would load an executable (ELF or whatever) in a separate address space (different Cr3 value), and jump there.

[ChocolateLoverRaj]

So now instead of a double fault there is a invalid opcode interrupt as soon as the syscall instruction is executed.

check_exception old: 0xffffffff new 0x6
  4636: v=06 e=0000 i=0 cpl=3 IP=002b:000000000001d546 pc=000000000001d546 SP=0033:000000000001f3e0 env->regs[R_EAX]=0000000000010033
RAX=0000000000010033 RBX=0000000007efa110 RCX=0000000000000000 RDX=000000000010002b
RSI=000000000001f3e8 RDI=000000000001d540 RBP=000000000001f3e0 RSP=000000000001f3e0
R8 =00000080000861e0 R9 =0000000000000002 R10=00000000000000ff R11=0000000000000000
R12=0000000007ef8e20 R13=0000000007efb5c8 R14=0000000007ef8d68 R15=0000000007efb4b0
RIP=000000000001d546 RFL=00000202 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 00000000 00000000
CS =002b 0000000000000000 ffffffff 00affb00 DPL=3 CS64 [-RA]
SS =0033 0000000000000000 ffffffff 00cff300 DPL=3 DS   [-WA]
DS =0033 0000000000000000 ffffffff 00cff300 DPL=3 DS   [-WA]
FS =0000 0000000000000000 00000000 00000000
GS =0000 0000000000000000 00000000 00000000
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0018 000000800008d870 00000067 00008900 DPL=0 TSS64-avl
GDT=     000000800008d8f8 00000037
IDT=     000000800008c860 00000fff
CR0=80010033 CR2=0000000000000000 CR3=0000000000101000 CR4=00000668
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=0000000000000000 CCD=000001000000fba0 CCO=EFLAGS
EFER=0000000000000d00
ERROR: panicked at kernel/src/modules/panicking_invalid_opcode_handler.rs:4:5:
Invalid opcode!

[tsatke]

So that means that the one problem of the interrupt handlers not being found is solved.

Are you still doing the sysret?

Anyways, I think that you should be able to see the exact instruction in lldb now. You can do instruction level steps with the si command.

[ChocolateLoverRaj]

Are you still doing the sysret?

Yes, but based on LLDB when I do si when the current instruction is syscall it instantly goes to the invalid opcode handler.

[tsatke]

Maybe this? https://stackoverflow.com/questions/76359244/syscall-causing-invalid-opcode-interruption

Other than that, I don't have much experience with syscall.

[ChocolateLoverRaj]

I think I need to put the syscall handler in the upper half and that's why it is not working. I will try it. nvm it's not the issue

[ChocolateLoverRaj]

So I was missing 2 things:

• Set a privilege stack table entry
• Enable sycalls by setting bit 0 to 1 in IA32_EFER:

let mut ia32_efer = Msr::new(0xC0000080);
let mut value = ia32_efer.read();
value |= 0b1;
ia32_efer.write(value);

And syscall works now. No need to have kernel in higher half addresses.

This is how I realized that the invalid opcode was caused by not setting the efer thing: https://shell-storm.org/x86doc/SYSCALL.html