Support syscall instruction

[nyh]

The libgomp library (gcc's implementation of OpenMP) does the strange thing of calling the "syscall" machine instruction, instead of the syscall() library function - perhaps in a misplaced attempt to save a few cycles.

In gcc4.9.2/libgomp/config/linux/x86/futex.h, there is:

#ifdef __x86_64__
...
static inline void
futex_wait (int *addr, int val)
{
  register long r10 __asm__("%r10");
  long res;

  r10 = 0;
  __asm volatile ("syscall"
                  : "=a" (res)
                  : "0" (SYS_futex), "D" (addr), "S" (gomp_futex_wait),
                    "d" (val), "r" (r10)
                  : "r11", "rcx", "memory");
...

And more of that ilk.

If we want to support an unmodified libgomp.so, we therefore need to support the syscall instruction, and not just the syscall() library function.

In November Pekka posted a patch to the mailing list, titled "[RFC PATCH 08/14] x64: 'syscall' instruction support", for supporting the syscall instruction. This was useful for issue #522 and issue #212, but we now see there are more situations why it can be useful, so I think we should just go ahead and support it.

[nyh]

FPC (Free Pascal) is another project which needs support of the syscall instruction.
http://wiki.lazarus.freepascal.org/Free_Pascal_on_OSv describes how to compile a simple pascal program for OSv, and explains that it is necessary to change the link line to avoid the fpc system library initialization - because that crashes.

If we do not change the link line, and look at what failed, we see this stack trace:

#4  0x0000000000394175 in invalid_opcode (ef=<optimized out>)
    at arch/x64/exceptions.cc:309
#5  <signal handler called>
#6  0x0000100000c04229 in SYSTEM_FPSYSCALL$INT64$INT64$INT64$$INT64 ()
#7  0x0000100000c04ad1 in SYSTEM_FPGETRLIMIT$LONGINT$PRLIMIT$$LONGINT ()
#8  0x00002000001ff9d0 in ?? ()
#9  0x0000100000c23fe7 in SYSTEM_CHECKINITIALSTKLEN$QWORD$$QWORD ()
#10 0x0000000000000000 in ?? ()

This SYSTEM_FPSYSCALL$INT64$INT64$INT64$$INT64 is the free-pascal's system library's FpSysCall() function, defined in fpc's source code in fpcsrc/rtl/linux/x86_64/syscall.inc. Sadly, it is assembly-language code, which calls the syscall instruction directly, which we can see in that source code, or by disassembling the place where we got the #UD:

(gdb) disassemble SYSTEM_FPSYSCALL$INT64$INT64$INT64$$INT64
Dump of assembler code for function SYSTEM_FPSYSCALL$INT64$INT64$INT64$$INT64:
   0x0000100000c04218 <+0>: push   %rbp
   0x0000100000c04219 <+1>: mov    %rsp,%rbp
   0x0000100000c0421c <+4>: sub    $0x10,%rsp
   0x0000100000c04220 <+8>: mov    %rdi,%rax
   0x0000100000c04223 <+11>:    mov    %rsi,%rdi
   0x0000100000c04226 <+14>:    mov    %rdx,%rsi
=> 0x0000100000c04229 <+17>:    syscall 
   0x0000100000c0422b <+19>:    cmp    $0xfffffffffffff001,%rax

[penberg]

Yes, I think we should support the syscall instruction. It should be easy to implement but we should first refactor the syscall() function to use a system call table instead of the switch statement. We can the reuse the table for syscall instruction support.

[VincentS]

Hey, I would be up for this task since I want to run application that utilize OpenMP on OSv.
Could you give me further references where to start on this?

[nyh]

You can see Pekka's (penberg) original proposed patch to support the "syscall" instruction in
https://groups.google.com/d/msg/osv-dev/PW3bkaVCuMg/qnFSu9xtD4UJ
See also his comment above about having a table instead of a switch.

This patch should work at least sometime (Pekka tested it somehow) but you might want to look into other people's concerns about that patch in:
https://groups.google.com/d/msg/osv-dev/PW3bkaVCuMg/-bePROMbWWEJ
https://groups.google.com/d/msg/osv-dev/PW3bkaVCuMg/91SDJwLiU0cJ
https://groups.google.com/d/msg/osv-dev/PW3bkaVCuMg/DoIEIWUGFfYJ

Some of those issues (like the FS switching) might only be relevant to static executables (which think they own the world) and not to normal shared objects who just happen to call the syscall instruction here and there like in the above examples, so if we try to solve only this issue and not #522 or #212, we might not have to solve all the problems just yet.