cannot checkpoint firefox on ubuntu 14 #2

skhavari · 2014-08-26T05:12:26Z

Linux ubuntu 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce

built latest criu from source
run firefox
run sudo ./criu dump -D checkpoint -t <pid> --file-locks --tcp-established

Expect: checkpoint firefox
Actual:
Error (parasite-syscall.c:387): si_code=1 si_pid=7723 si_status=0
Error (parasite-syscall.c:387): si_code=1 si_pid=7725 si_status=0
Error (parasite-syscall.c:387): si_code=1 si_pid=7727 si_status=0
Error (parasite-syscall.c:387): si_code=1 si_pid=7729 si_status=0
Error (parasite-syscall.c:387): si_code=1 si_pid=7731 si_status=0
Error (parasite-syscall.c:387): si_code=1 si_pid=7733 si_status=0
Error (parasite-syscall.c:387): si_code=1 si_pid=7735 si_status=0
Error (parasite-syscall.c:387): si_code=1 si_pid=7737 si_status=0
Error (parasite-syscall.c:387): si_code=1 si_pid=7739 si_status=0
Error (parasite-syscall.c:387): si_code=1 si_pid=7741 si_status=0
Error (parasite-syscall.c:387): si_code=1 si_pid=7743 si_status=0
Error (parasite-syscall.c:387): si_code=1 si_pid=7745 si_status=0
Error (parasite-syscall.c:387): si_code=4 si_pid=7507 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7508 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7509 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7510 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7511 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7512 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7513 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7514 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7518 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7519 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7521 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7522 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7523 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7524 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7525 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7527 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7528 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7529 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7530 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7533 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7534 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7535 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7536 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7537 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7539 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7541 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7542 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7543 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7545 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7546 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7550 si_status=5
Error (parasite-syscall.c:387): si_code=4 si_pid=7640 si_status=5
Error (cr-dump.c:408): Task 7501 with SysVIPC shmem map @7f473a185000 doesn't live in IPC ns
Error (cr-dump.c:1665): Dump mappings (pid: 7501) failed with -1
Error (cr-dump.c:1953): Dumping FAILED.

The text was updated successfully, but these errors were encountered:

xemul · 2014-08-26T08:35:08Z

@skhavari , X applications use SysVIPC shared memory segments to talk to X server and keep part of the state on the server itself. These two facts make it pretty hard to checkpoint and restore just and application and CRIU currently doesn't support this scenario.

If you're OK to launch Firefox via VNC, you can try playing with it -- http://criu.org/VNC

skhavari · 2014-08-26T15:13:35Z

Great, thanks!

CID 1168165 (#2 of 2): Untrusted array index read (TAINTED_SCALAR) 40. tainted_data: Using tainted variable "hoff" as an index into an array "str" $ man 3 scanf n Nothing is expected; instead, the number of characters consumed thus far from the input is stored through the next pointer, which must be a pointer to int. This is not a conversion, although it can be suppressed with the * assignment-suppression character. The C standard says: "Execution of a %n directive does not increment the assignment count returned at the comple‐ tion of execution" but the Corrigendum seems to contradict this. Probably it is wise not to make any assumptions on the effect of %n conversions on the return value. So it isn't not enough to check a return code from scanf(). Cc: Cyrill Gorcunov <[email protected]> Signed-off-by: Andrey Vagin <[email protected]> Acked-by: Cyrill Gorcunov <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

It is called from prepare_cgroup_sfd() and cr_restore_tasks(). + criu restore --file-locks --tcp-established --evasive-devices --link-remap --root /var/lib/vz/root/101 --restore-detached --action-script /usr/local/libexec/vzctl/scripts/vps-rst-env -D /vz/dump/Dump.101 -o restore.log -vvvv --pidfile /var/lib/vzctl/vepid/101 *** Error in `criu': double free or corruption (fasttop): 0x00000000006bcd40 *** Program terminated with signal 6, Aborted. Missing separate debuginfos, use: debuginfo-install glibc-2.17-20.fc19.x86_64 libgcc-4.8.3-1.fc19.x86_64 protobuf-c-0.15-7.fc19.x86_64 (gdb) bt #0 0x00007ffff72179e9 in raise () from /lib64/libc.so.6 #1 0x00007ffff72190f8 in abort () from /lib64/libc.so.6 #2 0x00007ffff7257d17 in __libc_message () from /lib64/libc.so.6 #3 0x00007ffff725f0b8 in _int_free () from /lib64/libc.so.6 #4 0x0000000000426971 in cr_restore_tasks () at cr-restore.c:1833 #5 0x0000000000418426 in main (argc=<optimized out>, argv=0x7fffffffeb38, envp=<optimized out>) at crtools.c:479 Signed-off-by: Andrey Vagin <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

If you call clone directly you are responsible for setting up the TLS area yourself. $ abrt-cli ls | grep different_creds | wc -l 39 $ gdb -c /var/spool/abrt/ccpp-2015-07-24-10\:21\:14-8014/coredump different_creds Core was generated by `./different_creds --pidfile=different_creds.pid --outfile=different_creds.out'. Program terminated with signal SIGILL, Illegal instruction. #0 0x00007f86e2d8c7d9 in _dl_x86_64_restore_sse () from /lib64/ld-linux-x86-64.so.2 Missing separate debuginfos, use: dnf debuginfo-install glibc-2.21-7.fc22.x86_64 libattr-2.4.47-9.fc22.x86_64 libcap-2.24-7.fc22.x86_64 (gdb) bt #0 0x00007f86e2d8c7d9 in _dl_x86_64_restore_sse () from /lib64/ld-linux-x86-64.so.2 #1 0x00007f86e2d84add in _dl_fixup () from /lib64/ld-linux-x86-64.so.2 #2 0x00007f86e2d8bbc0 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2 #3 0x0000000000402da3 in sys_futex (val3=0, uaddr2=0x0, timeout=0x0, val=0, op=0, uaddr=0x6063f0 <sig_received>) at lock.h:29 #4 futex_wait_while (f=0x6063f0 <sig_received>, v=0) at lock.h:121 #5 test_waitsig () at test.c:367 #6 0x0000000000401c4b in main (argc=<optimized out>, argv=0x7ffce16432f8) at different_creds.c:82 Reported-by: Mr Jenkins Cc: Tycho Andersen <[email protected]> Signed-off-by: Andrew Vagin <[email protected]> Acked-by: Tycho Andersen <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

This is fixlet to patch #2. Signed-off-by: Pavel Emelyanov <[email protected]> Looks-good-to-me: Andrew Vagin <[email protected]>

CID 159478 (#2 of 2): Resource leak (RESOURCE_LEAK) 8. leaked_handle: Handle variable sk going out of scope leaks the handle. Signed-off-by: Andrew Vagin <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

CID 159478 (checkpoint-restore#2 of 2): Resource leak (RESOURCE_LEAK) 8. leaked_handle: Handle variable sk going out of scope leaks the handle. Signed-off-by: Andrew Vagin <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

CID 159478 (#2 of 2): Resource leak (RESOURCE_LEAK) 8. leaked_handle: Handle variable sk going out of scope leaks the handle. Signed-off-by: Andrew Vagin <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

It can be dead-lokced: #0 0x00007fafbf49f6ac in __lll_lock_wait_private () from /lib64/libc.so.6 #1 0x00007fafbf44af1c in _L_lock_2460 () from /lib64/libc.so.6 #2 0x00007fafbf44ad57 in __tz_convert () from /lib64/libc.so.6 checkpoint-restore#3 0x00000000004022e2 in test_msg (format=0x404508 "Receive signal %d\n") at msg.c:51 checkpoint-restore#4 <signal handler called> checkpoint-restore#5 0x00007fafbf3f2483 in __GI__IO_vfscanf () from /lib64/libc.so.6 checkpoint-restore#6 0x00007fafbf408f27 in vsscanf () from /lib64/libc.so.6 checkpoint-restore#7 0x00007fafbf4032f7 in sscanf () from /lib64/libc.so.6 checkpoint-restore#8 0x00007fafbf449ba6 in __tzset_parse_tz () from /lib64/libc.so.6 checkpoint-restore#9 0x00007fafbf44c4cb in __tzfile_compute () from /lib64/libc.so.6 checkpoint-restore#10 0x00007fafbf44ae17 in __tz_convert () from /lib64/libc.so.6 checkpoint-restore#11 0x00000000004022e2 in test_msg (format=format@entry=0x40458c "PASS\n") at msg.c:51 checkpoint-restore#12 0x0000000000401ceb in main (argc=<optimized out>, argv=<optimized out>) at ptrace_sig.c:172 https://jira.sw.ru/browse/PSBM-47772 Signed-off-by: Andrey Vagin <[email protected]> Signed-off-by: Andrew Vagin <[email protected]> Tested-by: Cyrill Gorcunov <[email protected]>

It can be dead-lokced: #0 0x00007fafbf49f6ac in __lll_lock_wait_private () from /lib64/libc.so.6 #1 0x00007fafbf44af1c in _L_lock_2460 () from /lib64/libc.so.6 #2 0x00007fafbf44ad57 in __tz_convert () from /lib64/libc.so.6 #3 0x00000000004022e2 in test_msg (format=0x404508 "Receive signal %d\n") at msg.c:51 #4 <signal handler called> #5 0x00007fafbf3f2483 in __GI__IO_vfscanf () from /lib64/libc.so.6 #6 0x00007fafbf408f27 in vsscanf () from /lib64/libc.so.6 #7 0x00007fafbf4032f7 in sscanf () from /lib64/libc.so.6 #8 0x00007fafbf449ba6 in __tzset_parse_tz () from /lib64/libc.so.6 #9 0x00007fafbf44c4cb in __tzfile_compute () from /lib64/libc.so.6 #10 0x00007fafbf44ae17 in __tz_convert () from /lib64/libc.so.6 #11 0x00000000004022e2 in test_msg (format=format@entry=0x40458c "PASS\n") at msg.c:51 #12 0x0000000000401ceb in main (argc=<optimized out>, argv=<optimized out>) at ptrace_sig.c:172 Signed-off-by: Andrey Vagin <[email protected]> Signed-off-by: Andrew Vagin <[email protected]> Tested-by: Cyrill Gorcunov <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

Fix CID 163485 (#2 of 2): Dereference null return value (NULL_RETURNS) 7. dereference: Dereferencing a pointer that might be null dest when calling handle_user_fault. Signed-off-by: Mike Rapoport <[email protected]>

Fix CID 163485 (#2 of 2): Dereference null return value (NULL_RETURNS) 7. dereference: Dereferencing a pointer that might be null dest when calling handle_user_fault. Signed-off-by: Mike Rapoport <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

It can be dead-lokced: #0 0x00007fafbf49f6ac in __lll_lock_wait_private () from /lib64/libc.so.6 #1 0x00007fafbf44af1c in _L_lock_2460 () from /lib64/libc.so.6 #2 0x00007fafbf44ad57 in __tz_convert () from /lib64/libc.so.6 #3 0x00000000004022e2 in test_msg (format=0x404508 "Receive signal %d\n") at msg.c:51 #4 <signal handler called> #5 0x00007fafbf3f2483 in __GI__IO_vfscanf () from /lib64/libc.so.6 #6 0x00007fafbf408f27 in vsscanf () from /lib64/libc.so.6 #7 0x00007fafbf4032f7 in sscanf () from /lib64/libc.so.6 #8 0x00007fafbf449ba6 in __tzset_parse_tz () from /lib64/libc.so.6 #9 0x00007fafbf44c4cb in __tzfile_compute () from /lib64/libc.so.6 #10 0x00007fafbf44ae17 in __tz_convert () from /lib64/libc.so.6 #11 0x00000000004022e2 in test_msg (format=format@entry=0x40458c "PASS\n") at msg.c:51 #12 0x0000000000401ceb in main (argc=<optimized out>, argv=<optimized out>) at ptrace_sig.c:172 Signed-off-by: Andrey Vagin <[email protected]> Signed-off-by: Andrew Vagin <[email protected]> Tested-by: Cyrill Gorcunov <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000000000435744 in cr_pre_dump_finish (ret=0) at cr-dump.c:1452 1452 pr_info("\tPre-dumping %d\n", ctl->pid.virt); (gdb) bt #0 0x0000000000435744 in cr_pre_dump_finish (ret=0) at cr-dump.c:1452 #1 cr_pre_dump_tasks (pid=pid@entry=24) at cr-dump.c:1556 #2 0x000000000041f665 in main (argc=<optimized out>, argv=0x7ffda430e818, envp=<optimized out>) at crtools.c:753 https://github.com/xemul/criu/issues/189 Cc: Mike Rapoport <[email protected]> Signed-off-by: Andrew Vagin <[email protected]>

Fix CID 163485 (#2 of 2): Dereference null return value (NULL_RETURNS) 7. dereference: Dereferencing a pointer that might be null dest when calling handle_user_fault. Signed-off-by: Mike Rapoport <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000000000435744 in cr_pre_dump_finish (ret=0) at cr-dump.c:1452 1452 pr_info("\tPre-dumping %d\n", ctl->pid.virt); (gdb) bt #0 0x0000000000435744 in cr_pre_dump_finish (ret=0) at cr-dump.c:1452 #1 cr_pre_dump_tasks (pid=pid@entry=24) at cr-dump.c:1556 #2 0x000000000041f665 in main (argc=<optimized out>, argv=0x7ffda430e818, envp=<optimized out>) at crtools.c:753 https://github.com/xemul/criu/issues/189 Cc: Mike Rapoport <[email protected]> Signed-off-by: Andrew Vagin <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

Fix CID 163485 (#2 of 2): Dereference null return value (NULL_RETURNS) 7. dereference: Dereferencing a pointer that might be null dest when calling handle_user_fault. Signed-off-by: Mike Rapoport <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

phys_stat_resolve() call mount_resolve_path() which requires that mntinfo_tree in the ns_id struct is initialized. This is a problem we observed with sockets on btrfs volumes: Program received signal SIGSEGV, Segmentation fault. 0x00005555555bb6dd in mount_resolve_path (mntinfo_tree=<optimized out>, path=0x555555875790 "/var/lib/lxd/unix.socket") at criu/mount.c:213 213 criu/mount.c: No such file or directory. (gdb) bt #0 0x00005555555bb6dd in mount_resolve_path (mntinfo_tree=<optimized out>, path=0x555555875790 "/var/lib/lxd/unix.socket") at criu/mount.c:213 #1 0x00005555555be240 in phys_stat_resolve_dev (ns=<optimized out>, st_dev=43, path=<optimized out>) at criu/mount.c:240 #2 0x00005555555be2bb in phys_stat_dev_match (st_dev=<optimized out>, phys_dev=41, ns=ns@entry=0x5555558753a0, path=path@entry=0x555555875790 "/var/lib/lxd/unix.socket") at criu/mount.c:256 checkpoint-restore#3 0x00005555555e75ed in unix_process_name (d=d@entry=0x5555558756e0, tb=tb@entry=0x7fffffffe0c0, m=<optimized out>) at criu/sk-unix.c:565 checkpoint-restore#4 0x00005555555e9378 in unix_collect_one (tb=0x7fffffffe0c0, m=0x555555869f18 <buf+312>) at criu/sk-unix.c:620 checkpoint-restore#5 unix_receive_one (h=0x555555869f08 <buf+296>, arg=<optimized out>) at criu/sk-unix.c:692 checkpoint-restore#6 0x00005555555b85aa in nlmsg_receive (buf=<optimized out>, arg=<optimized out>, err_cb=<optimized out>, cb=<optimized out>, len=<optimized out>) at criu/libnetlink.c:45 checkpoint-restore#7 do_rtnl_req (nl=nl@entry=5, req=req@entry=0x7fffffffe220, size=size@entry=72, receive_callback=0x5555555e9290 <unix_receive_one>, error_callback=0x5555555b83d0 <rtnl_return_err>, error_callback@entry=0x0, arg=arg@entry=0x0) at criu/libnetlink.c:119 checkpoint-restore#8 0x00005555555e9cf7 in do_collect_req (nl=nl@entry=5, req=req@entry=0x7fffffffe220, receive_callback=<optimized out>, arg=arg@entry=0x0, size=72) at criu/sockets.c:610 checkpoint-restore#9 0x00005555555eb1d0 in collect_sockets (ns=ns@entry=0x7fffffffe300) at criu/sockets.c:636 checkpoint-restore#10 0x000055555559ddfc in check_sock_diag () at criu/cr-check.c:118 checkpoint-restore#11 cr_check () at criu/cr-check.c:999 checkpoint-restore#12 0x00005555555872d0 in main (argc=<optimized out>, argv=0x7fffffffe678, envp=<optimized out>) at criu/crtools.c:719 Signed-off-by: Christian Brauner <[email protected]>

Fix CID 163485 (#2 of 2): Dereference null return value (NULL_RETURNS) 7. dereference: Dereferencing a pointer that might be null dest when calling handle_user_fault. Signed-off-by: Mike Rapoport <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

I discovered that the scripts/ suffix is added to __nmk_dir despite the fact it already contains it, ending in obviously wrong filenames like scripts/nmk/scripts/scripts/msg.mk. As those files are non-existent, make tried to recreate every .mk file, spawninga child to execute 'true' command, like this (part of "make -dr" output): > Considering target file '../scripts/nmk/scripts/scripts/include.mk'. > File '../scripts/nmk/scripts/scripts/include.mk' does not exist. > Finished prerequisites of target file > '../scripts/nmk/scripts/scripts/include.mk'. > Must remake target '../scripts/nmk/scripts/scripts/include.mk'. > Putting child 0x564ec1768740 (../scripts/nmk/scripts/scripts/include.mk) > PID 21633 on the chain. > Live child 0x564ec1768740 (../scripts/nmk/scripts/scripts/include.mk) > PID 21633 > Reaping winning child 0x564ec1768740 PID 21633 > Removing child 0x564ec1768740 PID 21633 from chain. The fix was to remove the extra scripts/, but once I did it, I found out problem #2: these targets, being defined in contents that is often included in the beginning of Makefiles, hijacks the default make target (the first one in the Makefile), breaking the usual and expected make behavior, and forcing to use .DEFAULT_GOAL. Finally, I don't know why these targets are there, i.e. what purpose do they serve. Maybe it was done to exclude any implicit rules to re-make those files, but there are no such rules as far as I can see. So, in order to address problem #2, I have removed these targets. I don't see any harm in doing that; let me know if it breaks anything. Cc: Cyrill Gorcunov <[email protected]> Signed-off-by: Kir Kolyshkin <[email protected]> Signed-off-by: Pavel Emelyanov <[email protected]>

Move xfree() up Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]>

CID 181291 (#2 of 2): Resource leak (RESOURCE_LEAK) 15. leaked_storage: Variable buf going out of scope leaks the storage it points to. CID 181288 (#2 of 2): Resource leak (RESOURCE_LEAK) 15. leaked_storage: Variable buf going out of scope leaks the storage it points to. Signed-off-by: Andrei Vagin <[email protected]>

By exhaustive testing I understand a test suite that generates as much states to try to C/R as possible by trying all the possible sequences of system calls. Since such a generation, if done on all the Linux API we support in CRIU, would produce bazillions of process, I propose to start with something simple. As a starting point -- unix stream sockets with abstract names that can be created and used by a single process :) The script generates situations in which unix sockets can get into by using a pre-defined set of system calls. In this patch the syscalls are socket, listen, bind, accept, connect and send. Also the nummber of system calls to use (i.e. -- the depth of the tree) is limited by the --depth option. There are three things that can be done with a generated 'state': I) Generate :) and show Generation is done by recursively doing everything that is possible (and makes sence) in a given state. To reduce the size of the tree some meaningless branches are cut, e.g. creating a socket and closing it right after that, creating two similar sockets one-by-one and some more. Shown on the screen is a cryptic string, e.g. 'SA-CX-MX_SBL one, describing the sockets in the state. This is how it can be decoded: - sockets are delimited with _ - first goes type (S -- stream, D --datagram) - next goes name state (A -- no name, B with name, X socket is not in FD table, i.e. closed or not yet accepted) - next may go letter L meaning that the socket is listening - -Cx -- socket is connected and x is the peer's name state - -Ixyz -- socket has incoming connections queue and xyz are the connect()-ors name states - -Mxyz -- socket has messages and xyz is senders' name states The example above means, that we have two sockets: - SA-CX-MX: stream, with no name, connected to a dead one and with a message from a dead one - SBL: stream, with name, listening Next printed is the sequence of system calls to get into it, e.g. this is how to get into the state above: socket(S) = 1 bind(1, $name-1) listen(1) socket(S) = 2 connect(2, $name-1) accept(1) = 3 send(2, $message-0) send(3, $message-0) close(3) Program has created a stream socket, bound it, listened it, then created another stream socket, connected to the 1st one, then accepted the connection sent two messages vice-versa and closed the accepted end, so the 1st socket left connected to the dead socket with a message from it. II) Run the state This is when test actually creates a process that does the syscalls required to get into the generated state (and hopefully gets into it). III) Check C/R of the state This is the trickiest part when it comes to the R step -- it's not clear how to validate that the state restored is correct. But if only trying to dump the state -- it's just calling criu dump. As images dir the state string description is used. One may choose only to generate the states with --gen option. One may choose only to run the states with --run option. The latter is useful to verify that the states generator is actually producing valid states. If no options given, the state is also dump-ed (restore is to come later). For now the usage experience is like this: - Going --depth 10 --gen (i.e. just generating all possibles states that are acheivable with 10 syscalls) produces 44 unique states for 0.01 seconds. The generated result covers some static tests we have in zdtm :) More generation stats is like this: --depth 15 : 1.1 sec / 72 states --depth 18 : 13.2 sec / 89 states --depth 20 : 1 m 8 sec / 101 state - Running and trying with criu is checked with --depth 9. Criu fails to dump the state SA-CX-MX_SBL (shown above) with the error Error (criu/sk-queue.c:151): recvmsg fail: error: Connection reset by peer Nearest plans: 1. Add generators for on-disk sockets names (now oly abstract). Here an interesting case is when names overlap and one socket gets a name of another, but isn't accessible by it 2. Add datagram sockets. Here it'd be fun to look at how many-to-one connections are generated and checked. 3. Add socketpair()-s. Farther plans: 1. Cut the tree better to allow for deeper tree scan. 2. Add restore. 3. Add SCM-s 4. Have the exhaustive testing for other resources. Changes since v1: * Added DGRAM sockets :) Dgram sockets are trickier that STREAM, as they can reconnect from one peer to another. Thus just limiting the tree depth results in wierd states when socket just changes peer. In the v1 of this patch new sockets were added to the state only when old ones reported that there's nothing that can be done with them. This limited the amount of stupid branches, but this strategy doesn't work with dgram due to reconnect. Due to this, change #2: * Added the --sockets NR option to limit the amount of sockets. This allowed to throw new sockets into the state on each step, which made a lot of interesting states for DGRAM ones. * Added the 'restore' stage and checks after it. After the process is restore the script performs as much checks as possible having the expected state description in memory. The checks verify that the values below get from real sockets match the expectations in generated state: - socket itself - name - listen state - pending connections - messages in queue (sender is not checked) - connectivity The latter is checked last, after all queues should be empty, by sending control messages with socket.recv() method. * Added --keep option to run all tests even if one of them fails. And print nice summary at the end. So far the test found several issues: - Dump doesn't work for half-closed connection with unread messages - Pending half-closed connection is not restored - Socket name is not restored - Message is not restored New TODO: - Check listen state is still possible to accept connections (?) - Add socketpair()s - Add on-disk names - Add SCM-s - Exhaustive script for other resources Signed-off-by: Pavel Emelyanov <[email protected]> Signed-off-by: Andrei Vagin <[email protected]>

Move xfree() up Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]>

By exhaustive testing I understand a test suite that generates as much states to try to C/R as possible by trying all the possible sequences of system calls. Since such a generation, if done on all the Linux API we support in CRIU, would produce bazillions of process, I propose to start with something simple. As a starting point -- unix stream sockets with abstract names that can be created and used by a single process :) The script generates situations in which unix sockets can get into by using a pre-defined set of system calls. In this patch the syscalls are socket, listen, bind, accept, connect and send. Also the nummber of system calls to use (i.e. -- the depth of the tree) is limited by the --depth option. There are three things that can be done with a generated 'state': I) Generate :) and show Generation is done by recursively doing everything that is possible (and makes sence) in a given state. To reduce the size of the tree some meaningless branches are cut, e.g. creating a socket and closing it right after that, creating two similar sockets one-by-one and some more. Shown on the screen is a cryptic string, e.g. 'SA-CX-MX_SBL one, describing the sockets in the state. This is how it can be decoded: - sockets are delimited with _ - first goes type (S -- stream, D --datagram) - next goes name state (A -- no name, B with name, X socket is not in FD table, i.e. closed or not yet accepted) - next may go letter L meaning that the socket is listening - -Cx -- socket is connected and x is the peer's name state - -Ixyz -- socket has incoming connections queue and xyz are the connect()-ors name states - -Mxyz -- socket has messages and xyz is senders' name states The example above means, that we have two sockets: - SA-CX-MX: stream, with no name, connected to a dead one and with a message from a dead one - SBL: stream, with name, listening Next printed is the sequence of system calls to get into it, e.g. this is how to get into the state above: socket(S) = 1 bind(1, $name-1) listen(1) socket(S) = 2 connect(2, $name-1) accept(1) = 3 send(2, $message-0) send(3, $message-0) close(3) Program has created a stream socket, bound it, listened it, then created another stream socket, connected to the 1st one, then accepted the connection sent two messages vice-versa and closed the accepted end, so the 1st socket left connected to the dead socket with a message from it. II) Run the state This is when test actually creates a process that does the syscalls required to get into the generated state (and hopefully gets into it). III) Check C/R of the state This is the trickiest part when it comes to the R step -- it's not clear how to validate that the state restored is correct. But if only trying to dump the state -- it's just calling criu dump. As images dir the state string description is used. One may choose only to generate the states with --gen option. One may choose only to run the states with --run option. The latter is useful to verify that the states generator is actually producing valid states. If no options given, the state is also dump-ed (restore is to come later). For now the usage experience is like this: - Going --depth 10 --gen (i.e. just generating all possibles states that are acheivable with 10 syscalls) produces 44 unique states for 0.01 seconds. The generated result covers some static tests we have in zdtm :) More generation stats is like this: --depth 15 : 1.1 sec / 72 states --depth 18 : 13.2 sec / 89 states --depth 20 : 1 m 8 sec / 101 state - Running and trying with criu is checked with --depth 9. Criu fails to dump the state SA-CX-MX_SBL (shown above) with the error Error (criu/sk-queue.c:151): recvmsg fail: error: Connection reset by peer Nearest plans: 1. Add generators for on-disk sockets names (now oly abstract). Here an interesting case is when names overlap and one socket gets a name of another, but isn't accessible by it 2. Add datagram sockets. Here it'd be fun to look at how many-to-one connections are generated and checked. 3. Add socketpair()-s. Farther plans: 1. Cut the tree better to allow for deeper tree scan. 2. Add restore. 3. Add SCM-s 4. Have the exhaustive testing for other resources. Changes since v1: * Added DGRAM sockets :) Dgram sockets are trickier that STREAM, as they can reconnect from one peer to another. Thus just limiting the tree depth results in wierd states when socket just changes peer. In the v1 of this patch new sockets were added to the state only when old ones reported that there's nothing that can be done with them. This limited the amount of stupid branches, but this strategy doesn't work with dgram due to reconnect. Due to this, change #2: * Added the --sockets NR option to limit the amount of sockets. This allowed to throw new sockets into the state on each step, which made a lot of interesting states for DGRAM ones. * Added the 'restore' stage and checks after it. After the process is restore the script performs as much checks as possible having the expected state description in memory. The checks verify that the values below get from real sockets match the expectations in generated state: - socket itself - name - listen state - pending connections - messages in queue (sender is not checked) - connectivity The latter is checked last, after all queues should be empty, by sending control messages with socket.recv() method. * Added --keep option to run all tests even if one of them fails. And print nice summary at the end. So far the test found several issues: - Dump doesn't work for half-closed connection with unread messages - Pending half-closed connection is not restored - Socket name is not restored - Message is not restored New TODO: - Check listen state is still possible to accept connections (?) - Add socketpair()s - Add on-disk names - Add SCM-s - Exhaustive script for other resources Signed-off-by: Pavel Emelyanov <[email protected]> Signed-off-by: Andrei Vagin <[email protected]>

Move xfree() up Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]>

By exhaustive testing I understand a test suite that generates as much states to try to C/R as possible by trying all the possible sequences of system calls. Since such a generation, if done on all the Linux API we support in CRIU, would produce bazillions of process, I propose to start with something simple. As a starting point -- unix stream sockets with abstract names that can be created and used by a single process :) The script generates situations in which unix sockets can get into by using a pre-defined set of system calls. In this patch the syscalls are socket, listen, bind, accept, connect and send. Also the nummber of system calls to use (i.e. -- the depth of the tree) is limited by the --depth option. There are three things that can be done with a generated 'state': I) Generate :) and show Generation is done by recursively doing everything that is possible (and makes sence) in a given state. To reduce the size of the tree some meaningless branches are cut, e.g. creating a socket and closing it right after that, creating two similar sockets one-by-one and some more. Shown on the screen is a cryptic string, e.g. 'SA-CX-MX_SBL one, describing the sockets in the state. This is how it can be decoded: - sockets are delimited with _ - first goes type (S -- stream, D --datagram) - next goes name state (A -- no name, B with name, X socket is not in FD table, i.e. closed or not yet accepted) - next may go letter L meaning that the socket is listening - -Cx -- socket is connected and x is the peer's name state - -Ixyz -- socket has incoming connections queue and xyz are the connect()-ors name states - -Mxyz -- socket has messages and xyz is senders' name states The example above means, that we have two sockets: - SA-CX-MX: stream, with no name, connected to a dead one and with a message from a dead one - SBL: stream, with name, listening Next printed is the sequence of system calls to get into it, e.g. this is how to get into the state above: socket(S) = 1 bind(1, $name-1) listen(1) socket(S) = 2 connect(2, $name-1) accept(1) = 3 send(2, $message-0) send(3, $message-0) close(3) Program has created a stream socket, bound it, listened it, then created another stream socket, connected to the 1st one, then accepted the connection sent two messages vice-versa and closed the accepted end, so the 1st socket left connected to the dead socket with a message from it. II) Run the state This is when test actually creates a process that does the syscalls required to get into the generated state (and hopefully gets into it). III) Check C/R of the state This is the trickiest part when it comes to the R step -- it's not clear how to validate that the state restored is correct. But if only trying to dump the state -- it's just calling criu dump. As images dir the state string description is used. One may choose only to generate the states with --gen option. One may choose only to run the states with --run option. The latter is useful to verify that the states generator is actually producing valid states. If no options given, the state is also dump-ed (restore is to come later). For now the usage experience is like this: - Going --depth 10 --gen (i.e. just generating all possibles states that are acheivable with 10 syscalls) produces 44 unique states for 0.01 seconds. The generated result covers some static tests we have in zdtm :) More generation stats is like this: --depth 15 : 1.1 sec / 72 states --depth 18 : 13.2 sec / 89 states --depth 20 : 1 m 8 sec / 101 state - Running and trying with criu is checked with --depth 9. Criu fails to dump the state SA-CX-MX_SBL (shown above) with the error Error (criu/sk-queue.c:151): recvmsg fail: error: Connection reset by peer Nearest plans: 1. Add generators for on-disk sockets names (now oly abstract). Here an interesting case is when names overlap and one socket gets a name of another, but isn't accessible by it 2. Add datagram sockets. Here it'd be fun to look at how many-to-one connections are generated and checked. 3. Add socketpair()-s. Farther plans: 1. Cut the tree better to allow for deeper tree scan. 2. Add restore. 3. Add SCM-s 4. Have the exhaustive testing for other resources. Changes since v1: * Added DGRAM sockets :) Dgram sockets are trickier that STREAM, as they can reconnect from one peer to another. Thus just limiting the tree depth results in wierd states when socket just changes peer. In the v1 of this patch new sockets were added to the state only when old ones reported that there's nothing that can be done with them. This limited the amount of stupid branches, but this strategy doesn't work with dgram due to reconnect. Due to this, change #2: * Added the --sockets NR option to limit the amount of sockets. This allowed to throw new sockets into the state on each step, which made a lot of interesting states for DGRAM ones. * Added the 'restore' stage and checks after it. After the process is restore the script performs as much checks as possible having the expected state description in memory. The checks verify that the values below get from real sockets match the expectations in generated state: - socket itself - name - listen state - pending connections - messages in queue (sender is not checked) - connectivity The latter is checked last, after all queues should be empty, by sending control messages with socket.recv() method. * Added --keep option to run all tests even if one of them fails. And print nice summary at the end. So far the test found several issues: - Dump doesn't work for half-closed connection with unread messages - Pending half-closed connection is not restored - Socket name is not restored - Message is not restored New TODO: - Check listen state is still possible to accept connections (?) - Add socketpair()s - Add on-disk names - Add SCM-s - Exhaustive script for other resources Signed-off-by: Pavel Emelyanov <[email protected]> Signed-off-by: Andrei Vagin <[email protected]>

Segmentation fault was raised while trying to restore a process with tty. Coredump file says this is caused by uninitialized tty_mutex: (gdb) where #0 0x00000000004d7270 in atomic_add_return (i=1, v=0x0) at include/common/asm/atomic.h:34 #1 0x00000000004d7398 in mutex_lock (m=0x0) at include/common/lock.h:151 #2 0x00000000004d840c in __pty_open_ptmx_index (index=3, flags=2, cb=0x4dce50 <open_pty>, arg=0x11, path=0x5562e0 "ptmx") at criu/tty.c:603 #3 0x00000000004dced8 in pty_create_ptmx_index (dfd=17, index=3, flags=2) at criu/tty.c:2384 since init_tty_mutex() is reentrantable, just calling it before mutex_lock() Signed-off-by: Deng Guangxing <[email protected]> Reviewed-by: Cyrill Gorcunov <[email protected]> Signed-off-by: Andrei Vagin <[email protected]>

CID 302717 (#2 of 2): Resource leak (RESOURCE_LEAK) Variable dirnew going out of scope leaks the storage it points to. Signed-off-by: Adrian Reber <[email protected]>

CID 226486 (#1 of 2): Resource leak (RESOURCE_LEAK) Variable mi going out of scope leaks the storage it points to. CID 226486 (#2 of 2): Resource leak (RESOURCE_LEAK) Variable mi going out of scope leaks the storage it points to. Signed-off-by: Adrian Reber <[email protected]>

CID 226485 (#1 of 3): Resource leak (RESOURCE_LEAK) Variable events going out of scope leaks the storage it points to CID 226485 (#2 of 3): Resource leak (RESOURCE_LEAK) Variable events going out of scope leaks the storage it points to CID 226485 (#3 of 3): Resource leak (RESOURCE_LEAK) Variable events going out of scope leaks the storage it points to Also changed epoll_prepare() to check return value of epoll_create() against '< 0' instead if '== -1' to make coverity happy. Signed-off-by: Adrian Reber <[email protected]>

CID 226478 (#1 of 2): Double close (USE_AFTER_FREE) Calling close(int) closes handle fd which has already been closed. CID 226478 (#2 of 2): Double close (USE_AFTER_FREE) Calling close(int) closes handle fd which has already been closed. Signed-off-by: Adrian Reber <[email protected]>

CID 73358 (#2 of 2): Argument cannot be negative (NEGATIVE_RETURNS) sk is passed to a parameter that cannot be negative. Signed-off-by: Adrian Reber <[email protected]>

CID 302717 (#2 of 2): Resource leak (RESOURCE_LEAK) Variable dirnew going out of scope leaks the storage it points to. Signed-off-by: Adrian Reber <[email protected]>

CID 226486 (#1 of 2): Resource leak (RESOURCE_LEAK) Variable mi going out of scope leaks the storage it points to. CID 226486 (#2 of 2): Resource leak (RESOURCE_LEAK) Variable mi going out of scope leaks the storage it points to. Signed-off-by: Adrian Reber <[email protected]>

CID 226485 (#1 of 3): Resource leak (RESOURCE_LEAK) Variable events going out of scope leaks the storage it points to CID 226485 (#2 of 3): Resource leak (RESOURCE_LEAK) Variable events going out of scope leaks the storage it points to CID 226485 (#3 of 3): Resource leak (RESOURCE_LEAK) Variable events going out of scope leaks the storage it points to Also changed epoll_prepare() to check return value of epoll_create() against '< 0' instead if '== -1' to make coverity happy. Signed-off-by: Adrian Reber <[email protected]>

CID 226478 (#1 of 2): Double close (USE_AFTER_FREE) Calling close(int) closes handle fd which has already been closed. CID 226478 (#2 of 2): Double close (USE_AFTER_FREE) Calling close(int) closes handle fd which has already been closed. Signed-off-by: Adrian Reber <[email protected]>

CID 73358 (#2 of 2): Argument cannot be negative (NEGATIVE_RETURNS) sk is passed to a parameter that cannot be negative. Signed-off-by: Adrian Reber <[email protected]>

Since child's pid_ns may have user_ns not equal to parent's, and we do not want to lose parent's user_ns (as it's not possible to restore it back), create the child from a sub-process. v3: New Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]> +++ restore: Simplify do_fork_with_pid() memcpy() is not need here, as we rewrite all the fields later. Also, use PID_SIZE() helper. Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]> +++ restore: Simplify do_fork_with_pid() #2 Move xfree() up Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]> +++ restore: Delete excess code in call_clone_fn() We never call this function for root_item. It's for dropping user ns, which may happen with the rest of tasks only. Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]> +++ ns: Move forked task user_ns assignment Child reaper of a ns have initial user_ns equal to its pid_ns->user_ns. Keep all ns assignments together. v2: Delete the assignment from call_clone_fn() and rename patch. Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]> rebase: merge fixups, add some sanity BUG_ON's Signed-off-by: Pavel Tikhomirov <[email protected]>

Since child's pid_ns may have user_ns not equal to parent's, and we do not want to lose parent's user_ns (as it's not possible to restore it back), create the child from a sub-process. v3: New Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]> +++ restore: Simplify do_fork_with_pid() memcpy() is not need here, as we rewrite all the fields later. Also, use PID_SIZE() helper. Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]> +++ restore: Simplify do_fork_with_pid() checkpoint-restore#2 Move xfree() up Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]> +++ restore: Delete excess code in call_clone_fn() We never call this function for root_item. It's for dropping user ns, which may happen with the rest of tasks only. Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]> +++ ns: Move forked task user_ns assignment Child reaper of a ns have initial user_ns equal to its pid_ns->user_ns. Keep all ns assignments together. v2: Delete the assignment from call_clone_fn() and rename patch. Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]> rebase: merge fixups, add some sanity BUG_ON's Signed-off-by: Pavel Tikhomirov <[email protected]> Feature: Pid & User namespaces

skhavari closed this as completed Aug 26, 2014

eliams mentioned this issue Sep 11, 2015

Issue mounting memory cgroup in Docker #28

Closed

xemul added a commit that referenced this issue Dec 24, 2015

proc: Fix aio mapping detection

d487ad7

This is fixlet to patch #2. Signed-off-by: Pavel Emelyanov <[email protected]> Looks-good-to-me: Andrew Vagin <[email protected]>

kikosha mentioned this issue May 29, 2016

Segmentation fault in libnl-3.so when dumping a process #169

Closed

avagin pushed a commit that referenced this issue Oct 17, 2017

restore: Simplify do_fork_with_pid() #2

a98d8f5

Move xfree() up Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]>

This was referenced Nov 22, 2017

Restore failed on Oracle Database Server 12c R2 #412

Open

CRIU 3.6 make failure on oracle linux 7.3 #416

Closed

avagin pushed a commit that referenced this issue Nov 29, 2017

restore: Simplify do_fork_with_pid() #2

2208c2a

Move xfree() up Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]>

avagin pushed a commit that referenced this issue Nov 30, 2017

restore: Simplify do_fork_with_pid() #2

f7b1e3f

Move xfree() up Signed-off-by: Kirill Tkhai <[email protected]> Signed-off-by: Andrei Vagin <[email protected]>

sunny868 mentioned this issue Sep 16, 2020

mips64: implement vdso_redirect_calls() #1205

Merged

avagin pushed a commit that referenced this issue Sep 30, 2020

cgroup: Resource leak (RESOURCE_LEAK)

91d9c25

CID 302717 (#2 of 2): Resource leak (RESOURCE_LEAK) Variable dirnew going out of scope leaks the storage it points to. Signed-off-by: Adrian Reber <[email protected]>

avagin pushed a commit that referenced this issue Oct 20, 2020

cgroup: Resource leak (RESOURCE_LEAK)

19365c1

CID 302717 (#2 of 2): Resource leak (RESOURCE_LEAK) Variable dirnew going out of scope leaks the storage it points to. Signed-off-by: Adrian Reber <[email protected]>

This was referenced Mar 17, 2021

When name is null, it will coredump #1411

Closed

When name is null, it will coredump #1412

Open

throwbear mentioned this issue May 15, 2021

[Python Supervisord] criu restore is successful, But the supervisord program coredump #1477

Open

Carol-code2222 mentioned this issue Feb 14, 2023

docker checkpoint create failed #2082

Closed

chenguanxi12138 mentioned this issue Jan 25, 2024

CRIU dump failed in docker:Error (criu/util.c:626): execvp("iptables-restore", ...) failed: No such file or directory #2339

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cannot checkpoint firefox on ubuntu 14 #2

cannot checkpoint firefox on ubuntu 14 #2

skhavari commented Aug 26, 2014

xemul commented Aug 26, 2014

skhavari commented Aug 26, 2014

cannot checkpoint firefox on ubuntu 14 #2

cannot checkpoint firefox on ubuntu 14 #2

Comments

skhavari commented Aug 26, 2014

Steps to reproduce

xemul commented Aug 26, 2014

skhavari commented Aug 26, 2014