Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test suite assumes too much about host system #51

Open
Apteryks opened this issue Dec 8, 2022 · 0 comments
Open

test suite assumes too much about host system #51

Apteryks opened this issue Dec 8, 2022 · 0 comments

Comments

@Apteryks
Copy link

Apteryks commented Dec 8, 2022

Hi,

The test suite assumes many details from the host system it runs on, which makes it fragile. For example, running it in the Guix build container, it fails like:

starting phase `check'
----> cukinia_cmd <----
[PASS] Running "true" is successful
[PASS] Running "false" is NOT successful [!]
----> cukinia_user <----
[FAIL] Checking user "root" exists
[PASS] Checking user "nonexistent" NOT exists [!]
----> cukinia_group <----
[FAIL] Checking group "games" exists
[PASS] Checking group "nonexistent" NOT exists [!]
----> cukinia_process <----
[FAIL] Checking process "systemd" running as any user
[FAIL] Checking process "systemd" running as root
[PASS] Checking process "systemd" NOT running as nonexistent [!]
[PASS] Checking process "nosuchprocess" NOT running as any user [!]
----> cukinia_http_request <----
[FAIL] HTTP 200 on localhost:631 (cups)
[PASS] Checking http url "http://localhost:633/" does NOT return 200 [!]
----> cukinia_test <----
[PASS] Checking file "/etc/passwd" exists
[PASS] Checking "/etc/passwd" is NOT a symlink [!]
----> cukinia_python_pkg <----
[FAIL] cukinia_python_pkg: interpreter not found
[PASS] cukinia_python_pkg: interpreter not found [!]
----> cukinia_mount <----
[FAIL] Checking proc on /proc type any (any)  mounted
[FAIL] Checking proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)  mounted
[PASS] Checking /dev/nonexistent on /mnt type any (any) NOT mounted [!]
----> cukinia_symlink <----
[FAIL] Checking link "/tmp/cukinia.27.link" does point to "/tmp/guix-build-cukinia-0.6.0.drv-0/tmp.3KCYj4Oohg"
[PASS] Checking link "/tmp/guix-build-cukinia-0.6.0.drv-0/tmp.3KCYj4Oohg" does NOT point to "/tmp/cukinia.27.link" [!]
[PASS] Checking link "/tmp/cukinia.27.link" does NOT point to "/dev/null" [!]
----> cukinia_kmod <----
[FAIL] Checking kernel module "vfat" is loaded
[PASS] Checking kernel module "nonexistent" is NOT loaded [!]
----> cukinia_kconf <----
[PASS] Checking kernel config "CONFIG_NONEXISTENT" NOT set to "y" [!]
----> cukinia_kversion <----
[PASS] Checking if kernel version is 6.0
----> cukinia_listen4 <----
[FAIL] Checking tcp v4 port "631" is open
[PASS] Checking tcp v4 port "632" is NOT open [!]
[FAIL] Checking udp v4 port "5353" is open
[FAIL] Checking tcp|udp v4 port "67" is open
[PASS] Checking tcp|udp v4 port "63530" is NOT open [!]
[PASS] Checking tcp v4 port "63530" is NOT open [!]
----> cukinia_systemd_* <----
[FAIL] Checking if systemd unit "atd.service" is active
[PASS] Checking for no failed systemd units
----> cukinia_i2c <----
[PASS] Checking if i2c-2-0xXX exists [!]
----> cukinia_gpio_* <----
[FAIL] Checking if gpio pins are well configured via libgpiod
[FAIL] Checking if gpio pins are well configured via sysfs
----> cukinia_knoerror <----
[PASS] Checking that kernel has booted with no error (using grep)
----> _ver2int <----
[PASS] _ver2int test 44.10.5 > 44.6.5
[PASS] _ver2int test 43.6.5 < 43.6.6
[PASS] _ver2int test 44.3.0 == 44.3
[PASS] _ver2int test 0.99.1 < 1.0.0
[PASS] _ver2int test 1.99 > 1.98.999.999
----> cukinia_conf_include <----
cukinia: warning: can't include "conf.d/*.conf"
----> cukinia_run_dir <----
----> color tests <----
* _colorize red
* _colorize green
* _colorize blue
* _colorize cyan
* _colorize yellow
* _colorize gray
* _colorize purple
----> failure detection (must all FAIL) <----
[FAIL] Running "false" is successful
[FAIL] Running "true" is NOT successful [!]
[FAIL] Running "test 0 -eq 1" returns success
[FAIL] Checking process "nosuchprocess" running as any user
[FAIL] cukinia_python_pkg: interpreter not found
[FAIL] Checking link "/dev/zero" does point to "/dev/null"
[FAIL] Checking if systemd unit "nosuchunit.service" is active
----> tests/failures counters <----
ran 50 tests, 23 failed

The host doesn't use systemd, there's no root user, etc. It'd be nicer to have a setup script that creates props to be validated, so that everything is controlled and deterministic, on any system.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant