Home
This is a unit test library and configurable tester for ft_printf.
By default, it can check if your completed printf is pretty good or not pretty good.
It's more useful as a production tool while you're developing ft_prinf, because it lets you enable and disable entire blocks of tests at once, search and run tests by name, category, and previous run history, and in general perform quick regression testing. It's quick and easy to add your own tests, which I recommend on principle. It's built to be flexible and highly configurable, so you can use it how you wish.
I've added a lot of features recently, so if you notice any bugs let me know on slack --- @gfielder.
You have to have a Makefile in your project directory that will compile libftprintf.a as the default make option, and your libftprintf.a has to have ft_printf inside.
Other than this, it should be completely general to all ft_printf projects.
Requires PHP. All 42 lab computers should have PHP installed.
In the root of your repo, run this command:
git clone https://github.com/gavinfielder/pft.git pft && echo "pft/" >> .gitignore
If your libft.a is separate from libftprintf.a:
If you include all required .o files (including your libft) in libftprintf.a, this is not necessary. If you do NOT, and require your libft separate, you must set USE_SEPARATE_LIBFT=1 in options-config.ini. See options-config.ini, and it should be self-explanatory.
-
./test helpshows some help text and usage examples
The executable accepts the following as queries:
-
./test moulruns all the enabled tests whose name starts with a string, in this case 'moul' -
./test "d_*prec"is a wildcard search; this one runs all the enabled tests that have start with 'd_' and have 'prec' in the name. -
./test 42 84runs (enabled) test #42 through test #84 -
./test 42runs test #42 (also turns on debugger compatibility mode.) -
./testruns all the enabled tests -
./test -d [any of the above queries]runs the selected test(s) in debugger compatibility mode.
Wildcard-based searches have an implict '*' at the end. For example, ./test "*zeropad" runs all the tests that have 'zeropad' anywhere in the name.
unit_tests.c shows you all the tests that are available. Failing a test means that your output and/or return value was not the same as the libc printf. When this happens, there will be a new file, 'results.txt', that holds information about the failed test, the first line of code for the test (most of them are one line anyway), what printf printed, and what ft_printf printed.
You can add your own tests to unit_tests.c, following the same format as the existing tests. You do not need to do anything except write the function in this file and re-make. The new tests will be included in the test index and can be queried the same way.
Valid unit tests have the prototype int foo(void) and return the value returned by a call to the test function, which has the same prototype as printf.
PFT compiles with debugging symbols by default, and also by default, running a single test e.g. ./test 42 turns on debugger compatibility mode. You can force all tests to run in debugger compatibility mode by using the -d command line option e.g. ./test -d nospec. You can read more about command line options and debugger compatibility mode under Advanced Options.
tl;dr: To use PFT with lldb, use it like this: lldb ./test 42.
Of course, in order for lldb to read your own libftprintf.a, you must also use the -g flag in your own Makefile...
I have provided scripts that make it easy to enable and disable tests. The scripts accept the same basic queries as the ./test executable.
./disable-test s # All the tests that start with 's' are disabled
./enable-test s_null_ # All the tests that start with 's_null_' are enabled
./disable-test 42 84 # Tests 42 to 84 are disabled
./disable-test && ./enable-test s # Disables all tests except tests that start with 's'
Wildcard search:
./disable-test "*zeropad" # Disables all the tests that have 'zeropad' anywhere in the name
./enable-test "*null*prec" # Enables all the tests that have a 'null' followed by a 'prec'
./enable-test "s_*prec" # Enables all tests that start with 's_' and have a 'prec' in the nameYou can call ./enable-test (with no arguments) to enable all tests, but keep in mind that some tests are disabled by default because if you have not implemented certain bonuses, your ft_printf will segfault.
enable-test and disable-test also support the -[-|=|+][fnop] command line options to filter by test history. See Advanced Options for more on this.
options-config.ini includes an option to ignore return value checking. I included this because at the time of writing this, moulinette does not check return value, and from what I've seen, the return value is the #1 reason people fail a lot of PFT tests. I don't encourage ignoring the return value, but it is an option if you would like to.
The Makefile creates two versions of each unit test function, one that uses ft_printf, and one that uses printf. For each test, it redirects stdout to a file, then compares their return value. If the return value is identical, it opens both files and reads each one byte by byte until both reach EOF. If any single byte differs, the test fails.
These options are selected either in options-config.ini (to set the default run options) and/or as command line options.
In both cases, options are processed left to right, and can override previous selections.
| Option | On | Off | Note |
|---|---|---|---|
| Debugger compatibility Mode | -d |
default on for single tests, off otherwise | |
| Timeout | -t |
-T |
default on |
| Test history logging | -l |
-L |
default on; enables extra options |
| Include disabled tests | -a |
-A |
default off |
| Leaks test (BETA) | -k |
-K |
default off; turns off fork mode |
| Fork mode | -x |
-X |
default on |
| Handle Signals | -s |
-S |
default on (non-fork mode only) |
| Print run info before tests | -i |
-I |
default on |
| Refresh test results before each run | -r |
-R |
default on; off mean new test results are appended |
| Historical category | ||||
|---|---|---|---|---|
| Select only the following | = |
Recently failed tests | f |
|
| Include the following | + |
Recently passed tests | p |
|
| Exclude the following | - |
Outdated tests | o |
|
| Tests with no history | n |
| Option | On | Off | Note |
|---|---|---|---|
| Update test history log when complete | -w |
-W |
default on |
examples
-
./test --p xruns tests that start with 'x', but excludes tests that recently passed. -
./test -d=po nocrashruns tests in debug mode that start with 'nocrash' that are either passing or outdated. -
./test -=f sruns only recently failed tests that start with 's'.
On: -l ; Off: -L Default on.
By default, PFT will track the last time each test passes and fails, and print information about the most recent run of each test. Tests' history will also become 'outdated' if their age exceeds the value set in options-config.ini. This feature also allows selecting tests to run by whether they have recently passed or failed. See Additional options when test history logging is enabled.
By default, the PFT Makefile removes the test history whenever unit_tests.c is strictly newer. This prevents the test history from becoming corrupted when you add your own tests (potentially changing the test numbers). By default the enable-test and disable-test scripts, as they modify unit_tests.c, will trigger such removal of history.csv. There is an option in options-config.ini to disable the history removal behavior, as well as an option in each of these scripts to touch history.csv to prevent the removal trigger. Use these options at your own discretion.
On: -d Default off.
Debuggers tend to only work well on single-threaded single processes, so "debugger compatibility mode" means "disable forking and multithreading". It also disables signal handling. -d is identical to -XTS. By default, debugger compatibility mode is turned on when running a single test e.g. ./test 42.
On: -t ; Off: -T Default on.
Fails tests after a specified time interval. The timeout duration can be set in options-config.ini. Only available in fork mode.
On: -k ; Off: -K Default off.
When this option is on, a leaks test command will run after all tests are completed. This disables fork mode. Leaks test will not run when any test segfaulted or otherwise terminated abnormally (as in this case leaks can come from PFT).
The default leaks testing method is to call leaks(1) via system(3). You can specify a different method with the LEAKS_TEST_CMD option in options-config.ini.
This feature is still being tested. Do not rely on it completely.
On: -x ; Off: -X Default on.
By default, PFT calls ft_printf only on forked child processes to improve stability. Turning this option off means that tests will run in a single process and a single thread. Timeout is not available when fork mode is off.
On: -s ; Off: -S Default on.
(only applies to non-fork mode) When this option is off, signals will not be caught by PFT and PFT will stop immediately when a test aborts abnormally. When this option is on, abnormal terminations by certain signals will be caught by PFT and tests will continue running.
These options are defined in options-config.ini:
| Option | Description |
|---|---|
DEFAULT_RUN_OPTIONS |
(string) Determines default program behavior. These are the options that are selected by default at the start of the program; they are overridden by any command line options given at execution time. |
TIMEOUT_SECONDS |
(float) time in seconds for a test to time out, when timeouts are enabled |
TEST_OUTDATED_TIME |
(long) time in seconds for a test result history to become outdated. The default is 900 (15 minutes). You can set this value to 0 to always consider tests outdated, or to 9999999 to never consider test results outdated, or anything in between. |
MAKE_RE_ALSO_REBUILDS_LIBFTPRINTF |
(1 or 0) if 1, make re will also recompile your project |
IGNORE_RETURN_VALUE |
(1 or 0) When this option is 0, tests fail if ft_printf and printf have different return values. If 1, the return value is ignored. |
ENABLE_DISABLE_REMAKES_PFT |
(1 or 0) When this option is 1, the enable-test and disable-test scripts also re-make PFT when finished. |
ENABLE_DISABLE_TOUCHES_TEST_HISTORY |
(1 or 0) When this option is 1, the enable=test and disable-test scripts touch history.csv when finished so as to prevent a Makefile removal of the test history. |
REMOVE_HISTORY_WHEN_TESTS_NEW |
(1 or 0) If this option is 1, the makefile will remove history.csv whenever unit_tests.c is strictly newer. |
LEAKS_TEST_CMD |
(C macro) The command to run at the end of the test run when the leaks test (-k) is selected. The default is to invoke leaks(1) with system(3). |
TEST_FAIL_LOGGING_MAXBYTES |
(int) The maximum size of output strings to show in results.txt |
SINGLE_NUMBER_SINGLE_TEST |
(1 or 0) When this option is 1, single numeric arguments run a single test. If 0, single numeric arguments runs all tests starting at the given number. The latter is a legacy feature, but could still be useful if you are using only your own tests and are writing them as you develop |
SINGLE_TEST_TURNS_ON_LLDB_COMPAT_MODE |
(1 or 0) When this option is 1 (and SINGLE_NUMBER_SINGLE_TEST is also 1), single numeric arguments run the requested test in debugger compatibility mode unless explicitly overridden with command line options. |
LIBFTPRINTF_DIR |
(path) If you do not wish to have PFT inside your local repo, you can specify a different path for your project. |
I tried making and got a bunch of "undefined symbol" errors
Are these your libft functions? If so, you probably need to follow the installation directions under 'Installation' in order to include your libft.a separately from libftprintf.a
Help! Wildcard search isn't working!
For almost all shell terminals, the * needs to be escaped--usually, putting a string in double quotes is sufficient, but some terminals still treat it as a shell * even then. You can either escape it manually '\*', or, to make this feature compatible with all shells, I've made any character not valid for a C function name (alphanumeric + underscore) is now considered a wildcard. This means instead of \*, you can also use @, or anything else your terminal doesn't recognize as a special character. The same is true for the enable-test and disable-test scripts.
If something goes wrong--slack me @gfielder. I like testing, like people using good testing, and want to make this easier to use, so don't hesitate to contact me. And if you think it's a bug, let me know so I can fix it.
I encourage everyone to contribute to this, even if it's just adding tests to the library. To do this, fork and make pull requests.
Before making pull requests, please:
./enable-test && ./disable-test argnum && ./disable-test moul_notmandatory \
&& ./disable-test nocrash && ./disable-test moul_D && ./disable-test moul_F \
&& ./disable-test f_reserved_values_ && ./disable-test f_L_reserved_values_and if you add non-mandatory test cases or tests that can segfault, modify this block in the readme
I have a few ideas how to improve this:
- The unit test library could stand to be a little more consistent and concise with naming conventions.
- There are very few tests for the
*bonus. - Could add tests for the thousands separator optional format flag.
- Could add tests for the
nspecifier.
Feel free to give me suggestions, or code them yourself and make a pull request.
The test method itself was adapted from outdated moulinette test files a buddy gave me, from which the author was [email protected]. The vast majority of code was written by me. The tests prefixed moul_ were adapted from the moulinette test files, the tests with _ftfc_ were adapted from 42FileChecker, and all other tests (so far) were written by me.
Also thanks to: