Timing tests drift refactor #2588
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/morph test |
Result: FAILUREYour command has finished executing! Here's what you wrote!
Outputmbed Build Number: 772 Test failed! |
|
/morph test |
Result: FAILUREYour command has finished executing! Here's what you wrote!
Outputmbed Build Number: 773 Test failed! |
|
So good news, bad news, and then more good news: First good news: No timing test failures! 😄 We're currently looking into why the boards were reset, but in the meantime these tests seem to indicate that they at least help stabilize the timing tests. @mazimkhan Could you please sanity-check my changes? Thanks! |
|
There is actually an issue with the |
|
/morph test |
Result: SUCCESSYour command has finished executing! Here's what you wrote!
Outputmbed Build Number: 777 All builds and test passed! |
The timing host tests reported success even if the total drift was negative. This adds a check for this now. The wait_us test now does not use a timer and just waits for 100000 us between prints. This adds inherent drift, but it should still be well under the limit.
bridadan
force-pushed
the
timing-tests-drift-refactor
branch
from
d29fffe
to
1473240
Compare
|
I had to make one final test with the |
theotherjimmy
added a commit
that referenced
this pull request
Sep 19, 2016
Release mbed-os-5.1.4 Changes: New Targets: 2504: [Disco_F769NI] adding new target [#2504] 2654: DELTA_DFBM_NQ620 platform porting [#2654] 2615: [MTM_MTCONNECT04S] Added support for MTM_MTCONNECT04S [#2615] 2548: Nucleof303ze [#2548] Fixes: 2678: Fixing NCS36510 compile on Linux #2678 2657: [MAX326xx] Removed echoing of characters and carriage return. #2657 2651: Use lp_timer to count time in the deepsleep tests #2651 2645: NUCLEO_F446ZE - Enable mbed5 release version #2645 2643: Fix thread self termination #2643 2634: Updated USBHost for library changes #2634 2633: Updated USBDevice to use Callback #2633 2630: Test names not dependent on disk location of root #2630 2624: CFSTORE Bugfix for realloc() moving KV area and cfstore_file_t data structures not being updated correctly #2624 2623: DISCO_L476VG - Add Serial Flow Control pins + add SERIAL_FC macro #2623 2617: STM32F2xx - Enable Serial Flow Control #2617 2613: Correctly providing directories to build_apis #2613 2607: Fix uvisor memory tracing #2607 2604: Tools - Fix fill section size variation #2604 2601: Adding ON Semiconductor copyright notice to source and header files. #2601 2597: [HAL] Fixed "intrinsic is deprecated" warnings #2597 2596: [HAL] Improve memory tracer #2596 2594: Fix TCPServer constructor #2594 2593: Add app config command line switch for test and make #2593 2589: [NUC472] Fix heap configuration error with armcc #2589 2588: Timing tests drift refactor #2588 2587: add PTEx pins as option for SPI on Hexiwear - for SD Card Interface #2587 2584: Set size of callback irq array to IrqCnt #2584 2583: github issue and PR templates #2583 2582: [GCC_CR] fix runtime hang for baremetal build #2582 2580: lwip - Add check for previously-bound socket #2580 2579: lwip - Fix handling of max sockets in socket_accept #2579 2578: Fix double free in NanostackInterface #2578 2576: Add smoke test that builds example programs with mbed-cli #2576 2575: tools-config! - Allow an empty or mal-formed config to be passed to the config system #2575 2562: Fix GCC lazy init race condition and add test #2562 2559: [utest]: Allow the linker to remove any part of utest if not used #2559 2545: Added define guards for SEQUENTIAL_FLASH_JOURNAL_MAX_LOGGED_BLOBS so #2545 2538: STM32F4xx - Add support of ADC internal channels (Temp, VRef, VBat) #2538 2521: [NUCLEO_F207ZG] Add MBED5 capability #2521 2514: Updated FlexCan and SAI SDK drivers #2514 2487: Runtime dynamic memory tracing #2487 2442: Malloc heap info #2442 2419: [STM32F1] Add asynchronous serial #2419 2393: [tools] Prevent trace-backs from incomplete args #2393 2245: Refactor export subsystem #2245 2130: stm32 : reduce number of device.h files #2130
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR changes the way we typically do timing tests in the mbed-os tree.
Background
In the past, we've had tests that used the
wait_us_autohost test in our testing tools (for reference, this is located here in the htrun tool: https://github.com/ARMmbed/htrun/blob/master/mbed_host_tests/host_tests/wait_us_auto.py)The way this works is the device prints a key-value pair (
{{tick;n}}) that is picked up bywait_us_auto. It logs the timestamp at which it received this data over the serial line. The device never actually stops printing these "tick" events. Instead what happens is the test "times-out", but then the host test determines the result of the test by combing through the "ticks" and measuring the amount of deviation between each event. If any one of these events deviated by more that0.1seconds, the test fails.This approach is susceptible to the host PC being under load, especially if it can't read the serial port fast enough (ex. reliably within
0.1seconds). This can cause tests to fail when in fact the device was printing the ticks on time.New approach
This PR changes the approach a little bit. Instead of failing the test if any tick event deviates too much, it instead takes the average deviation and compares that against a maximum (
0.05seconds). It also compares the total time the test took against the following formula:If the above is true in addition to the average deviation being below the maximum, the test passes. This behavior is contained within a new host test called
timing_drift_auto.Changes
tests-integration-threaded_blinkywas removed since it was a near duplicate oftests-mbedmicro-rtos-mbed-basic. It also had a high rate of drift due to the way it was written.tests-mbed_drivers-tickerandtests-mbed_drivers-timeoutare no longer printing in ISR context.tests-mbed_drivers-wait_ususes a timer to measure how much longer it should "wait". This reduces the cumulative drift.tests-mbedmicro-rtos-mbed-basicuses a signal now to print in a different thread. This reduces the cumulative drift.timing_drift_auto:tests-mbed_drivers-tickertests-mbed_drivers-timeouttests-mbed_drivers-wait_ustests-mbedmicro-rtos-mbed-basictests-mbedmicro-rtos-mbed-timer