Verify tests do not allocate resources early and do not leave them behind #15165

findepi · 2022-11-23T13:30:51Z

When running TestNG tests with surefire, the test class instance
initialization (class initializer, test instance construction) happens
before tests are run, for all instances. Allocating resources during
this phase is bad for these reasons:

resources are allocated long before they are needed, so tests may
exhaust available memory even if every single test is not memory-hungry
failure reporting during this phase is imperfect (it was observed that
actual failure stacktrace may not be reported in the logs), so
diagnosing failures is hard.

This commit adds a runtime verification attempting to ensure that test
instances do not hold on to any "resourceful" classes before the test is
started. Of course, we don't want to prohibit any field initialization
in test instances (that would affect readability and would be
frustrating), so "resourceful" classes are some known ones. The list can
be extended in the future.

Besides checking test instances before test is run, it also checks
instances after test class is completed, to catch any resources that
were left behind and not closed.

Adds test coverage for #15133

Already approved at #15151

...trino-testing-services/src/main/java/io/trino/testng/services/ReportResourceHungryTests.java

testing/trino-testing-services/src/main/java/io/trino/testng/services/Listeners.java

client/trino-jdbc/src/test/java/io/trino/jdbc/TestJdbcVendorCompatibility.java

findepi · 2022-11-25T14:16:55Z

CI #14814 , #15195

findepi · 2022-11-28T08:45:19Z

(just rebased)

MiguelWeezardo · 2022-11-28T10:05:05Z

 [check-commits-dispatcher (master, 7b4aba6c18a0e22ec7696b6a01e6e429cd1a5afb)](https://github.com/trinodb/trino/actions/runs/3555387076/jobs/5972063982#logs)
failed 2 days ago in 25s
Search logs
1s
24s
0s
Error: Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/runner/work/trino/trino/.github/actions/compile-commit'. Did you forget to run actions/checkout before running your local action?

I probably beed to rebase? @nineinchnick @MiguelWeezardo

Sorry about that, I will try to fix this check (#15210) so that PRs opened before it was merged won't need to be rebased.

core/trino-main/src/test/java/io/trino/execution/TestSetSessionTask.java

core/trino-main/src/test/java/io/trino/execution/TestResetSessionTask.java

core/trino-main/src/test/java/io/trino/cost/TestStatsCalculator.java

losipiuk · 2022-11-28T10:29:23Z

core/trino-main/src/test/java/io/trino/execution/TestMemoryRevokingScheduler.java

@@ -107,8 +108,12 @@ public void tearDown()
    {
        queryContexts.clear();
        memoryPool = null;
+        taskExecutor.stop();


should we use closer here to make sure all cleanup routines are called?

I don't see a need to. @AfterClass failure is a failure for tests execution so we won't ignore it.

...o-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergGetTableStatisticsOperations.java

...metheus/src/test/java/io/trino/plugin/prometheus/TestPrometheusQueryMatrixResponseParse.java

core/trino-main/src/test/java/io/trino/execution/TestSqlTaskManager.java

core/trino-main/src/test/java/io/trino/execution/TestSqlTask.java

losipiuk

I agree this is somewhat fragile - but at the same time it proves helpful in current shape. If we notice logic in verifier class deteriorates over time we can either drop it or make it more up-to-date.

findepi · 2022-11-28T13:20:17Z

I agree this is somewhat fragile - but at the same time it proves helpful in current shape. If we notice logic in verifier class deteriorates over time we can either drop it or make it more up-to-date.

agreed

Ever after `TestingTrinoServer.close()` is called, the `TestingTrinoServer` may hold on to quite some memory. Let's clear references to these upon test completion. This opportunistically includes freeing of a few more things that are of lesser importance.

It's never a good idea, so make it this fail cleanly. As an added bonus, it prevents use of `closeAfterClass` after closing was called. That could lead to resources being left behind.

When running TestNG tests with surefire, the test class instance initialization (class initializer, test instance construction) happens before tests are run, for all instances. Allocating resources during this phase is bad for these reasons: - resources are allocated long before they are needed, so tests may exhaust available memory even if every single test is not memory-hungry - failure reporting during this phase is imperfect (it was observed that actual failure stacktrace may not be reported in the logs), so diagnosing failures is hard. This commit adds a runtime verification attempting to ensure that test instances do not hold on to any "resourceful" classes before the test is started. Of course, we don't want to prohibit any field initialization in test instances (that would affect readability and would be frustrating), so "resourceful" classes are some known ones. The list can be extended in the future. Besides checking test instances before test is run, it also checks instances after test class is completed, to catch any resources that were left behind and not closed.

This also makes `DistributedQueryRunner.close` explicitly idempotent. Indeed, some tests call it twice, for example once explicitly and once via `QueryAssertions.close`.

kokosing · 2022-12-07T12:47:58Z

@findepi Thank you for doing this! This is great!

findepi · 2022-12-09T22:39:18Z

@kokosing with all due modesty, i think i agree!

findepi requested review from nineinchnick, losipiuk, MiguelWeezardo, ebyhr and hashhar November 23, 2022 13:30

cla-bot bot added the cla-signed label Nov 23, 2022

findepi mentioned this pull request Nov 23, 2022

Verify tests do not allocate resources early #15151

Closed

nineinchnick approved these changes Nov 23, 2022

View reviewed changes

martint reviewed Nov 23, 2022

View reviewed changes

...trino-testing-services/src/main/java/io/trino/testng/services/ReportResourceHungryTests.java Outdated Show resolved Hide resolved

github-actions bot added the jdbc Relates to Trino JDBC driver label Nov 23, 2022

findepi force-pushed the findepi/resourceful-tests branch 2 times, most recently from a5deb2a to f5524c0 Compare November 23, 2022 16:26

github-actions bot added the tests:hive label Nov 23, 2022

martint force-pushed the master branch from d8b802c to 6238bdc Compare November 23, 2022 18:10

findepi force-pushed the findepi/resourceful-tests branch 2 times, most recently from cd9062e to 0f1693d Compare November 24, 2022 11:18

findepi mentioned this pull request Nov 24, 2022

Allow Iceberg materialized views to federate #15108

Merged

findepi force-pushed the findepi/resourceful-tests branch 8 times, most recently from db97397 to 5a32f0c Compare November 25, 2022 10:17

findepi changed the title ~~Verify tests do not allocate resources early~~ Verify tests do not allocate resources early and do not leave them behind Nov 25, 2022

ksobolew reviewed Nov 25, 2022

View reviewed changes

testing/trino-testing-services/src/main/java/io/trino/testng/services/Listeners.java Show resolved Hide resolved

ksobolew reviewed Nov 25, 2022

View reviewed changes

client/trino-jdbc/src/test/java/io/trino/jdbc/TestJdbcVendorCompatibility.java Show resolved Hide resolved

findepi force-pushed the findepi/resourceful-tests branch from 5a32f0c to 6da68d1 Compare November 25, 2022 11:18