Manual testing of build #232.8660.202

[tyuldashev]

UTBot release (IDEA 2023.1.5 + plugin 2023.3) vs UTBot main (IDEA 2023.2 + plugin 2023.8.5058)

Fuzzer

Set fuzzer only generation, run on utbot-fuzzing/src/test/kotlin/org/utbot/fuzzing/samples, compare coverage and generated results. Do not mock and mock everything outside the class.

New version, no mocks

Baseline, no mocks

New version, mock outside class

Baseline, mock outside class

Symbolic 95% + 5% fuzzing

Samples from org.utbot.examples:

• strings
• collections
• mock
• lambda
• recursion
• algorithms
• primitives
• structures
• primitives
• stream
• exceptions

Tested features

• Parametrized tests
• Smoke of various frameworks TestNG, JUnit4, JUnit5
• Smoke of different build systems: IntelliJ, Maven, Gradle.
• Smoke tests with JDK8, 11, 17.

[tyuldashev]

Found regression problems:

• NoSuchElementException: List is empty. during fuzzing samples.Integers  #2511
• ClassNotFoundException: org.mockito.MockedConstruction$Context in some Gradle projects when generate tests with Fuzzing #2523

Found pre-existing problems:

• Uncompilable test for inner class, due improper naming #2515
• Uncompilable test is generated by fuzzer for Collections test #2514
• Symbolic execution cannot generate tests in case of specific Random usage #2520
• Fuzzing is stopped because of an error IllegalStateException: unknown type of value abc (class kotlin.String) #2527