Introduce object creation using builders for fuzzers #2583

Markoutte · 2023-09-08T06:58:01Z

Description

We introduce new BuilderObjectValueProvider for Java that finds any method, that can create a value of given type. Together with AbstractObjectValueProvider it works only when ObjectValueProvider and others fails to build a value.

The provider searches any method from any class that can return object of this type or any type that can be inferenced from the given one. It doesn't matter if this method static or not, but if this method is not static, that we suppose that we have a builder class for it. As a builder class we try to find other methods that returns an instance of this builder. This logic added because of default way to create such builder, where any set or add method returns the instance of builder to support chain calling. This builder should work to find methods for factories and singletones as well.

To support this provider, we introduce new routine in fuzzer's API called Modify (the name maybe is not final). It is added for Seed.Recurisve and works as follows: it accepts a sequence of routines that changes or creates new object with given this instance and fuzzed values. For example, we firstly create builder's object and update it with builder method, then finally we call the method, that creates an object using this builder. This sequence is always called when creating an object and never changed its ordering.

In addition, you can use this method, apply some method to the instance and return the instance object itself, for example, to always close a Closable object.

As a result, ContestEstimator covers 53% of guava project code, before it covers only 43%.

How to test

Automated tests

All fuzzer tests should pass.

Manual tests

You can try to generate tests for this little project:

public interface Muzzi {
    int getA();
}

public class RealMuzzi implements Muzzi {

    private final int a;

    private RealMuzzi(int a) {
        this.a = a;
    }

    @Override
    public int getA() {
        return a;
    }

    public static class Builder {

        private int a;

        public Builder setA(int a) {
            this.a = a;
            return this;
        }

        public Muzzi build() {
            return new RealMuzzi(a);
        }
    }
}

public class MuzziTest {
    public int size(Muzzi muzzi) {
        if (muzzi.getA() < 0) {
            return -1;
        } else if (muzzi.getA() > 0) {
            return 1;
        }
        return 0;
    }
}

Self-check list

Check off the item if the statement is true. Hint: [x] is a marked item.

Please do not delete the list or its items.

I've set the proper labels for my PR (at least, for category and component).
PR title and description are clear and intelligible.
I've added enough comments to my code, particularly in hard-to-understand areas.
The functionality I've repaired, changed or added is covered with automated tests.
Manual tests have been provided optionally.
The documentation for the functionality I've been working on is up-to-date.

utbot-fuzzing/src/main/kotlin/org/utbot/fuzzing/Api.kt

utbot-java-fuzzing/src/main/kotlin/org/utbot/fuzzing/providers/Objects.kt

…oader

IlyaMuravjov · 2023-09-11T14:44:33Z

Now it finds bridge method org.springframework.format.number.money.CurrencyUnitFormatter.print(Object, Locale): String and latter crashes with ClassNotFoundException: javax.money.CurrencyUnit when trying to create a value for CurrencyUnitFormatter.

I think the codebase is not ready to deal with arbitrary classes that can be found on the classpath and the best solution for now is to make sure that we only create FuzzedType instances from classes that only mention classes that are on the classpath.

In other words some ValueProviders expect that using fuzzedType.classId.allMethods, fuzzedType.classId.allConstructors, and fuzzedType.classId.allDeclaredFieldIds won't throw an exception.

The easiest way to achieve that, that I can see is to make runJavaFuzzing pass ClassDependenciesCheckingValueProvider(delegate = ValueProvider.of(providers) to runFuzzing where ClassDependenciesCheckingValueProvider.generate(description, type):

returns sequenceOf(Seed.Simple(nullFuzzedValue(classClassId)) if type.classId.allMethods, type.classId.allConstructors, or type.classId.allDeclaredFieldIds throws any exceptions
otherwise it returns delegate.generate(description, type), preferably filtering out routines that accept classes, mentioning classes that are not on the classpath

Markoutte · 2023-09-12T05:09:21Z

@IlyaMuravjov I think that there're a many of other problems that we won't find by this review. Let's create separate issues with reproducers and stack traces for them and fix them. The main goal of this PR is to add this builder for Java, not for Spring. And I'd like to keep Java part as simple as possible without additional wrappers and delegators.

…iders # Conflicts: # utbot-spring-framework/src/main/kotlin/org/utbot/framework/context/spring/SpringApplicationContextImpl.kt

IlyaMuravjov · 2023-09-12T20:47:24Z

@Markoutte
I've pushed a commit that adds a decorator letting Spring test generation to work with BuilderObjectValueProvider without crashing with ClassNotFoundException. However, this problem is not Spring-specific, so I've also made an issue #2594 showing how it can be reproduced without Spring.

I merged main to resolve conflicts instead of rebasing onto main, because I didn't want to force push into your branch.

Also, this PR reveled a problem in instrumnted-process that marks executions as timed out even if the thing that actually timed out is the argument creation prior to method under test invocation, see #2595.

Next, it seems that Scene.v().classes doesn't contain all classes on the classpath, rather it only includes:

classes that are reachable from classes located in the .class files (as opposed to .jar files)
some classes from the JDK
classes that are reachable from org.utbot.framework.util.classesToLoad

Meaning that for large projects BuilderObjectValueProvider can get significantly slower and the result of 53% stated in the PR description depends on the project under test being provided as .jar file (similarly to how it's done in the ContestEstimator) and not .class files (similarly to how it's done when UtBot is used via Intellij plugin).

Finally, I tried to check by how much providing guava in form of .class files instead of .jar file affects performance, but when I tried generating tests for the following method, the fuzzer crashed with the following exception.

public static boolean isNull(LocalDateTime localDateTime) {
    if (localDateTime == null) return true;
    return false;
}

23:32:56.036 | INFO  | JavaLanguage              | Fuzzing is stopped because of an error
java.lang.ClassNotFoundException: com.google.common.io.TempFileCreator$JavaNioCreator$lambda_userPermissions_3__261
	at java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:587) ~[?:?]
	at org.utbot.common.FallbackClassLoader.loadClass(FallbackClassLoader.kt:41) ~[utbot-core-2023.09-SNAPSHOT.jar:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:520) ~[?:?]
	at org.utbot.framework.plugin.api.util.IdUtilKt.getJClass(IdUtil.kt:105) ~[utbot-framework-api-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.providers.BuilderObjectValueProvider$generate$1.invokeSuspend(Objects.kt:237) ~[utbot-java-fuzzing-2023.09-SNAPSHOT.jar:?]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.SequenceBuilderIterator.hasNext(SequenceBuilder.kt:129) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at org.utbot.fuzzing.ValueProvider$Combined$generate$1.invokeSuspend(Providers.kt:231) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.SequenceBuilderIterator.hasNext(SequenceBuilder.kt:129) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at org.utbot.fuzzing.ValueProvider$Combined$generate$1.invokeSuspend(Providers.kt:231) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.SequenceBuilderIterator.hasNext(SequenceBuilder.kt:129) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.FlatteningSequence$iterator$1.ensureItemIterator(Sequences.kt:307) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.FlatteningSequence$iterator$1.hasNext(Sequences.kt:303) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.SequencesKt___SequencesKt.toCollection(_Sequences.kt:787) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.SequencesKt___SequencesKt.toMutableList(_Sequences.kt:817) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.SequencesKt___SequencesKt.toList(_Sequences.kt:808) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at org.utbot.fuzzing.FuzzingApi$produce$seeds$1.apply(Api.kt:427) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi$produce$seeds$1.apply(Api.kt:426) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
	at org.utbot.fuzzing.FuzzingApi.produce(Api.kt:426) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz(Api.kt:399) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.reduce(Api.kt:484) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.produce(Api.kt:438) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz(Api.kt:399) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.reduce(Api.kt:519) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.produce(Api.kt:437) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz(Api.kt:399) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.reduce(Api.kt:519) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.produce(Api.kt:437) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz(Api.kt:399) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.reduce(Api.kt:539) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.produce(Api.kt:437) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz(Api.kt:399) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz$fuzzOne(Api.kt:331) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz(Api.kt:350) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.access$fuzz(Api.kt:1) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi$fuzz$2.invokeSuspend(Api.kt) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:284) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at org.utbot.common.ConcurrencyKt.runBlockingWithCancellationPredicate(Concurrency.kt:38) ~[utbot-core-2023.09-SNAPSHOT.jar:?]
	at org.utbot.framework.plugin.api.TestCaseGenerator$generate$2$3.invoke(TestCaseGenerator.kt:204) ~[utbot-framework-2023.09-SNAPSHOT.jar:?]
	at org.utbot.framework.plugin.api.TestCaseGenerator$generate$2$3.invoke(TestCaseGenerator.kt:203) ~[utbot-framework-2023.09-SNAPSHOT.jar:?]
	at org.utbot.common.ConcurrencyKt.runIgnoringCancellationException(Concurrency.kt:47) ~[utbot-core-2023.09-SNAPSHOT.jar:?]
	at org.utbot.framework.plugin.api.TestCaseGenerator.generate(TestCaseGenerator.kt:203) ~[utbot-framework-2023.09-SNAPSHOT.jar:?]
	at org.utbot.framework.process.EngineProcessMainKt$setup$4.invoke(EngineProcessMain.kt:127) ~[utbot-framework-2023.09-SNAPSHOT.jar:?]
	at org.utbot.framework.process.EngineProcessMainKt$setup$4.invoke(EngineProcessMain.kt:112) ~[utbot-framework-2023.09-SNAPSHOT.jar:?]
	at org.utbot.rd.IdleWatchdog$measureTimeForActiveCall$1$2$1.invoke(ClientProcessUtil.kt:115) ~[utbot-rd-2023.09-SNAPSHOT.jar:?]
	at org.utbot.rd.IdleWatchdog.wrapActive(ClientProcessUtil.kt:88) ~[utbot-rd-2023.09-SNAPSHOT.jar:?]
	at org.utbot.rd.IdleWatchdog$measureTimeForActiveCall$1.invoke(ClientProcessUtil.kt:114) ~[utbot-rd-2023.09-SNAPSHOT.jar:?]
	at com.jetbrains.rd.framework.IRdEndpoint$set$1.invoke(TaskInterfaces.kt:182) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.IRdEndpoint$set$1.invoke(TaskInterfaces.kt:182) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.impl.RdCall.onWireReceived(RdTask.kt:362) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.MessageBroker$invoke$2$2.invoke(MessageBroker.kt:57) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.MessageBroker$invoke$2$2.invoke(MessageBroker.kt:56) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.impl.ProtocolContexts.readMessageContextAndInvoke(ProtocolContexts.kt:148) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.MessageBroker$invoke$2.invoke(MessageBroker.kt:56) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.MessageBroker$invoke$2.invoke(MessageBroker.kt:54) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.util.threading.SingleThreadSchedulerBase.queue$lambda-3(SingleThreadScheduler.kt:41) ~[rd-core-2023.1.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]

…ing (#2561) * Support mock strategies and type replacement in Spring unit test fuzzing * Avoid mocking replaced types * Only use `InjectMockValueProvider` for `thisInstance` * Disallow non-mocks when mock can be used * Move all value provider decorators to one package * Improve adding of `ReplacedFuzzedTypeFlag` * Make `map` and `except` apply transformer/filter to ALL value providers * Avoid using `anyObjectValueProvider` that is removed in #2583 * Make `AnyObjectValueProvider` not extend any interfaces, remove unrelated comment * Fix compilation after rebase

Markoutte added 4 commits September 8, 2023 07:10

Introduce Builder value provider

a21fcfb

Rename finally to transformers

e5be238

Simplify code

4a45ad4

Fix problem when synthetic methods are used for test generation

510099e

Markoutte added ctg-enhancement New feature, improvement or change request comp-fuzzing Issue is related to the fuzzing labels Sep 8, 2023

Markoutte requested review from egiptipavel and IlyaMuravjov September 8, 2023 06:58

IlyaMuravjov mentioned this pull request Sep 8, 2023

Add executable generic parameter type resolving #2584

Merged

6 tasks

IlyaMuravjov requested changes Sep 8, 2023

View reviewed changes

Markoutte added 2 commits September 11, 2023 08:29

Add fuzzed summary

84d51c6

Remove unnecessary check

0852a76

Markoutte requested a review from IlyaMuravjov September 11, 2023 05:50

Preserve transformers when copying Seed.Recursive

59c99e5

IlyaMuravjov requested changes Sep 11, 2023

View reviewed changes

utbot-java-fuzzing/src/main/kotlin/org/utbot/fuzzing/providers/Objects.kt Show resolved Hide resolved

utbot-java-fuzzing/src/main/kotlin/org/utbot/fuzzing/providers/Objects.kt Outdated Show resolved Hide resolved

Markoutte added 2 commits September 11, 2023 14:43

Handle CNFE when class from method parameter is not present in classl…

1caa8de

…oader

Unify hierarchy calling method

19a9298

Markoutte requested a review from IlyaMuravjov September 11, 2023 11:49

IlyaMuravjov added a commit that referenced this pull request Sep 11, 2023

Avoid using anyObjectValueProvider that is removed in #2583

b130123

IlyaMuravjov mentioned this pull request Sep 12, 2023

Java fuzzer crashes when referencing classes not in the classpath #2594

Open

IlyaMuravjov added 2 commits September 12, 2023 22:11

Use null for partially unresolvable classes in fuzzed Spring tests

bbb102b

Merge remote-tracking branch 'origin/main' into pelevin/builders-prov…

b9de69d

…iders # Conflicts: # utbot-spring-framework/src/main/kotlin/org/utbot/framework/context/spring/SpringApplicationContextImpl.kt

IlyaMuravjov mentioned this pull request Sep 12, 2023

Multiple time out tests are generated by fuzzer #2595

Open

IlyaMuravjov mentioned this pull request Sep 15, 2023

Support mock strategies and type replacement in Spring unit test fuzzing #2561

Merged

6 tasks

IlyaMuravjov added a commit that referenced this pull request Sep 19, 2023

Avoid using anyObjectValueProvider that is removed in #2583

f64a5b7

IlyaMuravjov mentioned this pull request Sep 20, 2023

Fuzzer ignores generics of class StringMap extends HashMap<String, String> #2571

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce object creation using builders for fuzzers #2583

Introduce object creation using builders for fuzzers #2583

Markoutte commented Sep 8, 2023

IlyaMuravjov commented Sep 11, 2023

Markoutte commented Sep 12, 2023

IlyaMuravjov commented Sep 12, 2023

Introduce object creation using builders for fuzzers #2583

Are you sure you want to change the base?

Introduce object creation using builders for fuzzers #2583

Conversation

Markoutte commented Sep 8, 2023

Description

How to test

Automated tests

Manual tests

Self-check list

IlyaMuravjov commented Sep 11, 2023

Markoutte commented Sep 12, 2023

IlyaMuravjov commented Sep 12, 2023