Insertions to mongo databases

[hghianni]

Hi @arcuri82 , I think development is almost ready so I would like to know your thoughts regarding the implementation.

Here are the key points:

1. I have introduced a new replacement for the class MappingMongoEntityInformation from Spring. This class stores information about the type of documents in a repository and the corresponding mapped collection. To make this replacement work, I made a modification in the method initConstructors of the class ThirdPartyClassReplacements. As I understand it, the parameters of a constructor should be cast to the ThirdPartyClass annotated, similar to how it's done with methods. Hence, I implemented this modification.
   Additionally, this information is retrieved during the start of the SUT so to ensure it is not lost I added the method initMongoHandler to SutController. However, I'm uncertain if this is the optimal solution.

2. While calculating the heuristic distance, if the collection is empty, the query is considered as failed, and relevant information is stored to facilitate insertion into the collection at a later stage. This includes capturing the type of documents present in the collection, which might have been collected during step 1.

3. For each failed find operation, a MongoDbAction is created with a single Object gene. Each field of the document's type is mapped to a gene using the MongoGeneBuilder.

4. I have introduced a new output mode for genes called ExtendedJson. This format aligns with Bson's structure, ensuring that type information is preserved. Thus, EJSON (Extended JSON) is obtained from MongoDbAction.

5. The EJSON is then parsed, converted into a document, and finally inserted into the collection.

Additionally, I have made some modifications to handle another database. However, I believe further improvements can be made to allow EvoMaster to handle multiple databases more seamlessly.

I would greatly appreciate your feedback. Thank you!

[arcuri82]

@hghianni thanks! i ll start to have a look at it. indeed, so far we always assumed there was at most only 1 single SQL database. most likely i ll need to do some refactoring to enable different kinds of initializing actions

[arcuri82]

thx!

[arcuri82]

add comment to explain reason for this specific option

[hghianni]

I post the comment also here:

There is an issue that arises when running the MongoEMGenerationTest after other Mongo tests. The problem is that MappingMongoEntityInformationReplacement does not work in that case, causing the test to fail. This replacement relies on the flag "instrumentMR_MONGO" to be true when the SUT starts. However, in the other tests, except for MongoEMGenerationTest, "instrumentMR_MONGO" is set to false when the SUT starts. It seems that when the replacement does not occur in the other tests, some component is left in an inconsistent state, or there may be a related cache that prevents the replacement from happening in the MongoEMGenerationTest. The issue can be resolved by using separate JVM processes for each test, which solves the problem.

[arcuri82]

hi @hghianni ,
isn't this the same issue we discussed once, and that I did refer to how to address with example from ExternalServiceMockingFlakyEMTest? ie:

 @BeforeAll
    public static void initClass() throws Exception {
        ServiceController serviceController = new ServiceController();
        EMConfig config = new EMConfig();
        config.setInstrumentMR_NET(true);
        SpringTestBase.initClass(serviceController, config);
    }

can you please try the equivalent solution for MongoDB? to see if indeed it is the same problem or not. recall that when running tests from IDE those settings in maven plugins would be not applied by default

[hghianni]

Hi @arcuri82 , MongoEMGenerationTest is already configured like that. If not, MappingMongoEntityInformation replacement would not happen, and the test would fail as instrumentMR_MONGO would be set as false when starting the controller. This is the only test that needs this replacement, so the others are not configured the same way. If I configure all like this, all tests will pass, but it shouldn't be required. Maybe is it a better solution that forcing all to run on different processes. It seems like if the replacement happened in the first test run, then it will happen in the MongoEMGeneration test. If it didn't happen, it will not happen in the MongoEMGeneration test. Is there a cache related to replacements that may prevent them if they previously were not possible?. I was able to reproduce the issue both with Maven and the IDE by running all tests in the same process.

[arcuri82]

hi @hghianni the only cache is the classloader. a class can be loaded only once in a classloader. if for any reason a class is loaded before the flag for the instrumentation is set, then it cannot be changed.

[arcuri82]

@hghianni hi. are you still working on this branch? is it marked as "work in progress". if you think it is stable (ie no major changes), I can merge it, and you can continue to work an another branch/PR

[hghianni]

Hi @arcuri82, I found some issues while testing with External Controllers and I'm fixing them now. I'm gonna push new commits probably tomorrow. After that I think we can merge it. Thank you!

[hghianni]

Hi @arcuri82, I changed how document's type is stored. Previously I stored the Java class itself and now I'm using ClassToSchema to convert it to OAS and then RestActionBuilder to convert from OAS to genes. As I understand as during the conversion the fields of the type are not set as required by ClassToSchema they will end as OptionalGenes. I prefer genes to be not optional. Is there any way to do so? Thanks!

[arcuri82]

hi @hghianni if you do not want them as optional, you need to mark the schema as required. however, note that optional genes are on by default, and only with low probability they are deactivated

[arcuri82]

@hghianni thanks. once you are done with the changes, write a message with me tagged, so I can review them

[hghianni]

Hi @arcuri82! I think it's ready.