Add integration tests

[thesoftwarephilosopher]

This will catch things like incorrect logic when creating/uploading deals, which we recently got bit by (e.g. bug created at 1485339 and fixed at c6788ba).

This will involve:

• Making MemoryDownloader and MemoryUploader classes for tests
• Creating helper functions to create realistic sample data & scenarios
• Adjusting uploader/downloader behaviors to better match HubSpot
• Probably separating high from low level logic in upserting engine jobs
• Add GH action to not allow PR merge unless tests pass

[bberenberg]

How does this relate to jest in #42?

[thesoftwarephilosopher]

The testing framework is out of the way now.

This just has the 1 task left, which is to either:

1. generate sample data and sample expectations for the test suite, or
2. write tests against real data (store in git-ignored fixtures) with real expectations.
3. Or both.

[bberenberg]

Can you share an estimate of work for 1?

[thesoftwarephilosopher]

Breaking 1 down:

1. Write up maybe 10 scenarios for license/transaction patterns and how the engine should act on them
   1. What contacts should be generated/updated?
   2. What deals should be generated/updated?
   3. What contacts/companies should be associated with the deals?
   4. Add some variations for when deals exist and are in various states.
   5. Add some variations with different partner / mass-provider domains.
2. Run them by you and adjust as needed
3. Turn them into tests

Step 1 could take me a few hours of analyzing the data and engine dry-run output.
Step 2 is async so it's hard to estimate.
Step 3 could also take a few hours.
So overall probably a day, possibly 2.

[bberenberg]

Can we do this via one of the tools at https://towardsdatascience.com/free-resources-for-generating-realistic-fake-data-da63836be1a8 In particular I’d like to see a tool that can read our prod exports and spit out a deterministic output which could be added to the repo and tested using gh actions on PRs

…

On Mon, Nov 15, 2021 at 10:17 PM Steven Degutis ***@***.***> wrote: Breaking 1 down: 1. Write up maybe 10 scenarios for license/transaction patterns and how the engine should act on them 1. What contacts should be generated/updated? 2. What deals should be generated/updated? 3. What contacts/companies should be associated with the deals? 4. Add some variations for when deals exist and are in various states. 5. Add some variations with different partner / mass-provider domains. 2. Run them by you and adjust as needed 3. Turn them into tests Step 1 could take me a few hours of analyzing the data and engine dry-run output. Step 2 is async so it's hard to estimate. Step 3 could also take a few hours. So overall probably a day, possibly 2. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAP7P7SV45QGA2OUIWOQ7TUMHELJANCNFSM5C52XK7A> .

[thesoftwarephilosopher]

Those don't help much. Each test should have 4 levels of data:

1. Plain-English purpose: "if a customer gets eval, then buys, then renews a year later, there should be 2 deals." This should have as minimal details as possible to convey a tangible business purpose.

2. High-level: "1 eval, then 1 purchase, then 1 renewal in 12 months." These details are core to the test scenario, e.g. "purchase date = eval date + 1 day" and "renewal date = purchase date + 12-months."

3. Medium-level: SENs, ATs, sale dates, license types, specific domains, etc. These details will only be needed for verifying test results and creating relationships between things, but their contents aren't important. We can easily create fakes for these.

4. Low-level: Company names, first/last names, phone numbers, etc. This stuff literally doesn't matter at all, but it's necessary to create real models for the engine to act on. This part is what those tools are good at faking.

Some of the tools here (and other similar tools) are okay at generating data for level 3, but we have to do just as much work at feeding them schema info (if not more) as we would if we just wrote fake-data generators ourselves.

[thesoftwarephilosopher]

That said, the engine has a detailed-log-level that outputs extremely helpful structure of MPAC records, engine events, and engine actions. Here's a real example with redactions:

Records                                                                                       
  Hosting   AddonLicenseId   Date         LicenseType   SaleType   Transaction     Amount     
  -------   --------------   ----         -----------   --------   -----------     ------     
  Server    L1111111         2015-03-02   EVALUATION                                          
  Server    L2222222         2015-03-02   EVALUATION                                          
  Server    3333333          2015-06-09   COMMERCIAL                                          
  Server    4444444          2015-12-15   COMMERCIAL                                          
  Server    L55555555        2018-11-08   EVALUATION                                          
  Server    3333333          2019-11-08   COMMERCIAL    Renewal    AT-66666666    $111.00     
  Server    3333333          2020-11-08   COMMERCIAL    Renewal    AT-777777777   $222.00     
  Server    3333333          2021-11-08   COMMERCIAL    Renewal    AT-888888888   $333.00     

Events                                                                                        
  Type       Licenses                      Transactions                                       
  ----       --------                      ------------                                       
  purchase   L1111111, L2222222, 3333333                                                      
  purchase   4444444                                                                          
  renewal                                  AT-66666666[3333333]                               
  renewal                                  AT-777777777[3333333]                              
  renewal                                  AT-888888888[3333333]                              

Pulling sample scenarios from the logs when running on real data, and then feeding them into a script that auto-redacts info, should get us sample data pretty easily for levels 2 and 3.

[thesoftwarephilosopher]

Regarding GH actions and PRs, we can pretty easily set the repo up to run npm test on PR and not allow merge unless it passes. And the integration tests can run inside that command. So that's pretty much solved, but I'll add it to the task list.

[thesoftwarephilosopher]

A good portion of the work for data-levels 2 and 3 is now done in #56. Running engine in dev mode now saves all intermediate engine data in a similar format as above, plus deal actions. The main thing left to make it suitable for creating test cases is redacting IDs, which can be automated pretty easily. We might also want to automate generating test-data directly (in JSON format) instead of in this format, which we'd have to manually turn into tests.

[thesoftwarephilosopher]

Just did a few small PRs towards this between other projects over the past few days.

We can now generate realistic sample scenarios with fake data, like this (all IDs/amounts are fake):

Records
  Hosting   AddonLicenseId   Date         LicenseType   SaleType   Transaction     Amount
  -------   --------------   ----         -----------   --------   -----------     ------
  Server    L40139679        2019-07-23   EVALUATION                                     
  Server    56820547         2019-07-23   COMMERCIAL    New        AT-911512978   $335.00
Events
  Type       Licenses              Transactions          
  ----       --------              ------------          
  purchase   L40139679, 56820547   AT-911512978[56820547]
Actions
  Nothing: 6257492075, via AT-911512978[56820547], stage=CLOSED_WON, amount=335

This shows scenario-centric aspects of the data that the engine is looking at when running its logic.

We'll be able to:

1. Easily skim the generated file
2. Single out test cases that would make a good test, either
   a. Because it's an important/common scenario to keep working right, or
   b. Because it's currently incorrect and we need to fix it
3. And use this to generate a new test.

So what's next?

We actually skim these output files and find good scenarios that we want to turn into tests. Once we find them, we'll need to feed them into a script (which we need to write) that:

• Redacts the rest of the license/transaction info (we can reuse the new redaction code)
• Outputs test-skeletons using the record/event/action data in more computer-friendly format (e.g. JSON)
• A very tiny bit of the tests will still need to be edited by hand, like adding a test description.

I think once those tasks are done, we'll have super good tests 👌💯

[thesoftwarephilosopher]

Just added GH-Action-based CI testing.

[bberenberg]

@sdegutis should we merge in #62 and close this? Not sure what the current status is.

[thesoftwarephilosopher]

Not yet, I have a bit of work to do to integrate it first, and after that I need to fill in some real-world business cases.

[bberenberg]

@sdegutis how is this done with an open task in the description?

[thesoftwarephilosopher]

Forgot to toggle it. Oops.