Implement a Fuzzy CI to catch `ocamlmerlin` regressions #1716

pitag-ha · 2023-12-04T14:52:04Z

This PR implements a Fuzzy CI, that catches end-to-end regressions in Merlin by checking the responses of a fixed set of ocamlmerlin queries. That set of queries is determined by a deterministic but randomly chosen set of file location samples on the Irmin code base. The CI passes automatically if the responses on all samples remain the same before and after the PR. If there's a diff in the responses, the CI fails and makes the diff available as a GH action artifact. There's a high-level description of the CI in much more detail on a new wiki page about the CI.

Motivation for the CI

Merlin already has quite a few end-to-end tests (which are also run in CI). All of those are behavior-specific: We want to test one concrete behavior and write a tailored test for it. Additionally to those end-to-end tests, we now also want to test non-specifically on a large randomized input set.

Some implementation details

There are two reasons why the data of the ocamlmerlin responses is generated in the CI instead of living in the repo: 1. The data is over 20 MB big. 2. The generation of the data takes over 10 minutes, and we don't want our contributors to have to re-generate it for PRs that change the ocamlmerlin responses.

With the current implementation, the CI takes about 14-15 min to run. That's about 2 min longer than the mac-os CI (edit: Actually nvm, the mac-os CI is only one job of the main.yml workflow, which in total takes over 20 min, so clearly longer than this workflow! :)), which is the currently longest CI. There'd be a straightforward way to reduce the running time of this CI by another ~30 seconds. Let me know if I should do that.

I also think that we should observe over the coming months if a given ocamlmerlin response change will always be duplicated among multiple samples. If that's the case, we can reduce the sample size, which would speed up the CI.

We can also discuss whether we want to reduce the number of query types we test. Currently, it's type-enclosing, occurrences,locate, complete-prefix, and errors. I think all of them are useful to test. What do you think?

pitag-ha · 2023-12-04T14:56:10Z

Btw, the CI is expected to fail on this PR: The first commit changes the ocamlmerlin behavior by adding a query_num value to its response. That's necessary for the simplified/"distilled" diff generated by the CI (there's more info about the distilled diff on the new wiki page). Let me know if you'd like me to factor out the query_num addition into a separate PR, e.g. to discuss whether the query_num should always form part of the response or only on an opt-in basis via a CLI arg.

github-actions · 2023-12-21T09:22:23Z

This PR changes the response of some of the ocamlmerlin queries, that were run and analyzed by the Merlin Fuzzy CI. The change is not considered a regression, the analysis of this PR has been approved in its following state:

URL to download the generated data sets and their diffs between PR base branch and merge branch (at the moment of approval): https://github.com/ocaml/merlin/actions/runs/7281268850
256-sha of full responses diff: ad6b69213dc31fe5be9d6a9b48138daff64c8a0eaf3f47ecdfb0c8e792d6d352

voodoos

We can also discuss whether we want to reduce the number of query types we test. Currently, it's type-enclosing, occurrences,locate, complete-prefix, and errors. I think all of them are useful to test. What do you think?

I think that's a good starting point.

I made a review pass on the PR. My main worry is future maintenance: the actions are quite complex. Do you know what, in day-to-day Merlin work might require changes to the CI ? I guess upgrading to a new version of OCaml is one, but are there other things we should take in consideration?

.github/workflows/fuzzy-ci.yml

voodoos · 2023-12-22T13:10:15Z

.github/workflows/fuzzy-ci.yml

+        --sample-size=30 \
+        --data=${{ env.data_dir }} \
+        --merlin=ocamlmerlin \
+        --project=irmin/src/irmin,irmin/src/irmin-pack,irmin/test/irmin-pack


Maybe we could take advantage of the monorepo to also run on more diverse projects and not only Irmin ?

Yes, adding more diverse projects to the workflow should be relatively easy (I'd then decrease the number of samples per file, though, to avoid having the CI run for too long). Do you have projects in mind?

I would target something that is part of Irmin's dependencies but didn't have a look yet (and probably won't before next year...).

We'd need to have a look if anything in the Irmin dependency cone adds additional value to the test base. As mentioned in the comment about the CI execution time, it's better for the execution time to have few files with lots of samples than to have lots of files with few samples.

(and probably won't before next year...).

(same for me :) )

.github/workflows/fuzzy-ci.yml

pitag-ha · 2023-12-22T17:08:05Z

My main worry is future maintenance: the actions are quite complex. Do you know what, in day-to-day Merlin work might require changes to the CI ?

I'd say that there are the following main areas of attention in "day-to-day" Merlin work:

Whenever a non-deterministic field is added to the ocamlmerlin responses, those fields need to get filtered by the CI. That filtering already needs to be done for the merlin end-to-end tests, where it's done in tests/merlin-wrapper. If we merge this CI, it will also need to be done for the CI in its dedicated step for it.
Upgrading to a new version of OCaml will require several adaptations. Among others, it requires re-generating Irmin's lock file. I've decided to use opam-monorepo to generate the lock file rather than opam lock for several reasons, such as it supports depexts, and a universe is a good base for caching. We can talk about the point of adapting the CI when upgrading OCaml more, if you want to, since it will be the most significant point of the CI's maintenance, and it's currently not ideally supported by the CI.
For proactive maintenance, the CI can be improved in the future by adding support for new queries / queries that become more important. Also, the sample set could possibly be improved. That will be easier once we have a good overview of what the CI is capable of with the current sample size and current sample code base choice, and in which cases it fails to catch a regression.

Like every CI, there are also a few Merlin-external factors that might need maintenance:

The workflow uses a couple of external actions. If external actions get out of date, they'll need to be upgraded. I've made sure to only use official external actions from ocaml and from actions: ocaml/setup-ocaml, actions/checkout, actions/cache, actions/download-artifact and actions/upload-artifact.
A few things are done manually, e.g. some manual GH API calls. In the case of GH API breaking changes, the CI would need to adapt.
There's a few clumsy details, e.g.: When reading the comments on the PR to find and parse the comment containing information about the last approved diff state, only the last page with 100 comments is read. So if there were ever more than 99 comments after the approval comment, the CI wouldn't work. I can fix that, if it happens. Or I can fix it rn if you prefer. (not the case anymore)
Some things rely on current GH actions primitives: I share a binary between different runners, assuming that all runners are hosted on x86 machines (currently true); I've split the workflow for permission-handling (currently recommended and possible).

pitag-ha · 2024-01-08T19:11:00Z

I've just updated the comment above, crossing out one maintenance point (after implementing a fix). I think maintenance of this CI would be quite ok, but do let me know if you think differently! The main maintenance burden (I hope) will be updating the CI when Merlin bumps the compiler. As you've mentioned, that happens only twice a year. I'm also very happy to update the CI myself those two times per year. What will need to be done will be to edit an env variable in the CI and to generate a lock file for the test code base Irmin.

If you think similarly as me about maintenance, this might be ready to be merged.

voodoos · 2024-01-09T12:47:55Z

Thanks a lot Sonja !

Let's merge it and see how it goes :-)

voodoos · 2024-01-09T12:49:18Z

Could rebase and squash commits into relevant steps ?

This commit adds an iterator called query_num to the ocamlmerlin output. The motivation for that is to identify server crashes (the server has crashed iff the counter is back to 0).

This commit implements a Fuzzy CI to catch end-to-end regressions in Merlin by checking the responses of a set of ocamlmerlin queries. That set of queries is determined by a deterministic but randomly chosen set of file location samples on the Irmin code base. The CI passes automatically if the responses on all samples remain the same before and after the PR. If there's a diff in the responses, the CI fails and makes the diff available as a GH action artifact. There's a high-level description of the CI in much more detail on https://github.com/ocaml/merlin/wiki/Merlin-Fuzzy-CI. Co-authored-by: Enguerrand Decorne <[email protected]>

pitag-ha · 2024-01-09T17:12:14Z

Cool, thanks! I've just squashed the commits.

As a heads-up: I'm planning to transfer the merl-an repo, which this CI uses, from my GH account to the Tarides GH org this week. If you want, we can wait for that to happen before you merge. Otherwise, you can merge now and I open a one-line PR to adapt the CI once merl-an is transferred.

voodoos · 2024-01-10T09:55:39Z

As a heads-up: I'm planning to transfer the merl-an repo, which this CI uses, from my GH account to the Tarides GH org this week. If you want, we can wait for that to happen before you merge.

If your confident this will happen during the next few weeks then let's wait for it 🙂

pitag-ha · 2024-01-29T13:44:23Z

In the end, I'm not prioritizing transferring merl-an at all. If that sounds good to you, @voodoos , it might be best to merge this PR now, and once I transfer merl-an, I open a 1-liner PR. Wdyt?

CHANGES: Thu Feb 22 14:00:42 CET 2024 + merlin binary - Add a "heap_mbytes" field to Merlin server responses to report heap usage (ocaml/merlin#1717) - Add cache stats to telemetry (ocaml/merlin#1711) - Add new SyntaxDocument command to find information about the node under the cursor (ocaml/merlin#1706) - Fix `FLG -pp ppx.exe -as-pp/-dump-ast` use of invalid shell redirection when direct process launch on Windows. (ocaml/merlin#1723, fixes ocaml/merlin#1722) - Add a query_num field to the `ocamlmerlin` responses to detect server crashes (ocaml/merlin#1716) - Jump to cases within a match statement (ocaml/merlin#1726) - Jump to `module-type` (ocaml/merlin#1728, partially fixes ocaml/merlin#1656) - Exposes stable functions for configuration handling and pattern variable destruction. (ocaml/merlin#1730) + editor modes - vim: load merlin under the ocamlinterface and ocamllex filetypes (ocaml/merlin#1340) - Fix merlinpp not using binary file open (ocaml/merlin#1725, fixes ocaml/merlin#1724)

pitag-ha force-pushed the fuzzy-ci-squashed branch 2 times, most recently from 1aacfd4 to 5924649 Compare December 4, 2023 16:49

voodoos added fuzzy-diff-looks-good and removed fuzzy-diff-looks-good labels Dec 4, 2023

pitag-ha force-pushed the fuzzy-ci-squashed branch 2 times, most recently from 31a334c to cbc4daa Compare December 7, 2023 15:35

voodoos added fuzzy-diff-looks-good and removed fuzzy-diff-looks-good labels Dec 7, 2023

pitag-ha mentioned this pull request Dec 19, 2023

Lie the ground for the Fuzzy CI #1719

Merged

pitag-ha force-pushed the fuzzy-ci-squashed branch from 8ed7d3c to c1c8175 Compare December 20, 2023 22:04

voodoos added the fuzzy-diff-looks-good label Dec 21, 2023

pitag-ha force-pushed the fuzzy-ci-squashed branch from c1c8175 to 1205048 Compare December 21, 2023 11:13

voodoos self-requested a review December 21, 2023 13:49

voodoos reviewed Dec 22, 2023

View reviewed changes

Add a query_num field to the ocamlmerlin output

b3a9476

This commit adds an iterator called query_num to the ocamlmerlin output. The motivation for that is to identify server crashes (the server has crashed iff the counter is back to 0).

pitag-ha force-pushed the fuzzy-ci-squashed branch from a89039a to 38ee345 Compare January 9, 2024 17:02

pitag-ha force-pushed the fuzzy-ci-squashed branch from 38ee345 to 842515e Compare January 9, 2024 17:03

voodoos merged commit d982cfc into ocaml:master Jan 29, 2024
12 checks passed

pitag-ha deleted the fuzzy-ci-squashed branch January 29, 2024 17:24

This was referenced Jan 29, 2024

[RFC] Fuzzy behavior tests/CI for Merlin #1657

Closed

Specified Regression CIs #1635

Closed

General Behavior CI #1634

Closed

voodoos mentioned this pull request Feb 22, 2024

[new release] merlin-lib, merlin (4.14-501/500/414) ocaml/opam-repository#25315

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement a Fuzzy CI to catch `ocamlmerlin` regressions #1716

Implement a Fuzzy CI to catch `ocamlmerlin` regressions #1716

pitag-ha commented Dec 4, 2023 •

edited

Loading

pitag-ha commented Dec 4, 2023

github-actions bot commented Dec 21, 2023

voodoos left a comment

voodoos Dec 22, 2023

pitag-ha Dec 22, 2023

voodoos Dec 22, 2023

pitag-ha Dec 22, 2023

pitag-ha commented Dec 22, 2023 •

edited

Loading

pitag-ha commented Jan 8, 2024

voodoos commented Jan 9, 2024

voodoos commented Jan 9, 2024

pitag-ha commented Jan 9, 2024

voodoos commented Jan 10, 2024 •

edited

Loading

pitag-ha commented Jan 29, 2024 •

edited by voodoos

Loading

Implement a Fuzzy CI to catch ocamlmerlin regressions #1716

Implement a Fuzzy CI to catch ocamlmerlin regressions #1716

Conversation

pitag-ha commented Dec 4, 2023 • edited Loading

Motivation for the CI

Some implementation details

pitag-ha commented Dec 4, 2023

github-actions bot commented Dec 21, 2023

voodoos left a comment

Choose a reason for hiding this comment

voodoos Dec 22, 2023

Choose a reason for hiding this comment

pitag-ha Dec 22, 2023

Choose a reason for hiding this comment

voodoos Dec 22, 2023

Choose a reason for hiding this comment

pitag-ha Dec 22, 2023

Choose a reason for hiding this comment

pitag-ha commented Dec 22, 2023 • edited Loading

pitag-ha commented Jan 8, 2024

voodoos commented Jan 9, 2024

voodoos commented Jan 9, 2024

pitag-ha commented Jan 9, 2024

voodoos commented Jan 10, 2024 • edited Loading

pitag-ha commented Jan 29, 2024 • edited by voodoos Loading

Implement a Fuzzy CI to catch `ocamlmerlin` regressions #1716

Implement a Fuzzy CI to catch `ocamlmerlin` regressions #1716

pitag-ha commented Dec 4, 2023 •

edited

Loading

pitag-ha commented Dec 22, 2023 •

edited

Loading

voodoos commented Jan 10, 2024 •

edited

Loading

pitag-ha commented Jan 29, 2024 •

edited by voodoos

Loading