feat: render a simple markdown dashboard #88

laurentsenta · 2022-12-05T12:48:23Z

Closes #55

This PR adds a dashboard CI rendered in a table like seen in https://github.com/laurentsenta/test-plans/actions/runs/3620746503 and it adds a badge to the readme which links to the latest dashboard run:

.

(fail on purpose, for the sake of demonstration)

add a new action, generate-dashboard which loads a CSV file of testground results and outputs a markdown table
create a Dashboard workflow that calls the composer, generates tests, then generates the dashboard
create an Update Badge workflow, which updates the README badge with a link to the latest dashboard (based on @galargh's PR)

It assumes that a results.csv was generated after running the tests generated by the composer.
See: #55 (comment)

Todo

Getting the job summary URL from pl-strflt/job-summary-url-action doesn't work yet
Render the badge with failure if not all tests are passing
Remove the last commit wich drops other actions to help with debugging
Remove errors from the demo in results.csv so that CI is green again.

.github/workflows/update-badge.yml

- var substitution fixed - update names and jobs - simplif the escaping to make it easier to debug - fix the dirty detection (github uses nopipefail)

Co-authored-by: Piotr Galar <[email protected]>

MarcoPolo · 2022-12-06T22:16:27Z

Do we have to commit the files in dist/?

MarcoPolo

some small changes. But overall looks good, thanks! I think we're quite close here.

MarcoPolo · 2022-12-06T22:22:33Z

.github/actions/generate-dashboard/src/lib.ts

+  }
+
+  if (process.env.RUN_URL) {
+    result = `[${result}](${process.env.RUN_URL})`;


Is it possible to get a URL that drops you to the test output directly rather than just the run url?

MarcoPolo · 2022-12-06T22:29:24Z

.github/actions/generate-dashboard/tsconfig.json

+    // "moduleDetection": "auto",                        /* Control what method is used to detect module-format JS files. */
+
+    /* Modules */
+    "module": "commonjs" /* Specify what module code is generated. */,


Does this need to be commonjs or can it be es2020?

MarcoPolo · 2022-12-06T22:30:54Z

.github/actions/generate-dashboard/README.md

+
+## Usage
+
+`ts-node ./src/index.ts input_path.csv output_path.md`


Can we make this a script in package json so you can do npm run generate-dashboard input.csv output.md?

MarcoPolo · 2022-12-06T22:40:34Z

.github/actions/generate-dashboard/src/lib.ts

+export const listUniqPairs = (pairs: PairOfImplementation[]): string[] => {
+  const uniq = new Set<string>();
+
+  for (const [a, b] of pairs) {
+    uniq.add(a);
+    uniq.add(b);
+  }
+
+  return Array.from(uniq).sort();
+};


Suggested change

export const listUniqPairs = (pairs: PairOfImplementation[]): string[] => {

const uniq = new Set<string>();

for (const [a, b] of pairs) {

uniq.add(a);

uniq.add(b);

}

return Array.from(uniq).sort();

};

function uniqFromPairs<T>(pairs: Array<[T, T]>): Array<T> { return [...new Set(pairs.flat())] }

MarcoPolo · 2022-12-06T22:43:08Z

.github/actions/generate-dashboard/src/lib.ts

+// `     something x something-else    more info ignore`
+const RUN_ID_MATCHER = /\s*(\S+)\s+x\s+(\S+)\s*.*/;
+
+export const fromRunIdToCoordinate = (runId: string): PairOfImplementation => {


This isn't really a coordinate. This is more like fromRunIdToPair. Getting the coordinate is done separately.

MarcoPolo · 2022-12-06T22:45:51Z

.github/actions/generate-dashboard/src/lib.ts

+export const generateTable = (
+  results: ResultFile,
+  defaultValue: string = ":white_circle:",
+  testedCell: CellRender = defaultCellRender


Should this be called cellRenderer instead?

MarcoPolo · 2022-12-06T22:55:38Z

.github/actions/generate-dashboard/src/lib.ts

+
+    const cell = testedCell(a, b, result);
+
+    matrix[i + 1][j + 1] = cell;


This means that "A x B" is the same as "B x A" which may not be the case. This is true for our current "ping" test, but it's possible we may want "A x B" to mean A dials out to B.

MarcoPolo · 2022-12-06T22:56:58Z

.github/actions/generate-dashboard/src/lib.ts

+
+export type ResultLine = {
+  task_id: string;
+  run_id: string;


I would parse this and have the parsed version in here. So instead of run_id this would be test_pair: [string, string]. And rename this ResultLine to TestOutcome. This way these functions are separated from the actual semantics of the result.csv. This is useful because with the multidim interop the result.csv will look slightly different.

Then functions operate on test pairs instead of re-parsing with fromRunIdToCoordinate multiple times.

I know we agreed on this specific format of result.csv, I'm just asking for this change here so that we can handle slightly different result.csv formats. Basically one "parse" step between L128 & L129.

But this is a minor thing and easy to change later when we integrate this with the multidim interop work. So feel free to disregard.

laurentsenta · 2023-01-05T10:11:46Z

@MarcoPolo I noticed you use that code in https://github.com/libp2p/test-plans/blob/ed5893b0942401fe2351fdbd5240dd41f408d926/multidim-interop/src/lib.ts already, is it correct to assume we'll merge everything with #85 and we can close this PR?

MarcoPolo · 2023-01-05T16:49:01Z

Yes

laurentsenta and others added 10 commits December 5, 2022 10:35

chore(generate-dashboard): init repo

62d2e3b

generated-dashboard: add task id matcher

c47e814

generate-dashboard: move matcher to lib file

67baab5

generate-dashboard: prepare the main function

2326367

generate-dashboard: implement & test generation

0a9ef64

generate-dashboard: update main, add e2e test

5a0821e

generate-dashboard: add README.md

c4c573e

generate-dashboard: introduce RUN_URL option

155fdd2

ci: add generate-dashboard workflow

f33159e

ci: update badge whenever new interop dashboard is ready

721c7d3

laurentsenta marked this pull request as draft December 5, 2022 12:48

laurentsenta mentioned this pull request Dec 5, 2022

ci: update badge whenever new interop dashboard is ready #86

Closed

laurentsenta commented Dec 5, 2022

View reviewed changes

.github/workflows/update-badge.yml Show resolved Hide resolved

galargh reviewed Dec 5, 2022

View reviewed changes

.github/workflows/update-badge.yml Outdated Show resolved Hide resolved

laurentsenta and others added 6 commits December 5, 2022 14:55

ci: fix update badge tweak

04c092b

- var substitution fixed - update names and jobs - simplif the escaping to make it easier to debug - fix the dirty detection (github uses nopipefail)

.github/workflows/update-badge.yml: fix id

8eebac2

Co-authored-by: Piotr Galar <[email protected]>

generate-dashboard: add packaging & action.yml

dda4765

ci: render & output dashboard

c935360

README.md: add dashboard badge

529f083

ci: run dashboard nightly

1396e8f

laurentsenta force-pushed the feat/issue-55-simple-dashboard branch from 7d474af to 4634908 Compare December 5, 2022 14:03

laurentsenta added 2 commits December 5, 2022 15:05

ci: exit dashboard with eventual error

6863ad9

tmp: add coordination file results.csv

b742c2b

laurentsenta force-pushed the feat/issue-55-simple-dashboard branch from 4634908 to b742c2b Compare December 5, 2022 14:05

laurentsenta marked this pull request as ready for review December 5, 2022 14:13

generate-dashboard: fix typos in README.md

e14aad5

laurentsenta requested review from galargh, jxs and MarcoPolo December 5, 2022 14:16

laurentsenta mentioned this pull request Dec 5, 2022

Create a "simple" interop dashboard #55

Closed

13 tasks

p-shahi mentioned this pull request Dec 5, 2022

composer: add initial version #75

Closed

MarcoPolo requested changes Dec 6, 2022

View reviewed changes

MarcoPolo closed this Jan 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: render a simple markdown dashboard #88

feat: render a simple markdown dashboard #88

laurentsenta commented Dec 5, 2022 •

edited

Loading

MarcoPolo commented Dec 6, 2022

MarcoPolo left a comment

MarcoPolo Dec 6, 2022

MarcoPolo Dec 6, 2022

MarcoPolo Dec 6, 2022

MarcoPolo Dec 6, 2022

MarcoPolo Dec 6, 2022

MarcoPolo Dec 6, 2022

MarcoPolo Dec 6, 2022

MarcoPolo Dec 6, 2022

laurentsenta commented Jan 5, 2023

MarcoPolo commented Jan 5, 2023


		## Usage

		`ts-node ./src/index.ts input_path.csv output_path.md`


		const cell = testedCell(a, b, result);

		matrix[i + 1][j + 1] = cell;

feat: render a simple markdown dashboard #88

feat: render a simple markdown dashboard #88

Conversation

laurentsenta commented Dec 5, 2022 • edited Loading

Todo

MarcoPolo commented Dec 6, 2022

MarcoPolo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laurentsenta commented Jan 5, 2023

MarcoPolo commented Jan 5, 2023

laurentsenta commented Dec 5, 2022 •

edited

Loading