chore: initial Azure Pipelines support

[kaylangan]

Summary

As discussed in #7211, this PR is to add initial support for CI on Azure Pipelines across Mac, Windows, and Linux. Currently, Travis doesn't run master on Windows so there's a test gap.

This PR adds the azure-pipelines.yaml file to run test-ci-partial on the 3 OS's with Node 10. This PR can also be updated to run multiple versions of Node if that's desired. Some of the tests were updated as well.

The tests on Windows are currently failing. One appears to be path separator related and another fs.watch related (which can be finicky on Windows).

You can see the output of the fork @willsmythe set up. He also set up test reporting:

If this change looks good, here's how to proceed to set up Azure Pipelines:

1. Merge this PR.
2. Set up the Azure Pipelines app with this repo. You'll need to have the appropriate repo permissions for this. You can find more info in our docs. If granting write access to Azure Pipelines is a non-starter, you can instead create an Azure Pipelines org and set up the connection using OAuth.
3. Follow the set up flow that immediately follows the app set up to create a Pipeline.

I'll update the CHANGELOG.md once the PR is created.

Test plan

@willsmythe did most of the leg work here and can comment more on the testing he did.

[facebook-github-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need the corporate CLA signed.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

[SimenB]

@kaylangan thanks!

@rickhanlonii, do you know who's got the correct permissions? @JoelMarcey perhaps?

[SimenB]

The failing tests seems to mostly be a case of not being able to replace rootDir, thus getting long paths.

@kaylangan what's the easiest way to debug? I've got access to a windows box through a virtualbox image, and those tests pass there (at least the few I tested now).

I'd guess the issue is somehow with this: https://github.com/facebook/jest/blob/7b7fd013508de475e2151e50847373713fc97c7e/e2e/Utils.js#L73-L90

No real need for us to roll our own here either, should be possible to find some module which can do this for us. E.g. npm uses this: https://www.npmjs.com/package/tacks

If not that, maybe cwd is wrong when running from os.tmpdir()

[JoelMarcey]

I just gave Azure Pipelines access to the Jest repo.

[kaylangan]

@SimenB Sorry for the delayed reply. It looks like the issue with the Windows tests was that TEMP uses the short path name (e.g. ../VSSADM~1/AppData/Local/Temp) whereas jest computes and writes a relative path that uses the long name (e.g. ../VssAdministrator/AppData/Local/Temp). I've updated azure-pipelines.yml to account for this, though it may be worth considering updating jest itself to handle this case. You can find the latest run here.

[SimenB]

Thanks!

though it may be worth considering updating jest itself to handle this case.

Not sure I get this. We just use os.tmpdir(), I think? We might call realpath on it, not sure as I'm not sure where the issue is. Is it Jest's cacheDirectory or
when we do writeFiles in the tests? Or in the reporter when we calculate the relative path for printing?

Also, what's the next step now that the app has access to the repo? Is it possible to log in to Azure Pipelines using GH?

[kaylangan]

Not sure I get this. We just use os.tmpdir(), I think? We might call realpath on it, not sure as I'm not sure where the issue is. Is it Jest's cacheDirectory or
when we do writeFiles in the tests? Or in the reporter when we calculate the relative path for printing?

The reason the tests failed were partly due to an inconsistency on the Azure Pipelines end (which we're working on). On our Windows machines, TMP and TEMP use the shortened path (VSSADM~1) which is what is returned by os.tmpdir(). Other environment variables like HOME and USERPROFILE use the full name (VssAdministrator).

I'd have to do more digging but it appears that either in writeFiles or the reporter, the full path name VssAdministrator is getting written. Perhaps when using path.resolve or path.relative? It only affected 30 or so tests.

Also, what's the next step now that the app has access to the repo? Is it possible to log in to Azure Pipelines using GH?

Unfortunately, we don't yet have log in with GitHub. But we're working on it!

@JoelMarcey when you gave access to the Azure Pipelines app, did you also create an Azure Pipelines organization? If you already have an Azure Pipelines org, you can merge this PR, then in the Azure Pipelines org click "New Pipeline" and it will detect the .azure-pipelines.yml file. Otherwise, you'll want to merge the PR then go through the flow that follows giving the Azure Pipelines app access to the repo (https://github.com/apps/azure-pipelines).

[willsmythe]

@SimenB - some, if not all, of the tests were failing because of how Jest was reporting the test paths.

When Jest is run from the short/8.3 form of a directory on Windows, "cwd" will stay the short/8.3 name, but "rootDir" (and other paths in the config) get expanded to their long form. When Jest then calculates the relative path for a test (for reporting), the "from" argument is in the short/8.3 form, but the "to" argument is in the normal/expanded form. This results in Jest reporting a path that is technically correct, but not expected ;)

This problem isn't unique to Azure Pipelines. You can see it on Windows by running Jest from the short and normal forms of the same directory:

Running Jest from normal/expanded directory:

C:\work\willsmythe\simple-jest-test>yarn test
yarn run v1.12.3
$ jest
 PASS  ./sum.test.js
  √ adds 1 + 2 to equal 3 (3ms)

(./sum.test.js lives in this directory and its path is correctly reported)

Running Jest from the short/8.3 form of the same directory:

C:\work\WILLSM~1\SIMPLE~1>yarn test
yarn run v1.12.3
$ jest
 PASS  ../../willsmythe/simple-jest-test/sum.test.js
  √ adds 1 + 2 to equal 3 (3ms)

(the reported path is technically correct, but different from above)

I think the right solution requires changes in Jest itself --- either changes to how paths like rootDir are (or are not) normalized, or how relative paths are being calculated for reporting.

We "fixed" the problem in Azure Pipelines by setting a variable that causes TEMP to point to a normal/expanded path. Alternatively, we could have changed e2e/runJest.js to expand the working directory that Jest is run under (i.e. call reapath-native), but this didn't feel like the right fix and would have caused a lot of extra file system calls.

Make sense?

[SimenB]

Yeah, that makes sense, thanks! I was able to reproduce in a windows image I've got. Will take a look 🙂

[SimenB]

@willsmythe I opened up #7598 which calls realpath on cwd (and rootDir for consistency) during our initial config normalization at startup. Hopefully it passes all tests and doesn't mess anything up (thinking network drives or some symlinked setup).

Do you know of any way to normalize the short form to the long form on windows that does not use realpath? I tried googling, found nothing

[SimenB]

@kaylangan @willsmythe Merged that PR, mind rebasing and seeing if things work without the workaround you added?

Also, there's currently a linting error (see https://circleci.com/gh/facebook/jest/42898), a quick yarn eslint e2e/Utils.js --fix should resolve it 🙂

[willsmythe]

@SimenB - thanks for the quick fix. We removed the VSTS_OVERWRITE_TEMP workaround, re-ran the build, but still hit some path-related problems on Windows (much fewer than before).

The failing tests were findRelatedFiles.test.js, jestRequireActual.test.js, and jestRequireMock.test.js (see build 369 for more details). The root cause for these was code in jest-cli (SearchSource.js) and jest-config (getCacheDirectory.js) that gets the current working directory and temp directory by calling process.cwd and os.tmpdir directly . To get these tests working, I made a targeted fix to both. I am certainly cool with backing these changes out, but wanted to get the tests working so we could have the conversation.

In general, it seems like you might need (or need to enforce) a common way to get the current working and temp directories. This would avoid sprinkling realpath(process.cwd()) / os.tmpdir all over (there are 32 places were process.cwd is called directly in non-test code still).

There was also a new test failure on macOS that surfaced after rebasing. It was in e2e\__tests__\hasteMapSize.test.js (introduced in #7580). This test constructs a new HasteMap and sets the root directory to a directory under temp. Apparently there are issues with "watch" on macOS when watching a non-real path (this caused the file change event to never trigger, which caused the test to eventually time out). I made a targeted fix to get around this, but it's worth discussing whether calls to realpath should happen directly in HasteMap instead (probably should, but I wasn't sure about the lifecycle for a HasteMap object).

To answer your previous question, I'm not aware of an alternate way to get the normal/expanded path for a short directory on Windows.

[willsmythe]

See latest build results in Azure Pipelines:

There is one intermittent test failure on Linux ("can press "u" to update snapshots"). It appears an extra line is occasionally getting picked up in stdout for this test, which causes a mismatch with the snapshot. We will investigate ...

[SimenB]

We moved cwd assignment in #7146, so making sure to access that variable makes sense to me instead of figuring it out other places makes sense to me. Probably just an oversight

it's worth discussing whether calls to realpath should happen directly in HasteMap instead (probably should, but I wasn't sure about the lifecycle for a HasteMap object).

@rubennorte thoughts on that?

---

Thank you so much for sticking with me through this btw. Very much appreciated! 🎉

[SimenB]

Looks like everything is green, that's awesome 😀 I'll see if I can ping some FB peeps to do the necessary configuration so we can land this

[willsmythe]

Awesome! Thanks a lot. Will From: Simen Bekkhus <[email protected]> Sent: Monday, January 14, 2019 3:57 PM To: facebook/jest <[email protected]> Cc: Will Smythe <[email protected]>; Mention <[email protected]> Subject: Re: [facebook/jest] chore: initial Azure Pipelines support (#7556) Looks like everything is green, that's awesome 😀 I'll see if I can ping som FB peeps to do the necessary configuration so we can land this — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebook%2Fjest%2Fpull%2F7556%23issuecomment-454158412&data=02%7C01%7Cwismythe%40microsoft.com%7Cf4834b89d2fc45c6236b08d67a62d7b3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636830962326058605&sdata=Lg07VLmONURMAz6UdM2jkBbkPq0Q%2BjKpdoXEM%2Bx%2BnUU%3D&reserved=0>, or mute the thread<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACYxjEU3QjTQqfWLGSL_xNc9j7VWsdsfks5vDO8igaJpZM4Zi55H&data=02%7C01%7Cwismythe%40microsoft.com%7Cf4834b89d2fc45c6236b08d67a62d7b3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636830962326058605&sdata=qgwNSJnMmt0YJBvQ2f%2BFepjchpv9bF3KkLbVi0fwbL8%3D&reserved=0>.

[Daniel15]

I'll see if I can ping some FB peeps to do the necessary configuration so we can land this

@SimenB - What changes do you need? Feel free to ping me if you have trouble getting anything done.

[Daniel15]

As a side note, there's still some tests that are skipped on Windows (search for skipSuiteOnWindows in the repo)... It'd be good to clean those up one day.

[SimenB]

Thanks @Daniel15! See this from @kaylangan:

@JoelMarcey when you gave access to the Azure Pipelines app, did you also create an Azure Pipelines organization? If you already have an Azure Pipelines org, you can merge this PR, then in the Azure Pipelines org click "New Pipeline" and it will detect the .azure-pipelines.yml file. Otherwise, you'll want to merge the PR then go through the flow that follows giving the Azure Pipelines app access to the repo (github.com/apps/azure-pipelines).

Also, I'm not sure if FB wants an organization, or if Jest should have its own?

As a side note, there's still some tests that are skipped on Windows (search for skipSuiteOnWindows in the repo)... It'd be good to clean those up one day.

Yeah, for sure. I had one passthrough in May (#6310), might be worth having another one

[SimenB]

you can merge this PR, then in the Azure Pipelines org click "New Pipeline" and it will detect the .azure-pipelines.yml file.

Ergh, I guess that means we can merge this, so it's detected correctly

[Daniel15]

Also, I'm not sure if FB wants an organization, or if Jest should have its own?

It's probably fine to have a Jest org. The jest org name is taken, but jestjs is available if you want to create that.

[SimenB]

Ah, I can do that, cool! Yeah, jestjs works, that matches the website domain.

Did that, however I got an error when trying to confirm.

https://dev.azure.com/jestjs/jest

[Daniel15]

Hmm, we may need an admin of the repo to do it. I don't have admin access to this repo, unfortunately. @cpojer Do you know who has admin access to this repo?

[SimenB]

@JoelMarcey has (IIRC, @cpojer does not).

[willsmythe]

@SimenB / @Daniel15 - we (Microsoft) can help you get the Azure Pipelines (aka Azure DevOps) organization name you want. In general, we recommend that your Azure DevOps org name matches your GitHub org name (and the project names within this org match your GitHub repo names).

If you want to go this route, let us know and we can help. In the meantime, feel free to continue down the path of setting up a "jestjs" organization (it can be renamed later).

[JoelMarcey]

Jest should be approved -- it was before, I just approved a second time. Hopefully that doesn't mess anything up.

[Daniel15]

@JoelMarcey I think someone with admin access to the repo needs to actually configure it on the Azure Pipelines end, as it has to add a webhook to the repo.

[SimenB]

I'll send an invite to the org to your fb email. If I understand things correctly, go to https://dev.azure.com/jestjs/jest/_build, hit "new" and follow the prompts

[Daniel15]

@SimenB Could you please send me an invite to the org? I usually use [email protected] as my Windows Live account, but you could use my FB account [email protected]. Joel can give me temporary admin access to the Jest org to get everything configured.

[SimenB]

Done! The live one 🙂 you should be able to invite the fb one if you want

[Daniel15]

Looks like I could add it and it's running successfully: https://dev.azure.com/jestjs/jest/_build/results?buildId=1

[SimenB]

Yeah, seems like it works!

#6650

Need a badge for the readme, and give it a week or something then turn off appveyor?

[kaylangan]

@SimenB I just submitted #7630 for the build badge. 😄

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.