Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(watchman): Parallelize Watchman calls in crawler again #5640

Merged
merged 3 commits into from
Feb 22, 2018
Merged

fix(watchman): Parallelize Watchman calls in crawler again #5640

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
69c6d8c
fix(watchman): Parallelize Watchman calls in crawler again
BYK Feb 22, 2018
ce4773f
Refactor: pull things into functions
BYK Feb 22, 2018
2b062f4
Fix: typo
BYK Feb 22, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 39 additions & 35 deletions packages/jest-haste-map/src/crawlers/watchman.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,48 +51,52 @@ module.exports = async function watchmanCrawl(

try {
const watchmanRoots = new Map();
for (const root of roots) {
const response = await cmd('watch-project', root);
const existing = watchmanRoots.get(response.watch);
// A root can only be filtered if it was never seen with a relative_path before
const canBeFiltered = !existing || existing.length > 0;
await Promise.all(
roots.map(async root => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a function that you could hoist out :-)

const response = await cmd('watch-project', root);
const existing = watchmanRoots.get(response.watch);
// A root can only be filtered if it was never seen with a relative_path before
const canBeFiltered = !existing || existing.length > 0;

if (canBeFiltered) {
if (response.relative_path) {
watchmanRoots.set(
response.watch,
(existing || []).concat(response.relative_path),
);
} else {
// Make the filter directories an empty array to signal that this root
// was already seen and needs to be watched for all files/directories
watchmanRoots.set(response.watch, []);
if (canBeFiltered) {
if (response.relative_path) {
watchmanRoots.set(
response.watch,
(existing || []).concat(response.relative_path),
);
} else {
// Make the filter directories an empty array to signal that this root
// was already seen and needs to be watched for all files/directories
watchmanRoots.set(response.watch, []);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Slightly concerned about race conditions here but it should be fine since the main thread is memory-safe, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, this is completely fine

}
}
}
}
}),
);

let shouldReset = false;
const watchmanFileResults = new Map();
for (const [root, directoryFilters] of watchmanRoots) {
const expression = Array.from(defaultWatchExpression);
if (directoryFilters.length > 0) {
expression.push([
'anyof',
...directoryFilters.map(dir => ['dirname', dir]),
]);
}
const fields = ['name', 'exists', 'mtime_ms'];
await Promise.all(
Array.from(watchmanRoots).map(async ([root, directoryFilters]) => {
const expression = Array.from(defaultWatchExpression);
if (directoryFilters.length > 0) {
expression.push([
'anyof',
...directoryFilters.map(dir => ['dirname', dir]),
]);
}
const fields = ['name', 'exists', 'mtime_ms'];

const query = clocks[root]
? // Use the `since` generator if we have a clock available
{expression, fields, since: clocks[root]}
: // Otherwise use the `suffix` generator
{expression, fields, suffix: extensions};
const query = clocks[root]
? // Use the `since` generator if we have a clock available
{expression, fields, since: clocks[root]}
: // Otherwise use the `suffix` generator
{expression, fields, suffix: extensions};

const response = await cmd('query', root, query);
shouldReset = shouldReset || response.is_fresh_instance;
watchmanFileResults.set(root, response);
}
const response = await cmd('query', root, query);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of what I did above, I can also just return [root, cmd('query', root, query)] here and then do

const watchmanFileResults = new Map(await Promise.all(...));

That said I still need to loop through responses and determine if shouldReset needs to be set to true, which would probably be

const watcmanRawResults = await Promise.all(...);
const watchmanFileResults = new Map(watcmanRawResults);
const shouldReset = watcmanRawResults.some(([_root, response]) => response.is_fresh_instance);

I'm kind of liking this new version but what do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine tbh

shouldReset = shouldReset || response.is_fresh_instance;
watchmanFileResults.set(root, response);
}),
);

// Reset the file map if watchman was restarted and sends us a list of files.
if (shouldReset) {
Expand Down