Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: print stdout/stderr when dev server fails in tests #4918

Merged
merged 6 commits into from
Aug 10, 2022
Merged

Conversation

danez
Copy link
Contributor

@danez danez commented Aug 10, 2022

Summary

This adds a new log message once the edge function server is up and running. The tests wait for this before doing any requests, so we avoid timing issues when we request edge functions but the edge function server is not yet up.

The other improvement is that we log stdout and stderr when a test fails. This gives a lot more information and usually includes the reason why a request failed.

vector-illustration-little-cute-red-ant-vector-illustration-little-cute-red-ant-smiling-116379979

@danez danez added the type: chore work needed to keep the product and development running smoothly label Aug 10, 2022
@danez danez requested a review from a team August 10, 2022 11:44
@github-actions
Copy link

github-actions bot commented Aug 10, 2022

📊 Benchmark results

Comparing with 91f4ab2

Package size: 228 MB

⬇️ 0.00% decrease vs. 91f4ab2

^  230 MB  230 MB  230 MB  230 MB  230 MB  228 MB  228 MB  228 MB  228 MB  228 MB  228 MB  228 MB  228 MB 
│   ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
└───┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴──>
    T-12    T-11    T-10    T-9     T-8     T-7     T-6     T-5     T-4     T-3     T-2     T-1      T    
Legend

ps.stdout.on('data', (data) => {
outputBuffer.push(data)
if (!expectFailure && data.includes('Server now ready on')) {
if (!expectFailure && data.includes('Edge functions server running')) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that we were looking for Server now ready on because that's a line we always print in the dev server. We're now changing that to a string that only applies when we use edge functions? Can you add a bit more context here?

Copy link
Contributor Author

@danez danez Aug 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main problem here is that when we print Server now ready on the edge function server is not yet up and running, because the start function is async but we do not await it.. This makes total sense for users, not so much the tests where we pretty much instantly start the assertions/requests/etc of the tests once this is printed.

Somehow I had the perception yesterday that this message is printed after the edge function proxy server is started. Looking at the code today that is not the case and this change does not make any sense. The heat is slowly (or fastly) making my brain not really work the last days.🥵

Still the question is how can we wait in tests for edge functions to be up before starting requests? Conditionally awaiting the edge functions seems one possibility.

const server = prepareServer()

if (NODE_ENV === test) {
 await server;
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like the idea of adding test-specific logic to the runtime code. Yes, it would be cool to have a reliable way for the tests to wait for the edge functions server, but if we can't do it without changing the runtime code itself, I'd rather continue with polling and flaky tests, personally.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I reverted this change and now this PR is only printing stdout and stderr on dev server failure.

The only other thing I added is to resolve in the next event loop iteration, to give the edge function server slightly more time to boot.

@@ -12,9 +12,6 @@ const { normalize } = require('./utils/snapshots')

const content = 'Hello World!'

// Remove extra '.'s resulting from different load times which lead to flaky snapshots
const frameworkDetectionNormalizer = (output) => normalize(output.replace(/\.+(?=\.)/, ''))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed because we do not use the dot reporter in wait-port #4730

@danez danez changed the title chore: wait in tests till edge functions server is running and print stdout/stderr on failure chore: print stdout/stderr when dev server fails in tests Aug 10, 2022
@danez danez added the automerge Add to Kodiak auto merge queue label Aug 10, 2022
@kodiakhq kodiakhq bot merged commit 868ec26 into main Aug 10, 2022
@kodiakhq kodiakhq bot deleted the better-tests branch August 10, 2022 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge Add to Kodiak auto merge queue type: chore work needed to keep the product and development running smoothly
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants