Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

502 Bad Gateway when deploying large site #484

Closed
unwitting opened this issue Aug 20, 2019 · 17 comments
Closed

502 Bad Gateway when deploying large site #484

unwitting opened this issue Aug 20, 2019 · 17 comments
Labels
area: command: deploy type: bug code to address defects in shipped code

Comments

@unwitting
Copy link

unwitting commented Aug 20, 2019 •
edited
Loading

- Do you want to request a feature or report a bug?

Bug, sort of. We might be stretching what's classed as supported behaviour, in which case I'm requesting that that scope be widened to sites of this size.

- What is the current behavior?

When attempting to deploy a large site (upwards of 20000, sometimes 50000 total files) via the netlify CLI, we often see the following error either during the CDN diffing files phase or (as in this case) the Uploading files phase:

Deploying to live site URL...
- Hashing files...
✔ Finished hashing 57892 files
- CDN diffing files...
✔ CDN requesting 29854 files
- Uploading 29854 files
 ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
TextHTTPError: Bad Gateway
    at NetlifyAPI.uploadDeployFile (/usr/local/lib/node_modules/netlify-cli/node_modules/netlify/src/open-api/index.js:173:13)
    at <anonymous>
Exited with code 2

This is from a CI server environment, so one with a (presumably) pretty reliable and fast internet connection.

When run on a local machine, we get slightly more overview of the behaviour. The upload appears to work as expected right up until the end:

Deploying to live site URL...
✔ Finished hashing 57893 files
✔ CDN requesting 30907 files
⠋ (30906/30907) Uploading s/404.html...
 ›   Warning: TextHTTPError: 502 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
⠒ (30906/30907) Uploading s/404.html...TextHTTPError: Bad Gateway
    at NetlifyAPI.uploadDeployFile (~/.nvm/versions/node/v8.11.4/lib/node_modules/netlify-cli/node_modules/netlify/src/open-api/index.js:173:13)
    at <anonymous>

You can see above, the tool claims the bad gateway on the very last file (which in this case happens to be 404.html). Re-running the deploy at this point usually results in the CDN saying it doesn't need any of the files uploading again, and successfully finishing.

- If the current behavior is a bug, please provide the steps to reproduce.

I think reproduction steps might be to generate a site with around 50000 files and try to deploy it. I can try to form a simple Gatsby setup that might generate 50000 unique-looking files on each build if needed.

- What is the expected behavior?

A successful deploy.
@jtraulle
Copy link

I have the same issue for a very large site :

✔ Finished hashing 810170 files
- CDN diffing files...

 ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
TextHTTPError: Bad Gateway
    at NetlifyAPI.createSiteDeploy (~/.nvm/versions/node/v11.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/open-api/index.js:173:13)

@unwitting
Copy link
Author

Holy wow, 810k files 😅

@bwlng
Copy link

bwlng commented Oct 17, 2019

We have the same issue on a site w/ 5477 files. We generally get the Bad Gateway error the first time we run netlify deploy and then run the command again and the build runs without issue.

Deploying to draft URL...
✔ Finished hashing 5477 files
⧇ CDN diffing files... ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
⧆ CDN diffing files...TextHTTPError: Bad Gateway
    at NetlifyAPI.createSiteDeploy (~/.nvm/versions/node/v10.16.1/lib/node_modules/netlify-cli/node_modules/netlify/src/open-api/index.js:173:13)

@laurenskling
Copy link

I tried to upload a whopping 1.6m files, only got errors. Reached out to the support team, they said they would not be able to handle such a large site. They would only be able to handle about 40k files sites.

@devpatelwebdeveloper
Copy link

We are getting the same issue at 4000 files. Sometimes, it runs smooth with no issue and sometimes it keeps on failing whole day.

Deploying to live site URL...
- Hashing files...
 ✔️ Finished hashing 4496 files
- CDN diffing files...
 ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 › TextHTTPError: Bad Gateway at parseResponse (/usr/src/node_modules/netlifycli/node_modules/netlify/src/methods/response.js:9:11)
error Command failed with exit code 2.

info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

script returned exit code 1

@mkrn
Copy link

mkrn commented Mar 17, 2020

This is happening consistently on a website with about 2450 files, 1200 pages.
Curious if there's a way to deploy zip file using netlify deploy?

Command:

netlify deploy $DEPLOYFLAG --auth $NETLIFY_TOKEN --site=$NETLIFY_SITE_ID --json

  ›   Warning: TextHTTPError: 502
  ›   Warning: 
  ›   TextHTTPError: Bad Gateway
  ›
 TextHTTPError: Bad Gateway
     at parseResponse (/usr/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/response.js:9:11)
     at async callMethod (/usr/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/index.js:38:26)
     at async module.exports (/usr/lib/node_modules/netlify-cli/node_modules/netlify/src/deploy/index.js:78:16)
     at async NetlifyAPI.deploy (/usr/lib/node_modules/netlify-cli/node_modules/netlify/src/index.js:92:12)
     at async DeployCommand.run (/usr/lib/node_modules/netlify-cli/src/commands/deploy.js:158:17)
     at async DeployCommand._run (/usr/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:44:20)
     at async Config.runCommand (/usr/lib/node_modules/netlify-cli/node_modules/@oclif/config/lib/config.js:160:9)
     at async Main.run (/usr/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/main.js:21:9)
     at async Main._run (/usr/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:44:20)```

@lukehsiao
Copy link

I am also seeing this issue on a site with just over 4100 pages. However, I'm also not able to deploy manually with the drag/drop upload, meaning we are currently stuck and unable to deploy our site.

@devpatelwebdeveloper
Copy link

Netlify has open the issue #745 for this problem, but its medium priority

@xmflsct
Copy link

xmflsct commented May 13, 2020

Same here with 4k files.

@thandaanda
Copy link

We are having issues with 1000 files. Also, is there any way to check which particular file is failing?

@stephencweiss
Copy link

stephencweiss commented May 29, 2020 •
edited
Loading

I'm on the low end of site size here (~4800 pages), and noticed this just started happening:

$ netlify deploy
Deploy path: ~/code/blog/public
Deploying to draft URL...
✔ Finished hashing 4799 files
⊷ CDN diffing files... ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›

I got around it eventually by manually setting a 10 minute timeout: netlify deploy --timeout 6000

$ netlify deploy --timeout 6000 --message "with a timeout"
Deploy path: ~/code/blog/public
Deploying to draft URL...
✔ Finished hashing 4799 files✔ CDN requesting 2417 files
✔ Finished uploading 2417 assets✔ Draft deploy is live!

Not an ideal solution... but hopefully it helps others!
/shrug

@stephencweiss stephencweiss mentioned this issue May 31, 2020
Closed
danielgavrilov added a commit to owid/owid-grapher that referenced this issue Jul 8, 2020
@danielgavrilov
Increase netlify deploy timeout to 10min
cb5e5e7 
As suggested in netlify/cli#484.
@danielgavrilov
Copy link

We also started noticing flaky deploys 2 days ago, with all failures happening at the "CDN diffing" stage:

- Hashing files...
✔ Finished hashing 64624 files
- CDN diffing files...
 ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
TextHTTPError: Bad Gateway
    at parseResponse (~/live/node_modules/netlify/src/methods/response.js:12:11)
    at runMicrotasks (<anonymous>)
    at async callMethod (~/live/node_modules/netlify/src/methods/index.js:36:26)
    at async module.exports (~/live/node_modules/netlify/src/deploy/index.js:79:16)
    at async NetlifyAPI.deploy (~/live/node_modules/netlify/src/index.js:93:12)
    at async DeployCommand.run (~/live/node_modules/netlify-cli/src/commands/deploy.js:203:17)
    at async DeployCommand._run (~/live/node_modules/@oclif/command/lib/command.js:44:20)
    at async Config.runCommand (~/live/node_modules/@oclif/config/lib/config.js:151:9)
    at async Main.run (~/live/node_modules/@oclif/command/lib/main.js:21:9)

We upgraded to 2.58.0 yesterday, since there was a fix merged (#966) intended to address #745, which is related to this issue.

We still got another 2 deploy failures today at the “CDN diffing files…” stage, so it seems this is still an issue.

If it helps, our site is fully public: https://github.com/owid/owid-static

We can’t reliably reproduce the issue, but it was occurring between:

All the above were running netlify-cli 2.58.0 on a DigitalOcean droplet (London). We always run netlify deploy on the server instead of GitHub-triggered deploys just because it saves us a few minutes.

For now, we've increased the deploy timeout to 10min, as suggested here, and we will update this if we see any more failures.

@ViachaslauBohdan
Copy link

Guys do you know whether this kind of error can be related to this issue as well?

`✔ Client-side fallback created: 200.html
ℹ Ready to run nuxt serve or deploy dist/ directory
Deploy path: /root/repo/rgnuxt/dist
Functions path: /root/repo/rgnuxt/functions
Configuration path: /root/repo/rgnuxt/netlify.toml
Deploying to draft URL...

  • Hashing files...
    ✔ Finished hashing 29532 files and 1 functions
  • CDN diffing files...
    ✔ CDN requesting 19671 files and 0 functions
  • Uploading 19671 files
    › Warning: JSONHTTPError: undefined 500
    › Warning:
    › {
    › "name": "JSONHTTPError",
    › "status": 500,
    › "json": {
    › "status": 500,
    › "error": "Internal Server Error"
    › }
    › }
    ›
    JSONHTTPError: Internal Server Error
    at parseResponse (/repo/rgnuxt/node_modules/netlify/src/methods/response.js:12:11)
    at runMicrotasks ()
    at async callMethod (    /repo/rgnuxt/node_modules/netlify/src/methods/index.js:36:26)`

@erezrokah erezrokah self-assigned this Sep 3, 2020
@erezrokah erezrokah added area: command: deploy type: bug code to address defects in shipped code labels Sep 9, 2020
@erezrokah erezrokah mentioned this issue Sep 9, 2020
Open
@Auspicus
Copy link

Auspicus commented Sep 9, 2020

Setting a timeout as @stephencweiss suggested in #484 (comment) worked for me as a temporary workaround.

@actuallyakash
Copy link

A question! Does npm publish command also counts the hidden directories (.) and the node_modules files?

@erezrokah
Copy link
Contributor

A question! Does npm publish command also counts the hidden directories (.) and the node_modules files?

Hi @actuallyakash, do you mean netlify deploy? The logic for that is here:

cli/src/commands/deploy.js

Line 146 in 2b53f4a

const skipNodeModules = site.root === deployFolder

You can read about npm publish here.

If you have any other questions I suggest opening a new issue.

@erezrokah erezrokah removed their assignment Aug 12, 2021
@sarahetter
Copy link
Contributor

Closing as stale. Please re-open if needed

@MaxMood96 MaxMood96 mentioned this issue Jan 5, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area: command: deploy type: bug code to address defects in shipped code
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests