v2.2.4 Deploy failing on Amazon Elastic Beanstalk #1276
Comments
|
Searched a bit and this issue is probably same as this #https://github.com/christkv/kerberos/issues/33 Looks like this issue is limited to AWS Elastic beanstalk. I'm still not sure why I'm facing it now, been trying hard to figure out what changed between v2.2.2 and v2.2.4 that caused this. |
|
Are you running EB over CentOS 6? Maybe try changing the distro to debian, ubuntu or something else if you can. |
|
@flovilmart Elastic beanstalk provides only Amazon Linux as an option. As for the workaround mentioned in the link you posted, it seemed to work at first but subsequent deploys are still failing. It fails sometimes due to some issue with |
|
Ok... So the solution would be to remove the GCS adapter dependency from parse-server, and let the users that use it depend on it by themselves. That would be the quick/dirty fix. |
|
Yes and I feel all 3rd party integrations should be an external dependancy. OneSignal in case of push is already separated, Even Mailgun, S3, GCS and maybe if in future an Azure based file adapter or mandrill based mail adapter comes along, it makes sense to keep those external. |
|
They have been all externalizer already. We just kept them in as this is a breaking change, not backwards compatible. |
|
Ah I see. I don't think anyone is using parse-server yet for production so I guess the sooner you introduce these breaking changes the better since we are getting so close to a stable product. Most of the people are keeping up to date with releases. Thanks again for all the hard work. |
|
We may need to release a Major that's what's problematic... We can't really assume that. Not sure of the proportions using it as a npm module Opened #1339 |
|
Tried to deploy again and it failed. Here are the logs I'm afraid we have no choice but to make |
|
If you add gcc-c++.x86_64 to the yum packages elastic beanstalk config, does it help? |
|
How do I add that? I already have gcc. Here are the packages I'm installing according to the link you mentioned at the top: |
|
node needs a c++ compiler too... I'm testing now. that would be gcc-c++.noarch or gcc-c++ |
|
Adde this |
|
on my eb the package was not found, set to gcc-c++ instad |
|
still no luck for me, I start suspecting the problem is that node-gyp is missing. |
|
Same here, didn't work. I've noticed that the first time EB tries to deploy the app it works, any subsequent deployment fails. Right now I have to terminate the env and create it again to get 2.2.4 to work OR remove |
|
hmmmm that's interesting... we should put a not to the EB guys |
|
2.2.4 works for me (multiple deployments so far). This is my setup: It was failing for me first because the machine didn't have enough memory (t2.micro), |
|
@markuswinkler you're using elastic beanstalk? |
|
yes |
|
you gotta be kidding me.... |
|
t2.micro has about 1Gb of memory, are you saying that 1gb ain't enough? Surely something is not right. |
|
Looks like it needs more than 1GB during install. Found this in the install log, maybe this helps: |
|
That's entirely possible that the memory pressure is high during install, as this is a python script that orchestrates the deployment and that spawns the npm install / rebuild process. @markuswinkler even with those logs, the installation don't fail right? |
|
yes, works all the way through (takes about 5 minutes to deploy). just thought I share my findings. |
|
@markuswinkler that is definitely helpful! |
|
If true then it's all the more reason not to include unwanted deps and keep them external only. I am going to try monitoring memory when installing and get back. https://www.otreva.com/blog/monitoring-memory-ram-aws-elasticbeanstalk-ec2-instances-cloudwatch/ |
|
Are there any log files that could help? I am by no means an AWS expert. |
|
I had a custom .ebextension thing, I'll try without. I'll post the logs when I get them |
|
are your package requirements all the same as mine? maybe there is a weird dependency. |
|
@markuswinkler @flovilmart I monitored the memory and the memory utilisation is indeed very high. Peaked at about 927mb during installation. We definitely need to work on cutting down memory usage during installation.
That being said I'm still not sure if this is indeed the cause of failure. Will try with a t2.small and report back. |
|
@markuswinkler just parse-server as a dependency |
|
try my full stack with the version numbers below. so far that's the only main difference. |
|
Well what do you know, using a t2.small worked! Tried deploying 5 times and all of them succeeded! But there's a huge issue. With every subsequent deploy the memory required to install keeps increasing. Take a look at the memory usage below. All of those peak points are deployments. There seems to be a memory leak/cleanup issue. This explains why the
I'm going to re-deploy a few more times and check whether even this eventually fails. |
|
Update: Spoke too soon, thankfully the memory peaked at about 1.07Gb and never went above that. Average was about 900mb for the 7 deployments I tried. @flovilmart here are the packages I've specified in .ebextensions config file. Try adding them and then deploy. |
|
so that's great! I'm glad that's fixed for you! @tanmays @markuswinkler now, what would be the most awesome thing ever would be to write a little blog article :) |
|
For that I would need to have a blog first ;) |
|
Thanks for sharing! We'll keep that in mind. The PR that removes those dependencies is still open and would I believe dramatically reduce the memory usage when running install / rebuild. Not sure if we're gonna merge it as it introduces a breaking change. @tanmays I'll let you close that as you opened it when you feel it's good! |
|
well, if there is any good time to make a breaking change for the greater good it is now I guess. |
|
I've already externalized all the adapters (besides GridStoreAdapter, but it will go to) into external repositories, we decided to still depend on them to avoid the breaking changes. In the end, what matters most is the benefit for the community. Thanks for helping on that! That was a really meaningful contribution! |
|
@flovilmart I'll close this one, I hope the PR with breaking changes but improved memory efficiency gets merged, it'd be good in the long run. Alongside running a production server, a lot of people including me will also be using it for limited user beta releases, for which a micro instance would be perfect. I'm sure the community would present a lot of similar cases. |
|
@flovilmart, @tanmays, @markuswinkler I know this issue is now closed but wanted to update you all on this issue. I'm on the AWS Mobile team and worked with the Elastic Beanstalk engineers to identify the issue and we are working on more permanent solution. You all have figured out a workaround but I wanted to document this and let others know they have options. First of all, we (and you all) have identified TWO issues when upgrading Parse Server running on AWS Elastic Beanstalk. Problem #1: The persistent error: fatal error: gssapi/gssapi.h: No such file or directory Save and re-deploy. Tip: The easiest way to manage your Parse Server application on Elastic Beantalk is to install and configure the EB CLI here. I'm not a command-line user typically but this tool works great for both PC and Mac. Here's the quick overview for modifying an existing Parse Server application using the EB CLI:
Problem #2: As @markuswinkler found out, the default t1.micro instance type for Elastic Beanstalk is running out of memory on subsequent deploys or reboots and so we thought the solution to #1 wasn't working. Solution: If you are running a t1.micro and with the current Parse Server dependencies, you'll want to upgrade the instance type to something with more memory. This may not be an issue if the breaking PR gets pulled in, from what I hear. However, in the meantime, upgrade the instance type to avoid getting out-of-memory when the application starts up. To change your instance type, click on Configuration in the EB Console then click the edit icon for the "Instances" card. This will let you set the instance type that determines the processing power of the Parse Server in your environment. Conclusion: Applying the changes to the app.config for issue #1 and upgrading the instance type for issue #2 should resolve any "current" issues when attempting to modify, upgrade, or reboot your Parse Server application on AWS Elastic Beanstalk. Next Steps: We'll be submitting a PR to modify the .ebextenstions/app.config for parse-server-example repo so that any future deployments don't need to worry about applying the changes for issue #1. For issue #2, AWS will most likely specify (explicitly) a larger instance type in their solution offered on the AWS Mobile Blog here. Currently, the solution does not explicitly specify the type and it defaults to the free-tier t1.micro in the us-east-1 region. |
|
We've also removed parse-server-gcs-adapter that was attempting to compile a native module. This is likely that since 2.2.6, t1.micro deploy again. |
|
deployment of t2.micro (1 GB RAM) still fails on 2.2.6. I also added the .ebextensions/app.config file @dennisAWS suggested. btw, @dennisAWS, what's the best way to force go back to a working configuration? Once you select t2.micro instead of t2.small the configuration is locked in a reboot/install/fail cycle and aborting takes about 15 minutes. |
|
@markuswinkler looks like 'parse-server-gcs-adapter' was removed after 2.2.6. We probably have to wait for 2.2.7. |
|
yes, but that one should actually reduce the memory foot print (gcs is used for google cloud storage), but alas, still fails. |
|
Great news, 2.2.7 runs stable on a t2.micro instance on AWS. |
|
Hello, I tryed to add @dennisAWS or @tanmays packages to .ebextensions, tryed with t2.micro and t2.small but in every case I got an error. Here is the full log in case you have a hint. Thanks! `------------------------------------- /var/log/nodejs/npm-debug.log4268 verbose linkMans [email protected] /var/log/eb-activity.lognpm ERR! [email protected] install: npm ERR! Please include the following file with any support request:
gyp ERR! clean error npm ERR! Linux 4.4.23-31.54.amzn1.x86_64 npm ERR! Please include the following file with any support request: [2016-10-31T17:17:14.528Z] INFO [3420] - [Application update app-161031_181506@3/AppDeployStage0/AppDeployPreHook/50npm.sh] : Activity failed. /var/log/eb-commandprocessor.logSetting npm config jobs to 1
gyp ERR! clean error npm ERR! Linux 4.4.23-31.54.amzn1.x86_64 npm ERR! Please include the following file with any support request: ` |
|
Dear all, I have the same problem as @Helmikku - tried with Parse 2.2.25-beta.1. working under "64bit Amazon Linux 2016.09 v3.1.0 running Node.js" on t2.micro. Any ideas how to resolve it ? |
|
I had to switch to *.small instance with more memory, that worked for me. |
|
Hello @przemyslaw-szurmak, I "accidentally" managed to deploy it, even on t2.micro. Maybe the error comes from the local "npm_modules" folder? In the end, I have 2 files in my .ebextensions: 2 - packages.config And my package.json: |
|
@Helmikku hey, so finally i was able to do deployment with latest commit of parse-server (commit c8823f2) ! I did what you've suggested - removed npm_modules from my local repository and deployed without it. Now it works fine. Thanks for help ! |
I was trying out the latest release and I'm why not sure why but I'm not able to successfully deploy to elastic beanstalk anymore.
Last successfu eb deployment was on parse-server release v2.2.2
Local installation however is successful.
Here is the eb activity log:
Possibly related issue #1082
The text was updated successfully, but these errors were encountered: