Requirements for building the lambda function

[leviwilson]

I'm having issues being able to successfully build the ./build-s3-dist.sh source-bucket-base-name. The README doesn't specify what the base OS requirements are for building the solution, but given yum is mentioned I assumed that it was built on an Amazon Linux AMI or the like. To get around this I'm working on a Dockerfile that bases the image off of the amazonlinux based image.

pip

The first issue I ran into is that the latest version of pip does not expose req, so for pip > version 10 it needs to say pip._internal.req rather than pip.req. Here is what I've updated both setup.py to contain to get around it.

try: # for pip >= 10
    from pip._internal.req import parse_requirements
except ImportError: # for pip <= 9.0.3
    from pip.req import parse_requirements

However, when tornado_botocore is included, it suffers from the same problem:

Collecting tornado_botocore==1.0.2 (from image-handler==2.0)
  Downloading https://files.pythonhosted.org/packages/35/35/65434dd70a524c9bffc8c24c3c549573887c741f82bf33b8dc2bed696d88/tornado-botocore-1.0.2.tar.gz
    Complete output from command python setup.py egg_info:
    WARNING:root:It looks like some requirements are missing.
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-RY9v9C/tornado-botocore/setup.py", line 35, in <module>
        from pip.req import parse_requirements
    ImportError: No module named req

To get around this, I had to update 'tornado_botocore==1.3.2' as well as 'botocore==1.8' as those versions have updates around the pip issue.

pycurl

The next issue that I found was when installing pycurl. Here is the output:

Collecting pycurl<7.44.0,>=7.19.0 (from thumbor->thumbor-plugins==0.2.1->-r source/image-handler/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz (214kB)
    100% |################################| 215kB 17.4MB/s
    Complete output from command python setup.py egg_info:
    Using curl-config (libcurl 7.51.0)
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-a6Ym8A/pycurl/setup.py", line 913, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/tmp/pip-install-a6Ym8A/pycurl/setup.py", line 582, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/tmp/pip-install-a6Ym8A/pycurl/setup.py", line 99, in __init__
        self.configure()
      File "/tmp/pip-install-a6Ym8A/pycurl/setup.py", line 316, in configure_unix
        specify the SSL backend manually.''')
    __main__.ConfigurationError: Curl is configured to use SSL, but we have not been able to determine which SSL backend it is using. Please see PycURL documentation for how to specify the SSL backend manually.

So it'd seem that the amazonlinux base image isn't setup to be able to build pycurl. In order to get around that issue, I had to do the following:

yum install -y python27-devel openssl-devel

Success? ✅

...or so I thought. I was at least able to build the ZIP and deploy the function, however when I went to convert an image I got a 502. Here are a couple of things from the CloudWatch logs:

13:29:27 [WARNING] It looks like some requirements are missing.
13:29:27 [ERROR] start_thumbor error: cannot import name Botocore

So, the bump for botocore seemed to have broken some things with tc_aws, so in order to get past that I had to use 'tc_aws==6.2.10' per thumbor-community/aws#100.

Where I'm At

After bumping tc_aws and deploying the latest built package, now all I'm relegated is a 502 still and simply a message that says:

[WARNING] It looks like some requirements are missing.

At this point, I'm at a loss at what to look into, so any help would be greatly appreciated to point me in the right direction as to what I might be doing wrong. I'll probably try loading an actual AWS AMI that Lambda is using and see if I get the same results or not from there.

Ultimately, all I'm trying to do is bump thumbor to 6.5.2 but I've thus far been unsuccessful at even building a package that I can start with.

Here is the Dockerfile that I'm currently attempting this with:

FROM amazonlinux:2017.03.1.20170812
MAINTAINER Levi Wilson <[email protected]>

# lock yum to the same repository version
RUN sed -i 's/releasever=.*/releasever=2017.03/g' /etc/yum.conf

# base requirements
RUN yum install yum-utils zip -y && \
    yum-config-manager --enable epel && \
    yum install git libpng-devel libcurl-devel gcc python27-devel libjpeg-devel -y

# enable epel on Amazon Linux 2
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

# pip
RUN alias sudo='env PATH=$PATH' && \
    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
    python get-pip.py && rm get-pip.py && \
    pip install --upgrade setuptools && \
    pip install --upgrade virtualenv

# pycurl
RUN yum install -y nss-devel
ENV PYCURL_SSL_LIBRARY=nss

# ImageMagick
RUN yum install -y ImageMagick-devel

[leviwilson]

I'm noticing that some things are failing in the build command but not actually failing the build. Part of what I'm missing in my Dockerfile is that on Amazon Linux 2 you need to enable epel like so:

yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

This got me a little farther in the my errors in CloudWatch now are:

start_thumbor error: pycurl: libcurl link-time ssl backend (nss) is different from compile-time ssl backend (openssl)

I've updated the Dockerfile to reflect the changes for nss as well as I had to yum install -y ImageMagick-devel.

[leviwilson]

Closing this issue as I believe that I've gotten past them all. I intend on opening a pull request instead.

[timkelty]

@leviwilson do you have your Dockerfile pushed anywhere I can see? I'm close, but looks like I'm running into the same pycurl error you were: https://github.com/timkelty/serverless-image-handler/tree/docker/docker

Update: oh duh, its posted above…I was actually able to get mine building from amazonlinux, just had to add which make python2-pip to the yum install

[leviwilson]

@timkelty #45

☝️ that PR has my completed example if that helps

[timkelty]

@leviwilson awesome, thank you!

[leviwilson]

@timkelty note that I did that prior to this repo updating their latest release...there are merge conflicts vs. master that I hadn't bothered to fix since I've not gotten any feedback on my PR ¯_(ツ)_/¯

[timkelty]

I hadn't bothered to fix since I've not gotten any feedback on my PR ¯_(ツ)_/¯

@leviwilson that's a bummer…

@gsingh04 and ideas on how we can move some of these PRs along?

[rromanchuk]

@leviwilson do you have a recent workflow (current Dockerfile) for building this? I want to avoid a rabbit hole if possible.

[leviwilson]

@rromanchuk this is what I'd done the last time we needed to build it

https://github.com/northwoodspd/serverless-image-handler/blob/sharp-output-format/.gitlab-ci.yml