libpcre.so.1: cannot open shared object file:

[rsitro-chrono]

Getting this issue when running clamav update in lambda.
b'./bin/freshclam: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory\n'

I do see other that have had this issue in the past and the latest Dockerfile seems to resolve it for them, but not for me. Any ideas?

[allanlewis]

I had the same issue. I found I had to restrict ClamAV to v0.102.3: see #203.

[rsitro-chrono]

@allanlewis I was actually able to fix this without downgrading clamav by updating my Dockerfile to

FROM amazonlinux:2

# Set up working directories
RUN mkdir -p /opt/app
RUN mkdir -p /opt/app/build
RUN mkdir -p /opt/app/bin/

# Copy in the lambda source
WORKDIR /opt/app
COPY ./*.py /opt/app/
COPY requirements.txt /opt/app/requirements.txt

# Install packages
RUN yum update -y
RUN amazon-linux-extras install epel -y
RUN yum install -y cpio yum-utils tar.x86_64 gzip zip python3-pip

# This had --no-cache-dir, tracing through multiple tickets led to a problem in wheel
RUN pip3 install -r requirements.txt
RUN rm -rf /root/.cache/pip

# Download libraries we need to run in lambda
WORKDIR /tmp
RUN yumdownloader -x \*i686 --archlist=x86_64 clamav
RUN rpm2cpio clamav-0*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 clamav-lib
RUN rpm2cpio clamav-lib*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 clamav-update
RUN rpm2cpio clamav-update*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 json-c
RUN rpm2cpio json-c*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 pcre2
RUN rpm2cpio pcre*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 libtool-ltdl
RUN rpm2cpio libtool-ltdl*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 libxml2
RUN rpm2cpio libxml2*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 bzip2-libs
RUN rpm2cpio bzip2-libs*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 xz-libs
RUN rpm2cpio xz-libs*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 libprelude
RUN rpm2cpio libprelude*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 gnutls
RUN rpm2cpio gnutls*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 nettle
RUN rpm2cpio nettle*.rpm | cpio -vimd


# Copy over the binaries and libraries
RUN cp /tmp/usr/bin/clamscan /tmp/usr/bin/freshclam /tmp/usr/lib64/* /usr/lib64/libpcre.so.1 /opt/app/bin/

# Fix the freshclam.conf settings
RUN echo "DatabaseMirror database.clamav.net" > /opt/app/bin/freshclam.conf
RUN echo "CompressLocalDatabase yes" >> /opt/app/bin/freshclam.conf
RUN echo "ScriptedUpdates no" >> /opt/app/bin/freshclam.conf
RUN echo "DatabaseDirectory /var/lib/clamav" >> /opt/app/bin/freshclam.conf

RUN yum install shadow-utils.x86_64 -y

RUN groupadd clamav
RUN useradd -g clamav -s /bin/false -c "Clam Antivirus" clamav
RUN useradd -g clamav -s /bin/false -c "Clam Antivirus" clamupdate

ENV LD_LIBRARY_PATH=/opt/app/bin
RUN ldconfig

# Create the zip file
WORKDIR /opt/app
RUN zip -r9 --exclude="*test*" /opt/app/build/lambda.zip *.py bin

WORKDIR /usr/local/lib/python3.7/site-packages
RUN zip -r9 /opt/app/build/lambda.zip *

WORKDIR /opt/app

[allanlewis]

@allanlewis I was actually able to fix this without downgrading clamav by updating my Dockerfile...

Great - are you able to raise a PR with your changes so I can compare with #203?

[rsitro-chrono]

@allanlewis I don't have permissions to push my branch to create the PR. I also wanted to create a PR that adds terraform, copied from the cloudformation. We use terraform here at Chronograph, so I converted and wanted to push up here as well.

[allanlewis]

@rsitro-chrono you'll need to fork the repo to make a PR, just like I did with mine.

[michaelmelody91]

Hey folks, just wondering did anyone get to the point of understanding what changed under the hood for this issue to occur? I'm assuming a new version of a package/lib was pushed that no longer included libpcre.so.1?

@allanlewis - what prompted you to revert ClamAV to v0.102.3?
@rsitro-chrono - It's hard to identify the changes that were included in your PR, is there a particular set of packages/libs that you added or removed?

[allanlewis]

...what changed under the hood for this issue to occur? I'm assuming a new version of a package/lib was pushed that no longer included libpcre.so.1?

No, the underlying OS didn't change, it's that the Docker image used to build the Lambda package is using Amazon Linux 2 (AL2) whereas the Lamdba runtime for Python before 3.8 uses AL1. ClamAV in v0.103.0 updated their bindings such that they no longer support the version of libpcre provided by the (increasingly ancient) AL1.

@allanlewis - what prompted you to revert ClamAV to v0.102.3?

I found that this version works on AL1.

One solution here might be to explicitly drop support for Python <3.8, i.e. the AL1-based Lambda runtimes. Alternatively, we could select the base image for building the Lambda package depending on the target runtime.

Given that Python 3.6 is already obsolete, the only version worth supporting on AL1 is Python 3.7, and I don't think it would affect many people if they had to upgrade their AV scanner Lambda to Python 3.8 or 3.9. The issue is that we then hit issues with libmagic, which I'm not sure how to resolve - perhaps that's worth raising as a separate issue.

[bellis-ai]

I had the same issue. I found I had to restrict ClamAV to v0.102.3: see #203.

Not sure this is a good idea -- 0.102 is EOL and when I used it, freshclam got blocked: https://blog.clamav.net/2022/01/clamav-01035-and-01042-security-patch.html

[allanlewis]

I found a solution in the end - the problem was due to a change in my fork done by someone else years ago. We'd like to rebase on upstream at some point but in the meantime my fix was to use the official Docker image for Lambda Python 3.7 to build the deployment package.

[bellis-ai]

I thought it builds just using the Amazon Linux 2 Docker Image? Are you saying you built directly using the python lambda image instead?

[allanlewis]

The problem is that the Lambda runtime for Python <=3.7 uses Amazon Linux 1 whereas the runtime for Python >=3.8 uses Amazon Linux 2. If you build the package with the wrong one you'll end up trying to run the ClamAV packages from one OS on another, which won't work due to library dependencies, hence this issue.

[bellis-ai]

Strange... I've been running this with Python 3.7 for half a year now and only recently it's given me this error...
And clearly it's building with Amazon Linux 2 and running with the Python 3.7 image all this time. Did something change?

[allanlewis]

Did something change?

Yes: the version of ClamAV in the AL2 repos.

[bellis-ai]

@allanlewis How did you handle the case for the 'nettle' and 'pcre2' packages that aren't available on amazon linux 1?

[allanlewis]

@allanlewis How did you handle the case for the 'nettle' and 'pcre2' packages that aren't available on amazon linux 1?

I just didn't include them, they don't appear to be necessary.

[gchamon]

@rsitro-chrono the proposed Dockerfile still works as a fix for the current code in the master branch