Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(build): reduce build time for ingestion image #5225

Merged
merged 13 commits into from
Jun 29, 2022
Merged

feat(build): reduce build time for ingestion image #5225

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
f9e6571
feat(build): reduce build time for ingestion image
anshbansal Jun 22, 2022
ce7db89
change to get codegen working
anshbansal Jun 22, 2022
cb4077d
copy pip cache instead between stages
anshbansal Jun 22, 2022
b301546
remove code from prev commit
anshbansal Jun 22, 2022
4201751
attempt matrix to run both in parallel
anshbansal Jun 22, 2022
21f20ab
add change to see diff
anshbansal Jun 22, 2022
d1e53c4
try base image
anshbansal Jun 24, 2022
38a88e9
change to point to acryldata image
anshbansal Jun 24, 2022
fc6eaaf
temporary build on pull request
anshbansal Jun 24, 2022
6d21ad3
remove files moved to separate PR
anshbansal Jun 24, 2022
d57ffd8
revert testing changes
anshbansal Jun 24, 2022
8b4c2af
use base image directly
anshbansal Jun 27, 2022
d0ef37c
remove tmp change
anshbansal Jun 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 0 additions & 51 deletions .github/workflows/metadata-ingestion-slow.yml
View file
Open in desktop

This file was deleted.

12 changes: 2 additions & 10 deletions docker/datahub-ingestion/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,11 @@
# Defining environment
ARG APP_ENV=prod

FROM python:3.8 as base
FROM acryldata/datahub-ingestion-base as base
# ENV DOCKERIZE_VERSION v0.6.1
# RUN apk --no-cache add curl tar \
# && curl https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-runner/9.4.20.v20190813/jetty-runner-9.4.20.v20190813.jar --output jetty-runner.jar \
# && curl -L https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz | tar -C /usr/local/bin -xzv
RUN apt-get update && apt-get install -y \
jq \
librdkafka-dev \
python3-ldap \
libldap2-dev \
libsasl2-dev \
libsasl2-modules \
ldap-utils \
&& python -m pip install --upgrade pip wheel setuptools==57.5.0


FROM openjdk:8 as prod-build
Expand All @@ -29,6 +20,7 @@ RUN cd /datahub-src/metadata-ingestion && \

FROM base as prod-install
COPY --from=prod-codegen /datahub-src/metadata-ingestion /datahub-ingestion
COPY --from=prod-codegen /root/.cache/pip /root/.cache/pip
ARG RELEASE_VERSION
RUN cd /datahub-ingestion && \
sed -i.bak "s/__version__ = \"0.0.0.dev0\"/__version__ = \"$RELEASE_VERSION\"/" src/datahub/__init__.py && \
Expand Down