Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add production image support #7832

Merged
merged 1 commit into from
Apr 2, 2020

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Mar 23, 2020


Issue link: WILL BE INSERTED BY boring-cyborg

Make sure to mark the boxes below before creating PR: [x]


In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

Dockerfile Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
Dockerfile Show resolved Hide resolved
@potiuk potiuk force-pushed the add-production-docker-image branch from ff1d713 to ba1927a Compare March 23, 2020 13:26
Dockerfile Outdated Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
@mik-laj
Copy link
Member

mik-laj commented Mar 23, 2020

There is some error in the scripts.
When I ran following command:

bash breeze restart -a 1.10.6 && bash breeze -a 1.10.6

I got following error:

Installs version of Airflow: 1.10.6

Restarts the environment. Includes emptying the databases.

Fixing group permissions
Fixed group permissions

Docker image build is forced for CI image

Checking if the remote image needs to be pulled

/Users/kamilbregula/devel/google-airflow/airflow/scripts/ci/_utils.sh: line 628: AIRFLOW_REMOTE_MANIFEST_IMAGE: unbound variable

@potiuk
Copy link
Member Author

potiuk commented Mar 23, 2020

/Users/kamilbregula/devel/google-airflow/airflow/scripts/ci/_utils.sh: line 628: AIRFLOW_REMOTE_MANIFEST_IMAGE: unbound variable

will take a look thanks!

BREEZE.rst Outdated Show resolved Hide resolved
Dockerfile Show resolved Hide resolved
@potiuk potiuk force-pushed the add-production-docker-image branch from 550e8ef to 91a424e Compare March 23, 2020 21:56
Dockerfile Outdated Show resolved Hide resolved
Dockerfile Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
@potiuk potiuk force-pushed the add-production-docker-image branch 6 times, most recently from f0c94f8 to 7dce17b Compare March 25, 2020 05:22
entrypoint.sh Outdated Show resolved Hide resolved
BREEZE.rst Show resolved Hide resolved
@potiuk potiuk force-pushed the add-production-docker-image branch from 182a141 to dc4c5a5 Compare April 2, 2020 02:15
@potiuk
Copy link
Member Author

potiuk commented Apr 2, 2020

All corrected - please take another look

@potiuk potiuk force-pushed the add-production-docker-image branch from dc4c5a5 to f0ee1c7 Compare April 2, 2020 07:45
@potiuk
Copy link
Member Author

potiuk commented Apr 2, 2020

Hey @ashb @kaxil - I want to merge it to 1.10.10 before we release, Is there anything else you want to comment on?

@ashb
Copy link
Member

ashb commented Apr 2, 2020

Taking another look now

Dockerfile Outdated Show resolved Hide resolved
Dockerfile Show resolved Hide resolved
IMAGES.rst Show resolved Hide resolved
IMAGES.rst Show resolved Hide resolved
@potiuk potiuk force-pushed the add-production-docker-image branch from f0ee1c7 to 37014a4 Compare April 2, 2020 15:25
IMAGES.rst Show resolved Hide resolved
@potiuk potiuk force-pushed the add-production-docker-image branch from 37014a4 to 314f919 Compare April 2, 2020 15:48
Copy link
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do other minor changes in follow up PRs

@potiuk
Copy link
Member Author

potiuk commented Apr 2, 2020

🎉 🎉 !

@kaxil kaxil added this to the Airflow 1.10.10 milestone Apr 2, 2020
Dockerfile Show resolved Hide resolved
@dimberman
Copy link
Contributor

dimberman commented Apr 2, 2020

@potiuk I can't tell if I need to reload something but I'm not seeing build-image in auto-complete

(airflow) dimberman airflow ((HEAD detached at polidea/add-production-docker-image)) (⎈ |N/A|N/A)
$ ./breeze build-
build-docs  build-only
(airflow) dimberman airflow ((HEAD detached at polidea/add-production-docker-image)) (⎈ |N/A|N/A)
$ ./breeze build-image --production-image

@feluelle
Copy link
Member

feluelle commented Apr 2, 2020

@dimberman run breeze-complete script again. (. ./breeze-complete did it for me)

@dimberman
Copy link
Contributor

@feluelle works! had to run ./breeze setup-autocomplete and then restart shell. Might be worth mentioning in the docs or UPDATING.md?


.. code-block::

docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.6-slim-buster" \

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solved/.

Comment on lines +364 to +375
docker build . --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 --build-arg COPY_SOURCE=. \
--build-arg COPY_TARGET=/opt/airflow --build-arg AIRFLOW_SOURCES=/opt/airflow \
--build-arg CONSTRAINT_REQUIREMENTS=requirements/requirements-python3.7.txt" \
--build-arg ENTRYPOINT_FILE=entrypoint.sh \
--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \
--build-arg CONSTRAINT_REQUIREMENTS="https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt"
--build-arg ENTRYPOINT_FILE="https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh" \
--build-arg SOURCES_FROM="Dockerfile" \
--build-arg SOURCES_TO="/Dockerfile" \
--build-arg WWW_FOLDER="www_rbac"
Copy link
Member

@feluelle feluelle Apr 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is sth. wrong.

Suggested change
docker build . --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 --build-arg COPY_SOURCE=. \
--build-arg COPY_TARGET=/opt/airflow --build-arg AIRFLOW_SOURCES=/opt/airflow \
--build-arg CONSTRAINT_REQUIREMENTS=requirements/requirements-python3.7.txt" \
--build-arg ENTRYPOINT_FILE=entrypoint.sh \
--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \
--build-arg CONSTRAINT_REQUIREMENTS="https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt"
--build-arg ENTRYPOINT_FILE="https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh" \
--build-arg SOURCES_FROM="Dockerfile" \
--build-arg SOURCES_TO="/Dockerfile" \
--build-arg WWW_FOLDER="www_rbac"
docker build .
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \
--build-arg CONSTRAINT_REQUIREMENTS=requirements/requirements-python3.7.txt" \
--build-arg ENTRYPOINT_FILE=entrypoint.sh \
--build-arg AIRFLOW_SOURCES_FROM="Dockerfile" \
--build-arg AIRFLOW_SOURCES_TO="/Dockerfile" \
--build-arg WWW_FOLDER="www_rbac"

Like this @potiuk ?

IMAGES.rst Show resolved Hide resolved
@kaxil kaxil merged commit 07fd0d7 into apache:master Apr 2, 2020
kaxil pushed a commit that referenced this pull request Apr 2, 2020
@potiuk
Copy link
Member Author

potiuk commented Apr 2, 2020

@feluelle works! had to run ./breeze setup-autocomplete and then restart shell. Might be worth mentioning in the docs or UPDATING.md?

I think it really depend on your setup. simply '. ./breeze-complete' should do. Did you have another directory with airflow somewhere? Tejh setup-autocomplete sets up a symbolic link to breeze-complete and only one can be active at the same time ... It's a bit tricky to get it right if you have more than one breeze command... The command itself will work though (it will use correct definition). I keep the master v1_10_test in sync so most of the auto-complete is good for both (some values like python version are not) and eventually you will get it right next time you get into the shell...

Not sure if UPDATING.md is for those dev tools though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.