tilt up not working for local development

[CorsettiS]

When I tried to follow the guidelines of local development using tilt, I got the following error

Loading Tiltfile at: ***/astronomer-cosmos/Tiltfile
Traceback (most recent call last):
  ***/astronomer-cosmos/Tiltfile:1:15: in <toplevel>
Error in docker_compose: command ["docker" "compose" "--project-name" "dev" "--project-directory" "***/astronomer-cosmos/dev" "-f" "***/astronomer-cosmos/dev/docker-compose.yaml" "config"] failed.
error: exit status 14
stdout: ""
stderr: 'open ***/astronomer-cosmos/dev/.env: no such file or directory

[jlaneve]

Can you try creating an empty .env file and see if that fixes the issue?

[CorsettiS]

@jlaneve it fixes it partially, however there are still few errors coming up

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
       → google-cloud-workflows 1.7.1 requires protobuf<4.0.0dev,>=3.19.0, but you have protobuf 4.23.3 which is incompatible.
       → google-cloud-vision 1.0.2 requires protobuf<4.0.0dev, but you have protobuf 4.23.3 which is incompatible.
       → google-cloud-videointelligence 1.16.3 requires protobuf<4.0.0dev, but you have protobuf 4.23.3 which is incompatible.
       → google-cloud-translate 1.7.2 requires protobuf<4.0.0dev, but you have protobuf 4.23.3 which is incompatible.
       → google-cloud-texttospeech 1.0.3 requires protobuf<4.0.0dev, but you have protobuf 4.23.3 which is incompatible.
       → google-cloud-tasks 2.10.1 requires protobuf<4.0.0dev,>=3.19.0, but you have protobuf 4.23.3 which is incompatible.
...

STEP 2/2 — Deploying
Container dev-airflow-init-1  Creating
Container dev-airflow-init-1  Created
no such service: redis

got unexpected error during build/deploy: command ["docker" "compose" "--project-name" "dev" "--project-directory" "***/astronomer-cosmos/dev" "-f" "***/astronomer-cosmos/dev/docker-compose.yaml" "up" "--no-deps" "-d" "airflow-init"] failed.
error: exit status 1

Build Failed: command ["docker" "compose" "--project-name" "dev" "--project-directory" "***/astronomer-cosmos/dev" "-f" "***/astronomer-cosmos/dev/docker-compose.yaml" "up" "--no-deps" "-d" "airflow-init"] failed.
error: exit status 1

For the aforementioned one, it is strange though since the redis is available

[jlaneve]

@CorsettiS I actually just removed the dependency on redis (and flower) - the changes are now on main. Can you try again and see if it works now?

Also, does the pip error stop Airflow from running? Or does it still proceed, just with the (potentially) wrong dependencies?

[CorsettiS]

@jlaneve unfortunately there is still an error taking place, now on postgres side

STEP 3/3 — Deploying
Container dev-airflow-init-1  Creating
Container dev-airflow-init-1  Created
no such service: postgres

got unexpected error during build/deploy: command ["docker" "compose" "--project-name" "dev" "--project-directory" "/***/astronomer-cosmos/dev" "-f" "/***/astronomer-cosmos/dev/docker-compose.yaml" "up" "--no-deps" "-d" "airflow-init"] failed.
error: exit status 1

The postgres container is up & running tho. when I access it, I obtain the following logs

PostgreSQL Database directory appears to contain a database; Skipping initialization


2023-06-29 08:19:43.922 UTC [1] LOG:  starting PostgreSQL 13.11 (Debian 13.11-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit

2023-06-29 08:19:43.922 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432

2023-06-29 08:19:43.922 UTC [1] LOG:  listening on IPv6 address "::", port 5432

2023-06-29 08:19:43.925 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

2023-06-29 08:19:43.932 UTC [27] LOG:  database system was interrupted; last known up at 2023-06-29 08:16:01 UTC

2023-06-29 08:19:44.114 UTC [27] LOG:  database system was not properly shut down; automatic recovery in progress

2023-06-29 08:19:44.116 UTC [27] LOG:  redo starts at 0/15CFF48

2023-06-29 08:19:44.116 UTC [27] LOG:  invalid record length at 0/15CFF80: wanted 24, got 0

2023-06-29 08:19:44.116 UTC [27] LOG:  redo done at 0/15CFF48

2023-06-29 08:19:44.131 UTC [1] LOG:  database system is ready to accept connections

I then tried to run a simple psql command inside the container, but got the following error

# psql
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  role "root" does not exist

I am not a docker expert so it is hard for me to properly give a better feedback, sorry for that.

[tatiana]

@CorsettiS Are you able to use psql to connect to Postgres with the credentials defined in the Docker compose file?
https://github.com/astronomer/astronomer-cosmos/blob/main/dev/docker-compose.yaml#L36-L39

Also, which operating system are you using?

[CorsettiS]

@tatiana I am, it connects fine. I am currently using a Mac. Instead of using tilt, I just manually did a docker compose up and it worked fine. I do not know how this tilt command works, but from what I saw it is breaking the build process of the docker compose file. I would advocate for changing the contribute guide and tell people to set up the environment with a docker compose directly. It is more straightforward from my point of view.