Unable to use multiple build definition files with bake

[Apakottur]

I am unable to use docker buildx bake to build a docker image based on two docker-compose files.

To reproduce, create three files, all in the same directory:

# Dockerfile
FROM ubuntu:latest

# docker-compose.yml
services:
  test:
    build:
      context: .
    entrypoint: echo 1

# docker-compose.prod.yml
services:
  test:
    entrypoint: echo 2

The idea is to have the core configuration of an image in the base yml file, and then have another one with overrides (for a prod environment, for example).
I can then build and run the base/prod images like so:

$ docker-compose build
$ COMPOSE_FILE=docker-compose.yml docker-compose run --rm test
1
$ COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml docker-compose run --rm test
2

The buildx docs say that If multiple files are specified they are all read and configurations are combined.

However, when I try to use both files with buildx:

docker buildx bake -f docker-compose.yml -f docker-compose.prod.yml test

I am getting:

error: service "test" has neither an image nor a build context specified: invalid compose project

Am I missing something, or is this a bug?

[tonistiigi]

@crazy-max Is there something specific in the compose library that is blocking this?

[crazy-max]

Looks like we are loading compose files separately with bake:

buildx/bake/bake.go

Line 167 in a8bb25d

cfg, isCompose, composeErr := ParseComposeFile(f.Data, f.Name)

Compose seems to merge them together before loading the definition:

$ COMPOSE_FILE=docker-compose.yml docker-compose config test
name: "1172"
services:
  test:
    build:
      context: /home/src/buildx/.dev/1172
      dockerfile: Dockerfile
    entrypoint:
    - echo
    - "1"
    networks:
      default: null
networks:
  default:
    name: 1172_default

$ COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml docker-compose config test
name: "1172"
services:
  test:
    build:
      context: /home/src/buildx/.dev/1172
      dockerfile: Dockerfile
    entrypoint:
    - echo
    - "2"
    networks:
      default: null
networks:
  default:
    name: 1172_default

So I think we should produce a compose project from compose files found with bake. Not sure if there is a public api for this. Will take a look.

[crazy-max]

As I can see, consistency check added in #87 is already handled by the library: https://github.com/compose-spec/compose-go/blob/c45b40b31e66dbc3966642803d133bba33680b21/loader/validate.go#L31-L33

Now with compose v2, I think we can skip the consistency check and remove the dup check in bake as well and just use the default values. In this case it would produce:

# docker-compose.yml
services:
  test:
    build:
      context: .
      tags:
        - docker.io/username/webapp:latest
    entrypoint: echo 1

# docker-compose.prod.yml
services:
  test:
    entrypoint: echo 2

$ docker buildx bake -f docker-compose.yml test --print
{
  "group": {
    "default": {
      "targets": [
        "test"
      ]
    }
  },
  "target": {
    "test": {
      "context": ".",
      "dockerfile": "Dockerfile",
      "tags": [
        "docker.io/username/webapp:latest"
      ]
    }
  }
}

$ docker buildx bake -f docker-compose.prod.yml test --print
{
  "group": {
    "default": {
      "targets": [
        "test"
      ]
    }
  },
  "target": {
    "test": {
      "context": ".",
      "dockerfile": "Dockerfile"
    }
  }
}

$ docker buildx bake -f docker-compose.yml -f docker-compose.prod.yml test --print
{
  "group": {
    "default": {
      "targets": [
        "test"
      ]
    }
  },
  "target": {
    "test": {
      "context": ".",
      "dockerfile": "Dockerfile",
      "tags": [
        "docker.io/username/webapp:latest"
      ]
    }
  }
}

WDYT @tonistiigi?

[Apakottur]

@crazy-max @tonistiigi Thanks a lot for the fix!

I am using buildx via the GitHub Action, but it seems that pointing the buildx version in the action to master triggers a very lengthy build, presumably buildx is compiled from scratch?

Is there a testing release I can point the action to, so that I can use this fix?