Skip to content

Commit

Permalink
Make tilt and docker faster (#1920)
Browse files Browse the repository at this point in the history
## Description of changes

*Summarize the changes made by this PR.*
 - Improvements & Bug fixes
- Tighten up dockerfiles and tiltfile: Get rid of Alpine linux, make
dockerfiles a little friendlier to cache and make the final images
smaller, and tell tilt only to reload some of the dockerfiles on certain
path reloads.

## Test plan
*How are these changes tested?*

- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust

## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
  • Loading branch information
beggers committed Mar 24, 2024
1 parent 650c09c commit 8c7cc1b
Show file tree
Hide file tree
Showing 3 changed files with 18 additions and 29 deletions.
12 changes: 8 additions & 4 deletions Tiltfile
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,29 @@ update_settings(max_parallel_updates=6)

docker_build(
'local:sysdb-migration',
context='.',
'.',
only=['go/'],
dockerfile='./go/Dockerfile.migration'
)

docker_build(
'local:sysdb',
context='.',
'.',
only=['go/', 'idl/'],
dockerfile='./go/Dockerfile'
)

docker_build(
'local:frontend-service',
context='.',
'.',
only=['chromadb/', 'idl/', 'requirements.txt', 'bin/'],
dockerfile='./Dockerfile',
)

docker_build(
'local:query-service',
context='.',
'.',
only=["rust/", "idl/", "Cargo.toml", "Cargo.lock"],
dockerfile='./rust/worker/Dockerfile'
)

Expand Down
28 changes: 7 additions & 21 deletions go/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,33 +1,19 @@
FROM golang:1.20-alpine3.18 as build
WORKDIR /src/chroma-coordinator
RUN apk add --no-cache make git build-base bash
FROM golang:bookworm as builder
WORKDIR /build-dir
RUN apt-get update && apt-get install -y make git bash

ADD ./go/go.mod ./go.mod
ADD ./go/go.sum ./go.sum
ENV PATH=$PATH:/go/bin
RUN go mod download

ADD ./go/ ./
ENV GOCACHE=/root/.cache/go-build
RUN --mount=type=cache,target="/root/.cache/go-build" make

FROM alpine:3.17.3
FROM debian:bookworm-slim

RUN apk add --no-cache bash bash-completion findutils

# As of 6 Dec 2023, the atlas package isn't in Alpine's main package manager, only
# testing. So we have to add the testing repository to get it.
RUN apk add \
--no-cache \
--repository http://dl-cdn.alpinelinux.org/alpine/edge/testing \
--repository http://dl-cdn.alpinelinux.org/alpine/edge/main \
atlas

RUN mkdir /chroma-coordinator
WORKDIR /chroma-coordinator

COPY --from=build /src/chroma-coordinator/bin/coordinator /chroma-coordinator/bin/coordinator
COPY --from=build /src/chroma-coordinator/bin/logservice /chroma-coordinator/bin/logservice
ENV PATH=$PATH:/chroma-coordinator/bin
COPY --from=builder /build-dir/bin/coordinator .
COPY --from=builder /build-dir/bin/logservice .
ENV PATH=$PATH:./

CMD /bin/bash
7 changes: 3 additions & 4 deletions rust/worker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,6 @@ ARG CHROMA_KUBERNETES_INTEGRATION=0
ENV CHROMA_KUBERNETES_INTEGRATION $CHROMA_KUBERNETES_INTEGRATION

ARG RELEASE_MODE=
ENV BINARY_LOCATION=${RELEASE_MODE:+target/release/query_service}
ENV BINARY_LOCATION=${BINARY_LOCATION:-target/debug/query_service}

WORKDIR /
RUN git clone https://github.com/chroma-core/hnswlib.git
Expand All @@ -25,12 +23,13 @@ RUN curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v25.1

# We need to replace the query node's real main() with a dummy at the expected location.
RUN mkdir -p rust/worker/src/bin/ && echo "fn main() {}" > rust/worker/src/bin/query_service.rs
RUN cargo build
RUN if [ "$RELEASE_MODE" = "1" ]; then cargo build --release; else cargo build; fi
RUN rm -rf rust/

COPY rust/ rust/
RUN touch rust/worker/src/bin/query_service.rs
RUN cargo build
RUN if [ "$RELEASE_MODE" = "1" ]; then cargo build --release; else cargo build; fi
RUN if [ "$RELEASE_MODE" = "1" ]; then cp target/release/query_service .; else cp target/debug/query_service .; fi

FROM debian:bookworm-slim

Expand Down

0 comments on commit 8c7cc1b

Please sign in to comment.