Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker image build problem: couldn't find packages from bioconductor #52

Closed
YonghuiDong opened this issue Jan 19, 2023 · 11 comments
Closed

Comments

@YonghuiDong
Copy link

My dockerfile was build with

golem::add_dockerfile()

Here is the docker file:

FROM rocker/verse:4.2.0
RUN apt-get update && apt-get install -y  cmake git-core libcurl4-openssl-dev libgit2-dev libglpk-dev libgmp-dev libicu-dev libpng-dev libssl-dev libxml2-dev make pandoc zlib1g-dev && rm -rf /var/lib/apt/lists/*
RUN mkdir -p /usr/local/lib/R/etc/ /usr/lib/R/etc/
RUN echo "options(repos = c(CRAN = 'https://cran.rstudio.com/'), download.file.method = 'libcurl', Ncpus = 4)" | tee /usr/local/lib/R/etc/Rprofile.site | tee /usr/lib/R/etc/Rprofile.site
RUN R -e 'install.packages("remotes")'
RUN Rscript -e 'remotes::install_version("tidyr",upgrade="never", version = "1.2.1")'
RUN Rscript -e 'remotes::install_version("RColorBrewer",upgrade="never", version = "1.1-3")'
RUN Rscript -e 'remotes::install_version("dplyr",upgrade="never", version = "1.0.10")'
RUN Rscript -e 'remotes::install_version("shiny",upgrade="never", version = "1.7.4")'
RUN Rscript -e 'remotes::install_version("rmarkdown",upgrade="never", version = "2.19")'
RUN Rscript -e 'remotes::install_version("ggplot2",upgrade="never", version = "3.4.0")'
RUN Rscript -e 'remotes::install_version("igraph",upgrade="never", version = "1.3.5")'
RUN Rscript -e 'remotes::install_version("waiter",upgrade="never", version = "0.2.5")'
RUN Rscript -e 'remotes::install_version("shinydashboard",upgrade="never", version = "0.7.2")'
RUN Rscript -e 'remotes::install_version("config",upgrade="never", version = "0.3.1")'
RUN Rscript -e 'remotes::install_version("VIM",upgrade="never", version = "6.2.2")'
RUN Rscript -e 'remotes::install_version("shinydashboardPlus",upgrade="never", version = "2.0.3")'
RUN Rscript -e 'remotes::install_version("ropls",upgrade="never", version = "1.30.0")'
RUN Rscript -e 'remotes::install_version("readxl",upgrade="never", version = "1.4.1")'
RUN Rscript -e 'remotes::install_version("readr",upgrade="never", version = "2.1.3")'
RUN Rscript -e 'remotes::install_version("qgraph",upgrade="never", version = "1.9.3")'
RUN Rscript -e 'remotes::install_version("plotly",upgrade="never", version = "4.10.1")'
RUN Rscript -e 'remotes::install_version("golem",upgrade="never", version = "0.3.5")'
RUN Rscript -e 'remotes::install_version("ggraph",upgrade="never", version = "2.1.0")'
RUN Rscript -e 'remotes::install_version("DT",upgrade="never", version = "0.26")'
RUN Rscript -e 'remotes::install_version("ComplexHeatmap",upgrade="never", version = "2.12.1")'
RUN mkdir /build_zone
ADD . /build_zone
WORKDIR /build_zone
RUN R -e 'remotes::install_local(upgrade="never")'
RUN rm -rf /build_zone
EXPOSE 80
CMD R -e "options('shiny.port'=80,shiny.host='0.0.0.0');MetReport::run_app()"

It seems that remote::remotes::install_version() cannot install R packages from bioconductor.

The error message:

 > [18/31] RUN Rscript -e 'remotes::install_version("ropls",upgrade="never", version = "1.30.0")':
ThinkR-open/golem#21 9.175 Error in download_version_url(package, version, repos, type) : 
ThinkR-open/golem#21 9.175   couldn't find package 'ropls'
ThinkR-open/golem#21 9.175 Calls: <Anonymous> -> download_version_url
ThinkR-open/golem#21 9.176 Execution halted

My shinyapp Description file was generated by golem.

How should I solve this problem?

Thanks a lot.

Dong

@ColinFay ColinFay transferred this issue from ThinkR-open/golem Jan 19, 2023
@ColinFay
Copy link
Member

(transfering to Dockerfiler, as it handles the engineering of apps)

@VincentGuyader
Copy link
Member

can you tyr with golem::add_dockerfile_with_renv() ?

@YonghuiDong
Copy link
Author

YonghuiDong commented Jan 19, 2023

Dear @VincentGuyader and @ColinFay ,

Thanks a lot for your prompt and kind help. I have tried with golem::add_dockerfile_with_renv(output_dir = "deploy"), and I encountered the following error when I build the docker image:

invalid argument "MetReport_base" for "-t, --tag" flag: invalid reference format: repository name must be lowercase

Does that mean that I have to change my shinyapp name in lowercase?

Below is the generated dockerfile

FROM MetReport_base
COPY renv.lock.prod renv.lock
RUN R -e 'renv::restore()'
COPY MetReport_*.tar.gz /app.tar.gz
RUN R -e 'remotes::install_local("/app.tar.gz",upgrade="never")'
RUN rm /app.tar.gz
EXPOSE 80
CMD R -e "options('shiny.port'=80,shiny.host='0.0.0.0');MetReport::run_app()"

Best regards,
Dong

@VincentGuyader
Copy link
Member

what is your golem version ?
can you try to install the last golem version i have corrected this issue : ThinkR-open/golem#944
remotes::install_github("ThinkR-open/golem@dev")

@YonghuiDong
Copy link
Author

My golem version is 0.3.5.

After updating to 0.4.0 , I still get the same error message when I using the follow script to build the image.

docker build -f Dockerfile_base --progress=plain -t MetReport_base .
docker build -f Dockerfile --progress=plain -t MetReport:latest .
docker run -p 80:80 MetReport:latest
# then go to 127.0.0.1:80

Error message:

invalid argument "MetReport_base" for "-t, --tag" flag: invalid reference format: repository name must be lowercase

@VincentGuyader
Copy link
Member

VincentGuyader commented Jan 19, 2023

we made a mistake during our github merge request, I will check it right now, but please can you try with :
remotes::install_github("ThinkR-open/golem@89b40cb568ed2d8edd75909cbbf7dc693f25b82f")

edit :
after checking more deeper, are you sure to have restarted your R after golem installation ?

i made a test with remotes::install_github("ThinkR-open/golem@dev") on a golem called "MAJUSCULE" and I got a fully functionnal Dockerfile

@YonghuiDong
Copy link
Author

Thanks a lot @VincentGuyader , the image was built now using docker build -f Dockerfile_base --progress=plain -t MetReport_base .

  1. But it took 3.5 h to create the image, and the image size is 5.3G. I have about 25 direct dependency packages, but I did not expect the image size to be that big.

  2. I cannot continue with the next docker script docker build -f Dockerfile --progress=plain -t MetReport:latest ., and I got the following error:

failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed.

  1. I will try to build it manually, and update you later on.

Best regards,
Dong

@YonghuiDong
Copy link
Author

YonghuiDong commented Jan 20, 2023

Dear @VincentGuyader,

I have manually edited the dockerfile created using golem::add_dockerfile(), in which I used BiocManager to install R packages from Bioconductor.

FROM rocker/verse:4.2.0
RUN apt-get update && apt-get install -y  cmake git-core libcurl4-openssl-dev libgit2-dev libglpk-dev libgmp-dev libicu-dev libpng-dev libssl-dev libxml2-dev make pandoc zlib1g-dev && rm -rf /var/lib/apt/lists/*
RUN mkdir -p /usr/local/lib/R/etc/ /usr/lib/R/etc/
RUN echo "options(repos = c(CRAN = 'https://cran.rstudio.com/'), download.file.method = 'libcurl', Ncpus = 4)" | tee /usr/local/lib/R/etc/Rprofile.site | tee /usr/lib/R/etc/Rprofile.site

# Install bioconductor packages
FROM bioconductor/bioconductor_docker:RELEASE_3_16
RUN apt-get update
    RUN R -e 'BiocManager::install(ask = F)' && R -e 'BiocManager::install(c("ropls", "ComplexHeatmap", ask = F))'

# Install Cran and other packages
RUN R -e 'install.packages("remotes")'
RUN Rscript -e 'remotes::install_version("tidyr",upgrade="never", version = "1.2.1")'
RUN Rscript -e 'remotes::install_version("RColorBrewer",upgrade="never", version = "1.1-3")'
RUN Rscript -e 'remotes::install_version("dplyr",upgrade="never", version = "1.0.10")'
RUN Rscript -e 'remotes::install_version("shiny",upgrade="never", version = "1.7.4")'
RUN Rscript -e 'remotes::install_version("rmarkdown",upgrade="never", version = "2.19")'
RUN Rscript -e 'remotes::install_version("ggplot2",upgrade="never", version = "3.4.0")'
RUN Rscript -e 'remotes::install_version("igraph",upgrade="never", version = "1.3.5")'
RUN Rscript -e 'remotes::install_version("waiter",upgrade="never", version = "0.2.5")'
RUN Rscript -e 'remotes::install_version("shinydashboard",upgrade="never", version = "0.7.2")'
RUN Rscript -e 'remotes::install_version("config",upgrade="never", version = "0.3.1")'
RUN Rscript -e 'remotes::install_version("remotes",upgrade="never", version = "2.4.2")'
RUN Rscript -e 'remotes::install_version("VIM",upgrade="never", version = "6.2.2")'
RUN Rscript -e 'remotes::install_version("shinydashboardPlus",upgrade="never", version = "2.0.3")'
#RUN Rscript -e 'remotes::install_version("ropls",upgrade="never", version = "1.30.0")'
RUN Rscript -e 'remotes::install_version("readxl",upgrade="never", version = "1.4.1")'
RUN Rscript -e 'remotes::install_version("readr",upgrade="never", version = "2.1.3")'
RUN Rscript -e 'remotes::install_version("qgraph",upgrade="never", version = "1.9.3")'
RUN Rscript -e 'remotes::install_version("plotly",upgrade="never", version = "4.10.1")'
RUN Rscript -e 'remotes::install_version("ggraph",upgrade="never", version = "2.1.0")'
RUN Rscript -e 'remotes::install_version("DT",upgrade="never", version = "0.26")'
RUN Rscript -e 'remotes::install_version("markdown",upgrade="never", version = "1.4")'
#RUN Rscript -e 'remotes::install_version("ComplexHeatmap",upgrade="never", version = "2.12.1")'
RUN Rscript -e 'remotes::install_github("ThinkR-open/golem@89b40cb568ed2d8edd75909cbbf7dc693f25b82f")'
RUN mkdir /build_zone
ADD . /build_zone
WORKDIR /build_zone
RUN R -e 'remotes::install_local(upgrade="never")'
RUN rm -rf /build_zone
EXPOSE 80
#CMD R -e "options('shiny.port'=80,shiny.host='0.0.0.0');MetReport::run_app()"
CMD R -e "options('shiny.port'=80,shiny.host='0.0.0.0')"

This time it took about 50 min to build the image, and the image size is about 4.7G.

But when I tried to run the image docker run -p 80:80 metaboreport. No error was popped up, but the container stopped immediately.

I am using M1 Mac, so I used docker buildx build --platform linux/amd64 -t metaboreport . to build docker image.

Do you happen to know the possible reason?

Best regards,
Dong

@YonghuiDong
Copy link
Author

Dear @VincentGuyader,

I have finally rebuilt the docker image using golem::add_dockerfile_with_renv(), and it worked eventually.

I am using M1 Mac, so it is much slower in building a docker image, and I need to add --platform=linux/amd64 flag when building the image.

Thanks a lot again for your kind help.

Dong

@remlapmot
Copy link

RE: the point about Apple Silicon Macs running slowly under the old emulation. About 1 week ago Docker released a new beta feature for Apple Silicon Macs - it's called Rosetta for Linux - you need to update your Docker to version 4.16.0 or later to get it.

It's off by default. Once enabled in Docker settings (select the box in the screenshot below and click "Apply & restart") I have experienced amd64 containers build in literally one third the time compared to the old emulation.

docker-beta-feature

@YonghuiDong
Copy link
Author

Hi Tom @remlapmot,

Thanks a lot for the tip, it is very helpful.

Dong

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants