Hi All,
I found your mailing list via a stack overflow mention. I?m currently on my second full day of trying to build a docker container to use as an AWS lambda fcn and am feeling pretty stuck - I was hoping someone here could help. It needs to support python+R+rstan.
I currently construct the docker image from a python/debian base image. I can get R working fine, but when I try to run an r-stan script within the lambda function I get a ?there is no package called Stan?, despite a good 5 minutes of the image build being dedicated to the download of rstan. Am looking for advice here on how to proceed. My current dockerfile follows?
################
FROM python:3.8-slim-bullseye
ENV DEBIAN_FRONTEND=noninteractive
ENV LC_ALL=C.UTF-8
ENV LANG=en_US.UTF-8
ENV TZ=:/etc/localtime
ENV PATH=/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin
ENV LD_LIBRARY_PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib
ENV LAMBDA_TASK_ROOT=/var/task
ENV LAMBDA_RUNTIME_DIR=/var/runtime
# Define custom function directory
ARG FUNCTION_DIR="/var/task"
WORKDIR ${FUNCTION_DIR}
# Install aws-lambda-cpp build dependencies + r
RUN apt-get update && \
apt-get install -y \
g++ \
make \
cmake \
unzip \
libcurl4-openssl-dev \
r-base
COPY requirements.txt .
COPY irt_dim_red.R .
COPY . .
#install the python requirements
RUN pip install -r requirements.txt
COPY scripts/install_stan.R install_stan.R
RUN Rscript "install_stan.R"
RUN R -e "install.packages('stringr')"
RUN R -e "install.packages('gtools')"
RUN R -e "install.packages('tidyr')"
# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]
CMD [ "main.handler"]
Where the script install_rstan.R is:
# Following instructions from
# https://github.com/stan-dev/rstan/wiki/Installing-RStan-on-Linux
print("Running install_stan.R")
## Creating /home/rstudio/.R/Makevars
dotR <- file.path(Sys.getenv("HOME"), ".R")
if (!file.exists(dotR)) dir.create(dotR)
M <- file.path(dotR, "Makevars")
if (!file.exists(M)) file.create(M)
cat("\nCXX14FLAGS=-O3 -march=native -mtune=native -fPIC",
"CXX14=clang++",
file = M, sep = "\n", append = TRUE)
## Installing rstan
install.packages("rstan", dependencies=TRUE, type = "source")
Any help would be massively appreciated.
Jesse
?
Jesse McMullen-Crummey, PhD (he/him)
Data Engineer
Trying to install r-stan without using rocker
6 messages · Jesse McMullen-Crummey, Dirk Eddelbuettel, Elizabeth Tighe
Jesse,
On 13 June 2022 at 12:19, Jesse McMullen-Crummey wrote:
| Hi All, | | I found your mailing list via a stack overflow mention. I?m currently on my second full day of trying to build a docker container to use as an AWS lambda fcn and am feeling pretty stuck - I was hoping someone here could help. It needs to support python+R+rstan. If something does not work, I usually backtrack. You start from python:3.8-slim-bullseye. That is your right. But this is a Debian-and-R mailing list, so we may not have started from there. I maintain and curate several sets of Rocker containers building either on - plain Debian as our rocker/r-base which is also the official r-base - Ubuntu because I can then use Ubuntu binaries as from c2d4u or now also r2u If I were, I'd the same. With either base set you should get to 'apt install r-cran-rstan' easily, and adding your other package is easy. (If you look at work I worked on recently via its site https://eddelbuettel.github.io/r2u/ you will see that staring from eg eddelbuettel/r2u:22.04 you can run a single Rscript -e 'install.packages(c("rstan", "stringr", "gtools", "tidyr"))' without issues giving your the R side to which you can then add your Python requirements. I usually try that first interactively in the same Docker container. If all that fails, maybe you want to emphasize Python first and use Anaconda via mamba. Some folks had good luck with that (but don't mix and match, only tears result from that). We cannot help with *conda here. Dirk | I currently construct the docker image from a python/debian base image. I can get R working fine, but when I try to run an r-stan script within the lambda function I get a ?there is no package called Stan?, despite a good 5 minutes of the image build being dedicated to the download of rstan. Am looking for advice here on how to proceed. My current dockerfile follows? | | ################ | FROM python:3.8-slim-bullseye | | ENV DEBIAN_FRONTEND=noninteractive | | ENV LC_ALL=C.UTF-8 | ENV LANG=en_US.UTF-8 | ENV TZ=:/etc/localtime | ENV PATH=/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin | ENV LD_LIBRARY_PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib | ENV LAMBDA_TASK_ROOT=/var/task | ENV LAMBDA_RUNTIME_DIR=/var/runtime | | # Define custom function directory | ARG FUNCTION_DIR="/var/task" | WORKDIR ${FUNCTION_DIR} | | # Install aws-lambda-cpp build dependencies + r | RUN apt-get update && \ | apt-get install -y \ | g++ \ | make \ | cmake \ | unzip \ | libcurl4-openssl-dev \ | r-base | | COPY requirements.txt . | COPY irt_dim_red.R . | COPY . . | | #install the python requirements | RUN pip install -r requirements.txt | | COPY scripts/install_stan.R install_stan.R | | RUN Rscript "install_stan.R" | RUN R -e "install.packages('stringr')" | RUN R -e "install.packages('gtools')" | RUN R -e "install.packages('tidyr')" | | # Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile) | ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ] | CMD [ "main.handler"] | Where the script install_rstan.R is: | | # Following instructions from | # https://github.com/stan-dev/rstan/wiki/Installing-RStan-on-Linux | | print("Running install_stan.R") | | ## Creating /home/rstudio/.R/Makevars | | dotR <- file.path(Sys.getenv("HOME"), ".R") | if (!file.exists(dotR)) dir.create(dotR) | M <- file.path(dotR, "Makevars") | if (!file.exists(M)) file.create(M) | cat("\nCXX14FLAGS=-O3 -march=native -mtune=native -fPIC", | "CXX14=clang++", | file = M, sep = "\n", append = TRUE) | | ## Installing rstan | | install.packages("rstan", dependencies=TRUE, type = "source") | | Any help would be massively appreciated. | | Jesse | | | ? | | | Jesse McMullen-Crummey, PhD (he/him) | Data Engineer | | | [[alternative HTML version deleted]] | | _______________________________________________ | R-SIG-Debian mailing list | R-SIG-Debian at r-project.org | https://stat.ethz.ch/mailman/listinfo/r-sig-debian
dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
Hi All,
I?ve followed Dirk?s advice and am now starting from:
FROM eddelbuettel/r2u:22.04
While installing rstan using
RUN Rscript -e 'install.packages(c("rstan", "stringr", "gtools", "tidyr"))'
I can now get my lambda function running, but it throws an error when opening my Stan model, namely:
"C++ compiler not found on system.?
I would have thought that it would be part of this base image.
Any thoughts would be appreciated.
Jesse
un 13, 2022, at 12:36 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
Jesse, On 13 June 2022 at 12:19, Jesse McMullen-Crummey wrote: | Hi All, | | I found your mailing list via a stack overflow mention. I?m currently on my second full day of trying to build a docker container to use as an AWS lambda fcn and am feeling pretty stuck - I was hoping someone here could help. It needs to support python+R+rstan. If something does not work, I usually backtrack. You start from python:3.8-slim-bullseye. That is your right. But this is a Debian-and-R mailing list, so we may not have started from there. I maintain and curate several sets of Rocker containers building either on - plain Debian as our rocker/r-base which is also the official r-base - Ubuntu because I can then use Ubuntu binaries as from c2d4u or now also r2u If I were, I'd the same. With either base set you should get to 'apt install r-cran-rstan' easily, and adding your other package is easy. (If you look at work I worked on recently via its site https://eddelbuettel.github.io/r2u/ you will see that staring from eg eddelbuettel/r2u:22.04 you can run a single Rscript -e 'install.packages(c("rstan", "stringr", "gtools", "tidyr"))' without issues giving your the R side to which you can then add your Python requirements. I usually try that first interactively in the same Docker container. If all that fails, maybe you want to emphasize Python first and use Anaconda via mamba. Some folks had good luck with that (but don't mix and match, only tears result from that). We cannot help with *conda here. Dirk | I currently construct the docker image from a python/debian base image. I can get R working fine, but when I try to run an r-stan script within the lambda function I get a ?there is no package called Stan?, despite a good 5 minutes of the image build being dedicated to the download of rstan. Am looking for advice here on how to proceed. My current dockerfile follows? | | ################ | FROM python:3.8-slim-bullseye | | ENV DEBIAN_FRONTEND=noninteractive | | ENV LC_ALL=C.UTF-8 | ENV LANG=en_US.UTF-8 | ENV TZ=:/etc/localtime | ENV PATH=/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin | ENV LD_LIBRARY_PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib | ENV LAMBDA_TASK_ROOT=/var/task | ENV LAMBDA_RUNTIME_DIR=/var/runtime | | # Define custom function directory | ARG FUNCTION_DIR="/var/task" | WORKDIR ${FUNCTION_DIR} | | # Install aws-lambda-cpp build dependencies + r | RUN apt-get update && \ | apt-get install -y \ | g++ \ | make \ | cmake \ | unzip \ | libcurl4-openssl-dev \ | r-base | | COPY requirements.txt . | COPY irt_dim_red.R . | COPY . . | | #install the python requirements | RUN pip install -r requirements.txt | | COPY scripts/install_stan.R install_stan.R | | RUN Rscript "install_stan.R" | RUN R -e "install.packages('stringr')" | RUN R -e "install.packages('gtools')" | RUN R -e "install.packages('tidyr')" | | # Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile) | ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ] | CMD [ "main.handler"] | Where the script install_rstan.R is: | | # Following instructions from | # https://github.com/stan-dev/rstan/wiki/Installing-RStan-on-Linux | | print("Running install_stan.R") | | ## Creating /home/rstudio/.R/Makevars | | dotR <- file.path(Sys.getenv("HOME"), ".R") | if (!file.exists(dotR)) dir.create(dotR) | M <- file.path(dotR, "Makevars") | if (!file.exists(M)) file.create(M) | cat("\nCXX14FLAGS=-O3 -march=native -mtune=native -fPIC", | "CXX14=clang++", | file = M, sep = "\n", append = TRUE) | | ## Installing rstan | | install.packages("rstan", dependencies=TRUE, type = "source") | | Any help would be massively appreciated. | | Jesse | | | ? | | | Jesse McMullen-Crummey, PhD (he/him) | Data Engineer | | | [[alternative HTML version deleted]] | | _______________________________________________ | R-SIG-Debian mailing list | R-SIG-Debian at r-project.org | https://stat.ethz.ch/mailman/listinfo/r-sig-debian -- dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
On 13 June 2022 at 14:25, Jesse McMullen-Crummey wrote:
| Hi All,
|
| I?ve followed Dirk?s advice and am now starting from:
|
| FROM eddelbuettel/r2u:22.04
| While installing rstan using
| RUN Rscript -e 'install.packages(c("rstan", "stringr", "gtools", "tidyr"))'
(BTW I would actually use
RUN install.r rstan stringr gtools tidyr
which is more idiomatic _for me_ using littler, but equivalent <wink> )
| I can now get my lambda function running,
Nice!
| but it throws an error when opening my Stan model, namely:
| "C++ compiler not found on system.?
|
| I would have thought that it would be part of this base image.
Indeed. That r2u image has g++-11.
So (r)stan may be configured differently / may have seen / expected a
different compiler? I would check its docs. You should be able to square
that.
Dirk
dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
I'm not an expert, but I do use rstan on Ubuntu. The install instructions here: https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started include info on configuring the C++ Toolchain with a link for configuring on linux here: ?https://github.com/stan-dev/rstan/wiki/Configuring-C-Toolchain-for-Linux in case this helps ... Best, Liz Tighe
On 6/13/2022 2:36 PM, Dirk Eddelbuettel wrote:
On 13 June 2022 at 14:25, Jesse McMullen-Crummey wrote:
| Hi All,
|
| I?ve followed Dirk?s advice and am now starting from:
|
| FROM eddelbuettel/r2u:22.04
| While installing rstan using
| RUN Rscript -e 'install.packages(c("rstan", "stringr", "gtools", "tidyr"))'
(BTW I would actually use
RUN install.r rstan stringr gtools tidyr
which is more idiomatic _for me_ using littler, but equivalent <wink> )
| I can now get my lambda function running,
Nice!
| but it throws an error when opening my Stan model, namely:
| "C++ compiler not found on system.?
|
| I would have thought that it would be part of this base image.
Indeed. That r2u image has g++-11.
So (r)stan may be configured differently / may have seen / expected a
different compiler? I would check its docs. You should be able to square
that.
Dirk
Thanks Liz. I had that toolchain running as an install_stan.R script. I ended up reverting to 20.04 and installing build-essential as well. One of those seems to have done the trick! Thanks! Jesse
On Jun 13, 2022, at 4:01 PM, Elizabeth Tighe <tighe at brandeis.edu> wrote: I'm not an expert, but I do use rstan on Ubuntu. The install instructions here: https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started <https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started> include info on configuring the C++ Toolchain with a link for configuring on linux here: https://github.com/stan-dev/rstan/wiki/Configuring-C-Toolchain-for-Linux <https://github.com/stan-dev/rstan/wiki/Configuring-C-Toolchain-for-Linux> in case this helps ... Best, Liz Tighe On 6/13/2022 2:36 PM, Dirk Eddelbuettel wrote:
On 13 June 2022 at 14:25, Jesse McMullen-Crummey wrote:
| Hi All,
|
| I?ve followed Dirk?s advice and am now starting from:
|
| FROM eddelbuettel/r2u:22.04
| While installing rstan using
| RUN Rscript -e 'install.packages(c("rstan", "stringr", "gtools", "tidyr"))'
(BTW I would actually use
RUN install.r rstan stringr gtools tidyr
which is more idiomatic _for me_ using littler, but equivalent <wink> )
| I can now get my lambda function running,
Nice!
| but it throws an error when opening my Stan model, namely:
| "C++ compiler not found on system.?
|
| I would have thought that it would be part of this base image.
Indeed. That r2u image has g++-11.
So (r)stan may be configured differently / may have seen / expected a
different compiler? I would check its docs. You should be able to square
that.
Dirk
_______________________________________________ R-SIG-Debian mailing list R-SIG-Debian at r-project.org <mailto:R-SIG-Debian at r-project.org> https://stat.ethz.ch/mailman/listinfo/r-sig-debian <https://stat.ethz.ch/mailman/listinfo/r-sig-debian>