[Bioc-devel] Python module "tensorflow_probability" not found

Thank you all so much for your input and the references!

@Kasper: I mostly rely on tensorflow and tensorflow-probability, so I might
somehow get rid of the keras dependency but it would require some work.

After being inspired by the lovely orthos package (thanks Charlotte!), I
decided to play around further in the basilisk direction and updated my
project (https://github.com/cbg-ethz/pareg/tree/feature-basilisk).
The hardest part was figuring out a set of package versions which satisfy
conda's package manager (@Vincent, I feel you!).
But then it just magically worked on my local machine.

When testing with GitHub Actions, the windows runner crashes with
PackagesNotFoundError: The following packages are not available from
current channels:
  - tensorflow=2.11.1
A look at conda-forge (https://anaconda.org/conda-forge/tensorflow)
reveals
that for win-64, there's only v1.14.0 available... I guess ignoring the
windows build is not an option for my bioc package?

For the ubuntu runner, my vignettes were created successfully ("* creating
vignettes ... OK"). My tests still fail, but that is expected because I
have not wrapped them with basiliskRun. Do I have to do this manually for
every function call which may interact with tensorflow (so much
boilerplate), or can I somehow implicitly use the created conda env for
every function in my package?

On Thu, Jul 6, 2023 at 2:08?PM Vincent Carey <stvjc at channing.harvard.edu>
wrote:

That's great news.  FWIW I am finding that the advice at
https://rstudio.github.io/reticulate/articles/python_dependencies.html
can work to produce properly resolved python dependencies.  Just don't
follow the example literally; the requested
scipy version may not exist.  Version 1.11.1 does. Stay tuned.

On Thu, Jul 6, 2023 at 7:43?AM Charlotte Soneson <
charlottesoneson at gmail.com>
wrote:

Hi,

in case it's useful: we have a package (orthos) in review (
https://github.com/Bioconductor/Contributions/issues/3042) which uses
basilisk to set up a conda environment with tensorflow and keras. It

builds

and runs fine both on GitHub Actions (GitHub repo here:
https://github.com/fmicompbio/orthos) and on the Single Package

Builder.

We have also tested (locally) that it will use GPUs if available, and

that

the GPU configuration can be controlled from the R session (outside of

the

package), e.g. by setting the CUDA_VISIBLE_DEVICES environment

variable.

Charlotte

On 5 Jul 2023, at 23:12, Kasper Daniel Hansen <

kasperdanielhansen at gmail.com> wrote:

So I think Kim is interfacing to tensorflow by using the keras

package

from

CRAN (partly authored by the Rstudio people). This package leaves it

to

the

user to install tensorflow, which is a highly non-trivial

installation

task. There is some partly helpful instructions for using conda

together

with reticulate (see the macOS tab on
https://tensorflow.rstudio.com/install/local_gpu.html). This is the

job

that basilisk handles for you. In essence, basilisk allows the

developer

to

specify an R-package-specific conda. Tensorflow can be run on a

CPUor a

GPU. Getting it to run on a user-GPU is extra complicated and I am

not

sure

basilisk can handle this.

Going forward, we (Bioc) want to decide if we want to support keras

on

our

build system. This will require some work, because it is definitely

not

trivial to get to work (but much more possible if we limit ourselves

to

running on CPU). If we decide to support keras, we should try to

figure

out

how to wrap keras into a basilisk container; perhaps something like
creating a keras-basilisk R package, because IF we decide to support

keras,

this is going to be a major headache (to add to the frustration,

tensorflow

often rearranges everything so I future issues keeping it

operational).

For Kim: I think you should consider if there are any alternatives to
keras. Even if we get it to work on our build system, users will have

major

headache getting this to work (I think). If there are no alternatives

to

keras, you should perhaps think about doing the keras-basilisk

option I

outline above (assuming that is feasible; I don't know how keras

interfaces

with tensorflow. You might also have major headaches in your future:

I

have

found quite big differences in convergence and optimizers over time

in

TF

and you're basically saying it should work with every version of

tensorflow

= 2.2. That's a .... strong promise considering also the difference

between GPU and CPU.

In the meantime, in case the core build team wants some input, I have

some

experience with tensorflow, although so far my experience is mostly
frustration.

Best,
Kasper


On Wed, Jul 5, 2023 at 3:12?PM Vincent Carey <

stvjc at channing.harvard.edu

wrote:

I'll try to get clearer on the basilisk situation; I forked your

repo

and will plunge in soon.
It may take a while.  In the mean time I hope the BBS python stack

can

be looked at
to see what the issue might be.  @Jennifer Wokaty is it possible

with

the recent R upgrades that
tensorflow and tensorflow-probability might need to be

installed/updated?

On Wed, Jul 5, 2023 at 3:01?PM Kim Philipp Jablonski
<kim.philipp.jablonski at gmail.com> wrote:

Thanks a lot for your response!

The Config/reticulate approach in DESCRIPTION looks very neat. I

hope

we

can use it at some point.

Could the issue you are facing with the conda install be, that the

package is called "tensorflow-probability" instead of
"tensorflow_probability" (
https://anaconda.org/conda-forge/tensorflow-probability)?

I have followed the basilisk advice and incorporated it into my

package

here: https://github.com/cbg-ethz/pareg/tree/feature-basilisk.

The basilisk docs say "Any R functions that use Python code should

do

so

via basiliskRun()".

This does not seem possible in my case as I am using

tensorflow(probability) wrappers instead of calling them directly.

Building the vignettes with Github Actions fails because tensorflow

could not be found:

https://github.com/cbg-ethz/pareg/actions/runs/5466980003/jobs/9952690137#step:15:39

(I might have to activate basilisk somehow, but using basiliskStart

seems

to only make sense with basiliskRun). I could also not find any

advice

on

this elsewhere.

Do you have a suggestion how to best handle this (or should it

*just*

work)?

On Tue, Jul 4, 2023 at 1:04?PM Vincent Carey <

stvjc at channing.harvard.edu>

wrote:

Sorry to hear about this.  Our informal outlook on python
interoperation in Bioconductor packages is that
the basilisk discipline should be used.  This allows you to pin
specific versions of all python dependencies
and use python for your R functions from an insulated conda

environment.

There is also information about specifying python dependencies in

the

DESCRIPTION file at

https://rstudio.github.io/reticulate/articles/python_dependencies.html

I do not know if our build
system or R CMD install take effective advantage of that approach

at

this time.

I was surprised to see that my attempt to use

reticulate::py_install()

to install tensorflow_probabiliity
on my laptop failed:

'/home/stvjc/.local/share/r-miniconda/bin/conda' 'install' '--yes'
'--prefix'

'/home/stvjc/.local/share/r-miniconda/envs/r-reticulate'

'-c' 'conda-forge' 'tensorflow_probability'
Retrieving notices: ...working... done
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen

solve.

Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: unsuccessful initial attempt using frozen

solve.

Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available

from

current channels:

 - tensorflow_probability

Current channels:

 - https://conda.anaconda.org/conda-forge/linux-64
 - https://conda.anaconda.org/conda-forge/noarch
 - https://repo.anaconda.com/pkgs/main/linux-64
 - https://repo.anaconda.com/pkgs/main/noarch
 - https://repo.anaconda.com/pkgs/r/linux-64
 - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda

package

you're

looking for, navigate to

   https://anaconda.org

and use the search bar at the top of the page.


Error: one or more Python packages failed to install [error code

1]

Enter a frame number, or 0 to exit

1: py_install("tensorflow_probability")

So the "current channels" used by reticulate in my pretty stock
installation of R/reticulate seem flawed
for this purpose.  I had hoped to write a bit of code that would

check

for the desired module and install
it if missing, that you could include in your package.

get_pippath = function() {
  pypath = reticulate::conda_list() |>

(\(x)x[x$name=="r-reticulate",])()

  gsub("python$", "pip", pypath$python)
}

check_tfp = function() {
  pippath = get_pippath()
  peek = system(paste0(pippath, " list | grep tensor"),

intern=TRUE)

  peek = gsub(" +", " ", peek)
  tfdf = do.call(rbind, lapply(strsplit(peek, " "), function(x)
data.frame(pkg=x[1], version=x[2])))
  "tensorflow_probability" %in% tfdf$pkg
}

install_tfp = function() {
  pippath = get_pippath()
  chk = system(paste(pippath, "install tensorflow_probability"),

intern=TRUE)

  chk2 = check_tfp()
  list(sysout=chk, tfp_installed=chk2)
}

The first two functions seem to do what I want, the latter does

not.

Suggestion -- read the basilisk vignettes, use it if at all

possible.

Certainly we can take
care of this in our build system when the holiday ends, but it

would

be great to have the software
arrange the solution when necessary, and I don't see a way to
accomplish this at the moment.



On Tue, Jul 4, 2023 at 5:32?AM Kim Philipp Jablonski
<kim.philipp.jablonski at gmail.com> wrote:

Hello,

The latest builds of my package 'pareg' started failing on BioC

3.17

for

Linux and macOS but not Windows:

https://master.bioconductor.org/checkResults/3.17/bioc-LATEST/pareg/nebbiolo1-buildsrc.html

.
In both cases, the error messages is "Python module

tensorflow_probability

was not found". This wasn't an issue in earlier releases.

Do you know what the reason for this is and how I can fix it?
Thanks a lot for your help!

Best regards,
Kim

       [[alternative HTML version deleted]]

--
The information in this e-mail is intended only for the person to

whom

it

is
addressed. If you believe this e-mail was sent to you in error and

the

e-mail
contains patient information, please contact the Partners

Compliance

HelpLine at
http://www.partners.org/complianceline
<http://www.partners.org/complianceline> . If the e-mail was sent

to

you in

error
but does not contain patient information, please contact the

sender

and properly
dispose of the e-mail.

--
The information in this e-mail is intended only for

th...{{dropped:27}}

--
The information in this e-mail is intended only for th...{{dropped:10}}