Skip to content

[Bioc-devel] Problems with BiocParallel on Windows

8 messages · Jiefei Wang, Giulia Pais

Original
Giulia Pais
# 
Hello,
I?m the developer of the package ISAnalytics. Since the last build we?re having issues with BiocParallel on the Windows platform only. In particular, trying to debug what the issue might be on a local Windows machine, it seems BiocParallel isn?t able to find internal package functions and fails at several jobs that did just fine a few versions ago.
We?re using MulticoreParam for Linux and Mac and SnowParam for Windows as suggested, I?m not really sure how to fix this issue and I?ll appreciate any help,
Thank you
Giulia Pais
Jiefei Wang
# 
Hi Giulia,

One of your errors is raised from L1288 in the file "analysis-functions.R"

if (getOption("ISAnalytics.verbose") == TRUE) { ... }

The error message is "argument is of length zero". Usually, this means
something is wrong with the global option. Then I see at L4812 in
"internal-functions.R", you have

p <- BiocParallel::SnowParam(
                tasks = length(common_names),
                progressbar = getOption("ISAnalytics.verbose"),
                exportglobals = FALSE
            )

Please keep in mind that if you use SnowParam, the worker is a fresh R
process and independent with your master. Since you have
`exportglobals = FALSE`, the worker has no idea about the option
"ISAnalytics.verbose" in your master process and hence the error.

To fix that, you can set  `exportglobals = TRUE` to let the worker
know the option. If you do not want to export the globals, you can
also add the global option "ISAnalytics.verbose" manually at the
beginning of the function in bplapply.

Best,
Jiefei
On Mon, Oct 25, 2021 at 5:42 PM Giulia Pais <giuliapais1 at gmail.com> wrote:
Giulia Pais
# 
I tried that also on other functions but it seems also exported functions are invisible to SnowParam workers. That is really odd since it worked just fine before. For now since we're approaching the deadline for the new Bioc release we may opt to switch everything to MulticoreParam despite being just a serial param on Windows. We'll try to figure out what causes this after the new release, thanks
?On 10/25/21, 12:57, "Jiefei Wang" <szwjf08 at gmail.com> wrote:
Hi Giulia,

    One of your errors is raised from L1288 in the file "analysis-functions.R"

    if (getOption("ISAnalytics.verbose") == TRUE) { ... }

    The error message is "argument is of length zero". Usually, this means
    something is wrong with the global option. Then I see at L4812 in
    "internal-functions.R", you have

    p <- BiocParallel::SnowParam(
                    tasks = length(common_names),
                    progressbar = getOption("ISAnalytics.verbose"),
                    exportglobals = FALSE
                )

    Please keep in mind that if you use SnowParam, the worker is a fresh R
    process and independent with your master. Since you have
    `exportglobals = FALSE`, the worker has no idea about the option
    "ISAnalytics.verbose" in your master process and hence the error.

    To fix that, you can set  `exportglobals = TRUE` to let the worker
    know the option. If you do not want to export the globals, you can
    also add the global option "ISAnalytics.verbose" manually at the
    beginning of the function in bplapply.

    Best,
    Jiefei
On Mon, Oct 25, 2021 at 5:42 PM Giulia Pais <giuliapais1 at gmail.com> wrote:
>
    > Hello,
    > I?m the developer of the package ISAnalytics. Since the last build we?re having issues with BiocParallel on the Windows platform only. In particular, trying to debug what the issue might be on a local Windows machine, it seems BiocParallel isn?t able to find internal package functions and fails at several jobs that did just fine a few versions ago.
    > We?re using MulticoreParam for Linux and Mac and SnowParam for Windows as suggested, I?m not really sure how to fix this issue and I?ll appreciate any help,
    > Thank you
    > Giulia Pais
    >
    >         [[alternative HTML version deleted]]
    >
    > _______________________________________________
    > Bioc-devel at r-project.org mailing list
    > https://stat.ethz.ch/mailman/listinfo/bioc-devel
Jiefei Wang
# 
What do you mean by "exported functions are invisible to SnowParam
workers"? Please provide more details. If you call 'bplapply' with a
function from your package, it should be visible to the worker.
Giulia Pais
# 
I mean that trying to modify the code of one of the functions that raises errors and setting the option "exportglobals = TRUE" on SnowParam, somehow it still fails and says it doesn't find the function "mandatory_is_vars()" which is a function exported by the package
?On 10/25/21, 13:49, "Jiefei Wang" <szwjf08 at gmail.com> wrote:
What do you mean by "exported functions are invisible to SnowParam
    workers"? Please provide more details. If you call 'bplapply' with a
    function from your package, it should be visible to the worker.
Jiefei Wang
# 
I do not see it from the build report here:
http://bioconductor.org/checkResults/devel/bioc-LATEST/ISAnalytics/riesling1-checksrc.html

Can you please provide a reproducible example? It can be very helpful
to locate the issue
On Mon, Oct 25, 2021 at 7:53 PM Giulia Pais <giuliapais1 at gmail.com> wrote:
Giulia Pais
# 
Sorry, my bad, I actually changed the code but for it work properly it is mandatory to rebuild the package completely (not only loading it through devtools). It seems to work for 1 function, I'll change the option export global in all the code and rebuild it and see if it works. Do you know if it is necessary also for MulticoreParam? It doesn't seem to be necessary and when I previously set the option to TRUE I always got warning messages for the stats package, that?s way we set it to FALSE.
Thank you so much for your help
?On 10/25/21, 13:58, "Jiefei Wang" <szwjf08 at gmail.com> wrote:
I do not see it from the build report here:
    http://bioconductor.org/checkResults/devel/bioc-LATEST/ISAnalytics/riesling1-checksrc.html

    Can you please provide a reproducible example? It can be very helpful
    to locate the issue
On Mon, Oct 25, 2021 at 7:53 PM Giulia Pais <giuliapais1 at gmail.com> wrote:
>
    > I mean that trying to modify the code of one of the functions that raises errors and setting the option "exportglobals = TRUE" on SnowParam, somehow it still fails and says it doesn't find the function "mandatory_is_vars()" which is a function exported by the package
    >
> ?On 10/25/21, 13:49, "Jiefei Wang" <szwjf08 at gmail.com> wrote:
>
    >     What do you mean by "exported functions are invisible to SnowParam
    >     workers"? Please provide more details. If you call 'bplapply' with a
    >     function from your package, it should be visible to the worker.
Jiefei Wang
# 
There is no need to do it in MulticoreParam as it will simply fork
your master process and therefore inherit your global setting. I think
the warning you get is due to a bug in RStudio, I hope they can fix it
but it might not be that fast...

Best,
Jiefei
On Mon, Oct 25, 2021 at 8:16 PM Giulia Pais <giuliapais1 at gmail.com> wrote: