Skip to content

[Bioc-devel] Bioconductor package build fails

4 messages · Hoffman, Gabriel, Hervé Pagès, Martin Morgan

Original
Hoffman, Gabriel
# 
I am the developer of the variancePartition package, and I recently made some very small changes to the documentation of my package.  The latest version 1.10.3 builds correctly on Bioconductor's Windows and OS X machines but fails on Ubuntu:

http://bioconductor.org/checkResults/release/bioc-LATEST/variancePartition/malbec2-buildsrc.html

The code in the package has been stable, and I am not able to reproduce the error on by Ubuntu machine 14.04.5 LTS with R 3.5.1.

The issue has something to do with multithreading using doParallel.  The error is:

Warning in socketConnection("localhost", port = port, server = TRUE, blocking = TRUE,  :
  port 11104 cannot be opened
Quitting from lines 102-154 (variancePartition.Rnw)
Error: processing vignette 'variancePartition.Rnw' failed with diagnostics:
cannot open the connection
Execution halted

With the Bioconductor 3.7 Freeze coming, I?m concerned about getting stuck with a broken package.

Thanks,
 - Gabriel
Hervé Pagès
# 
Hi Gabriel,

I'm not able to reproduce this either. See my sessionInfo()
at the end (obtained after running all the code in the
variancePartition.Rnw vignette).

Did this error just appear today or yesterday, so would follow
your last commit to the RELEASE_3_7 branch? Or did it appear
before e.g. a few days ago?

If the former, then let's wait a couple more day and see if
the error reproduces. Could be a build machine hiccup and maybe
it will just go away.

If the latter, then it could be related to the new version of
doParallel (1.0.14) that showed up on CRAN on Sept 24 (18 days
ago). Our build machines picked it up and installed it so now
all builds are using this new version (as you can see by clicking
on the links in the rightmost column of the small table at the
top of the build report).

As a side note: you use pander in the vignette so would need to
add it to the Suggests field of the package.

Cheers,
H.

 > sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.5 LTS

Matrix products: default
BLAS: /home/hpages/R/R-3.5.1/lib/libRblas.so
LAPACK: /home/hpages/R/R-3.5.1/lib/libRlapack.so

locale:
  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
  [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats4    parallel  stats     graphics  grDevices utils     datasets
[8] methods   base

other attached packages:
  [1] dendextend_1.8.0            ballgown_2.12.0
  [3] DESeq2_1.20.0               SummarizedExperiment_1.10.1
  [5] DelayedArray_0.6.6          BiocParallel_1.14.2
  [7] matrixStats_0.54.0          GenomicRanges_1.32.7
  [9] GenomeInfoDb_1.16.0         IRanges_2.14.12
[11] S4Vectors_0.18.3            edgeR_3.22.5
[13] lme4_1.1-18-1               Matrix_1.2-14
[15] doParallel_1.0.14           iterators_1.0.10
[17] variancePartition_1.10.3    Biobase_2.40.0
[19] BiocGenerics_0.26.0         scales_1.0.0
[21] foreach_1.4.4               limma_3.36.5
[23] ggplot2_3.0.0               knitr_1.20

loaded via a namespace (and not attached):
   [1] minqa_1.2.4              colorspace_1.3-2         class_7.3-14 

   [4] modeltools_0.2-22        colorRamps_2.3           mclust_5.4.1 

   [7] rprojroot_1.3-2          htmlTable_1.12           XVector_0.20.0 

  [10] base64enc_0.1-3          rstudioapi_0.8           flexmix_2.3-14 

  [13] bit64_0.9-7              mvtnorm_1.0-8 
AnnotationDbi_1.42.1
  [16] codetools_0.2-15         splines_3.5.1 
robustbase_0.93-3
  [19] geneplotter_1.58.0       Formula_1.2-3            nloptr_1.2.1 

  [22] Rsamtools_1.32.3         pbkrtest_0.4-7           annotate_1.58.0 

  [25] kernlab_0.9-27           cluster_2.0.7-1          compiler_3.5.1 

  [28] backports_1.1.2          assertthat_0.2.0         lazyeval_0.2.1 

  [31] acepack_1.4.1            htmltools_0.3.6          tools_3.5.1 

  [34] bindrcpp_0.2.2           gtable_0.2.0             glue_1.3.0 

  [37] GenomeInfoDbData_1.1.0   reshape2_1.4.3           dplyr_0.7.6 

  [40] Rcpp_0.12.19             trimcluster_0.1-2.1 
Biostrings_2.48.0
  [43] gdata_2.18.0             nlme_3.1-137 
rtracklayer_1.40.6
  [46] fpc_2.1-11.1             stringr_1.3.1            gtools_3.8.1 

  [49] XML_3.98-1.16            DEoptimR_1.0-8           zlibbioc_1.26.0 

  [52] MASS_7.3-50              BiocStyle_2.8.2 
RColorBrewer_1.1-2
  [55] yaml_2.2.0               memoise_1.1.0            gridExtra_2.3 

  [58] rpart_4.1-13             latticeExtra_0.6-28      stringi_1.2.4 

  [61] RSQLite_2.1.1            genefilter_1.62.0        checkmate_1.8.5 

  [64] caTools_1.17.1.1         prabclus_2.2-6           rlang_0.2.2 

  [67] pkgconfig_2.0.2          bitops_1.0-6             evaluate_0.12 

  [70] lattice_0.20-35          purrr_0.2.5              bindr_0.1.1 

  [73] GenomicAlignments_1.16.0 htmlwidgets_1.3          labeling_0.3 

  [76] bit_1.1-14               tidyselect_0.2.5         plyr_1.8.4 

  [79] magrittr_1.5             R6_2.3.0                 gplots_3.0.1 

  [82] Hmisc_4.1-1              DBI_1.0.0                pillar_1.3.0 

  [85] whisker_0.3-2            foreign_0.8-71           withr_2.1.2 

  [88] mgcv_1.8-24              survival_2.42-6          RCurl_1.95-4.11 

  [91] nnet_7.3-12              tibble_1.4.2             crayon_1.3.4 

  [94] KernSmooth_2.23-15       rmarkdown_1.10           viridis_0.5.1 

  [97] locfit_1.5-9.1           grid_3.5.1               sva_3.28.0 

[100] data.table_1.11.8        blob_1.1.1               diptest_0.75-7 

[103] digest_0.6.18            xtable_1.8-3             munsell_0.5.0 

[106] viridisLite_0.3.0
 >
On 10/11/2018 11:38 AM, Hoffman, Gabriel wrote:

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Hoffman, Gabriel
    

    

      
      #
    

  


  

      
Thanks for your help.  This just showed up last night.  I will wait few
more days and see if it continues to be an issue.  Also, I?ll add pander

- Gabriel

        
On 10/11/18, 7:01 PM, "Herv? Pag?s" <hpages at fredhutch.org> wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Martin Morgan
    

    

      
      #
    

  


  

      
It might make sense to seriously throttle your use of parallel evaluation in your vignette / example code, perhaps even defaulting to serial evaluation. This is also relevant to resource use on the build machines. I'm not really sure of the details but it seems like the problem is that the socket cluster is trying to use a particular port, maybe chosen randomly from a range, and it turns out that the port is already in use, e.g., by another package being built.

Martin

?On 10/11/18, 7:11 PM, "Bioc-devel on behalf of Hoffman, Gabriel" <bioc-devel-bounces at r-project.org on behalf of gabriel.hoffman at mssm.edu> wrote:

    Thanks for your help.  This just showed up last night.  I will wait few
    more days and see if it continues to be an issue.  Also, I?ll add pander
    
    - Gabriel

        
On 10/11/18, 7:01 PM, "Herv? Pag?s" <hpages at fredhutch.org> wrote:

        
>Hi Gabriel,
    >
    >I'm not able to reproduce this either. See my sessionInfo()
    >at the end (obtained after running all the code in the
    >variancePartition.Rnw vignette).
    >
    >Did this error just appear today or yesterday, so would follow
    >your last commit to the RELEASE_3_7 branch? Or did it appear
    >before e.g. a few days ago?
    >
    >If the former, then let's wait a couple more day and see if
    >the error reproduces. Could be a build machine hiccup and maybe
    >it will just go away.
    >
    >If the latter, then it could be related to the new version of
    >doParallel (1.0.14) that showed up on CRAN on Sept 24 (18 days
    >ago). Our build machines picked it up and installed it so now
    >all builds are using this new version (as you can see by clicking
    >on the links in the rightmost column of the small table at the
    >top of the build report).
    >
    >As a side note: you use pander in the vignette so would need to
    >add it to the Suggests field of the package.
    >
    >Cheers,
    >H.
    >
    > > sessionInfo()
    >R version 3.5.1 (2018-07-02)
    >Platform: x86_64-pc-linux-gnu (64-bit)
    >Running under: Ubuntu 16.04.5 LTS
    >
    >Matrix products: default
    >BLAS: 
    >/home/hpages/R/R-3.5.1/lib/https://urldefense.proofpoint.com/v2/url?u=http
    >-3A__libRblas.so&d=DwIFaQ&c=shNJtf5dKgNcPZ6Yh64b-A&r=KdYcmw5SdXylMrTGSuNVk
    >NJulowod64k0PTDC5BHZkk&m=KuNqEpeopBuZGxLxcykyI2_IwkJE_oSorni83Ae717E&s=3jM
    >Ia0up64DNhDmFncfoVMDawXZIKv8OCyswMc8VWQ0&e=
    >LAPACK: 
    >/home/hpages/R/R-3.5.1/lib/https://urldefense.proofpoint.com/v2/url?u=http
    >-3A__libRlapack.so&d=DwIFaQ&c=shNJtf5dKgNcPZ6Yh64b-A&r=KdYcmw5SdXylMrTGSuN
    >VkNJulowod64k0PTDC5BHZkk&m=KuNqEpeopBuZGxLxcykyI2_IwkJE_oSorni83Ae717E&s=M
    >8hFMwj_eBOoEvdwmXC3h7J_VmhrPPNwg5B7giSrSSs&e=
    >
    >locale:
    >  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
    >  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
    >  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
    >  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
    >  [9] LC_ADDRESS=C               LC_TELEPHONE=C
    >[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
    >
    >attached base packages:
    >[1] stats4    parallel  stats     graphics  grDevices utils     datasets
    >[8] methods   base
    >
    >other attached packages:
    >  [1] dendextend_1.8.0            ballgown_2.12.0
    >  [3] DESeq2_1.20.0               SummarizedExperiment_1.10.1
    >  [5] DelayedArray_0.6.6          BiocParallel_1.14.2
    >  [7] matrixStats_0.54.0          GenomicRanges_1.32.7
    >  [9] GenomeInfoDb_1.16.0         IRanges_2.14.12
    >[11] S4Vectors_0.18.3            edgeR_3.22.5
    >[13] lme4_1.1-18-1               Matrix_1.2-14
    >[15] doParallel_1.0.14           iterators_1.0.10
    >[17] variancePartition_1.10.3    Biobase_2.40.0
    >[19] BiocGenerics_0.26.0         scales_1.0.0
    >[21] foreach_1.4.4               limma_3.36.5
    >[23] ggplot2_3.0.0               knitr_1.20
    >
    >loaded via a namespace (and not attached):
    >   [1] minqa_1.2.4              colorspace_1.3-2         class_7.3-14
    >
    >   [4] modeltools_0.2-22        colorRamps_2.3           mclust_5.4.1
    >
    >   [7] rprojroot_1.3-2          htmlTable_1.12           XVector_0.20.0
    >
    >  [10] base64enc_0.1-3          rstudioapi_0.8           flexmix_2.3-14
    >
    >  [13] bit64_0.9-7              mvtnorm_1.0-8
    >AnnotationDbi_1.42.1
    >  [16] codetools_0.2-15         splines_3.5.1
    >robustbase_0.93-3
    >  [19] geneplotter_1.58.0       Formula_1.2-3            nloptr_1.2.1
    >
    >  [22] Rsamtools_1.32.3         pbkrtest_0.4-7           annotate_1.58.0
    >
    >  [25] kernlab_0.9-27           cluster_2.0.7-1          compiler_3.5.1
    >
    >  [28] backports_1.1.2          assertthat_0.2.0         lazyeval_0.2.1
    >
    >  [31] acepack_1.4.1            htmltools_0.3.6          tools_3.5.1
    >
    >  [34] bindrcpp_0.2.2           gtable_0.2.0             glue_1.3.0
    >
    >  [37] GenomeInfoDbData_1.1.0   reshape2_1.4.3           dplyr_0.7.6
    >
    >  [40] Rcpp_0.12.19             trimcluster_0.1-2.1
    >Biostrings_2.48.0
    >  [43] gdata_2.18.0             nlme_3.1-137
    >rtracklayer_1.40.6
    >  [46] fpc_2.1-11.1             stringr_1.3.1            gtools_3.8.1
    >
    >  [49] XML_3.98-1.16            DEoptimR_1.0-8           zlibbioc_1.26.0
    >
    >  [52] MASS_7.3-50              BiocStyle_2.8.2
    >RColorBrewer_1.1-2
    >  [55] yaml_2.2.0               memoise_1.1.0            gridExtra_2.3
    >
    >  [58] rpart_4.1-13             latticeExtra_0.6-28      stringi_1.2.4
    >
    >  [61] RSQLite_2.1.1            genefilter_1.62.0        checkmate_1.8.5
    >
    >  [64] 
    >caTools_https://urldefense.proofpoint.com/v2/url?u=http-3A__1.17.1.1&d=DwI
    >FaQ&c=shNJtf5dKgNcPZ6Yh64b-A&r=KdYcmw5SdXylMrTGSuNVkNJulowod64k0PTDC5BHZkk
    >&m=KuNqEpeopBuZGxLxcykyI2_IwkJE_oSorni83Ae717E&s=c5XRWImsmPTHqu8vDjbP_MZ-k
    >E4pZylRmN0EZMQIqlA&e=         prabclus_2.2-6           rlang_0.2.2
    >
    >  [67] pkgconfig_2.0.2          bitops_1.0-6             evaluate_0.12
    >
    >  [70] lattice_0.20-35          purrr_0.2.5              bindr_0.1.1
    >
    >  [73] GenomicAlignments_1.16.0 htmlwidgets_1.3          labeling_0.3
    >
    >  [76] bit_1.1-14               tidyselect_0.2.5         plyr_1.8.4
    >
    >  [79] magrittr_1.5             R6_2.3.0                 gplots_3.0.1
    >
    >  [82] Hmisc_4.1-1              DBI_1.0.0                pillar_1.3.0
    >
    >  [85] whisker_0.3-2            foreign_0.8-71           withr_2.1.2
    >
    >  [88] mgcv_1.8-24              survival_2.42-6          RCurl_1.95-4.11
    >
    >  [91] nnet_7.3-12              tibble_1.4.2             crayon_1.3.4
    >
    >  [94] KernSmooth_2.23-15       rmarkdown_1.10           viridis_0.5.1
    >
    >  [97] locfit_1.5-9.1           grid_3.5.1               sva_3.28.0
    >
    >[100] data.table_1.11.8        blob_1.1.1               diptest_0.75-7
    >
    >[103] digest_0.6.18            xtable_1.8-3             munsell_0.5.0
    >
    >[106] viridisLite_0.3.0
    > >
    >
    >

        
>On 10/11/2018 11:38 AM, Hoffman, Gabriel wrote:

        
>> I am the developer of the variancePartition package, and I recently
    >>made some very small changes to the documentation of my package.  The
    >>latest version 1.10.3 builds correctly on Bioconductor's Windows and OS
    >>X machines but fails on Ubuntu:
    >> 
    >> 
    >>https://urldefense.proofpoint.com/v2/url?u=http-3A__bioconductor.org_chec
    >>kResults_release_bioc-2DLATEST_variancePartition_malbec2-2Dbuildsrc.html&
    >>d=DwIFaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKa
    >>aPhzWA&m=zy-TmtsjB4a5Mwg313kvmlN0-6yO-6JDYZsT1nV6cCw&s=RBYxTA_a5mlzemSuuk
    >>87tom-Q10dMclvh6KjhOGULDw&e=
    >> 
    >> The code in the package has been stable, and I am not able to reproduce
    >>the error on by Ubuntu machine 14.04.5 LTS with R 3.5.1.
    >> 
    >> The issue has something to do with multithreading using doParallel.
    >>The error is:
    >> 
    >> Warning in socketConnection("localhost", port = port, server = TRUE,
    >>blocking = TRUE,  :
    >>    port 11104 cannot be opened
    >> Quitting from lines 102-154 (variancePartition.Rnw)
    >> Error: processing vignette 'variancePartition.Rnw' failed with
    >>diagnostics:
    >> cannot open the connection
    >> Execution halted
    >> 
    >> With the Bioconductor 3.7 Freeze coming, I?m concerned about getting
    >>stuck with a broken package.
    >> 
    >> Thanks,
    >>   - Gabriel
    >> 
    >> 	[[alternative HTML version deleted]]
    >> 
    >> 
    >> 
    >> _______________________________________________
    >> Bioc-devel at r-project.org mailing list
    >> 
    >>https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman
    >>_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWd
    >>GbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=zy-TmtsjB4a5Mwg313kvmlN0-6yO-6JDYZsT1nV6c
    >>Cw&s=5WZzTZGK1jX2QI5fKn_INmjLYyv876INVFqftwFxlT4&e=
    >> 
    >
    >-- 
    >Herv? Pag?s
    >
    >Program in Computational Biology
    >Division of Public Health Sciences
    >Fred Hutchinson Cancer Research Center
    >1100 Fairview Ave. N, M1-B514
    >P.O. Box 19024
    >Seattle, WA 98109-1024
    >
    >E-mail: hpages at fredhutch.org
    >Phone:  (206) 667-5791
    >Fax:    (206) 667-1319
    
    _______________________________________________
    Bioc-devel at r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/bioc-devel