[Bioc-devel] Controlling vignette compilation order

In a number of my workflow packages (e.g., simpleSingleCell), I rely on a
specific compilation order for my vignettes. This is because some vignettes
set up resources or objects that are to be used by later vignettes.

From what I understand, vignettes are compiled in alphanumeric ordering of
their file names. As such, I give my vignettes fairly structured names,
e.g., ?work-1-reads.Rmd?, ?work-2-umi.Rmd? and so on.

However, it becomes rather annoying when I want to add a new vignette in
the middle somewhere. This results in some unnatural numberings, e.g.,
?work-0?, ?3b?, which are ugly and unintuitive. This is relevant as
BiocStyle::Biocpkg() links between vignettes require you to use the
destination vignette?s file name; so difficult names complicate linking,
especially if the names continually change to reflect new orderings.

Is there an easier way to control vignette compilation order? WRE provides
no (obvious) guidance, so I would like to know what non-standard hacks are
known to work on the build machines. I can imagine something dirty whereby
one ?reference? vignette contains code to ?rmarkdown::render" all other
vignettes in the specified order? ugh.

-A

On 18 Dec 2018, at 14:14, Martin Morgan <mtmorgan.bioc at gmail.com> wrote:

Also perhaps using BiocFileCache so that the result object is only generated once, then cached for future (different session) use.

?On 12/18/18, 8:35 AM, "Bioc-devel on behalf of Michael Lawrence" <bioc-devel-bounces at r-project.org on behalf of lawrence.michael at gene.com> wrote:

   I would recommend against dependencies across vignettes. Ideally someone
   can pick up a vignette and execute the code independently of any other
   documentation. Perhaps you could move the code generating those shared
   resources to the package. They could behave lazily, only generating the
   resource if necessary, otherwise reusing it. That would also make it easy
   for people to write their own documents using those resources.

   Michael

   On Tue, Dec 18, 2018 at 5:22 AM Aaron Lun <
   infinite.monkeys.with.keyboards at gmail.com> wrote:

In a number of my workflow packages (e.g., simpleSingleCell), I rely on a
specific compilation order for my vignettes. This is because some vignettes
set up resources or objects that are to be used by later vignettes.

From what I understand, vignettes are compiled in alphanumeric ordering of
their file names. As such, I give my vignettes fairly structured names,
e.g., ?work-1-reads.Rmd?, ?work-2-umi.Rmd? and so on.

However, it becomes rather annoying when I want to add a new vignette in
the middle somewhere. This results in some unnatural numberings, e.g.,
?work-0?, ?3b?, which are ugly and unintuitive. This is relevant as
BiocStyle::Biocpkg() links between vignettes require you to use the
destination vignette?s file name; so difficult names complicate linking,
especially if the names continually change to reflect new orderings.

Is there an easier way to control vignette compilation order? WRE provides
no (obvious) guidance, so I would like to know what non-standard hacks are
known to work on the build machines. I can imagine something dirty whereby
one ?reference? vignette contains code to ?rmarkdown::render" all other
vignettes in the specified order? ugh.

-A

   	[[alternative HTML version deleted]]

@Michael In this case, the resource produced by vignette X is a SingleCellExperiment object containing the results of various processing steps (normalization, clustering, etc.) described in that vignette.

I can imagine a lazy evaluation model for this, but it wouldn?t be pretty. If I had another vignette Y that depended on the SCE produced by vignette X, I would need Y to execute all of the steps in X if X hadn?t already been run before Y. This gets us into the territory of Makefile-like dependencies, which seems even more complicated than simply specifying a compilation order.

You might ask why X and Y are split into two separate vignettes. The use of different vignettes is motivated by the complexity of the workflows:

- Vignette 1 demonstrates core processing steps for one read-based single-cell RNAseq dataset.
- Vignette 2 demonstrates (slightly different) core steps for a UMI-based dataset.
- ? so on for a bunch of other core steps for different types of data.
- Vignette 6 demonstrates extra optional steps for the two SCEs produced by vignettes 1 & 3.
- ? and so on for a bunch of other optional steps.

The separation between core and optional steps into separate documents is desirable. From a pedagogical perspective, I would very much like to get the reader through all the core steps before even considering the extra steps, which would just be confusing if presented so early on. Previously, everything was in a single document, which was difficult to read (for users) and to debug (for me), especially because I had to use contrived variable names to avoid clashes between different sections of the workflow that did similar things.

@Martin I?ve been using BiocFileCache for all of the online resources that are used in the workflow. However, this is only for my (and the reader?s) convenience. I use a local cache rather than the system default, to ensure that the downloaded files are removed after package build. This is intentional as it forces the package builder to try to re-download resources when compiling the vignette, thus ensuring the validity of the URLs. For a similar reason, I would prefer not to cache the result objects for use in different R sessions. I could imagine caching the result objects for use by a different vignette in the same build session, but this gets back to the problem of ensuring that the result object is generated by one vignette before it is needed by another vignette.

-A

On 18 Dec 2018, at 14:14, Martin Morgan <mtmorgan.bioc at gmail.com> wrote:

Also perhaps using BiocFileCache so that the result object is only generated once, then cached for future (different session) use.

?On 12/18/18, 8:35 AM, "Bioc-devel on behalf of Michael Lawrence" <bioc-devel-bounces at r-project.org on behalf of lawrence.michael at gene.com> wrote:

    I would recommend against dependencies across vignettes. Ideally someone
    can pick up a vignette and execute the code independently of any other
    documentation. Perhaps you could move the code generating those shared
    resources to the package. They could behave lazily, only generating the
    resource if necessary, otherwise reusing it. That would also make it easy
    for people to write their own documents using those resources.

    Michael

    On Tue, Dec 18, 2018 at 5:22 AM Aaron Lun <
    infinite.monkeys.with.keyboards at gmail.com> wrote:

In a number of my workflow packages (e.g., simpleSingleCell), I rely on a
specific compilation order for my vignettes. This is because some vignettes
set up resources or objects that are to be used by later vignettes.

 From what I understand, vignettes are compiled in alphanumeric ordering of
their file names. As such, I give my vignettes fairly structured names,
e.g., ?work-1-reads.Rmd?, ?work-2-umi.Rmd? and so on.

However, it becomes rather annoying when I want to add a new vignette in
the middle somewhere. This results in some unnatural numberings, e.g.,
?work-0?, ?3b?, which are ugly and unintuitive. This is relevant as
BiocStyle::Biocpkg() links between vignettes require you to use the
destination vignette?s file name; so difficult names complicate linking,
especially if the names continually change to reflect new orderings.

Is there an easier way to control vignette compilation order? WRE provides
no (obvious) guidance, so I would like to know what non-standard hacks are
known to work on the build machines. I can imagine something dirty whereby
one ?reference? vignette contains code to ?rmarkdown::render" all other
vignettes in the specified order? ugh.

-A

    	[[alternative HTML version deleted]]

@Michael In this case, the resource produced by vignette X is a
SingleCellExperiment object containing the results of various processing
steps (normalization, clustering, etc.) described in that vignette.

I can imagine a lazy evaluation model for this, but it wouldn?t be pretty.
If I had another vignette Y that depended on the SCE produced by vignette
X, I would need Y to execute all of the steps in X if X hadn?t already been
run before Y. This gets us into the territory of Makefile-like
dependencies, which seems even more complicated than simply specifying a
compilation order.

You might ask why X and Y are split into two separate vignettes. The use
of different vignettes is motivated by the complexity of the workflows:

- Vignette 1 demonstrates core processing steps for one read-based
single-cell RNAseq dataset.
- Vignette 2 demonstrates (slightly different) core steps for a UMI-based
dataset.
- ? so on for a bunch of other core steps for different types of data.
- Vignette 6 demonstrates extra optional steps for the two SCEs produced
by vignettes 1 & 3.
- ? and so on for a bunch of other optional steps.

The separation between core and optional steps into separate documents is
desirable. From a pedagogical perspective, I would very much like to get
the reader through all the core steps before even considering the extra
steps, which would just be confusing if presented so early on. Previously,
everything was in a single document, which was difficult to read (for
users) and to debug (for me), especially because I had to use contrived
variable names to avoid clashes between different sections of the workflow
that did similar things.

@Martin I?ve been using BiocFileCache for all of the online resources that
are used in the workflow. However, this is only for my (and the reader?s)
convenience. I use a local cache rather than the system default, to ensure
that the downloaded files are removed after package build. This is
intentional as it forces the package builder to try to re-download
resources when compiling the vignette, thus ensuring the validity of the
URLs. For a similar reason, I would prefer not to cache the result objects
for use in different R sessions. I could imagine caching the result objects
for use by a different vignette in the same build session, but this gets
back to the problem of ensuring that the result object is generated by one
vignette before it is needed by another vignette.

-A

On 18 Dec 2018, at 14:14, Martin Morgan <mtmorgan.bioc at gmail.com> wrote:

Also perhaps using BiocFileCache so that the result object is only

generated once, then cached for future (different session) use.

?On 12/18/18, 8:35 AM, "Bioc-devel on behalf of Michael Lawrence" <

bioc-devel-bounces at r-project.org on behalf of lawrence.michael at gene.com>
wrote:

   I would recommend against dependencies across vignettes. Ideally

someone

   can pick up a vignette and execute the code independently of any other
   documentation. Perhaps you could move the code generating those shared
   resources to the package. They could behave lazily, only generating

the

   resource if necessary, otherwise reusing it. That would also make it

easy

   for people to write their own documents using those resources.

   Michael

   On Tue, Dec 18, 2018 at 5:22 AM Aaron Lun <
   infinite.monkeys.with.keyboards at gmail.com> wrote:

In a number of my workflow packages (e.g., simpleSingleCell), I rely on

a

specific compilation order for my vignettes. This is because some

vignettes

set up resources or objects that are to be used by later vignettes.

From what I understand, vignettes are compiled in alphanumeric ordering

of

their file names. As such, I give my vignettes fairly structured names,
e.g., ?work-1-reads.Rmd?, ?work-2-umi.Rmd? and so on.

However, it becomes rather annoying when I want to add a new vignette in
the middle somewhere. This results in some unnatural numberings, e.g.,
?work-0?, ?3b?, which are ugly and unintuitive. This is relevant as
BiocStyle::Biocpkg() links between vignettes require you to use the
destination vignette?s file name; so difficult names complicate linking,
especially if the names continually change to reflect new orderings.

Is there an easier way to control vignette compilation order? WRE

provides

no (obvious) guidance, so I would like to know what non-standard hacks

are

known to work on the build machines. I can imagine something dirty

whereby

one ?reference? vignette contains code to ?rmarkdown::render" all other
vignettes in the specified order? ugh.

-A

      [[alternative HTML version deleted]]

On 18 Dec 2018, at 17:41, Michael Lawrence <lawrence.michael at gene.com> wrote:

Sounds like a use case for drake...

On Tue, Dec 18, 2018 at 6:58 AM Aaron Lun <infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com>> wrote:
@Michael In this case, the resource produced by vignette X is a SingleCellExperiment object containing the results of various processing steps (normalization, clustering, etc.) described in that vignette. 

I can imagine a lazy evaluation model for this, but it wouldn?t be pretty. If I had another vignette Y that depended on the SCE produced by vignette X, I would need Y to execute all of the steps in X if X hadn?t already been run before Y. This gets us into the territory of Makefile-like dependencies, which seems even more complicated than simply specifying a compilation order.

You might ask why X and Y are split into two separate vignettes. The use of different vignettes is motivated by the complexity of the workflows:

- Vignette 1 demonstrates core processing steps for one read-based single-cell RNAseq dataset. 
- Vignette 2 demonstrates (slightly different) core steps for a UMI-based dataset.
- ? so on for a bunch of other core steps for different types of data.
- Vignette 6 demonstrates extra optional steps for the two SCEs produced by vignettes 1 & 3.
- ? and so on for a bunch of other optional steps.

The separation between core and optional steps into separate documents is desirable. From a pedagogical perspective, I would very much like to get the reader through all the core steps before even considering the extra steps, which would just be confusing if presented so early on. Previously, everything was in a single document, which was difficult to read (for users) and to debug (for me), especially because I had to use contrived variable names to avoid clashes between different sections of the workflow that did similar things.

@Martin I?ve been using BiocFileCache for all of the online resources that are used in the workflow. However, this is only for my (and the reader?s) convenience. I use a local cache rather than the system default, to ensure that the downloaded files are removed after package build. This is intentional as it forces the package builder to try to re-download resources when compiling the vignette, thus ensuring the validity of the URLs. For a similar reason, I would prefer not to cache the result objects for use in different R sessions. I could imagine caching the result objects for use by a different vignette in the same build session, but this gets back to the problem of ensuring that the result object is generated by one vignette before it is needed by another vignette.

-A

On 18 Dec 2018, at 14:14, Martin Morgan <mtmorgan.bioc at gmail.com <mailto:mtmorgan.bioc at gmail.com>> wrote:

Also perhaps using BiocFileCache so that the result object is only generated once, then cached for future (different session) use.

?On 12/18/18, 8:35 AM, "Bioc-devel on behalf of Michael Lawrence" <bioc-devel-bounces at r-project.org <mailto:bioc-devel-bounces at r-project.org> on behalf of lawrence.michael at gene.com <mailto:lawrence.michael at gene.com>> wrote:

   I would recommend against dependencies across vignettes. Ideally someone
   can pick up a vignette and execute the code independently of any other
   documentation. Perhaps you could move the code generating those shared
   resources to the package. They could behave lazily, only generating the
   resource if necessary, otherwise reusing it. That would also make it easy
   for people to write their own documents using those resources.

   Michael

   On Tue, Dec 18, 2018 at 5:22 AM Aaron Lun <
   infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com>> wrote:

In a number of my workflow packages (e.g., simpleSingleCell), I rely on a
specific compilation order for my vignettes. This is because some vignettes
set up resources or objects that are to be used by later vignettes.

From what I understand, vignettes are compiled in alphanumeric ordering of
their file names. As such, I give my vignettes fairly structured names,
e.g., ?work-1-reads.Rmd?, ?work-2-umi.Rmd? and so on.

However, it becomes rather annoying when I want to add a new vignette in
the middle somewhere. This results in some unnatural numberings, e.g.,
?work-0?, ?3b?, which are ugly and unintuitive. This is relevant as
BiocStyle::Biocpkg() links between vignettes require you to use the
destination vignette?s file name; so difficult names complicate linking,
especially if the names continually change to reflect new orderings.

Is there an easier way to control vignette compilation order? WRE provides
no (obvious) guidance, so I would like to know what non-standard hacks are
known to work on the build machines. I can imagine something dirty whereby
one ?reference? vignette contains code to ?rmarkdown::render" all other
vignettes in the specified order? ugh.

-A

      [[alternative HTML version deleted]]

I gave it a shot:

https://github.com/LTLA/DrakeTest <https://github.com/LTLA/DrakeTest>

This uses a single ?controller? Rmd file to trigger Drake::make. Running this file will instruct Drake to compile all of the other vignettes following the desired dependency structure.

The current sticking point is that I need to move the Drake-controlled Rmd files out of ?vignettes/?, otherwise they?ll just be compiled as usual without consideration of their dependencies. This causes problems as R CMD BUILD only recognizes the controller Rmd file as the sole vignette, and doesn?t retain or index the HTML files produced from the other Rmd files as side-effects of running the controller.

Are there any better ways to subvert the vignette building procedure to get the desired effect of running drake::make() and recognition of the resulting HTMLs as vignettes?

-A

On 18 Dec 2018, at 17:41, Michael Lawrence <lawrence.michael at gene.com> wrote:

Sounds like a use case for drake...

On Tue, Dec 18, 2018 at 6:58 AM Aaron Lun <infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com>> wrote:
@Michael In this case, the resource produced by vignette X is a SingleCellExperiment object containing the results of various processing steps (normalization, clustering, etc.) described in that vignette.

I can imagine a lazy evaluation model for this, but it wouldn?t be pretty. If I had another vignette Y that depended on the SCE produced by vignette X, I would need Y to execute all of the steps in X if X hadn?t already been run before Y. This gets us into the territory of Makefile-like dependencies, which seems even more complicated than simply specifying a compilation order.

You might ask why X and Y are split into two separate vignettes. The use of different vignettes is motivated by the complexity of the workflows:

- Vignette 1 demonstrates core processing steps for one read-based single-cell RNAseq dataset.
- Vignette 2 demonstrates (slightly different) core steps for a UMI-based dataset.
- ? so on for a bunch of other core steps for different types of data.
- Vignette 6 demonstrates extra optional steps for the two SCEs produced by vignettes 1 & 3.
- ? and so on for a bunch of other optional steps.

The separation between core and optional steps into separate documents is desirable. From a pedagogical perspective, I would very much like to get the reader through all the core steps before even considering the extra steps, which would just be confusing if presented so early on. Previously, everything was in a single document, which was difficult to read (for users) and to debug (for me), especially because I had to use contrived variable names to avoid clashes between different sections of the workflow that did similar things.

@Martin I?ve been using BiocFileCache for all of the online resources that are used in the workflow. However, this is only for my (and the reader?s) convenience. I use a local cache rather than the system default, to ensure that the downloaded files are removed after package build. This is intentional as it forces the package builder to try to re-download resources when compiling the vignette, thus ensuring the validity of the URLs. For a similar reason, I would prefer not to cache the result objects for use in different R sessions. I could imagine caching the result objects for use by a different vignette in the same build session, but this gets back to the problem of ensuring that the result object is generated by one vignette before it is needed by another vignette.

-A

On 18 Dec 2018, at 14:14, Martin Morgan <mtmorgan.bioc at gmail.com <mailto:mtmorgan.bioc at gmail.com>> wrote:

Also perhaps using BiocFileCache so that the result object is only generated once, then cached for future (different session) use.

?On 12/18/18, 8:35 AM, "Bioc-devel on behalf of Michael Lawrence" <bioc-devel-bounces at r-project.org <mailto:bioc-devel-bounces at r-project.org> on behalf of lawrence.michael at gene.com <mailto:lawrence.michael at gene.com>> wrote:

   I would recommend against dependencies across vignettes. Ideally someone
   can pick up a vignette and execute the code independently of any other
   documentation. Perhaps you could move the code generating those shared
   resources to the package. They could behave lazily, only generating the
   resource if necessary, otherwise reusing it. That would also make it easy
   for people to write their own documents using those resources.

   Michael

   On Tue, Dec 18, 2018 at 5:22 AM Aaron Lun <
   infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com>> wrote:

In a number of my workflow packages (e.g., simpleSingleCell), I rely on a
specific compilation order for my vignettes. This is because some vignettes
set up resources or objects that are to be used by later vignettes.

From what I understand, vignettes are compiled in alphanumeric ordering of
their file names. As such, I give my vignettes fairly structured names,
e.g., ?work-1-reads.Rmd?, ?work-2-umi.Rmd? and so on.

However, it becomes rather annoying when I want to add a new vignette in
the middle somewhere. This results in some unnatural numberings, e.g.,
?work-0?, ?3b?, which are ugly and unintuitive. This is relevant as
BiocStyle::Biocpkg() links between vignettes require you to use the
destination vignette?s file name; so difficult names complicate linking,
especially if the names continually change to reflect new orderings.

Is there an easier way to control vignette compilation order? WRE provides
no (obvious) guidance, so I would like to know what non-standard hacks are
known to work on the build machines. I can imagine something dirty whereby
one ?reference? vignette contains code to ?rmarkdown::render" all other
vignettes in the specified order? ugh.

-A

      [[alternative HTML version deleted]]

        [[alternative HTML version deleted]]

Anything that eventually lands in inst/doc is a vignette, I think, so
there might be a hack around that.

readRDS(system.file(package = "utils", "Meta", "vignette.rds"))

On Fri, Dec 21, 2018 at 11:26 PM Aaron Lun
<infinite.monkeys.with.keyboards at gmail.com> wrote:

I gave it a shot:

https://github.com/LTLA/DrakeTest <https://github.com/LTLA/DrakeTest>

This uses a single ?controller? Rmd file to trigger Drake::make. Running this file will instruct Drake to compile all of the other vignettes following the desired dependency structure.

The current sticking point is that I need to move the Drake-controlled Rmd files out of ?vignettes/?, otherwise they?ll just be compiled as usual without consideration of their dependencies. This causes problems as R CMD BUILD only recognizes the controller Rmd file as the sole vignette, and doesn?t retain or index the HTML files produced from the other Rmd files as side-effects of running the controller.

Are there any better ways to subvert the vignette building procedure to get the desired effect of running drake::make() and recognition of the resulting HTMLs as vignettes?

-A

On 18 Dec 2018, at 17:41, Michael Lawrence <lawrence.michael at gene.com> wrote:

Sounds like a use case for drake...

On Tue, Dec 18, 2018 at 6:58 AM Aaron Lun <infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com>> wrote:
@Michael In this case, the resource produced by vignette X is a SingleCellExperiment object containing the results of various processing steps (normalization, clustering, etc.) described in that vignette.

I can imagine a lazy evaluation model for this, but it wouldn?t be pretty. If I had another vignette Y that depended on the SCE produced by vignette X, I would need Y to execute all of the steps in X if X hadn?t already been run before Y. This gets us into the territory of Makefile-like dependencies, which seems even more complicated than simply specifying a compilation order.

You might ask why X and Y are split into two separate vignettes. The use of different vignettes is motivated by the complexity of the workflows:

- Vignette 1 demonstrates core processing steps for one read-based single-cell RNAseq dataset.
- Vignette 2 demonstrates (slightly different) core steps for a UMI-based dataset.
- ? so on for a bunch of other core steps for different types of data.
- Vignette 6 demonstrates extra optional steps for the two SCEs produced by vignettes 1 & 3.
- ? and so on for a bunch of other optional steps.

The separation between core and optional steps into separate documents is desirable. From a pedagogical perspective, I would very much like to get the reader through all the core steps before even considering the extra steps, which would just be confusing if presented so early on. Previously, everything was in a single document, which was difficult to read (for users) and to debug (for me), especially because I had to use contrived variable names to avoid clashes between different sections of the workflow that did similar things.

@Martin I?ve been using BiocFileCache for all of the online resources that are used in the workflow. However, this is only for my (and the reader?s) convenience. I use a local cache rather than the system default, to ensure that the downloaded files are removed after package build. This is intentional as it forces the package builder to try to re-download resources when compiling the vignette, thus ensuring the validity of the URLs. For a similar reason, I would prefer not to cache the result objects for use in different R sessions. I could imagine caching the result objects for use by a different vignette in the same build session, but this gets back to the problem of ensuring that the result object is generated by one vignette before it is needed by another vignette.

-A

On 18 Dec 2018, at 14:14, Martin Morgan <mtmorgan.bioc at gmail.com <mailto:mtmorgan.bioc at gmail.com>> wrote:

Also perhaps using BiocFileCache so that the result object is only generated once, then cached for future (different session) use.

?On 12/18/18, 8:35 AM, "Bioc-devel on behalf of Michael Lawrence" <bioc-devel-bounces at r-project.org <mailto:bioc-devel-bounces at r-project.org> on behalf of lawrence.michael at gene.com <mailto:lawrence.michael at gene.com>> wrote:

   I would recommend against dependencies across vignettes. Ideally someone
   can pick up a vignette and execute the code independently of any other
   documentation. Perhaps you could move the code generating those shared
   resources to the package. They could behave lazily, only generating the
   resource if necessary, otherwise reusing it. That would also make it easy
   for people to write their own documents using those resources.

   Michael

   On Tue, Dec 18, 2018 at 5:22 AM Aaron Lun <
   infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com>> wrote:

In a number of my workflow packages (e.g., simpleSingleCell), I rely on a
specific compilation order for my vignettes. This is because some vignettes
set up resources or objects that are to be used by later vignettes.

From what I understand, vignettes are compiled in alphanumeric ordering of
their file names. As such, I give my vignettes fairly structured names,
e.g., ?work-1-reads.Rmd?, ?work-2-umi.Rmd? and so on.

However, it becomes rather annoying when I want to add a new vignette in
the middle somewhere. This results in some unnatural numberings, e.g.,
?work-0?, ?3b?, which are ugly and unintuitive. This is relevant as
BiocStyle::Biocpkg() links between vignettes require you to use the
destination vignette?s file name; so difficult names complicate linking,
especially if the names continually change to reflect new orderings.

Is there an easier way to control vignette compilation order? WRE provides
no (obvious) guidance, so I would like to know what non-standard hacks are
known to work on the build machines. I can imagine something dirty whereby
one ?reference? vignette contains code to ?rmarkdown::render" all other
vignettes in the specified order? ugh.

-A

      [[alternative HTML version deleted]]

        [[alternative HTML version deleted]]

On 22 Dec 2018, at 19:22, Henrik Bengtsson <henrik.bengtsson at gmail.com> wrote:

On Sat, Dec 22, 2018 at 10:56 AM Michael Lawrence
<lawrence.michael at gene.com <mailto:lawrence.michael at gene.com>> wrote:

Anything that eventually lands in inst/doc is a vignette, I think, so
there might be a hack around that.

Just so this is not misread - it's *not* possible to just hack your
vignette "product" files (PDF or HTML) into inst/doc and thinking
you're good.  R keeps track of package vignettes in a "vignette
index", e.g.

readRDS(system.file(package = "utils", "Meta", "vignette.rds"))

       File              Title        PDF        R Depends Keywords
1 Sweave.Rnw Sweave User Manual Sweave.pdf Sweave.R   tools

which is created during 'R CMD build' by parsing and compiling the
vignettes (https://github.com/wch/r-source/blob/tags/R-3-5-2/src/library/tools/R/build.R#L283-L393 <https://github.com/wch/r-source/blob/tags/R-3-5-2/src/library/tools/R/build.R#L283-L393>).
This vignette index is used to find package vignettes (e.g.
utils::vignette()) and build the HTML vignette index.

Also, one vignette source (e.g. Rnw, Rmd, ...) can only produce one
vignette product (PDF or HTML) in the vignette index.  You can output
other files (e.g. image files) in a relative folder that the vignette
references, which is why for instance non-self-contained HTML files
work.  Thus, one ad-hoc, not-so-nice hack that OP could do is to have
a single main vignette that produces and links to all child vignettes.
However, personally, I'd aim for using memoization/caching (to file)
such that each vignette can be compiled independently of the others
(and in any order), while still reusing intermediate
results/calculations produced by earlier vignettes.

/Henrik

On Fri, Dec 21, 2018 at 11:26 PM Aaron Lun
<infinite.monkeys.with.keyboards at gmail.com> wrote:

I gave it a shot:

https://github.com/LTLA/DrakeTest <https://github.com/LTLA/DrakeTest>

This uses a single ?controller? Rmd file to trigger Drake::make. Running this file will instruct Drake to compile all of the other vignettes following the desired dependency structure.

The current sticking point is that I need to move the Drake-controlled Rmd files out of ?vignettes/?, otherwise they?ll just be compiled as usual without consideration of their dependencies. This causes problems as R CMD BUILD only recognizes the controller Rmd file as the sole vignette, and doesn?t retain or index the HTML files produced from the other Rmd files as side-effects of running the controller.

Are there any better ways to subvert the vignette building procedure to get the desired effect of running drake::make() and recognition of the resulting HTMLs as vignettes?

-A

On 18 Dec 2018, at 17:41, Michael Lawrence <lawrence.michael at gene.com> wrote:

Sounds like a use case for drake...

On Tue, Dec 18, 2018 at 6:58 AM Aaron Lun <infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com>> wrote:
@Michael In this case, the resource produced by vignette X is a SingleCellExperiment object containing the results of various processing steps (normalization, clustering, etc.) described in that vignette.

I can imagine a lazy evaluation model for this, but it wouldn?t be pretty. If I had another vignette Y that depended on the SCE produced by vignette X, I would need Y to execute all of the steps in X if X hadn?t already been run before Y. This gets us into the territory of Makefile-like dependencies, which seems even more complicated than simply specifying a compilation order.

You might ask why X and Y are split into two separate vignettes. The use of different vignettes is motivated by the complexity of the workflows:

- Vignette 1 demonstrates core processing steps for one read-based single-cell RNAseq dataset.
- Vignette 2 demonstrates (slightly different) core steps for a UMI-based dataset.
- ? so on for a bunch of other core steps for different types of data.
- Vignette 6 demonstrates extra optional steps for the two SCEs produced by vignettes 1 & 3.
- ? and so on for a bunch of other optional steps.

The separation between core and optional steps into separate documents is desirable. From a pedagogical perspective, I would very much like to get the reader through all the core steps before even considering the extra steps, which would just be confusing if presented so early on. Previously, everything was in a single document, which was difficult to read (for users) and to debug (for me), especially because I had to use contrived variable names to avoid clashes between different sections of the workflow that did similar things.

@Martin I?ve been using BiocFileCache for all of the online resources that are used in the workflow. However, this is only for my (and the reader?s) convenience. I use a local cache rather than the system default, to ensure that the downloaded files are removed after package build. This is intentional as it forces the package builder to try to re-download resources when compiling the vignette, thus ensuring the validity of the URLs. For a similar reason, I would prefer not to cache the result objects for use in different R sessions. I could imagine caching the result objects for use by a different vignette in the same build session, but this gets back to the problem of ensuring that the result object is generated by one vignette before it is needed by another vignette.

-A

On 18 Dec 2018, at 14:14, Martin Morgan <mtmorgan.bioc at gmail.com <mailto:mtmorgan.bioc at gmail.com>> wrote:

Also perhaps using BiocFileCache so that the result object is only generated once, then cached for future (different session) use.

?On 12/18/18, 8:35 AM, "Bioc-devel on behalf of Michael Lawrence" <bioc-devel-bounces at r-project.org <mailto:bioc-devel-bounces at r-project.org> on behalf of lawrence.michael at gene.com <mailto:lawrence.michael at gene.com>> wrote:

  I would recommend against dependencies across vignettes. Ideally someone
  can pick up a vignette and execute the code independently of any other
  documentation. Perhaps you could move the code generating those shared
  resources to the package. They could behave lazily, only generating the
  resource if necessary, otherwise reusing it. That would also make it easy
  for people to write their own documents using those resources.

  Michael

  On Tue, Dec 18, 2018 at 5:22 AM Aaron Lun <
  infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com>> wrote:

In a number of my workflow packages (e.g., simpleSingleCell), I rely on a
specific compilation order for my vignettes. This is because some vignettes
set up resources or objects that are to be used by later vignettes.

From what I understand, vignettes are compiled in alphanumeric ordering of
their file names. As such, I give my vignettes fairly structured names,
e.g., ?work-1-reads.Rmd?, ?work-2-umi.Rmd? and so on.

However, it becomes rather annoying when I want to add a new vignette in
the middle somewhere. This results in some unnatural numberings, e.g.,
?work-0?, ?3b?, which are ugly and unintuitive. This is relevant as
BiocStyle::Biocpkg() links between vignettes require you to use the
destination vignette?s file name; so difficult names complicate linking,
especially if the names continually change to reflect new orderings.

Is there an easier way to control vignette compilation order? WRE provides
no (obvious) guidance, so I would like to know what non-standard hacks are
known to work on the build machines. I can imagine something dirty whereby
one ?reference? vignette contains code to ?rmarkdown::render" all other
vignettes in the specified order? ugh.

-A

     [[alternative HTML version deleted]]

       [[alternative HTML version deleted]]

On 23 Dec 2018, at 01:24, Aaron Lun <infinite.monkeys.with.keyboards at gmail.com> wrote:

Yes, I had noticed the vignettes.rds as well, and I figured that would be a problem.

I just tried setting set cache=TRUE in my vignettes, implemented such that BUILDing each downstream vignette will also run all upstream vignettes on which it depends (that haven?t already been compiled). If an upstream vignette is run in this manner, it caches the results of each code chunk to avoid repeated work when it gets compiled ?for real? by R CMD BUILD.

This seems to work on initial inspection (the caches are produced for the upstream vignettes upon running one downstream vignette). I?ll have to check whether this plays nice with R CMD BUILD. I will probably have to write a function to isolate the scope of the execution of each upstream vignette, to avoid polluting the namespace and cache of each downstream vignette.

-A

On 22 Dec 2018, at 19:22, Henrik Bengtsson <henrik.bengtsson at gmail.com <mailto:henrik.bengtsson at gmail.com>> wrote:

On Sat, Dec 22, 2018 at 10:56 AM Michael Lawrence
<lawrence.michael at gene.com <mailto:lawrence.michael at gene.com>> wrote:

Anything that eventually lands in inst/doc is a vignette, I think, so
there might be a hack around that.

Just so this is not misread - it's *not* possible to just hack your
vignette "product" files (PDF or HTML) into inst/doc and thinking
you're good.  R keeps track of package vignettes in a "vignette
index", e.g.

readRDS(system.file(package = "utils", "Meta", "vignette.rds"))

       File              Title        PDF        R Depends Keywords
1 Sweave.Rnw Sweave User Manual Sweave.pdf Sweave.R   tools

which is created during 'R CMD build' by parsing and compiling the
vignettes (https://github.com/wch/r-source/blob/tags/R-3-5-2/src/library/tools/R/build.R#L283-L393 <https://github.com/wch/r-source/blob/tags/R-3-5-2/src/library/tools/R/build.R#L283-L393>).
This vignette index is used to find package vignettes (e.g.
utils::vignette()) and build the HTML vignette index.

Also, one vignette source (e.g. Rnw, Rmd, ...) can only produce one
vignette product (PDF or HTML) in the vignette index.  You can output
other files (e.g. image files) in a relative folder that the vignette
references, which is why for instance non-self-contained HTML files
work.  Thus, one ad-hoc, not-so-nice hack that OP could do is to have
a single main vignette that produces and links to all child vignettes.
However, personally, I'd aim for using memoization/caching (to file)
such that each vignette can be compiled independently of the others
(and in any order), while still reusing intermediate
results/calculations produced by earlier vignettes.

/Henrik

On Fri, Dec 21, 2018 at 11:26 PM Aaron Lun
<infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com>> wrote:

I gave it a shot:

https://github.com/LTLA/DrakeTest <https://github.com/LTLA/DrakeTest> <https://github.com/LTLA/DrakeTest <https://github.com/LTLA/DrakeTest>>

This uses a single ?controller? Rmd file to trigger Drake::make. Running this file will instruct Drake to compile all of the other vignettes following the desired dependency structure.

The current sticking point is that I need to move the Drake-controlled Rmd files out of ?vignettes/?, otherwise they?ll just be compiled as usual without consideration of their dependencies. This causes problems as R CMD BUILD only recognizes the controller Rmd file as the sole vignette, and doesn?t retain or index the HTML files produced from the other Rmd files as side-effects of running the controller.

Are there any better ways to subvert the vignette building procedure to get the desired effect of running drake::make() and recognition of the resulting HTMLs as vignettes?

-A

On 18 Dec 2018, at 17:41, Michael Lawrence <lawrence.michael at gene.com <mailto:lawrence.michael at gene.com>> wrote:

Sounds like a use case for drake...

On Tue, Dec 18, 2018 at 6:58 AM Aaron Lun <infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com> <mailto:infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com>>> wrote:
@Michael In this case, the resource produced by vignette X is a SingleCellExperiment object containing the results of various processing steps (normalization, clustering, etc.) described in that vignette.

I can imagine a lazy evaluation model for this, but it wouldn?t be pretty. If I had another vignette Y that depended on the SCE produced by vignette X, I would need Y to execute all of the steps in X if X hadn?t already been run before Y. This gets us into the territory of Makefile-like dependencies, which seems even more complicated than simply specifying a compilation order.

You might ask why X and Y are split into two separate vignettes. The use of different vignettes is motivated by the complexity of the workflows:

- Vignette 1 demonstrates core processing steps for one read-based single-cell RNAseq dataset.
- Vignette 2 demonstrates (slightly different) core steps for a UMI-based dataset.
- ? so on for a bunch of other core steps for different types of data.
- Vignette 6 demonstrates extra optional steps for the two SCEs produced by vignettes 1 & 3.
- ? and so on for a bunch of other optional steps.

The separation between core and optional steps into separate documents is desirable. From a pedagogical perspective, I would very much like to get the reader through all the core steps before even considering the extra steps, which would just be confusing if presented so early on. Previously, everything was in a single document, which was difficult to read (for users) and to debug (for me), especially because I had to use contrived variable names to avoid clashes between different sections of the workflow that did similar things.

@Martin I?ve been using BiocFileCache for all of the online resources that are used in the workflow. However, this is only for my (and the reader?s) convenience. I use a local cache rather than the system default, to ensure that the downloaded files are removed after package build. This is intentional as it forces the package builder to try to re-download resources when compiling the vignette, thus ensuring the validity of the URLs. For a similar reason, I would prefer not to cache the result objects for use in different R sessions. I could imagine caching the result objects for use by a different vignette in the same build session, but this gets back to the problem of ensuring that the result object is generated by one vignette before it is needed by another vignette.

-A

On 18 Dec 2018, at 14:14, Martin Morgan <mtmorgan.bioc at gmail.com <mailto:mtmorgan.bioc at gmail.com> <mailto:mtmorgan.bioc at gmail.com <mailto:mtmorgan.bioc at gmail.com>>> wrote:

Also perhaps using BiocFileCache so that the result object is only generated once, then cached for future (different session) use.

?On 12/18/18, 8:35 AM, "Bioc-devel on behalf of Michael Lawrence" <bioc-devel-bounces at r-project.org <mailto:bioc-devel-bounces at r-project.org> <mailto:bioc-devel-bounces at r-project.org <mailto:bioc-devel-bounces at r-project.org>> on behalf of lawrence.michael at gene.com <mailto:lawrence.michael at gene.com> <mailto:lawrence.michael at gene.com <mailto:lawrence.michael at gene.com>>> wrote:

  I would recommend against dependencies across vignettes. Ideally someone
  can pick up a vignette and execute the code independently of any other
  documentation. Perhaps you could move the code generating those shared
  resources to the package. They could behave lazily, only generating the
  resource if necessary, otherwise reusing it. That would also make it easy
  for people to write their own documents using those resources.

  Michael

  On Tue, Dec 18, 2018 at 5:22 AM Aaron Lun <
  infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com> <mailto:infinite.monkeys.with.keyboards at gmail.com <mailto:infinite.monkeys.with.keyboards at gmail.com>>> wrote:

In a number of my workflow packages (e.g., simpleSingleCell), I rely on a
specific compilation order for my vignettes. This is because some vignettes
set up resources or objects that are to be used by later vignettes.

From what I understand, vignettes are compiled in alphanumeric ordering of
their file names. As such, I give my vignettes fairly structured names,
e.g., ?work-1-reads.Rmd?, ?work-2-umi.Rmd? and so on.

However, it becomes rather annoying when I want to add a new vignette in
the middle somewhere. This results in some unnatural numberings, e.g.,
?work-0?, ?3b?, which are ugly and unintuitive. This is relevant as
BiocStyle::Biocpkg() links between vignettes require you to use the
destination vignette?s file name; so difficult names complicate linking,
especially if the names continually change to reflect new orderings.

Is there an easier way to control vignette compilation order? WRE provides
no (obvious) guidance, so I would like to know what non-standard hacks are
known to work on the build machines. I can imagine something dirty whereby
one ?reference? vignette contains code to ?rmarkdown::render" all other
vignettes in the specified order? ugh.

-A

     [[alternative HTML version deleted]]

       [[alternative HTML version deleted]]