[Bioc-devel] shiny, Bioconductor, and reproducible research

My perspective,

Clearly we are very interested in supporting reproducible research in
Bioconductor.  Indeed, it is a core mission of the project.  However, I
feel that Dan is placing a very high burden of proof on shiny apps in his
email, since it essentially says "because it is possible to potentially
conduct non-reproducible research in shiny, we should not allow it".  The
logical conclusion of such a statement, is that we should never allow GUIs
nor interactive code.

Without having a lot of experience with shiny, I am mostly excited because
of the potential for (semi) interactive graphics, especially for exploring
your data (here I am like very biased since that is what we are using it
for right now).  For such an application, there should not be any
modification of the user's R objects.
It is true that an interactive
exploration of your data might be considered non-reproducible, but that is
also the case if I inspects 100's of pdfs.

In my opinion we should try to get people to adhere to best practices, and
(once we have something) we could provide example code for these practices.
 Perhaps a starting point would be a function to tag all objects in the
user environment and keep track of whether they are modified during the
shiny session.  In the meantime (unless Michael and Tengfei comes up with
an awesome solution in the next week :), I do believe we should consider
how to host such apps, including how do we do code checking.  Ultimately,
this would be better for the research community than to force people to
host it on their own webpages.  If we cannot find a suitable way to do code
checking or to package up the app, perhaps we should not do it.  But that
would be a better reason than the potential for writing non-reproducible
code.

Best,
Kasper


On Sat, Jul 27, 2013 at 9:17 PM, Levi Waldron <lwaldron at hsph.harvard.edu>wrote:

I agree with Dan on the need to be able to "save" analysis done in a Shiny
Bioconductor app, allowing someone later to reload the same state, and to
inspect a session and see what was done.  Perhaps "save" and "load" buttons
should be present in the user interface of any Bioconductor Shiny app that
is not "read-only".

If interactive user input can be saved (as an .RData file?), could that
file be used as input for unit tests either the usual console way or
something like an http POST?

-Levi





On Fri, Jul 26, 2013 at 5:42 PM, Michael Lawrence <
lawrence.michael at gene.com

wrote:

An answer to both of these issues is the use of R code. It should be
possible to drive shiny apps through the server, without the user's code
knowing the difference. And you've already thought of recording user
actions as R code. This would be an application of the command pattern,

as

with undo/redo stacks, except each object boils down to a
function/expression (Tengfei and I have a CRAN package called commandr
related to this). Someone should try to make a prototype to explore these
issues then we can start thinking about common infrastructure.


On Fri, Jul 26, 2013 at 12:23 PM, Dan Tenenbaum <dtenenba at fhcrc.org>
wrote:

Hi everyone,

There was a lot of talk about shiny at BioC2013 and it seems like a
lot of people are using it for lots of different cool stuff.

Now we are at the point where some contributors want to add shiny apps
to Bioconductor as packages.

We had some discussion internally and decided that there are a few
issues we wanted to discuss with BioC developers as well as some of
the shiny developers. Please feel free to share your thoughts.

1) Testing shiny apps

Typically, bioconductor packages have man page examples, vignettes,
and (sometimes) unit tests, so when we build the packages every day on
our build system, the code in all of these is evaluated and we have
some idea of whether the package is working as it's supposed to.

I'm not clear on how to do similar testing of a shiny application.
Since launching a shiny app takes away the R console (until the app is
closed), shiny apps should probably not be launched in example or test
code (unless interactive() is TRUE).

Do the shiny folks (or anyone else) have thoughts on testing shiny

apps?

2) Reproducible research

Reproducible research is really important in our community, but shiny
apps are sort of a black box as far as reproducibility is concerned.

If shiny apps are "read-only" (that is, if they are just used to view
an object) then this is not really a problem. But shiny has the
ability now to change objects back in your R session, so we need to be
able to track what is done inside the shiny app.

shiny apps can return some sort of object that tells you what was done
in the shiny session (insofar as it modifies any objects in the
users's session) and this object can provide a way to reproduce those
steps without the shiny app. So maybe the object would consist of
lines of code. This means that if there is a shiny app, the package
author must also provide ways to do the same transformations on
objects without shiny.

There are probably other approaches as well. But the end goal would be
a scriptable, reproducible shiny session.

Another thought was that a shiny app could emit some sort of state
object, and then be restarted with that object.

Not sure if the place for such infrastructure is inside shiny itself,
or perhaps in a BioC infrastructure package.

What are your thoughts?
Dan

        [[alternative HTML version deleted]]

--
Levi Waldron
Post-doctoral fellow
Department of Biostatistics, Harvard School of Public Health
Department of Biostatistics and Computational Biology, Dana-Farber Cancer
Institute
Building 1, room 412C
655 Huntington Avenue
Boston, Massachusetts 02115
mobile: (617) 851-6849
fax: (617) 432-5619
http://www.hsph.harvard.edu/research/levi-waldron/

        [[alternative HTML version deleted]]

        [[alternative HTML version deleted]]

My perspective,

Clearly we are very interested in supporting reproducible research in
Bioconductor.  Indeed, it is a core mission of the project.  However, I
feel that Dan is placing a very high burden of proof on shiny apps in his
email, since it essentially says "because it is possible to potentially
conduct non-reproducible research in shiny, we should not allow it".  The
logical conclusion of such a statement, is that we should never allow GUIs
nor interactive code.

Without having a lot of experience with shiny, I am mostly excited because
of the potential for (semi) interactive graphics, especially for exploring
your data (here I am like very biased since that is what we are using it
for right now).  For such an application, there should not be any
modification of the user's R objects.  It is true that an interactive
exploration of your data might be considered non-reproducible, but that is
also the case if I inspects 100's of pdfs.

In my opinion we should try to get people to adhere to best practices, and
(once we have something) we could provide example code for these practices.
Perhaps a starting point would be a function to tag all objects in the
user environment and keep track of whether they are modified during the
shiny session.  In the meantime (unless Michael and Tengfei comes up with
an awesome solution in the next week :), I do believe we should consider
how to host such apps, including how do we do code checking.  Ultimately,
this would be better for the research community than to force people to
host it on their own webpages.  If we cannot find a suitable way to do code
checking or to package up the app, perhaps we should not do it.  But that
would be a better reason than the potential for writing non-reproducible
code.

Best,
Kasper


On Sat, Jul 27, 2013 at 9:17 PM, Levi Waldron <lwaldron at hsph.harvard.edu>wrote:

I agree with Dan on the need to be able to "save" analysis done in a Shiny
Bioconductor app, allowing someone later to reload the same state, and to
inspect a session and see what was done.  Perhaps "save" and "load" buttons
should be present in the user interface of any Bioconductor Shiny app that
is not "read-only".

If interactive user input can be saved (as an .RData file?), could that
file be used as input for unit tests either the usual console way or
something like an http POST?

-Levi





On Fri, Jul 26, 2013 at 5:42 PM, Michael Lawrence <
lawrence.michael at gene.com

wrote:

An answer to both of these issues is the use of R code. It should be
possible to drive shiny apps through the server, without the user's code
knowing the difference. And you've already thought of recording user
actions as R code. This would be an application of the command pattern,

as

with undo/redo stacks, except each object boils down to a
function/expression (Tengfei and I have a CRAN package called commandr
related to this). Someone should try to make a prototype to explore these
issues then we can start thinking about common infrastructure.


On Fri, Jul 26, 2013 at 12:23 PM, Dan Tenenbaum <dtenenba at fhcrc.org>
wrote:

Hi everyone,

There was a lot of talk about shiny at BioC2013 and it seems like a
lot of people are using it for lots of different cool stuff.

Now we are at the point where some contributors want to add shiny apps
to Bioconductor as packages.

We had some discussion internally and decided that there are a few
issues we wanted to discuss with BioC developers as well as some of
the shiny developers. Please feel free to share your thoughts.

1) Testing shiny apps

Typically, bioconductor packages have man page examples, vignettes,
and (sometimes) unit tests, so when we build the packages every day on
our build system, the code in all of these is evaluated and we have
some idea of whether the package is working as it's supposed to.

I'm not clear on how to do similar testing of a shiny application.
Since launching a shiny app takes away the R console (until the app is
closed), shiny apps should probably not be launched in example or test
code (unless interactive() is TRUE).

Do the shiny folks (or anyone else) have thoughts on testing shiny

apps?

2) Reproducible research

Reproducible research is really important in our community, but shiny
apps are sort of a black box as far as reproducibility is concerned.

If shiny apps are "read-only" (that is, if they are just used to view
an object) then this is not really a problem. But shiny has the
ability now to change objects back in your R session, so we need to be
able to track what is done inside the shiny app.

shiny apps can return some sort of object that tells you what was done
in the shiny session (insofar as it modifies any objects in the
users's session) and this object can provide a way to reproduce those
steps without the shiny app. So maybe the object would consist of
lines of code. This means that if there is a shiny app, the package
author must also provide ways to do the same transformations on
objects without shiny.

There are probably other approaches as well. But the end goal would be
a scriptable, reproducible shiny session.

Another thought was that a shiny app could emit some sort of state
object, and then be restarted with that object.

Not sure if the place for such infrastructure is inside shiny itself,
or perhaps in a BioC infrastructure package.

What are your thoughts?
Dan

       [[alternative HTML version deleted]]

--
Levi Waldron
Post-doctoral fellow
Department of Biostatistics, Harvard School of Public Health
Department of Biostatistics and Computational Biology, Dana-Farber Cancer
Institute
Building 1, room 412C
655 Huntington Avenue
Boston, Massachusetts 02115
mobile: (617) 851-6849
fax: (617) 432-5619
http://www.hsph.harvard.edu/research/levi-waldron/

       [[alternative HTML version deleted]]

   [[alternative HTML version deleted]]

So here is something I came across in my code:



--t