[Bioc-devel] Use and Usability metrics / shields

On May 9, 2015, at 19:15 GMT+2, Martin Morgan <mtmorgan at fredhutch.org> wrote:

Bioc developers!

It's important that our users be able to identify packages that are suitable for their research question. Obviously a first step is to identify packages in the appropriate research domain, for instance through biocViews.

 http://bioconductor.org/packages/release/

We'd like to help users further prioritize their efforts by summarizing use and usability. Metrics include:

- Cross-platform availability -- biocLite()-able from all or only some platforms
- Support forum activity -- questions and comments / responses, 6 month window
- Download percentile -- top 5, 20, 50%, or 'available'
- Current build status -- errors or warnings on some or all platforms
- Developer activity -- commits in the last 6 months
- Historical presence -- years in Bioconductor

Obviously the metrics are imperfect, so constructive feedback welcome -- we think the above capture in a more-or-less objective and computable way the major axes influencing use and usability.

We initially intend to prominently display 'shields' (small graphical icons) on package landing pages.

Thanks in advance for your comments,

Martin Morgan
Bioconductor
-- 
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

On May 9, 2015, at 19:15 GMT+2, Martin Morgan <mtmorgan at fredhutch.org> wrote:

Bioc developers!

It's important that our users be able to identify packages that are suitable for their research question. Obviously a first step is to identify packages in the appropriate research domain, for instance through biocViews.

 http://bioconductor.org/packages/release/

We'd like to help users further prioritize their efforts by summarizing use and usability. Metrics include:

- Cross-platform availability -- biocLite()-able from all or only some platforms
- Support forum activity -- questions and comments / responses, 6 month window
- Download percentile -- top 5, 20, 50%, or 'available'
- Current build status -- errors or warnings on some or all platforms
- Developer activity -- commits in the last 6 months
- Historical presence -- years in Bioconductor

Obviously the metrics are imperfect, so constructive feedback welcome -- we think the above capture in a more-or-less objective and computable way the major axes influencing use and usability.

We initially intend to prominently display 'shields' (small graphical icons) on package landing pages.

Thanks in advance for your comments,

Martin Morgan
Bioconductor
--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

From: "Martin Morgan" <mtmorgan at fredhutch.org>
To: bioc-devel at r-project.org
Sent: Saturday, May 9, 2015 10:15:13 AM
Subject: [Bioc-devel] Use and Usability metrics / shields

Bioc developers!

It's important that our users be able to identify packages that are
suitable for
their research question. Obviously a first step is to identify
packages in the
appropriate research domain, for instance through biocViews.

   http://bioconductor.org/packages/release/

We'd like to help users further prioritize their efforts by
summarizing use and
usability. Metrics include:

- Cross-platform availability -- biocLite()-able from all or only
some platforms
- Support forum activity -- questions and comments / responses, 6
month window
- Download percentile -- top 5, 20, 50%, or 'available'
- Current build status -- errors or warnings on some or all platforms
- Developer activity -- commits in the last 6 months
- Historical presence -- years in Bioconductor

Obviously the metrics are imperfect, so constructive feedback welcome
-- we
think the above capture in a more-or-less objective and computable
way the major
axes influencing use and usability.

We initially intend to prominently display 'shields' (small graphical
icons) on
package landing pages.

Dear Martin,

All of these suggestions sound good.

Wolfgang's suggestion regarding possible associated papers might be also great.

Another useful information would be to point to other publications where a given package was used, and cited.
I don't know if it's technically possible, but it would be greatly informative to know how frequently a package is used, and how it performs, in real contexts.

Frederic Commo
Bioinformatics, U981
Gustave Roussy

From: "Henrik Bengtsson" <henrik.bengtsson at ucsf.edu>
To: "COMMO Frederic" <Frederic.COMMO at gustaveroussy.fr>
Cc: bioc-devel at r-project.org
Sent: Wednesday, May 13, 2015 12:28:54 PM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields

Sweet; you went live with the badges/shields, e.g.

  http://bioconductor.org/packages/release/bioc/html/affxparser.html

A positive side effect is that now there's a link from the package
page to to the package's check results, which I always wanted :)

Thanks for adding this

/Henrik


On Sun, May 10, 2015 at 11:39 AM, COMMO Frederic
<Frederic.COMMO at gustaveroussy.fr> wrote:

Dear Martin,

All of these suggestions sound good.

Wolfgang's suggestion regarding possible associated papers might be
also great.

Another useful information would be to point to other publications
where a given package was used, and cited.
I don't know if it's technically possible, but it would be greatly
informative to know how frequently a package is used, and how it
performs, in real contexts.

Frederic Commo
Bioinformatics, U981
Gustave Roussy

On May 13, 2015, at 21:36 GMT+2, Dan Tenenbaum <dtenenba at fredhutch.org> wrote:



----- Original Message -----

From: "Henrik Bengtsson" <henrik.bengtsson at ucsf.edu>
To: "COMMO Frederic" <Frederic.COMMO at gustaveroussy.fr>
Cc: bioc-devel at r-project.org
Sent: Wednesday, May 13, 2015 12:28:54 PM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields

Sweet; you went live with the badges/shields, e.g.

 http://bioconductor.org/packages/release/bioc/html/affxparser.html

A positive side effect is that now there's a link from the package
page to to the package's check results, which I always wanted :)

That was there before (and still is, see the bottom of the Details section). But yes, it was not very visible.

Dan

Thanks for adding this

/Henrik


On Sun, May 10, 2015 at 11:39 AM, COMMO Frederic
<Frederic.COMMO at gustaveroussy.fr> wrote:

Dear Martin,

All of these suggestions sound good.

Wolfgang's suggestion regarding possible associated papers might be
also great.

Another useful information would be to point to other publications
where a given package was used, and cited.
I don't know if it's technically possible, but it would be greatly
informative to know how frequently a package is used, and how it
performs, in real contexts.

Frederic Commo
Bioinformatics, U981
Gustave Roussy

From: "Wolfgang Huber" <whuber at embl.de>
To: "Dan Tenenbaum" <dtenenba at fredhutch.org>
Cc: bioc-devel at r-project.org
Sent: Thursday, May 14, 2015 4:03:54 AM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields



Can it be that the ?in Bioc? shield is incorrect?
For instance, it says ?9.98 years? for vsn but the first commit was
in Oct 2002
Curiously ?9.98 years? is stated for many old packages - surely we
can use R for more precise date arithmetic?

	Cheers
	Wolfgang

On May 13, 2015, at 21:36 GMT+2, Dan Tenenbaum
<dtenenba at fredhutch.org> wrote:



----- Original Message -----

From: "Henrik Bengtsson" <henrik.bengtsson at ucsf.edu>
To: "COMMO Frederic" <Frederic.COMMO at gustaveroussy.fr>
Cc: bioc-devel at r-project.org
Sent: Wednesday, May 13, 2015 12:28:54 PM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields

Sweet; you went live with the badges/shields, e.g.

 http://bioconductor.org/packages/release/bioc/html/affxparser.html

A positive side effect is that now there's a link from the package
page to to the package's check results, which I always wanted :)

That was there before (and still is, see the bottom of the Details
section). But yes, it was not very visible.

Dan

Thanks for adding this

/Henrik


On Sun, May 10, 2015 at 11:39 AM, COMMO Frederic
<Frederic.COMMO at gustaveroussy.fr> wrote:

Dear Martin,

All of these suggestions sound good.

Wolfgang's suggestion regarding possible associated papers might
be
also great.

Another useful information would be to point to other
publications
where a given package was used, and cited.
I don't know if it's technically possible, but it would be
greatly
informative to know how frequently a package is used, and how it
performs, in real contexts.

Frederic Commo
Bioinformatics, U981
Gustave Roussy

From: "Dan Tenenbaum" <dtenenba at fredhutch.org>
To: "Wolfgang Huber" <whuber at embl.de>
Cc: bioc-devel at r-project.org
Sent: Thursday, May 14, 2015 7:33:04 AM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields



----- Original Message -----

From: "Wolfgang Huber" <whuber at embl.de>
To: "Dan Tenenbaum" <dtenenba at fredhutch.org>
Cc: bioc-devel at r-project.org
Sent: Thursday, May 14, 2015 4:03:54 AM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields



Can it be that the ?in Bioc? shield is incorrect?
For instance, it says ?9.98 years? for vsn but the first commit was
in Oct 2002
Curiously ?9.98 years? is stated for many old packages - surely we
can use R for more precise date arithmetic?

It is not counting from the first commit (as it says in the tooltip
text) but from the day of the first release in which the package
appeared.

So it does not count the time that the package is in devel only.

Dan

	Cheers
	Wolfgang

On May 13, 2015, at 21:36 GMT+2, Dan Tenenbaum
<dtenenba at fredhutch.org> wrote:



----- Original Message -----

From: "Henrik Bengtsson" <henrik.bengtsson at ucsf.edu>
To: "COMMO Frederic" <Frederic.COMMO at gustaveroussy.fr>
Cc: bioc-devel at r-project.org
Sent: Wednesday, May 13, 2015 12:28:54 PM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields

Sweet; you went live with the badges/shields, e.g.

 http://bioconductor.org/packages/release/bioc/html/affxparser.html

A positive side effect is that now there's a link from the
package
page to to the package's check results, which I always wanted :)

That was there before (and still is, see the bottom of the
Details
section). But yes, it was not very visible.

Dan

Thanks for adding this

/Henrik


On Sun, May 10, 2015 at 11:39 AM, COMMO Frederic
<Frederic.COMMO at gustaveroussy.fr> wrote:

Dear Martin,

All of these suggestions sound good.

Wolfgang's suggestion regarding possible associated papers
might
be
also great.

Another useful information would be to point to other
publications
where a given package was used, and cited.
I don't know if it's technically possible, but it would be
greatly
informative to know how frequently a package is used, and how
it
performs, in real contexts.

Frederic Commo
Bioinformatics, U981
Gustave Roussy

Dear Martin,

All of these suggestions sound good.

Wolfgang's suggestion regarding possible associated papers might be also great.

Another useful information would be to point to other publications where a given package was used, and cited.
I don't know if it's technically possible, but it would be greatly informative to know how frequently a package is used, and how it performs, in real contexts.

Frederic Commo
Bioinformatics, U981
Gustave Roussy

PS  Martin you?ll like Fig. 2 of their paper.

Wolfgang

On May 9, 2015, at 19:15 GMT+2, Martin Morgan <mtmorgan at fredhutch.org> wrote:

Bioc developers!

It's important that our users be able to identify packages that are suitable for their research question. Obviously a first step is to identify packages in the appropriate research domain, for instance through biocViews.

  http://bioconductor.org/packages/release/

We'd like to help users further prioritize their efforts by summarizing use and usability. Metrics include:

- Cross-platform availability -- biocLite()-able from all or only some platforms
- Support forum activity -- questions and comments / responses, 6 month window
- Download percentile -- top 5, 20, 50%, or 'available'
- Current build status -- errors or warnings on some or all platforms
- Developer activity -- commits in the last 6 months
- Historical presence -- years in Bioconductor

Obviously the metrics are imperfect, so constructive feedback welcome -- we think the above capture in a more-or-less objective and computable way the major axes influencing use and usability.

We initially intend to prominently display 'shields' (small graphical icons) on package landing pages.

Thanks in advance for your comments,

Martin Morgan
Bioconductor
--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

On 05/10/2015 11:39 AM, COMMO Frederic wrote:

Dear Martin,

All of these suggestions sound good.

Wolfgang's suggestion regarding possible associated papers might be also
great.

Another useful information would be to point to other publications where
a given package was used, and cited.
I don't know if it's technically possible, but it would be greatly
informative to know how frequently a package is used, and how it performs,
in real contexts.

Frederic Commo
Bioinformatics, U981
Gustave Roussy

Thanks for these suggestions.

I like the idea of linking into the scientific literature, initially as
part of the 'Citation' section on each landing page rather than as a shield
(maybe a shield in the long term). As Wolfgang mentions it is a little more
challenging than a one-liner to match the information available from a
CITATION file (or automatically generated) to an appropriate search in
PubMed, and because citations are an important formal metric it seems
important to get this more-or-less right.

For what it's worth the more-or-less continuous stream of papers citing
'Biocondcutor' are listed at

  http://bioconductor.org/help/publications/

The links in the 'Literature Search' box query various resources for use
of the term 'Bioconductor'.

We have so far kept the distinction between 'available' and 'build',
partly because builds sometimes fail for transient (e.g., connectivity)
reasons or, in devel, because of an incomplete check-in that does not
compromise the end-user availability and functionality of the version
available via biocLite().

It's kind of amusing that (a) most of the information was already
available, often on the landing page (like the links to build reports that
Henrik mentions, or years in bioc), so the shields are serving just to
emphasize these; and (b) the 'green' implies some-how 'good', but many of
the shields (e.g., years in Bioc, posts, commits, downloads) are actually
never not green. Maybe these shields should be white?

Thanks again for the feedback; initial response seems to be positive.

Martin


 PS  Martin you?ll like Fig. 2 of their paper.

Wolfgang





 On May 9, 2015, at 19:15 GMT+2, Martin Morgan <mtmorgan at fredhutch.org>

wrote:

Bioc developers!

It's important that our users be able to identify packages that are
suitable for their research question. Obviously a first step is to identify
packages in the appropriate research domain, for instance through biocViews.

  http://bioconductor.org/packages/release/

We'd like to help users further prioritize their efforts by summarizing
use and usability. Metrics include:

- Cross-platform availability -- biocLite()-able from all or only some
platforms
- Support forum activity -- questions and comments / responses, 6 month
window
- Download percentile -- top 5, 20, 50%, or 'available'
- Current build status -- errors or warnings on some or all platforms
- Developer activity -- commits in the last 6 months
- Historical presence -- years in Bioconductor

Obviously the metrics are imperfect, so constructive feedback welcome --
we think the above capture in a more-or-less objective and computable way
the major axes influencing use and usability.

We initially intend to prominently display 'shields' (small graphical
icons) on package landing pages.

Thanks in advance for your comments,

Martin Morgan
Bioconductor
--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

From: "Jim Hester" <james.f.hester at gmail.com>
To: "Martin Morgan" <mtmorgan at fredhutch.org>
Cc: bioc-devel at r-project.org
Sent: Thursday, May 14, 2015 7:53:03 AM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields

The common shield convention is to use blue or orange when the
information
is not qualitatively good or bad, but the color choice is just
subjective
in the end.

On Thu, May 14, 2015 at 10:47 AM, Martin Morgan
<mtmorgan at fredhutch.org>
wrote:

On 05/10/2015 11:39 AM, COMMO Frederic wrote:

Dear Martin,

All of these suggestions sound good.

Wolfgang's suggestion regarding possible associated papers might
be also
great.

Another useful information would be to point to other publications
where
a given package was used, and cited.
I don't know if it's technically possible, but it would be greatly
informative to know how frequently a package is used, and how it
performs,
in real contexts.

Frederic Commo
Bioinformatics, U981
Gustave Roussy

Thanks for these suggestions.

I like the idea of linking into the scientific literature,
initially as
part of the 'Citation' section on each landing page rather than as
a shield
(maybe a shield in the long term). As Wolfgang mentions it is a
little more
challenging than a one-liner to match the information available
from a
CITATION file (or automatically generated) to an appropriate search
in
PubMed, and because citations are an important formal metric it
seems
important to get this more-or-less right.

For what it's worth the more-or-less continuous stream of papers
citing
'Biocondcutor' are listed at

  http://bioconductor.org/help/publications/

The links in the 'Literature Search' box query various resources
for use
of the term 'Bioconductor'.

We have so far kept the distinction between 'available' and
'build',
partly because builds sometimes fail for transient (e.g.,
connectivity)
reasons or, in devel, because of an incomplete check-in that does
not
compromise the end-user availability and functionality of the
version
available via biocLite().

It's kind of amusing that (a) most of the information was already
available, often on the landing page (like the links to build
reports that
Henrik mentions, or years in bioc), so the shields are serving just
to
emphasize these; and (b) the 'green' implies some-how 'good', but
many of
the shields (e.g., years in Bioc, posts, commits, downloads) are
actually
never not green. Maybe these shields should be white?

Thanks again for the feedback; initial response seems to be
positive.

Martin


 PS  Martin you?ll like Fig. 2 of their paper.

Wolfgang





 On May 9, 2015, at 19:15 GMT+2, Martin Morgan
 <mtmorgan at fredhutch.org>

wrote:

Bioc developers!

It's important that our users be able to identify packages that
are
suitable for their research question. Obviously a first step is
to identify
packages in the appropriate research domain, for instance through
biocViews.

  http://bioconductor.org/packages/release/

We'd like to help users further prioritize their efforts by
summarizing
use and usability. Metrics include:

- Cross-platform availability -- biocLite()-able from all or only
some
platforms
- Support forum activity -- questions and comments / responses, 6
month
window
- Download percentile -- top 5, 20, 50%, or 'available'
- Current build status -- errors or warnings on some or all
platforms
- Developer activity -- commits in the last 6 months
- Historical presence -- years in Bioconductor

Obviously the metrics are imperfect, so constructive feedback
welcome --
we think the above capture in a more-or-less objective and
computable way
the major axes influencing use and usability.

We initially intend to prominently display 'shields' (small
graphical
icons) on package landing pages.

Thanks in advance for your comments,

Martin Morgan
Bioconductor
--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

	[[alternative HTML version deleted]]

----- Original Message -----

From: "Jim Hester" <james.f.hester at gmail.com>
To: "Martin Morgan" <mtmorgan at fredhutch.org>
Cc: bioc-devel at r-project.org
Sent: Thursday, May 14, 2015 7:53:03 AM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields

The common shield convention is to use blue or orange when the
information
is not qualitatively good or bad, but the color choice is just
subjective
in the end.

It does seem though that we should indicate the non-changing nature of these shields with some kind of color change. Perhaps we can come up with one that works with the other design elements on the page.

BTW, the 'posts' tag does change color; if there are 0 posts tagged with a package name, the shield is yellow; otherwise it's green.

Dan

On Thu, May 14, 2015 at 10:47 AM, Martin Morgan
<mtmorgan at fredhutch.org>
wrote:

On 05/10/2015 11:39 AM, COMMO Frederic wrote:

Dear Martin,

All of these suggestions sound good.

Wolfgang's suggestion regarding possible associated papers might
be also
great.

Another useful information would be to point to other publications
where
a given package was used, and cited.
I don't know if it's technically possible, but it would be greatly
informative to know how frequently a package is used, and how it
performs,
in real contexts.

Frederic Commo
Bioinformatics, U981
Gustave Roussy

Thanks for these suggestions.

I like the idea of linking into the scientific literature,
initially as
part of the 'Citation' section on each landing page rather than as
a shield
(maybe a shield in the long term). As Wolfgang mentions it is a
little more
challenging than a one-liner to match the information available
from a
CITATION file (or automatically generated) to an appropriate search
in
PubMed, and because citations are an important formal metric it
seems
important to get this more-or-less right.

For what it's worth the more-or-less continuous stream of papers
citing
'Biocondcutor' are listed at

  http://bioconductor.org/help/publications/

The links in the 'Literature Search' box query various resources
for use
of the term 'Bioconductor'.

We have so far kept the distinction between 'available' and
'build',
partly because builds sometimes fail for transient (e.g.,
connectivity)
reasons or, in devel, because of an incomplete check-in that does
not
compromise the end-user availability and functionality of the
version
available via biocLite().

It's kind of amusing that (a) most of the information was already
available, often on the landing page (like the links to build
reports that
Henrik mentions, or years in bioc), so the shields are serving just
to
emphasize these; and (b) the 'green' implies some-how 'good', but
many of
the shields (e.g., years in Bioc, posts, commits, downloads) are
actually
never not green. Maybe these shields should be white?

Thanks again for the feedback; initial response seems to be
positive.

Martin


 PS  Martin you?ll like Fig. 2 of their paper.

Wolfgang





 On May 9, 2015, at 19:15 GMT+2, Martin Morgan
 <mtmorgan at fredhutch.org>

wrote:

Bioc developers!

It's important that our users be able to identify packages that
are
suitable for their research question. Obviously a first step is
to identify
packages in the appropriate research domain, for instance through
biocViews.

  http://bioconductor.org/packages/release/

We'd like to help users further prioritize their efforts by
summarizing
use and usability. Metrics include:

- Cross-platform availability -- biocLite()-able from all or only
some
platforms
- Support forum activity -- questions and comments / responses, 6
month
window
- Download percentile -- top 5, 20, 50%, or 'available'
- Current build status -- errors or warnings on some or all
platforms
- Developer activity -- commits in the last 6 months
- Historical presence -- years in Bioconductor

Obviously the metrics are imperfect, so constructive feedback
welcome --
we think the above capture in a more-or-less objective and
computable way
the major axes influencing use and usability.

We initially intend to prominently display 'shields' (small
graphical
icons) on package landing pages.

Thanks in advance for your comments,

Martin Morgan
Bioconductor
--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

      [[alternative HTML version deleted]]

From: "Leonardo Collado Torres" <lcollado at jhu.edu>
To: "Dan Tenenbaum" <dtenenba at fredhutch.org>
Cc: "Jim Hester" <james.f.hester at gmail.com>, bioc-devel at r-project.org
Sent: Tuesday, May 19, 2015 12:37:18 PM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields

Regarding the 'posts' tag, I can see that it includes a "closed
questions" component. For example,
http://www.bioconductor.org/packages/release/bioc/html/derfinder.html
is 3/1/9/0 right now meaning that 0 questions are closed. From
https://support.bioconductor.org/info/faq/, only moderators can close
questions. That seems like quite a bit of work for the moderators. So
maybe it would be best to drop the "closed questions" component. Or
alternatively, can the author of a package moderate the posts that
have a tag corresponding to their package?

As for 'build: warnings', it seems like it will show for some devel
packages all the time. For example,
http://www.bioconductor.org/packages/devel/bioc/html/regionReport.html
always has a warning in the Windows build machine due to a mismatch
in
the version of Rtools installed.

I do like these changes and the addition of shields =)


On Thu, May 14, 2015 at 10:56 AM, Dan Tenenbaum
<dtenenba at fredhutch.org> wrote:

----- Original Message -----

From: "Jim Hester" <james.f.hester at gmail.com>
To: "Martin Morgan" <mtmorgan at fredhutch.org>
Cc: bioc-devel at r-project.org
Sent: Thursday, May 14, 2015 7:53:03 AM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields

The common shield convention is to use blue or orange when the
information
is not qualitatively good or bad, but the color choice is just
subjective
in the end.

It does seem though that we should indicate the non-changing nature
of these shields with some kind of color change. Perhaps we can
come up with one that works with the other design elements on the
page.

BTW, the 'posts' tag does change color; if there are 0 posts tagged
with a package name, the shield is yellow; otherwise it's green.

Dan

On Thu, May 14, 2015 at 10:47 AM, Martin Morgan
<mtmorgan at fredhutch.org>
wrote:

On 05/10/2015 11:39 AM, COMMO Frederic wrote:

Dear Martin,

All of these suggestions sound good.

Wolfgang's suggestion regarding possible associated papers
might
be also
great.

Another useful information would be to point to other
publications
where
a given package was used, and cited.
I don't know if it's technically possible, but it would be
greatly
informative to know how frequently a package is used, and how
it
performs,
in real contexts.

Frederic Commo
Bioinformatics, U981
Gustave Roussy

Thanks for these suggestions.

I like the idea of linking into the scientific literature,
initially as
part of the 'Citation' section on each landing page rather than
as
a shield
(maybe a shield in the long term). As Wolfgang mentions it is a
little more
challenging than a one-liner to match the information available
from a
CITATION file (or automatically generated) to an appropriate
search
in
PubMed, and because citations are an important formal metric it
seems
important to get this more-or-less right.

For what it's worth the more-or-less continuous stream of papers
citing
'Biocondcutor' are listed at

  http://bioconductor.org/help/publications/

The links in the 'Literature Search' box query various resources
for use
of the term 'Bioconductor'.

We have so far kept the distinction between 'available' and
'build',
partly because builds sometimes fail for transient (e.g.,
connectivity)
reasons or, in devel, because of an incomplete check-in that
does
not
compromise the end-user availability and functionality of the
version
available via biocLite().

It's kind of amusing that (a) most of the information was
already
available, often on the landing page (like the links to build
reports that
Henrik mentions, or years in bioc), so the shields are serving
just
to
emphasize these; and (b) the 'green' implies some-how 'good',
but
many of
the shields (e.g., years in Bioc, posts, commits, downloads) are
actually
never not green. Maybe these shields should be white?

Thanks again for the feedback; initial response seems to be
positive.

Martin


 PS  Martin you?ll like Fig. 2 of their paper.

Wolfgang





 On May 9, 2015, at 19:15 GMT+2, Martin Morgan
 <mtmorgan at fredhutch.org>

wrote:

Bioc developers!

It's important that our users be able to identify packages
that
are
suitable for their research question. Obviously a first step
is
to identify
packages in the appropriate research domain, for instance
through
biocViews.

  http://bioconductor.org/packages/release/

We'd like to help users further prioritize their efforts by
summarizing
use and usability. Metrics include:

- Cross-platform availability -- biocLite()-able from all or
only
some
platforms
- Support forum activity -- questions and comments /
responses, 6
month
window
- Download percentile -- top 5, 20, 50%, or 'available'
- Current build status -- errors or warnings on some or all
platforms
- Developer activity -- commits in the last 6 months
- Historical presence -- years in Bioconductor

Obviously the metrics are imperfect, so constructive feedback
welcome --
we think the above capture in a more-or-less objective and
computable way
the major axes influencing use and usability.

We initially intend to prominently display 'shields' (small
graphical
icons) on package landing pages.

Thanks in advance for your comments,

Martin Morgan
Bioconductor
--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

      [[alternative HTML version deleted]]

----- Original Message -----

From: "Leonardo Collado Torres" <lcollado at jhu.edu>
To: "Dan Tenenbaum" <dtenenba at fredhutch.org>
Cc: "Jim Hester" <james.f.hester at gmail.com>, bioc-devel at r-project.org
Sent: Tuesday, May 19, 2015 12:37:18 PM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields

Regarding the 'posts' tag, I can see that it includes a "closed
questions" component. For example,
http://www.bioconductor.org/packages/release/bioc/html/derfinder.html
is 3/1/9/0 right now meaning that 0 questions are closed. From
https://support.bioconductor.org/info/faq/, only moderators can close
questions. That seems like quite a bit of work for the moderators. So
maybe it would be best to drop the "closed questions" component. Or
alternatively, can the author of a package moderate the posts that
have a tag corresponding to their package?

Perhaps the wording is wrong; what 'closed' is supposed to mean is that the original poster has accepted an answer. I'll change 'closed' to 'accepted'.

As for 'build: warnings', it seems like it will show for some devel
packages all the time. For example,
http://www.bioconductor.org/packages/devel/bioc/html/regionReport.html
always has a warning in the Windows build machine due to a mismatch
in
the version of Rtools installed.

This is a bug in devtools and may have already been fixed (but not yet propagated to CRAN).
IMO this should be reflected in the build shield.

Dan

I do like these changes and the addition of shields =)


On Thu, May 14, 2015 at 10:56 AM, Dan Tenenbaum
<dtenenba at fredhutch.org> wrote:

----- Original Message -----

From: "Jim Hester" <james.f.hester at gmail.com>
To: "Martin Morgan" <mtmorgan at fredhutch.org>
Cc: bioc-devel at r-project.org
Sent: Thursday, May 14, 2015 7:53:03 AM
Subject: Re: [Bioc-devel] Use and Usability metrics / shields

The common shield convention is to use blue or orange when the
information
is not qualitatively good or bad, but the color choice is just
subjective
in the end.

It does seem though that we should indicate the non-changing nature
of these shields with some kind of color change. Perhaps we can
come up with one that works with the other design elements on the
page.

BTW, the 'posts' tag does change color; if there are 0 posts tagged
with a package name, the shield is yellow; otherwise it's green.

Dan

On Thu, May 14, 2015 at 10:47 AM, Martin Morgan
<mtmorgan at fredhutch.org>
wrote:

On 05/10/2015 11:39 AM, COMMO Frederic wrote:

Dear Martin,

All of these suggestions sound good.

Wolfgang's suggestion regarding possible associated papers
might
be also
great.

Another useful information would be to point to other
publications
where
a given package was used, and cited.
I don't know if it's technically possible, but it would be
greatly
informative to know how frequently a package is used, and how
it
performs,
in real contexts.

Frederic Commo
Bioinformatics, U981
Gustave Roussy

Thanks for these suggestions.

I like the idea of linking into the scientific literature,
initially as
part of the 'Citation' section on each landing page rather than
as
a shield
(maybe a shield in the long term). As Wolfgang mentions it is a
little more
challenging than a one-liner to match the information available
from a
CITATION file (or automatically generated) to an appropriate
search
in
PubMed, and because citations are an important formal metric it
seems
important to get this more-or-less right.

For what it's worth the more-or-less continuous stream of papers
citing
'Biocondcutor' are listed at

  http://bioconductor.org/help/publications/

The links in the 'Literature Search' box query various resources
for use
of the term 'Bioconductor'.

We have so far kept the distinction between 'available' and
'build',
partly because builds sometimes fail for transient (e.g.,
connectivity)
reasons or, in devel, because of an incomplete check-in that
does
not
compromise the end-user availability and functionality of the
version
available via biocLite().

It's kind of amusing that (a) most of the information was
already
available, often on the landing page (like the links to build
reports that
Henrik mentions, or years in bioc), so the shields are serving
just
to
emphasize these; and (b) the 'green' implies some-how 'good',
but
many of
the shields (e.g., years in Bioc, posts, commits, downloads) are
actually
never not green. Maybe these shields should be white?

Thanks again for the feedback; initial response seems to be
positive.

Martin


 PS  Martin you?ll like Fig. 2 of their paper.

Wolfgang





 On May 9, 2015, at 19:15 GMT+2, Martin Morgan
 <mtmorgan at fredhutch.org>

wrote:

Bioc developers!

It's important that our users be able to identify packages
that
are
suitable for their research question. Obviously a first step
is
to identify
packages in the appropriate research domain, for instance
through
biocViews.

  http://bioconductor.org/packages/release/

We'd like to help users further prioritize their efforts by
summarizing
use and usability. Metrics include:

- Cross-platform availability -- biocLite()-able from all or
only
some
platforms
- Support forum activity -- questions and comments /
responses, 6
month
window
- Download percentile -- top 5, 20, 50%, or 'available'
- Current build status -- errors or warnings on some or all
platforms
- Developer activity -- commits in the last 6 months
- Historical presence -- years in Bioconductor

Obviously the metrics are imperfect, so constructive feedback
welcome --
we think the above capture in a more-or-less objective and
computable way
the major axes influencing use and usability.

We initially intend to prominently display 'shields' (small
graphical
icons) on package landing pages.

Thanks in advance for your comments,

Martin Morgan
Bioconductor
--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

      [[alternative HTML version deleted]]