Competing with one's own work

Dear Duncan,

What constitutes a convincing argument for making significant changes?
Taking the example of optimization algorithms (say, for smooth objective
functions), how does one make a convincing argument that a particular class
of algorithms are "better" than another class? This can be a difficult task,
but quite doable with good benchmarking practices.

Supposing for the moment that such a convincing argument has been made, is
that sufficient to get the R-core to act upon it? ?Are there compelling
factors other than just "algorithm A is better than algorithm B"?

I'd think that the argument is relatively easy if the need for the change is
driven by consumer demand. But, even here I am not sure how to make an
argument to the R-core to consider the big changes. ?For example, there is a
reasonable demand for constrained (smooth) optimization algorithms in R
(based on R-help queries). ?Currently, there are only 3 packages that can
handle this. ?However, in the base distribution only `constrOptim' function
is provided, which cannot handle anything more than linear, inequality
constraints. ?I think that the base distribution needs to have a package for
constrained optimization that can handle linear/nonlinear and
equality/inequality constraints.

John, thanks for raising an important issue.

Thanks & Best,
Ravi.

-------------------------------------------------------
Ravi Varadhan, Ph.D.
Assistant Professor,
Division of Geriatric Medicine and Gerontology School of Medicine Johns
Hopkins University

Ph. (410) 502-2619
email: rvaradhan at jhmi.edu


-----Original Message-----
From: r-devel-bounces at r-project.org [mailto:r-devel-bounces at r-project.org]
On Behalf Of Duncan Murdoch
Sent: Friday, December 03, 2010 11:13 AM
To: nashjc at uottawa.ca
Cc: r-devel at r-project.org
Subject: Re: [Rd] Competing with one's own work

On 03/12/2010 10:57 AM, Prof. John C Nash wrote:

No, this is not about Rcpp, but a comment in that overly long discussion

raised a question

that has been in my mind for a while.

This is that one may have work that is used in R in the base functionality

and there are

improvements that should be incorporated.

For me, this concerns the BFGS, Nelder-Mead and CG options of optim(),

which are based on

the 1990 edition (Pascal codes) of my 1979 book "Compact numerical

methods...", which were

themselves derived from other people's work. By the time Brian Ripley took

that work (with

permission, even though not strictly required. Thanks!) there were already

some

improvements to these same algorithms (mainly bounds and masks) in the

BASIC codes of the

1987 book by Mary Walker-Smith and I. However, BASIC to R is not something

I'd wish on

anyone.

Now there are some R packages, including some I've been working on, that

do offer

improvements on the optim() offerings. I would not say mine are yet fully

ready for

incorporation into the base, but they are pretty close. Equally I think

some of the tools

in the base should be deprecated and users encouraged to try other

routines. It is also

getting more and more important that novice users be provided with

sensible guidance and

robust default settings and choices. In many areas, users are faced with

more choice than

is efficient for the majority of problems.

My question is: How should such changes be suggested / assisted? It seems

to me that this

is beyond a simple feature request. Some discussion on pros and cons would

be appropriate,

and those like myself who are familiar with particular tools can and

should offer help.

Alternatively, is there a document available in the style "Writing R

Extensions" that has

a title like "How the R Base Packages are Updated"? A brief search was

negative.

I'm happy to compete with my own prior work to provide improvements. It

would be nice to

see some of those improvements become the benchmark for further progress.

There are answers at many different levels to your questions. ?The
simplest is that base packages are part of R, so they get updated when a
member of R Core updates them, and the updates get released when a new
version of R is released.

So if you want a change, you need to convince a member of the core to
make it. ?Pointing out a bug is the easiest way to do this: ?bugs
usually get fixed quickly, if they are clearly demonstrated.

If you want a bigger change, you need to make a convincing argument in
favour of it. ?If you pick a topic that is of particular interest to one
core member, and you can convince him to make the change, then it will
happen. ?If pick some obscure topic that's not of interest to anyone,
you'll need a very strong argument to make it interesting. ?Part of any
of these arguments is explaining why the change needs to be made to the
base, why it can't just be published in a contributed package. ?(That's
why bug fixes are easy, and big additions to the base packages are not.)

Duncan Murdoch