Skip to content

application to mentor syrfr package development for Google Summer of Code 2010

10 messages · Chidambaram Annamalai, Michael Schmidt, James Salsman

Original
James Salsman
# 
Per http://rwiki.sciviews.org/doku.php?id=developers:projects:gsoc2010
-- and http://rwiki.sciviews.org/doku.php?id=developers:projects:gsoc2010:syrfr
-- I am applying to mentor the "Symbolic Regression for R" (syrfr)
package for the Google Summer of Code 2010.

I propose the following test which an applicant would have to pass in
order to qualify for the topic:

1. Describe each of the following terms as they relate to statistical
regression: categorical, periodic, modular, continuous, bimodal,
log-normal, logistic, Gompertz, and nonlinear.

2. Explain which parts of http://bit.ly/tablecurve were adopted in
SigmaPlot and which weren't.

3. Use the 'outliers' package to improve a regression fit maintaining
the correct extrapolation confidence intervals as are between those
with and without outlier exclusions in proportion to the confidence
that the outliers were reasonably excluded.  (Show your R transcript.)

4. Explain the relationship between degrees of freedom and correlated
independent variables.

Best regards,

James Salsman
jsalsman at talknicer.com
http://talknicer.com
James Salsman
# 
Chillu,

If I understand your concern, you want to lay the foundation for
derivatives so that you can implement the search strategies described
in Schmidt and Lipson (2010) --
http://www.springerlink.com/content/l79v2183725413w0/ -- is that
right? It is not clear to me how well this generalized approach will
work in practice, but there is no reason not to proceed in parallel to
establish a framework under which you could implement the metrics
proposed by Schmidt and Lipson in the contemplated syrfr package.

I have expanded the test I proposed with two more questions -- at
http://rwiki.sciviews.org/doku.php?id=developers:projects:gsoc2010:syrfr
-- specifically:

5. Critique http://sites.google.com/site/gptips4matlab/

6. Use anova to compare the goodness-of-fit of a SSfpl nls fit with a
linear model of your choice. How can your characterize the
degree-of-freedom-adjusted goodness of fit of nonlinear models?

I believe pairwise anova.nls is the optimal comparison for nonlinear
models, but there are several good choices for approximations,
including the residual standard error, which I believe can be adjusted
for degrees of freedom, as can the F statistic which TableCurve uses;
see: http://en.wikipedia.org/wiki/F-test#Regression_problems

Best regards,
James Salsman


On Sun, Mar 7, 2010 at 7:35 PM, Chidambaram Annamalai
<quantumelixir at gmail.com> wrote:
James Salsman
# 
Chillu, I meant that development on both a syrfr R package capable of
using either F statistics or parametric derivatives should proceed in
parallel with your work on such a derivatives package. You are right
that genetic algorithm search (and general best-first search --
http://en.wikipedia.org/wiki/Best-first_search -- of which genetic
algorithms are various special cases) can be very effectively
parallelized, too.

In any case, thank you for pointing out Eureqa --
http://ccsl.mae.cornell.edu/eureqa -- but I can see no evidence there
or in the user manual or user forums that Eureqa is considering
degrees of freedom in its goodness-of-fit estimation.  That is a
serious problem which will typically result in invalid symbolic
regression.  I am sending this message also to Michael Schmidt so that
he might be able to comment on the extent to which Eureqa adjusts for
degrees of freedom in his fit evaluations.

Best regards,
James Salsman

On Sun, Mar 7, 2010 at 10:39 PM, Chidambaram Annamalai
<quantumelixir at gmail.com> wrote:
James Salsman
# 
Michael,

Thanks for your reply:
On Mon, Mar 8, 2010 at 12:41 AM, Michael Schmidt <mds47 at cornell.edu> wrote:
That's very good, but I wonder whether we can perform automatic
outlier exclusion that way.  We would need to keep the confidence
interval, or at least the information necessary to derive it, accurate
in every step of the genetic beam search.  Since the confidence
intervals of extrapolation depend so heavily on the number of degrees
of freedom of the fit (along with the residual standard error) it's a
good idea to use a degree-of-freedom-adjusted F statistic instead of a
post-hoc combination of equation size and residual standard error, I
would think.  You might want to try it and see how it improves things.
 Confidence intervals, by representing the goodness of fit in the
original units and domain of the dependent variable, are tremendously
useful and sometimes make many kinds of tests which would otherwise be
very laborious easy to eyeball.

Being able to fit curves to one-to-many relations instead of strict
one-to-one functions appeals to those working in the imaging domain,
but not to as many traditional non-image statisticians. Regressing
functions usually results in well-defined confidence intervals, but
regressing general relations with derivatives produces confidence
intervals which can also be relations.  Trying to figure out a
spiral-shaped confidence interval probably appeals to astronomers more
than most people.  So I am proposing that, for R's contemplated
'syrfr' symbolic regression package, we do functions in a general
genetic beam search framework, Chillu and John Nash can do derivatives
in the new 'adinr' package, and then we can try to put them together,
extend the syrfr package with a parameter indicating to fit relations
with derivatives instead of functions, to try to replicate your work
on Eureqa using d.o.f-adjusted F statistics as a heuristic beam search
evaluation function.

Have you quantified the extent to which using the crossover rule in
the equation tree search is an improvement over mutation alone in
symbolic regression?  I am glad that Chillu and Dirk have already
supported that; there is no denying its utility.

Would you like to co-mentor this project?
http://rwiki.sciviews.org/doku.php?id=developers:projects:gsoc2010:syrfr
I've already stepped forward, so you could do as much or as little as
you like if you wanted to co-mentor and Dirk agreed to that
arrangement.

Best regards,
James Salsman
1 day later
James Salsman
# 
Michael,

Thanks for your reply with the information about the Eureqa API -- I
am forwarding it to the r-devel list below.

Dirk,

Will you please agree to referring to the syrfr package as symbolic
genetic algorithm regression of functions but not (yet) general
relations?  It would be best to refer to general relation regression
as a future package, something like 'syrr' and leave the
parametrization of the derivatives to that package.

May I please mentor, in consultation with Michael if necessary, work
on general function regressions while Chillu and John Nash work on the
derivative package necessary for general relation regressions?  Thank
you for your kind consideration.

Best regards,
James Salsman
On Tue, Mar 9, 2010 at 11:06 AM, Michael Schmidt <mds47 at cornell.edu> wrote: