Backtest trading strategies

Hello,

I understand the utility of MySQL for data storage.  But why is Python 
essential?  What does it do that R can't do for system 
creation/calculation?

Thanks
Python is great for parsing data from wherever you get it and populating 
databases.  MySQL is ideal for the write-once-read-thereafter scenario 
that research implies.  You can use R for the initial data marshaling 
if you'd rather not learn another language but Python seems like a 
better fit for that sort of thing.  It's a scripting language that 
integrates more naturally into its host environment.  For analysis and 
visualization however, R absolutely rules.
Neuro LeSuperH?ros wrote:
Hello,

I understand the utility of MySQL for data storage.  But why is Python
essential?  What does it do that R can't do for system
creation/calculation?

Thanks

Python is great for parsing data from wherever you get it and populating
databases.  MySQL is ideal for the write-once-read-thereafter scenario
that research implies.  You can use R for the initial data marshaling
if you'd rather not learn another language but Python seems like a
better fit for that sort of thing.  It's a scripting language that
integrates more naturally into its host environment.  For analysis and
visualization however, R absolutely rules.
I don't use MySQL so won't comment on that part but for parsing
data I have found R to have everything I need.  I used to use perl
but now use R exclusively.    R's string manipulation includes
regular expressions and the vector processing often simplifies
string manipulation by eliminating loops over lines or vectors
of strings.

To me its much easier to maintain code if its all in one language and
moving to R has enabled me to replace a bunch of perl, batch files
and other statistical software with R which really helps clean it
all up.  (Actually I still have some Windows batch files, see
http://cran.r-project.org/contrib/extra/batchfiles/, but they are only
for generic configuration utilities and nothing specific to any application.)
On 11/26/05, Rob Steele <rfin.20.phftt at xoxy.net> wrote:

Neuro LeSuperH?ros wrote:

Hello,

I understand the utility of MySQL for data storage.  But why is Python
essential?  What does it do that R can't do for system
creation/calculation?

Thanks

Python is great for parsing data from wherever you get it and populating
databases.  MySQL is ideal for the write-once-read-thereafter scenario
that research implies.  You can use R for the initial data marshaling
if you'd rather not learn another language but Python seems like a
better fit for that sort of thing.  It's a scripting language that
integrates more naturally into its host environment.  For analysis and
visualization however, R absolutely rules.

I don't use MySQL so won't comment on that part but for parsing
data I have found R to have everything I need.  I used to use perl
but now use R exclusively.    R's string manipulation includes
regular expressions and the vector processing often simplifies
string manipulation by eliminating loops over lines or vectors
of strings.

To me its much easier to maintain code if its all in one language and
moving to R has enabled me to replace a bunch of perl, batch files
and other statistical software with R which really helps clean it
all up.  (Actually I still have some Windows batch files, see
http://cran.r-project.org/contrib/extra/batchfiles/, but they are only
for generic configuration utilities and nothing specific to any application.)

Yeah, well, Perl.  :)

I take your point though.  Maybe someday R will play more nicely with 
shells and the command line so there will be no reason not to use it 
even for little jobs.

It's not totally bad now.  You can do stuff like:

#! /bin/sh

x=5

tmp=`R --vanilla --quiet << EOF
options(echo = FALSE)
y = $x * 1024 ^ 2
cat('\n', as.character(y), '\n')
q('no')
EOF`

y=`echo $tmp | awk '{print $NF}'`

echo $y

But you see what I mean.
On 11/26/05, Rob Steele <rfin.20.phftt at xoxy.net> wrote:

Neuro LeSuperH?ros wrote:

Hello,

I understand the utility of MySQL for data storage.  But why is Python
essential?  What does it do that R can't do for system
creation/calculation?

Thanks

Python is great for parsing data from wherever you get it and populating
databases.  MySQL is ideal for the write-once-read-thereafter scenario
that research implies.  You can use R for the initial data marshaling
if you'd rather not learn another language but Python seems like a
better fit for that sort of thing.  It's a scripting language that
integrates more naturally into its host environment.  For analysis and
visualization however, R absolutely rules.

I don't use MySQL so won't comment on that part but for parsing
data I have found R to have everything I need.  I used to use perl
but now use R exclusively.    R's string manipulation includes
regular expressions and the vector processing often simplifies
string manipulation by eliminating loops over lines or vectors
of strings.

To me its much easier to maintain code if its all in one language and
moving to R has enabled me to replace a bunch of perl, batch files
and other statistical software with R which really helps clean it
all up.  (Actually I still have some Windows batch files, see
http://cran.r-project.org/contrib/extra/batchfiles/, but they are only
for generic configuration utilities and nothing specific to any application.)
Each to their own I guess.  I happen to be much more familiar with 
Python than R and often use it to grab data in various formats which R 
won't read.  I wouldn't dream of using an MSDOS batch file.  As I learn 
more about R, I tend to do more in it but I couldn't imagine myself 
parsing dodgy HTML, for example, with it.
Gabor Grothendieck wrote:
On 11/26/05, Rob Steele <rfin.20.phftt at xoxy.net> wrote:

Neuro LeSuperH?ros wrote:

Hello,

I understand the utility of MySQL for data storage.  But why is Python
essential?  What does it do that R can't do for system
creation/calculation?

Thanks

Python is great for parsing data from wherever you get it and populating
databases.  MySQL is ideal for the write-once-read-thereafter scenario
that research implies.  You can use R for the initial data marshaling
if you'd rather not learn another language but Python seems like a
better fit for that sort of thing.  It's a scripting language that
integrates more naturally into its host environment.  For analysis and
visualization however, R absolutely rules.

I don't use MySQL so won't comment on that part but for parsing
data I have found R to have everything I need.  I used to use perl
but now use R exclusively.    R's string manipulation includes
regular expressions and the vector processing often simplifies
string manipulation by eliminating loops over lines or vectors
of strings.

To me its much easier to maintain code if its all in one language and
moving to R has enabled me to replace a bunch of perl, batch files
and other statistical software with R which really helps clean it
all up.  (Actually I still have some Windows batch files, see
http://cran.r-project.org/contrib/extra/batchfiles/, but they are only
for generic configuration utilities and nothing specific to any application.)
Each to their own I guess.  I happen to be much more familiar with
Python than R and often use it to grab data in various formats which R
won't read.  I wouldn't dream of using an MSDOS batch file.  As I learn
more about R, I tend to do more in it but I couldn't imagine myself
parsing dodgy HTML, for example, with it.
Actually I use R for parsing HTML and for parsing XML too.  I do
agree by Rob that it would be nice if R worked better with shells
and also wish I could write small self contained executables in R
like one can in tcl and Python.

MySQL is ideal for the write-once-read-thereafter scenario 
that research implies.
True (that's what MySQL was originally designed for), except that it's
SQL dialect is pretty limited, and the database has various
misfeatures, e.g.:

  http://sql-info.de/mysql/gotchas.html

The limited SQL support might or might not matter to you, depending on
just what sort of data you're loading in and how complicated your data
model and queries are, but it's certainly something you should keep in
mind when picking a RDBMS.  Maybe that's been improved in MySQL 5.0
(which was just released a month or so ago), I don't know.

If I didn't want to use PostgreSQL or Oracle, I'd be tempted to try a
lightweight embedded database like SQLite or Metakit:

  http://www.sqlite.org/
  http://www.equi4.com/metakit.html

PostgreSQL's SQL tends to be more powerful than MySQL's, but even
PostgreSQL doesn't yet support the (very useful) SQL:2003 windowing
functions (aka, the "OLAP" features T611 and T612 in the spec).
Oracle and DB2 both do.

  http://www.wintercorp.com/rwintercolumns/SQL_99snewolapfunctions.html
  http://www.ncb.ernet.in/education/modules/dbms/SQL99/OLAP-99-154r2.pdf
  http://www.wiscorp.com/sql/SQL2003Features.pdf
  http://troels.arvin.dk/db/rdbms/#select-limit-offset
  http://www.postgresql.org/docs/8.0/interactive/features.html
  http://en.wikipedia.org/wiki/SQL
  http://www.sigmod.org/sigmod/record/issues/0403/E.JimAndrew-standard.pdf
  http://www.oracle.com/oramag/oracle/01-jul/o41industry.html

Of course, with PostgreSQL you can also run R (or a whole bunch of
other languages) inside the RDBMS if you want:

  http://www.joeconway.com/plr/
  http://gborg.postgresql.org/project/plr/projdisplay.php
  http://archives.postgresql.org/pgsql-hackers/2003-02/msg00142.php
  http://www.omegahat.org/RSPostgres/
Andrew Piskorski <atp at piskorski.com>
http://www.piskorski.com/
On Sat, Nov 26, 2005 at 12:11:27AM -0500, Rob Steele wrote:

MySQL is ideal for the write-once-read-thereafter scenario 
that research implies

True (that's what MySQL was originally designed for), except that it's
SQL dialect is pretty limited, and the database has various
misfeatures, e.g.:

  http://sql-info.de/mysql/gotchas.html

The limited SQL support might or might not matter to you, depending on
just what sort of data you're loading in and how complicated your data
model and queries are, but it's certainly something you should keep in
mind when picking a RDBMS.  Maybe that's been improved in MySQL 5.0
(which was just released a month or so ago), I don't know.

If I didn't want to use PostgreSQL or Oracle, I'd be tempted to try a
lightweight embedded database like SQLite or Metakit:

One  of the advantages of PostgreSQL is that it is relatively easy to 
port a PostgreSQL database to Oracle if needed.  Moving MySQL to Oracle 
is a much bigger challenge.
My company delivers broad business intelligence solutions on foundations of
data warehouses and marts that grow to hundreds of gigabytes. It is
therefore critical that we optimize data storage and the ETL -- extract,
transform, and load -- processes for very large data. Our minimal open
source analytics solution consists of: MySql/Postgres (or Oracle, et. al. if
standard) repositories that house the sourcing data, assuring one version of
the truth; Perl/Python for "munging" potentially very large, complicated,
and messy input files, consolidating data across disparate sources,
performing array and date calculations on millions of records,
inserting/updating the databases, reporting, and sourcing data sets for
input to analytics; and R for graphics, statistics, and analytical
calculations. While R is a very powerful and rich language, we do not see
its strength in parsing ugly and error-prone data files, nor do we find its
interpreted speed adequate for very large data management. Once the data
stores are built and validated, we turn them over to users of R and other BI
tools such as Pentaho. We are very encouraged by both the acceptance of this
approach and R that we're starting to experience in the commercial world.

Steve Miller 

-----Original Message-----
From: r-sig-finance-bounces at stat.math.ethz.ch
[mailto:r-sig-finance-bounces at stat.math.ethz.ch] On Behalf Of Gabor
Grothendieck
Sent: Saturday, November 26, 2005 7:14 AM
To: paul sorenson
Cc: r-sig-finance at stat.math.ethz.ch
Subject: Re: [R-sig-finance] Backtest trading strategies
Gabor Grothendieck wrote:
On 11/26/05, Rob Steele <rfin.20.phftt at xoxy.net> wrote:

Neuro LeSuperH?ros wrote:

Hello,

I understand the utility of MySQL for data storage.  But why is Python
essential?  What does it do that R can't do for system
creation/calculation?

Thanks

Python is great for parsing data from wherever you get it and populating
databases.  MySQL is ideal for the write-once-read-thereafter scenario
that research implies.  You can use R for the initial data marshaling
if you'd rather not learn another language but Python seems like a
better fit for that sort of thing.  It's a scripting language that
integrates more naturally into its host environment.  For analysis and
visualization however, R absolutely rules.

I don't use MySQL so won't comment on that part but for parsing
data I have found R to have everything I need.  I used to use perl
but now use R exclusively.    R's string manipulation includes
regular expressions and the vector processing often simplifies
string manipulation by eliminating loops over lines or vectors
of strings.

To me its much easier to maintain code if its all in one language and
moving to R has enabled me to replace a bunch of perl, batch files
and other statistical software with R which really helps clean it
all up.  (Actually I still have some Windows batch files, see
http://cran.r-project.org/contrib/extra/batchfiles/, but they are only
for generic configuration utilities and nothing specific to any
application.)
Each to their own I guess.  I happen to be much more familiar with
Python than R and often use it to grab data in various formats which R
won't read.  I wouldn't dream of using an MSDOS batch file.  As I learn
more about R, I tend to do more in it but I couldn't imagine myself
parsing dodgy HTML, for example, with it.
Actually I use R for parsing HTML and for parsing XML too.  I do
agree by Rob that it would be nice if R worked better with shells
and also wish I could write small self contained executables in R
like one can in tcl and Python.

_______________________________________________
R-sig-finance at stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance