on ptr_R_WriteErrConsole

I fully support Duncan M (though it's painless for me).

What I wanted to do, and explored, when I was visiting Peter D in
Copenhagen nearly 2 years ago, was to XML-ize the output stream, so
that I could mark it up.  From that, it would be reasonably simple to
parse output (provided you marked it up on the way out, which was the
difficult part, I fully admit, which was where I failed to get a
distributable implementation).

(the goal was to provide multimedia output streams, i.e. text, audio,
static graphics, or dynamic graphics, as required).

The person asking about a log4j a while back (or what I'm looking at
right now, log4cl) probably wanted the same thing.


On 10/18/05, Duncan Murdoch <murdoch at stats.uwo.ca> wrote:

I'll volunteer to do your testing on Windows, provided nobody  
suggests a
way to do what you want without your patches.  That is, if there's  
a way
for a custom front-end to separate the streams of text coming to
ptr_R_WriteConsole through REvprintf from that coming through  
Rvprintf,
then your changes are not needed.  If there is currently no way to do
that, then I think there should be.

The justification for this change is to facilitate experimentation  
with
different front ends, as you describe below.  That's something I  
think
we should support.

Forcing both streams through ptr_R_WriteConsole without giving
information about their origin is a decision about a particular  
kind of
user interface.  Decisions like that should be made by the writer  
of the
front-end.

I agree with Luke and Brian that your second suggestion (which would
allow separation of text into regular output, warnings, and errors)
would tie us down too much in the internal details.

Duncan Murdoch




Thomas Friedrichsmeier wrote:

I second the point about not having made a case, and also that  
(from the
part deleted below) about needing to understand the internals  
before
proposing changes.

I'll try to make the case below point d). I fully accept  
rejection of the more
daring part of my proposal, but the part about  
ptr_R_WriteErrConsole is
really straightforward, and I have really made sure to understand  
the
internal going ons, here.
My reply will be lengthy, because this really is important to me.

a) I took a quick look at the patch, and it seems to be only  
half the
story.  R_WriteConsole is also supported under Windows, in a  
different
(and more flexible) way.  I think the part not covered is a lot  
more
problematic, as I don't see how to introduce R_WriteErrConsole in a
backwards compatible way.

I admit I've been neglectant to the Windows code. I don't have a  
windows
machine to compile and test any changes I make in my local  
sources, and that
lead me to be careless. so probably the patch is not correct.
Do I believe there is still a backwards compatible way to introduce
ptr_R_WriteErrConsole / R_WriteErrConsole?

void R_WriteErrConsole(char *buf, int len) {ptr_R_WriteErrConsole 
(buf, len);}
...
/* standard initialization: */
ptr_R_WriteErrConsole = R_WriteConsole;

*R_WriteConsole*. This ensures that if you do not touch the new
ptr_R_WriteErrConsole, you get *exactly* the old behavior. I  
can't be much
easier than that.

b) There are several inaccuracies in the account.  First,  
`channels' are
not what we are talking about here.  sink() allow output and  
messages to
be sent to different connections, but the console is connected  
to fixed
terminal connections stdout() and stderr().  There is a long- 
standing
design for R consoles (sketched in src/unix/system.txt) that has  
all the
output going to the console.  This is even true of the standard  
Unix
terminal front end.  The only circumstances in which stdout()  
and stderr()
are separated are when R is used for scripting.  Part of the  
motivation
for this is to ensure that stdout() and stderr() appear in an  
integrated
way (in particular, interleaved in the correct order) on a console.

Ok, so I have been simplifying, and using my own terminology.  
What I'm saying
is, output generally goes two routes *throughout* R sources:

"message":
[warnings]->...->REprintf->REvprintf
[errors]->...->REprintf->REvprintf
REprintf->REvprintf

"output":
[print]->...->Rprintf->Rvprintf
Rprintf->Rvprintf

So it is not like I'm making up any new distinction, here. I'm  
referring to an
existing one, avaible via already public API. See also this  
comment from
printutils.c:

/* =========
 * Printing:
 * =========
 *
 * All printing in R is done via the functions Rprintf and REprintf
 * or their (v) versions Rvprintf and REvprintf.
 * These routines work exactly like (v)printf(3).  Rprintf writes to
 * ``standard output''.        It is redirected by the sink()  
function,
 * and is suitable for ordinary output.        REprintf writes to
 * ``standard error'' and is useful for error messages and warnings.
 * It is not redirected by sink().

All I'm dealing with is, what should happen in REvprintf, the  
last station in
what I call the "message route":

Currently it is basically (and yes, I'm simplifying again):
void REvprintf(const char *format, va_list arg)
{
      /* A: if there is a sink(type="message"), print to that and  
return */
      /* B: if RConsoleFile is not NULL, print to that and return */
      /* C: if none of the above, call R_WriteConsole */
}

All I'm talking about is case C, and effectively, my proposal  
would change
this to
      /* C: ptr_R_WriteErrConsole has been modified by stubborn  
developer, print to
that and return */
      /* D: if none of the above, call R_WriteConsole */

No other console designer has seen a need for this, and there is  
at least
the potential for presenting very misleading information to end  
users.

Let me assure you this: The interface pointers give me enough  
rope already to
shoot myself in the foot. I have enough opportunity to present  
extremely
misleading information to end users. Re-read my answer to a) to  
see the
change really, really, really would not affect any code that does  
not
explicitely ask for this extra inch of rope.
Also let me assure you this: Other GUI designers are much  
interested in an
easy way to find out which portions of the output have come the  
"message"
route, as well. We do crazy stuff like using separate file-sinks,  
grepping
for "warning", or "error", and things like that. I've seen hacks  
and hacks,
and I've used hack after hack. But I think there is a good  
opportunity for a
solid solution.

So
one ramification of allowing consoles to present stdout() and  
stderr()
separately would have to be a review of how they are used.  This  
is not
hypothetical: I have struggled many times over the years with an  
R-like
system which when scripted wrote error messages in inappropriate  
places on
a file, at one point sending prompts to stderr yet echoed input  
to stdout.
We've worked hard in R to avoid that kind of thing, mainly by  
having a
single route to the console.  (There are still residual problems  
if C++ or
Fortran I/O is used, BTW, and note that R_ReadConsole also  
*writes* to the
console.)

And the single route to the console will remain intact using  
R_WriteConsole.
I'm only asking for the *opportunity*, not the *obligation* to  
intercept the
one call to R_WriteConsole in REvprintf.

c) Anything in R involving more than one of the three main  
families of
platforms is NOT a `small addition': it involves testing and  
subsequent
maintenance by two or three people.

So, yes, somebody would have to add corresponding code to windows.
Unfortunately, I don't think I qualify for this, as I just can't  
test on
windows. I'm fairly confident, the change I did in gnuwin32 will  
ensure
nothing is broken, but you would want a parallel to  
ptr_R_WriteErrConsole in
windows for consistency's sake.
But please: Don't conjure up a maintenance nightmare for this  
simple change.

d) There is an existing mechanism that could be used.  If you want
file-like stderr and stdout, you could drive R via a file-like  
interface
(e.g. ptys).  That is not easy on non-Unix-alike platforms (and was
probably impossible on classic MacOS R), but I understood Thomas  
was using
KDE.  (There are live examples of this route, even on Windows.)

And it's not like I haven't ventured along that route. Do you  
know how much
fun it is to use two separate ptys, then try to make sure the  
output arrives
in a sensible order, i.e. you don't get all warnings before all  
output or
vice versa, or some strange mixture? I'm not the infailable  
programmer, and
I'm not an expert in R internals. But before I go into lengthy  
discussions, I
have checked my options.

So again: Why do I want something like this?
A GUI may want to do some things, which are not needed on the  
console. One
such thing is to identify "message" output. There are several  
uses for this:
1) Highlight "message" output to bring it to the users attention
2) Offer context help on "message"s. Of course this is easier  
said that done,
but the first step in this, would be to find out which portions  
of the output
are messages.
3) Show "message"s that come up in operations that would usually  
redirect the
output elsewhere, and not show it to the user directly. Much like  
the
scripting situation you depicted in b)

And again: Why can't I just use current facilities?
1) Condition Handling: Probably the way to go for my advanced  
needs. But
totally useless, if I want to catch messages generated using  
direct calls to
REprintf/REvprintf. Those are abundant. They have to be dealt  
with. The only
way to do this is to use a mechanism available in/below/after  
REvprintf.
2) Ptys, sinks: See above
3) Grepping: come on now. Could not even solve most needs,  
impossible due to
internationalization...
4) Using mind reading? When I first posted about these matters to  
r-devel (and
yes, before that I tried my luck on r-help), it was a plain  
support question:
How can I do this? See also here:
https://stat.ethz.ch/pipermail/r-devel/2005-October/034975.html
I did not receive any reply on this. What should I conclude?

I'll gladly accept alternative solutions. They are not the above  
though. And
I've written why they are not before.

I have spent far longer (and written more) than I intended on  
this.  The
length of correspondence so far (and much in a prolix style) is  
all part
of the support costs.  One thing the R project can not afford is to
explain to individual users how internals work -- we have not  
even been
able to find the time to write down for the core team how a lot  
of the
internals work, and some developments are being held up by  
that.  So this
has to weighed against considering proposals which would appear  
to help
just one user.

Sorry about writing more and more lengthy mails. I don't really  
want to. I
have better things to do as well. But this is important, and -  
sorry to say -
IMO you've simply overlooked a number of points. All I can do is  
restate
them, trying to make extra sure to get my point across.

I suspect that we will only want to go forward if a concise and  
strong
case can be made from a group of developers who have tested it  
out on all
three main families of platforms.

And where would you think, I could conjure such a group of  
developers, if not
on this list?

Regards
Thomas Friedrichsmeier

--
best,
-tony

blindglobe at gmail.com
Muttenz, Switzerland.
"Commit early,commit often, and commit in a repository from which  
we can easily
roll-back your mistakes" (AJR, 4Jan05).