serial connection patch

Matt,

thanks for you efforts. We cannot directly use your patch due to licensing issues (please note the licenses in the files you are modifying - I don't think anyone can currently use the patch and redistribute the resulting R; also/or we may possibly need to clarify that the work is either done on behalf of the R-foundation or the copyright is assigned to R-foundation or something similar so future licensing issues can be solved by the Foundation since that is the copyright holder of most of R).

Apart from that, the overall structure looks good, but the devil is in the details. It does not quite work for me on OS X (any read/writes just hang in the same way that it would if you used a file connection on the tty device) - I don't think it is as simple (AFAIR you have to make sure you open it such that it doesn't become the controlling terminal). Also it would definitely need "blocking" argument like most other connections have.

In general I don't think serial is the right name (and scope) here - what we're really looking for a is a tty connection (and serial just an application of it). What you have so far is a file connection with setserial functionality on it - and I think that may be more easily done by really using setserial on the already implemented file connection. The difference with tty is the other parts such as flow control, modes etc. It's much more work to expose those, but I think that is what the direction would be. In fact it could well be that tty is simply file connection + termio so you don't have to re-invent the wheel (under the hood). Also that would give you many aspects for free since you could inherit from file at the R level.

I hope it helps. I was thinking about the above but in the end decided to back off once I was looking at the flood of parameters you have to take care of in tty. But it would be nice if someone did that ;).

Cheers,
Simon


On Apr 26, 2010, at 12:28 PM, Matt Shotwell wrote:

All,

Our discussion of serial interfaces last week motivated me to dig into
the R connections API. In short, I spent the weekend writing a patch for
version 2.11.0 that adds a serial connection. I posted a blog entry that
gives instructions for applying, configuring, and compiling the patched
version <http://biostatmatt.com/archives/112>.

The patch currently only enables a serial connection for POSIX compliant
OSs. It should NOT be considered well tested, though I feel it is fairly
safe. I would really like to hear from someone who can try it on Mac OS
X (and Windows to ensure that it has the correct 'null' behavior).

Briefly, the connection utilizes (and checks for) the termios.h header.
I opted not to extend the 'file' connection in a manner similar to the
'pipe' connection because this might not be suitable under Win32. Hence,
there are new read, write, flush, open, and close methods and a struct
serialconn for the serial connection. Opens are non-blocking and binary
only. Serial parameters can be passed when the connection is created. I
overloaded the summary method in order for the user to see the serial
settings.

I've done my best to be consistent and conforming with the connections
API and coding style. However, I would be great if someone with more
experience with the R API (Simon? :)) would take a critical look at the
design and coding of the new connection. If this is something that the R
community would like to pursue, I'd like to work on support for Windows
serial interfaces. Below is a link to the patch file and the output of
diffstat.

<http://biostatmatt.com/R/R-2.11.0-serial.patch>


file                                |  mod    ins     del
------------------------------------------------------------
configure.ac                        |    2    1 +     1 -
src/include/Internal.h              |    2    2 +     0 -
src/library/base/R/connections.R    |   10    10 +    0 -
src/library/base/man/connections.Rd |   60    49 +    11 -
src/main/connections.c              |  379    379 +   0 -
src/main/names.c                    |    2    2 +     0 -


P.S. connections.c is the largest file in src/main/, before and after
the patch. Making the 379 additions increased the file size by 6.9%
(10.7kB). Adding Windows support should increase the file size by a
lesser amount.

-Matt

On Wed, 2010-04-21 at 08:07 -0400, Simon Urbanek wrote:

Philippe,

unfortunately that approach has one major drawback - it does not give you a connection. As I said in the previous e-mail it is fairly easy to talk to a tty directly, but the challenge is to turn it into a connection. I don't like the Tcl approach for several reasons, one of them being that it's entirely unnatural since you have to construct strings of code -- you could as well use rJava and Java serial connections which has a little more friendly syntax (but I wouldn't recommend that, either) or any other language R interfaces to.

I was thinking about it a bit and I may be tempted to have a dab at a tty connection, but I still would not want to mess with ioctl (the other part Matt mentioned).

Cheers,
Simon



On Apr 21, 2010, at 6:30 AM, Philippe Grosjean wrote:

There is another option I use since a couple of years to pilot scientific devices, to program my chemostats, etc. and that is platform-independent: Tcl.

Tcl i/o design is excellent and very robust, and Tcl/Tk is integrated in R with the tcltk package.

It is really just a mather of a few lines of code in R to communicate through a serial port from R using Tcl. Something like:

require(tcltk)
.Tcl('set R_com1 [open "com1" r+]') # Works on Windows too!

# There are many config parameters available here... just an example
.Tcl('fconfigure $R_com1 -mode "9600,n,8,1" -buffering none -blocking 0')

# Send a command to your device through the serial port
.Tcl('puts -nonewline $R_com1 {my_cmd}')

# Read a line of text from the serial port
line <- tclvalue(.Tcl('gets $R_com1'))

With a little bit more code, one can program Tcl to call R code automatically everytime new data is pushed by the connected device through the serial port. This is done using something like:

.Tcl('fileevent $R_com1 readable [list Handler_com1 $R_com1]')

Here, "Handler_com1" is a Tcl function. So, some care must be taken using tcltk's .Tcl.callback() to trigger the event on the R side. One way to deal with this easily is by using tclFun() from the tcltk2 package.

In tcltk2, there is also ?tclTaskSchedule that can be of interest in the context of serial port communication to trigger a R function in the background regularly and collect data actively from the serial port.

All these tools give me a lot a flexibility to communicate through the serial port from R,... and most importantly, to write my code in a portable way (tested on Windows XP and Linux Ubuntu).

If there is some interest in this approach, I could initiate a 'tclcom' R package on R-Forge and place there the code I have.

Best,

Philippe
..............................................<?}))><........
) ) ) ) )
( ( ( ( (    Prof. Philippe Grosjean
) ) ) ) )
( ( ( ( (    Numerical Ecology of Aquatic Systems
) ) ) ) )   Mons University, Belgium
( ( ( ( (
..............................................................

On 21/04/10 03:17, shotwelm wrote:

Simon is right of course, there are plenty of sensors that would work
just fine at 9600 baud (like a thermistor rigged to an ADC). There's a
theorem along these lines (Nyquist sampling theorem?). I think piping
the output to R is a clever solution. I added a few lines to the ttys.c
program so that the baud rate is a command line option (i.e. -B9600)
<http://biostatmatt.com/temp/ttys.c>  and confirmed it will compile in
Linux (2.6.30). Maybe it will save a step. Microcontrollers really are
addictive!

For an ioctl package, I was originally thinking of using file
descriptors directly. However, I agree this feels like subverting what
could be an extension of the connections API. Given that "everything is
a file" in POSIX systems, there may be an argument for an ioctl package
that is independent of the connections implementation, say to do things
that connections were not designed to do. For example, interfacing with
V4L2 devices usually involves many ioctl calls, an mmap call, but rarely
read or write calls. But maybe it would just be better to pipe this type
of output to R also...

-Matt

On Tue, 2010-04-20 at 16:42 -0400, Simon Urbanek wrote:

On Apr 20, 2010, at 11:51 AM, shotwelm wrote:

I've done some microcontroller work over serial also. Unfortunately, interfacing with a serial port is system dependent, and the mechanisms can be quite different, as you probably know. It appears that Simon has a solution below that will work if you are willing to accept the default baud rate (9600 is way too slow for good sensor data

[OT: define "good" ;) - good doesn't mean fast - besides it won't be any good if it is too fast to be meaningfully processed -- that's a different story, though :P - and it is trivial to change so the solution works in general]

), parity, etc.. or use external tools. On POSIX systems, you would need access to the termios.h header and the system ioctl function in order to change these settings. Although I'm not 100% sure, I don't think R has this capability ... yet.

I'm new to the list, but I'd be surprised if the R developers that have been around awhile haven't already considered adding support for ioctls and the POSIX terminal interface. This makes me wonder why it's not there. If there is no good reason, I'm starting to see a series of R packages (or core extensions) developing.

Good luck ;). The issue is that connections are inherently backend-independent which implies that packages have no access to connection internals as they can change at any time. This means that you can't enhance them without putting the enhancements into R itself. This implies that you have to make a strong case since you need a volunteer in R-core to maintain that code etc.

With a package for ioctls, we could use all sorts of cool stuff, like Video4Linux2 (webcams, HAM radio, tuners)...

Ioctls are highly system-specific which is orthogonal to the design of connections. You could probably hack together a FD-based access system but it would not be compatible with connections (unless you exploit undocumented things if possible at all ...). Also ioctls can change the stream semantics entirely thus breaking anything that deals with the FD assuming some defined state ...

When I collect sensor data over serial, I do it in python or write a small C program to dump a single-column csv. Of course, R is excellent for digital signal processing after that. Check out the DSP ( http://biostatmatt.com/archives/78 ) I did in R with some ECG data I collected with an Atmel uC.

Well, we're back to calling tools to do the interfacing like the ttys (I do prefer pipe to intermediate files)... It's not that complicated and has several benefits (implicit parallelization, process separation in case things go wrong etc.) so it is not obvious that it's a bad thing ...

I suspect that we're simply suck until the connection API is either exposed or re-written so packages can provide new connections types or extend existing one. Again, this is not trivial especially when you start messing with ioctl since it's easy to depart from defined behavior in that case ... That said, I agree that expanding connections is useful so some progress there would be desirable - but the "how" and "who" is not clear to me ...

That's just my $0.02, though ...

Cheers,
Simon

On Tue, 2010-04-20 at 11:05 -0400, Simon Urbanek wrote:

On Apr 20, 2010, at 10:33 AM, Blair Christian wrote:

Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data...

Yes (I have Arduinos reporting measurements from all sensors in the house to R on my iMac which produces plots that are synchronized with my webserver). In principle you can simply use /dev/tty.usb... and read from it. In most cases the default setting is already fine (9600,n,8,1 on Mac) or you can use tools the set it up in advance (setserial on Linux etc.) so you don't have to worry about setting up the serial from R.

Depending on your OS you may be able to read from the serial device directly with a regular file connection or you can use a pipe connection to a tool which pipes out from the tty to stdout (written for a Mac but may work on other unices):

https://svn.rforge.net/C/trunk/tools/ttys.c

and then use something like

f=pipe("ttys /dev/tty.usbserial-X1234")

A rather handy option -d prepends current time to each line so you can track output over time. I have some more tools for this (even allowing you to share form Arduino output with several computers or even send remote commands to your Arduino including encryption etc ...).

Cheers,
Simon

PS: From experience I can say that Arduinos are highly addictive so beware ;).

In base::connections documentation, it's not clear if there's an easy
way to do this?  Any ideas on hacking it?  I'm open to win/linux/mac
solutions.  I'm not sure how sockets work, but possibly there is a way
to pipe things to a buffer and read from a buffer in bash (in my linux
mind I have the thought of trying to redirect /dev/something to a
file, or symlinking a file to point to the hardware, but know that
there has to be some secret sauce to go from streaming in to a
readable file, but don't know what the missing components are).

Thoughts?