Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data... In base::connections documentation, it's not clear if there's an easy way to do this? Any ideas on hacking it? I'm open to win/linux/mac solutions. I'm not sure how sockets work, but possibly there is a way to pipe things to a buffer and read from a buffer in bash (in my linux mind I have the thought of trying to redirect /dev/something to a file, or symlinking a file to point to the hardware, but know that there has to be some secret sauce to go from streaming in to a readable file, but don't know what the missing components are). Thoughts?
Serial connections?
14 messages · Blair Christian, Philippe GROSJEAN, Simon Urbanek +3 more
On Apr 20, 2010, at 10:33 AM, Blair Christian wrote:
Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data...
Yes (I have Arduinos reporting measurements from all sensors in the house to R on my iMac which produces plots that are synchronized with my webserver). In principle you can simply use /dev/tty.usb... and read from it. In most cases the default setting is already fine (9600,n,8,1 on Mac) or you can use tools the set it up in advance (setserial on Linux etc.) so you don't have to worry about setting up the serial from R. Depending on your OS you may be able to read from the serial device directly with a regular file connection or you can use a pipe connection to a tool which pipes out from the tty to stdout (written for a Mac but may work on other unices): https://svn.rforge.net/C/trunk/tools/ttys.c and then use something like f=pipe("ttys /dev/tty.usbserial-X1234") A rather handy option -d prepends current time to each line so you can track output over time. I have some more tools for this (even allowing you to share form Arduino output with several computers or even send remote commands to your Arduino including encryption etc ...). Cheers, Simon PS: From experience I can say that Arduinos are highly addictive so beware ;).
In base::connections documentation, it's not clear if there's an easy way to do this? Any ideas on hacking it? I'm open to win/linux/mac solutions. I'm not sure how sockets work, but possibly there is a way to pipe things to a buffer and read from a buffer in bash (in my linux mind I have the thought of trying to redirect /dev/something to a file, or symlinking a file to point to the hardware, but know that there has to be some secret sauce to go from streaming in to a readable file, but don't know what the missing components are). Thoughts?
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
I've done some microcontroller work over serial also. Unfortunately, interfacing with a serial port is system dependent, and the mechanisms can be quite different, as you probably know. It appears that Simon has a solution below that will work if you are willing to accept the default baud rate (9600 is way too slow for good sensor data), parity, etc.. or use external tools. On POSIX systems, you would need access to the termios.h header and the system ioctl function in order to change these settings. Although I'm not 100% sure, I don't think R has this capability ... yet. I'm new to the list, but I'd be surprised if the R developers that have been around awhile haven't already considered adding support for ioctls and the POSIX terminal interface. This makes me wonder why it's not there. If there is no good reason, I'm starting to see a series of R packages (or core extensions) developing. With a package for ioctls, we could use all sorts of cool stuff, like Video4Linux2 (webcams, HAM radio, tuners)... When I collect sensor data over serial, I do it in python or write a small C program to dump a single-column csv. Of course, R is excellent for digital signal processing after that. Check out the DSP ( http://biostatmatt.com/archives/78 ) I did in R with some ECG data I collected with an Atmel uC. -Matt
On Tue, 2010-04-20 at 11:05 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 10:33 AM, Blair Christian wrote:
Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data...
Yes (I have Arduinos reporting measurements from all sensors in the house to R on my iMac which produces plots that are synchronized with my webserver). In principle you can simply use /dev/tty.usb... and read from it. In most cases the default setting is already fine (9600,n,8,1 on Mac) or you can use tools the set it up in advance (setserial on Linux etc.) so you don't have to worry about setting up the serial from R. Depending on your OS you may be able to read from the serial device directly with a regular file connection or you can use a pipe connection to a tool which pipes out from the tty to stdout (written for a Mac but may work on other unices): https://svn.rforge.net/C/trunk/tools/ttys.c and then use something like f=pipe("ttys /dev/tty.usbserial-X1234") A rather handy option -d prepends current time to each line so you can track output over time. I have some more tools for this (even allowing you to share form Arduino output with several computers or even send remote commands to your Arduino including encryption etc ...). Cheers, Simon PS: From experience I can say that Arduinos are highly addictive so beware ;).
In base::connections documentation, it's not clear if there's an easy way to do this? Any ideas on hacking it? I'm open to win/linux/mac solutions. I'm not sure how sockets work, but possibly there is a way to pipe things to a buffer and read from a buffer in bash (in my linux mind I have the thought of trying to redirect /dev/something to a file, or symlinking a file to point to the hardware, but know that there has to be some secret sauce to go from streaming in to a readable file, but don't know what the missing components are). Thoughts?
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
On Apr 20, 2010, at 11:51 AM, shotwelm wrote:
I've done some microcontroller work over serial also. Unfortunately, interfacing with a serial port is system dependent, and the mechanisms can be quite different, as you probably know. It appears that Simon has a solution below that will work if you are willing to accept the default baud rate (9600 is way too slow for good sensor data
[OT: define "good" ;) - good doesn't mean fast - besides it won't be any good if it is too fast to be meaningfully processed -- that's a different story, though :P - and it is trivial to change so the solution works in general]
), parity, etc.. or use external tools. On POSIX systems, you would need access to the termios.h header and the system ioctl function in order to change these settings. Although I'm not 100% sure, I don't think R has this capability ... yet. I'm new to the list, but I'd be surprised if the R developers that have been around awhile haven't already considered adding support for ioctls and the POSIX terminal interface. This makes me wonder why it's not there. If there is no good reason, I'm starting to see a series of R packages (or core extensions) developing.
Good luck ;). The issue is that connections are inherently backend-independent which implies that packages have no access to connection internals as they can change at any time. This means that you can't enhance them without putting the enhancements into R itself. This implies that you have to make a strong case since you need a volunteer in R-core to maintain that code etc.
With a package for ioctls, we could use all sorts of cool stuff, like Video4Linux2 (webcams, HAM radio, tuners)...
Ioctls are highly system-specific which is orthogonal to the design of connections. You could probably hack together a FD-based access system but it would not be compatible with connections (unless you exploit undocumented things if possible at all ...). Also ioctls can change the stream semantics entirely thus breaking anything that deals with the FD assuming some defined state ...
When I collect sensor data over serial, I do it in python or write a small C program to dump a single-column csv. Of course, R is excellent for digital signal processing after that. Check out the DSP ( http://biostatmatt.com/archives/78 ) I did in R with some ECG data I collected with an Atmel uC.
Well, we're back to calling tools to do the interfacing like the ttys (I do prefer pipe to intermediate files)... It's not that complicated and has several benefits (implicit parallelization, process separation in case things go wrong etc.) so it is not obvious that it's a bad thing ... I suspect that we're simply suck until the connection API is either exposed or re-written so packages can provide new connections types or extend existing one. Again, this is not trivial especially when you start messing with ioctl since it's easy to depart from defined behavior in that case ... That said, I agree that expanding connections is useful so some progress there would be desirable - but the "how" and "who" is not clear to me ... That's just my $0.02, though ... Cheers, Simon
On Tue, 2010-04-20 at 11:05 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 10:33 AM, Blair Christian wrote:
Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data...
Yes (I have Arduinos reporting measurements from all sensors in the house to R on my iMac which produces plots that are synchronized with my webserver). In principle you can simply use /dev/tty.usb... and read from it. In most cases the default setting is already fine (9600,n,8,1 on Mac) or you can use tools the set it up in advance (setserial on Linux etc.) so you don't have to worry about setting up the serial from R. Depending on your OS you may be able to read from the serial device directly with a regular file connection or you can use a pipe connection to a tool which pipes out from the tty to stdout (written for a Mac but may work on other unices): https://svn.rforge.net/C/trunk/tools/ttys.c and then use something like f=pipe("ttys /dev/tty.usbserial-X1234") A rather handy option -d prepends current time to each line so you can track output over time. I have some more tools for this (even allowing you to share form Arduino output with several computers or even send remote commands to your Arduino including encryption etc ...). Cheers, Simon PS: From experience I can say that Arduinos are highly addictive so beware ;).
In base::connections documentation, it's not clear if there's an easy way to do this? Any ideas on hacking it? I'm open to win/linux/mac solutions. I'm not sure how sockets work, but possibly there is a way to pipe things to a buffer and read from a buffer in bash (in my linux mind I have the thought of trying to redirect /dev/something to a file, or symlinking a file to point to the hardware, but know that there has to be some secret sauce to go from streaming in to a readable file, but don't know what the missing components are). Thoughts?
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Simon is right of course, there are plenty of sensors that would work just fine at 9600 baud (like a thermistor rigged to an ADC). There's a theorem along these lines (Nyquist sampling theorem?). I think piping the output to R is a clever solution. I added a few lines to the ttys.c program so that the baud rate is a command line option (i.e. -B9600) <http://biostatmatt.com/temp/ttys.c> and confirmed it will compile in Linux (2.6.30). Maybe it will save a step. Microcontrollers really are addictive! For an ioctl package, I was originally thinking of using file descriptors directly. However, I agree this feels like subverting what could be an extension of the connections API. Given that "everything is a file" in POSIX systems, there may be an argument for an ioctl package that is independent of the connections implementation, say to do things that connections were not designed to do. For example, interfacing with V4L2 devices usually involves many ioctl calls, an mmap call, but rarely read or write calls. But maybe it would just be better to pipe this type of output to R also... -Matt
On Tue, 2010-04-20 at 16:42 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 11:51 AM, shotwelm wrote:
I've done some microcontroller work over serial also. Unfortunately, interfacing with a serial port is system dependent, and the mechanisms can be quite different, as you probably know. It appears that Simon has a solution below that will work if you are willing to accept the default baud rate (9600 is way too slow for good sensor data
[OT: define "good" ;) - good doesn't mean fast - besides it won't be any good if it is too fast to be meaningfully processed -- that's a different story, though :P - and it is trivial to change so the solution works in general]
), parity, etc.. or use external tools. On POSIX systems, you would need access to the termios.h header and the system ioctl function in order to change these settings. Although I'm not 100% sure, I don't think R has this capability ... yet. I'm new to the list, but I'd be surprised if the R developers that have been around awhile haven't already considered adding support for ioctls and the POSIX terminal interface. This makes me wonder why it's not there. If there is no good reason, I'm starting to see a series of R packages (or core extensions) developing.
Good luck ;). The issue is that connections are inherently backend-independent which implies that packages have no access to connection internals as they can change at any time. This means that you can't enhance them without putting the enhancements into R itself. This implies that you have to make a strong case since you need a volunteer in R-core to maintain that code etc.
With a package for ioctls, we could use all sorts of cool stuff, like Video4Linux2 (webcams, HAM radio, tuners)...
Ioctls are highly system-specific which is orthogonal to the design of connections. You could probably hack together a FD-based access system but it would not be compatible with connections (unless you exploit undocumented things if possible at all ...). Also ioctls can change the stream semantics entirely thus breaking anything that deals with the FD assuming some defined state ...
When I collect sensor data over serial, I do it in python or write a small C program to dump a single-column csv. Of course, R is excellent for digital signal processing after that. Check out the DSP ( http://biostatmatt.com/archives/78 ) I did in R with some ECG data I collected with an Atmel uC.
Well, we're back to calling tools to do the interfacing like the ttys (I do prefer pipe to intermediate files)... It's not that complicated and has several benefits (implicit parallelization, process separation in case things go wrong etc.) so it is not obvious that it's a bad thing ... I suspect that we're simply suck until the connection API is either exposed or re-written so packages can provide new connections types or extend existing one. Again, this is not trivial especially when you start messing with ioctl since it's easy to depart from defined behavior in that case ... That said, I agree that expanding connections is useful so some progress there would be desirable - but the "how" and "who" is not clear to me ... That's just my $0.02, though ... Cheers, Simon
On Tue, 2010-04-20 at 11:05 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 10:33 AM, Blair Christian wrote:
Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data...
Yes (I have Arduinos reporting measurements from all sensors in the house to R on my iMac which produces plots that are synchronized with my webserver). In principle you can simply use /dev/tty.usb... and read from it. In most cases the default setting is already fine (9600,n,8,1 on Mac) or you can use tools the set it up in advance (setserial on Linux etc.) so you don't have to worry about setting up the serial from R. Depending on your OS you may be able to read from the serial device directly with a regular file connection or you can use a pipe connection to a tool which pipes out from the tty to stdout (written for a Mac but may work on other unices): https://svn.rforge.net/C/trunk/tools/ttys.c and then use something like f=pipe("ttys /dev/tty.usbserial-X1234") A rather handy option -d prepends current time to each line so you can track output over time. I have some more tools for this (even allowing you to share form Arduino output with several computers or even send remote commands to your Arduino including encryption etc ...). Cheers, Simon PS: From experience I can say that Arduinos are highly addictive so beware ;).
In base::connections documentation, it's not clear if there's an easy way to do this? Any ideas on hacking it? I'm open to win/linux/mac solutions. I'm not sure how sockets work, but possibly there is a way to pipe things to a buffer and read from a buffer in bash (in my linux mind I have the thought of trying to redirect /dev/something to a file, or symlinking a file to point to the hardware, but know that there has to be some secret sauce to go from streaming in to a readable file, but don't know what the missing components are). Thoughts?
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
There is another option I use since a couple of years to pilot
scientific devices, to program my chemostats, etc. and that is
platform-independent: Tcl.
Tcl i/o design is excellent and very robust, and Tcl/Tk is integrated in
R with the tcltk package.
It is really just a mather of a few lines of code in R to communicate
through a serial port from R using Tcl. Something like:
require(tcltk)
.Tcl('set R_com1 [open "com1" r+]') # Works on Windows too!
# There are many config parameters available here... just an example
.Tcl('fconfigure $R_com1 -mode "9600,n,8,1" -buffering none -blocking 0')
# Send a command to your device through the serial port
.Tcl('puts -nonewline $R_com1 {my_cmd}')
# Read a line of text from the serial port
line <- tclvalue(.Tcl('gets $R_com1'))
With a little bit more code, one can program Tcl to call R code
automatically everytime new data is pushed by the connected device
through the serial port. This is done using something like:
.Tcl('fileevent $R_com1 readable [list Handler_com1 $R_com1]')
Here, "Handler_com1" is a Tcl function. So, some care must be taken
using tcltk's .Tcl.callback() to trigger the event on the R side. One
way to deal with this easily is by using tclFun() from the tcltk2 package.
In tcltk2, there is also ?tclTaskSchedule that can be of interest in the
context of serial port communication to trigger a R function in the
background regularly and collect data actively from the serial port.
All these tools give me a lot a flexibility to communicate through the
serial port from R,... and most importantly, to write my code in a
portable way (tested on Windows XP and Linux Ubuntu).
If there is some interest in this approach, I could initiate a 'tclcom'
R package on R-Forge and place there the code I have.
Best,
Philippe
..............................................<?}))><........
) ) ) ) )
( ( ( ( ( Prof. Philippe Grosjean
) ) ) ) )
( ( ( ( ( Numerical Ecology of Aquatic Systems
) ) ) ) ) Mons University, Belgium
( ( ( ( (
..............................................................
On 21/04/10 03:17, shotwelm wrote:
Simon is right of course, there are plenty of sensors that would work just fine at 9600 baud (like a thermistor rigged to an ADC). There's a theorem along these lines (Nyquist sampling theorem?). I think piping the output to R is a clever solution. I added a few lines to the ttys.c program so that the baud rate is a command line option (i.e. -B9600) <http://biostatmatt.com/temp/ttys.c> and confirmed it will compile in Linux (2.6.30). Maybe it will save a step. Microcontrollers really are addictive! For an ioctl package, I was originally thinking of using file descriptors directly. However, I agree this feels like subverting what could be an extension of the connections API. Given that "everything is a file" in POSIX systems, there may be an argument for an ioctl package that is independent of the connections implementation, say to do things that connections were not designed to do. For example, interfacing with V4L2 devices usually involves many ioctl calls, an mmap call, but rarely read or write calls. But maybe it would just be better to pipe this type of output to R also... -Matt On Tue, 2010-04-20 at 16:42 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 11:51 AM, shotwelm wrote:
I've done some microcontroller work over serial also. Unfortunately, interfacing with a serial port is system dependent, and the mechanisms can be quite different, as you probably know. It appears that Simon has a solution below that will work if you are willing to accept the default baud rate (9600 is way too slow for good sensor data
[OT: define "good" ;) - good doesn't mean fast - besides it won't be any good if it is too fast to be meaningfully processed -- that's a different story, though :P - and it is trivial to change so the solution works in general]
), parity, etc.. or use external tools. On POSIX systems, you would need access to the termios.h header and the system ioctl function in order to change these settings. Although I'm not 100% sure, I don't think R has this capability ... yet. I'm new to the list, but I'd be surprised if the R developers that have been around awhile haven't already considered adding support for ioctls and the POSIX terminal interface. This makes me wonder why it's not there. If there is no good reason, I'm starting to see a series of R packages (or core extensions) developing.
Good luck ;). The issue is that connections are inherently backend-independent which implies that packages have no access to connection internals as they can change at any time. This means that you can't enhance them without putting the enhancements into R itself. This implies that you have to make a strong case since you need a volunteer in R-core to maintain that code etc.
With a package for ioctls, we could use all sorts of cool stuff, like Video4Linux2 (webcams, HAM radio, tuners)...
Ioctls are highly system-specific which is orthogonal to the design of connections. You could probably hack together a FD-based access system but it would not be compatible with connections (unless you exploit undocumented things if possible at all ...). Also ioctls can change the stream semantics entirely thus breaking anything that deals with the FD assuming some defined state ...
When I collect sensor data over serial, I do it in python or write a small C program to dump a single-column csv. Of course, R is excellent for digital signal processing after that. Check out the DSP ( http://biostatmatt.com/archives/78 ) I did in R with some ECG data I collected with an Atmel uC.
Well, we're back to calling tools to do the interfacing like the ttys (I do prefer pipe to intermediate files)... It's not that complicated and has several benefits (implicit parallelization, process separation in case things go wrong etc.) so it is not obvious that it's a bad thing ... I suspect that we're simply suck until the connection API is either exposed or re-written so packages can provide new connections types or extend existing one. Again, this is not trivial especially when you start messing with ioctl since it's easy to depart from defined behavior in that case ... That said, I agree that expanding connections is useful so some progress there would be desirable - but the "how" and "who" is not clear to me ... That's just my $0.02, though ... Cheers, Simon
On Tue, 2010-04-20 at 11:05 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 10:33 AM, Blair Christian wrote:
Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data...
Yes (I have Arduinos reporting measurements from all sensors in the house to R on my iMac which produces plots that are synchronized with my webserver). In principle you can simply use /dev/tty.usb... and read from it. In most cases the default setting is already fine (9600,n,8,1 on Mac) or you can use tools the set it up in advance (setserial on Linux etc.) so you don't have to worry about setting up the serial from R. Depending on your OS you may be able to read from the serial device directly with a regular file connection or you can use a pipe connection to a tool which pipes out from the tty to stdout (written for a Mac but may work on other unices): https://svn.rforge.net/C/trunk/tools/ttys.c and then use something like f=pipe("ttys /dev/tty.usbserial-X1234") A rather handy option -d prepends current time to each line so you can track output over time. I have some more tools for this (even allowing you to share form Arduino output with several computers or even send remote commands to your Arduino including encryption etc ...). Cheers, Simon PS: From experience I can say that Arduinos are highly addictive so beware ;).
In base::connections documentation, it's not clear if there's an easy way to do this? Any ideas on hacking it? I'm open to win/linux/mac solutions. I'm not sure how sockets work, but possibly there is a way to pipe things to a buffer and read from a buffer in bash (in my linux mind I have the thought of trying to redirect /dev/something to a file, or symlinking a file to point to the hardware, but know that there has to be some secret sauce to go from streaming in to a readable file, but don't know what the missing components are). Thoughts?
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Philippe, unfortunately that approach has one major drawback - it does not give you a connection. As I said in the previous e-mail it is fairly easy to talk to a tty directly, but the challenge is to turn it into a connection. I don't like the Tcl approach for several reasons, one of them being that it's entirely unnatural since you have to construct strings of code -- you could as well use rJava and Java serial connections which has a little more friendly syntax (but I wouldn't recommend that, either) or any other language R interfaces to. I was thinking about it a bit and I may be tempted to have a dab at a tty connection, but I still would not want to mess with ioctl (the other part Matt mentioned). Cheers, Simon
On Apr 21, 2010, at 6:30 AM, Philippe Grosjean wrote:
There is another option I use since a couple of years to pilot scientific devices, to program my chemostats, etc. and that is platform-independent: Tcl.
Tcl i/o design is excellent and very robust, and Tcl/Tk is integrated in R with the tcltk package.
It is really just a mather of a few lines of code in R to communicate through a serial port from R using Tcl. Something like:
require(tcltk)
.Tcl('set R_com1 [open "com1" r+]') # Works on Windows too!
# There are many config parameters available here... just an example
.Tcl('fconfigure $R_com1 -mode "9600,n,8,1" -buffering none -blocking 0')
# Send a command to your device through the serial port
.Tcl('puts -nonewline $R_com1 {my_cmd}')
# Read a line of text from the serial port
line <- tclvalue(.Tcl('gets $R_com1'))
With a little bit more code, one can program Tcl to call R code automatically everytime new data is pushed by the connected device through the serial port. This is done using something like:
.Tcl('fileevent $R_com1 readable [list Handler_com1 $R_com1]')
Here, "Handler_com1" is a Tcl function. So, some care must be taken using tcltk's .Tcl.callback() to trigger the event on the R side. One way to deal with this easily is by using tclFun() from the tcltk2 package.
In tcltk2, there is also ?tclTaskSchedule that can be of interest in the context of serial port communication to trigger a R function in the background regularly and collect data actively from the serial port.
All these tools give me a lot a flexibility to communicate through the serial port from R,... and most importantly, to write my code in a portable way (tested on Windows XP and Linux Ubuntu).
If there is some interest in this approach, I could initiate a 'tclcom' R package on R-Forge and place there the code I have.
Best,
Philippe
..............................................<?}))><........
) ) ) ) )
( ( ( ( ( Prof. Philippe Grosjean
) ) ) ) )
( ( ( ( ( Numerical Ecology of Aquatic Systems
) ) ) ) ) Mons University, Belgium
( ( ( ( (
..............................................................
On 21/04/10 03:17, shotwelm wrote:
Simon is right of course, there are plenty of sensors that would work just fine at 9600 baud (like a thermistor rigged to an ADC). There's a theorem along these lines (Nyquist sampling theorem?). I think piping the output to R is a clever solution. I added a few lines to the ttys.c program so that the baud rate is a command line option (i.e. -B9600) <http://biostatmatt.com/temp/ttys.c> and confirmed it will compile in Linux (2.6.30). Maybe it will save a step. Microcontrollers really are addictive! For an ioctl package, I was originally thinking of using file descriptors directly. However, I agree this feels like subverting what could be an extension of the connections API. Given that "everything is a file" in POSIX systems, there may be an argument for an ioctl package that is independent of the connections implementation, say to do things that connections were not designed to do. For example, interfacing with V4L2 devices usually involves many ioctl calls, an mmap call, but rarely read or write calls. But maybe it would just be better to pipe this type of output to R also... -Matt On Tue, 2010-04-20 at 16:42 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 11:51 AM, shotwelm wrote:
I've done some microcontroller work over serial also. Unfortunately, interfacing with a serial port is system dependent, and the mechanisms can be quite different, as you probably know. It appears that Simon has a solution below that will work if you are willing to accept the default baud rate (9600 is way too slow for good sensor data
[OT: define "good" ;) - good doesn't mean fast - besides it won't be any good if it is too fast to be meaningfully processed -- that's a different story, though :P - and it is trivial to change so the solution works in general]
), parity, etc.. or use external tools. On POSIX systems, you would need access to the termios.h header and the system ioctl function in order to change these settings. Although I'm not 100% sure, I don't think R has this capability ... yet. I'm new to the list, but I'd be surprised if the R developers that have been around awhile haven't already considered adding support for ioctls and the POSIX terminal interface. This makes me wonder why it's not there. If there is no good reason, I'm starting to see a series of R packages (or core extensions) developing.
Good luck ;). The issue is that connections are inherently backend-independent which implies that packages have no access to connection internals as they can change at any time. This means that you can't enhance them without putting the enhancements into R itself. This implies that you have to make a strong case since you need a volunteer in R-core to maintain that code etc.
With a package for ioctls, we could use all sorts of cool stuff, like Video4Linux2 (webcams, HAM radio, tuners)...
Ioctls are highly system-specific which is orthogonal to the design of connections. You could probably hack together a FD-based access system but it would not be compatible with connections (unless you exploit undocumented things if possible at all ...). Also ioctls can change the stream semantics entirely thus breaking anything that deals with the FD assuming some defined state ...
When I collect sensor data over serial, I do it in python or write a small C program to dump a single-column csv. Of course, R is excellent for digital signal processing after that. Check out the DSP ( http://biostatmatt.com/archives/78 ) I did in R with some ECG data I collected with an Atmel uC.
Well, we're back to calling tools to do the interfacing like the ttys (I do prefer pipe to intermediate files)... It's not that complicated and has several benefits (implicit parallelization, process separation in case things go wrong etc.) so it is not obvious that it's a bad thing ... I suspect that we're simply suck until the connection API is either exposed or re-written so packages can provide new connections types or extend existing one. Again, this is not trivial especially when you start messing with ioctl since it's easy to depart from defined behavior in that case ... That said, I agree that expanding connections is useful so some progress there would be desirable - but the "how" and "who" is not clear to me ... That's just my $0.02, though ... Cheers, Simon
On Tue, 2010-04-20 at 11:05 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 10:33 AM, Blair Christian wrote:
Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data...
Yes (I have Arduinos reporting measurements from all sensors in the house to R on my iMac which produces plots that are synchronized with my webserver). In principle you can simply use /dev/tty.usb... and read from it. In most cases the default setting is already fine (9600,n,8,1 on Mac) or you can use tools the set it up in advance (setserial on Linux etc.) so you don't have to worry about setting up the serial from R. Depending on your OS you may be able to read from the serial device directly with a regular file connection or you can use a pipe connection to a tool which pipes out from the tty to stdout (written for a Mac but may work on other unices): https://svn.rforge.net/C/trunk/tools/ttys.c and then use something like f=pipe("ttys /dev/tty.usbserial-X1234") A rather handy option -d prepends current time to each line so you can track output over time. I have some more tools for this (even allowing you to share form Arduino output with several computers or even send remote commands to your Arduino including encryption etc ...). Cheers, Simon PS: From experience I can say that Arduinos are highly addictive so beware ;).
In base::connections documentation, it's not clear if there's an easy way to do this? Any ideas on hacking it? I'm open to win/linux/mac solutions. I'm not sure how sockets work, but possibly there is a way to pipe things to a buffer and read from a buffer in bash (in my linux mind I have the thought of trying to redirect /dev/something to a file, or symlinking a file to point to the hardware, but know that there has to be some secret sauce to go from streaming in to a readable file, but don't know what the missing components are). Thoughts?
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Simon, I see what you mean, and I agree it would be nice to have a connection. That would be the natural way in S. Yet, connections are driven by R. You cannot open a connection and let the connected device trigger some R code when data is available, don't you? Otherwise, I don't understand your problem with "strings of code". A .R file also contains a series of "strings of code" interpreted by the R parser when you source() it. Just that code happens to be S. Here the code is Tcl. You can source as well a .tcl file with the same code if you prefer... Best, Philippe
On 21/04/10 14:07, Simon Urbanek wrote:
Philippe, unfortunately that approach has one major drawback - it does not give you a connection. As I said in the previous e-mail it is fairly easy to talk to a tty directly, but the challenge is to turn it into a connection. I don't like the Tcl approach for several reasons, one of them being that it's entirely unnatural since you have to construct strings of code -- you could as well use rJava and Java serial connections which has a little more friendly syntax (but I wouldn't recommend that, either) or any other language R interfaces to. I was thinking about it a bit and I may be tempted to have a dab at a tty connection, but I still would not want to mess with ioctl (the other part Matt mentioned). Cheers, Simon On Apr 21, 2010, at 6:30 AM, Philippe Grosjean wrote:
There is another option I use since a couple of years to pilot scientific devices, to program my chemostats, etc. and that is platform-independent: Tcl.
Tcl i/o design is excellent and very robust, and Tcl/Tk is integrated in R with the tcltk package.
It is really just a mather of a few lines of code in R to communicate through a serial port from R using Tcl. Something like:
require(tcltk)
.Tcl('set R_com1 [open "com1" r+]') # Works on Windows too!
# There are many config parameters available here... just an example
.Tcl('fconfigure $R_com1 -mode "9600,n,8,1" -buffering none -blocking 0')
# Send a command to your device through the serial port
.Tcl('puts -nonewline $R_com1 {my_cmd}')
# Read a line of text from the serial port
line<- tclvalue(.Tcl('gets $R_com1'))
With a little bit more code, one can program Tcl to call R code automatically everytime new data is pushed by the connected device through the serial port. This is done using something like:
.Tcl('fileevent $R_com1 readable [list Handler_com1 $R_com1]')
Here, "Handler_com1" is a Tcl function. So, some care must be taken using tcltk's .Tcl.callback() to trigger the event on the R side. One way to deal with this easily is by using tclFun() from the tcltk2 package.
In tcltk2, there is also ?tclTaskSchedule that can be of interest in the context of serial port communication to trigger a R function in the background regularly and collect data actively from the serial port.
All these tools give me a lot a flexibility to communicate through the serial port from R,... and most importantly, to write my code in a portable way (tested on Windows XP and Linux Ubuntu).
If there is some interest in this approach, I could initiate a 'tclcom' R package on R-Forge and place there the code I have.
Best,
Philippe
..............................................<?}))><........
) ) ) ) )
( ( ( ( ( Prof. Philippe Grosjean
) ) ) ) )
( ( ( ( ( Numerical Ecology of Aquatic Systems
) ) ) ) ) Mons University, Belgium
( ( ( ( (
..............................................................
On 21/04/10 03:17, shotwelm wrote:
Simon is right of course, there are plenty of sensors that would work just fine at 9600 baud (like a thermistor rigged to an ADC). There's a theorem along these lines (Nyquist sampling theorem?). I think piping the output to R is a clever solution. I added a few lines to the ttys.c program so that the baud rate is a command line option (i.e. -B9600) <http://biostatmatt.com/temp/ttys.c> and confirmed it will compile in Linux (2.6.30). Maybe it will save a step. Microcontrollers really are addictive! For an ioctl package, I was originally thinking of using file descriptors directly. However, I agree this feels like subverting what could be an extension of the connections API. Given that "everything is a file" in POSIX systems, there may be an argument for an ioctl package that is independent of the connections implementation, say to do things that connections were not designed to do. For example, interfacing with V4L2 devices usually involves many ioctl calls, an mmap call, but rarely read or write calls. But maybe it would just be better to pipe this type of output to R also... -Matt On Tue, 2010-04-20 at 16:42 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 11:51 AM, shotwelm wrote:
I've done some microcontroller work over serial also. Unfortunately, interfacing with a serial port is system dependent, and the mechanisms can be quite different, as you probably know. It appears that Simon has a solution below that will work if you are willing to accept the default baud rate (9600 is way too slow for good sensor data
[OT: define "good" ;) - good doesn't mean fast - besides it won't be any good if it is too fast to be meaningfully processed -- that's a different story, though :P - and it is trivial to change so the solution works in general]
), parity, etc.. or use external tools. On POSIX systems, you would need access to the termios.h header and the system ioctl function in order to change these settings. Although I'm not 100% sure, I don't think R has this capability ... yet. I'm new to the list, but I'd be surprised if the R developers that have been around awhile haven't already considered adding support for ioctls and the POSIX terminal interface. This makes me wonder why it's not there. If there is no good reason, I'm starting to see a series of R packages (or core extensions) developing.
Good luck ;). The issue is that connections are inherently backend-independent which implies that packages have no access to connection internals as they can change at any time. This means that you can't enhance them without putting the enhancements into R itself. This implies that you have to make a strong case since you need a volunteer in R-core to maintain that code etc.
With a package for ioctls, we could use all sorts of cool stuff, like Video4Linux2 (webcams, HAM radio, tuners)...
Ioctls are highly system-specific which is orthogonal to the design of connections. You could probably hack together a FD-based access system but it would not be compatible with connections (unless you exploit undocumented things if possible at all ...). Also ioctls can change the stream semantics entirely thus breaking anything that deals with the FD assuming some defined state ...
When I collect sensor data over serial, I do it in python or write a small C program to dump a single-column csv. Of course, R is excellent for digital signal processing after that. Check out the DSP ( http://biostatmatt.com/archives/78 ) I did in R with some ECG data I collected with an Atmel uC.
Well, we're back to calling tools to do the interfacing like the ttys (I do prefer pipe to intermediate files)... It's not that complicated and has several benefits (implicit parallelization, process separation in case things go wrong etc.) so it is not obvious that it's a bad thing ... I suspect that we're simply suck until the connection API is either exposed or re-written so packages can provide new connections types or extend existing one. Again, this is not trivial especially when you start messing with ioctl since it's easy to depart from defined behavior in that case ... That said, I agree that expanding connections is useful so some progress there would be desirable - but the "how" and "who" is not clear to me ... That's just my $0.02, though ... Cheers, Simon
On Tue, 2010-04-20 at 11:05 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 10:33 AM, Blair Christian wrote:
Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data...
Yes (I have Arduinos reporting measurements from all sensors in the house to R on my iMac which produces plots that are synchronized with my webserver). In principle you can simply use /dev/tty.usb... and read from it. In most cases the default setting is already fine (9600,n,8,1 on Mac) or you can use tools the set it up in advance (setserial on Linux etc.) so you don't have to worry about setting up the serial from R. Depending on your OS you may be able to read from the serial device directly with a regular file connection or you can use a pipe connection to a tool which pipes out from the tty to stdout (written for a Mac but may work on other unices): https://svn.rforge.net/C/trunk/tools/ttys.c and then use something like f=pipe("ttys /dev/tty.usbserial-X1234") A rather handy option -d prepends current time to each line so you can track output over time. I have some more tools for this (even allowing you to share form Arduino output with several computers or even send remote commands to your Arduino including encryption etc ...). Cheers, Simon PS: From experience I can say that Arduinos are highly addictive so beware ;).
In base::connections documentation, it's not clear if there's an easy way to do this? Any ideas on hacking it? I'm open to win/linux/mac solutions. I'm not sure how sockets work, but possibly there is a way to pipe things to a buffer and read from a buffer in bash (in my linux mind I have the thought of trying to redirect /dev/something to a file, or symlinking a file to point to the hardware, but know that there has to be some secret sauce to go from streaming in to a readable file, but don't know what the missing components are). Thoughts?
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
5 days later
All, Our discussion of serial interfaces last week motivated me to dig into the R connections API. In short, I spent the weekend writing a patch for version 2.11.0 that adds a serial connection. I posted a blog entry that gives instructions for applying, configuring, and compiling the patched version <http://biostatmatt.com/archives/112>. The patch currently only enables a serial connection for POSIX compliant OSs. It should NOT be considered well tested, though I feel it is fairly safe. I would really like to hear from someone who can try it on Mac OS X (and Windows to ensure that it has the correct 'null' behavior). Briefly, the connection utilizes (and checks for) the termios.h header. I opted not to extend the 'file' connection in a manner similar to the 'pipe' connection because this might not be suitable under Win32. Hence, there are new read, write, flush, open, and close methods and a struct serialconn for the serial connection. Opens are non-blocking and binary only. Serial parameters can be passed when the connection is created. I overloaded the summary method in order for the user to see the serial settings. I've done my best to be consistent and conforming with the connections API and coding style. However, I would be great if someone with more experience with the R API (Simon? :)) would take a critical look at the design and coding of the new connection. If this is something that the R community would like to pursue, I'd like to work on support for Windows serial interfaces. Below is a link to the patch file and the output of diffstat. <http://biostatmatt.com/R/R-2.11.0-serial.patch> file | mod ins del ------------------------------------------------------------ configure.ac | 2 1 + 1 - src/include/Internal.h | 2 2 + 0 - src/library/base/R/connections.R | 10 10 + 0 - src/library/base/man/connections.Rd | 60 49 + 11 - src/main/connections.c | 379 379 + 0 - src/main/names.c | 2 2 + 0 - P.S. connections.c is the largest file in src/main/, before and after the patch. Making the 379 additions increased the file size by 6.9% (10.7kB). Adding Windows support should increase the file size by a lesser amount. -Matt
On Wed, 2010-04-21 at 08:07 -0400, Simon Urbanek wrote:
Philippe, unfortunately that approach has one major drawback - it does not give you a connection. As I said in the previous e-mail it is fairly easy to talk to a tty directly, but the challenge is to turn it into a connection. I don't like the Tcl approach for several reasons, one of them being that it's entirely unnatural since you have to construct strings of code -- you could as well use rJava and Java serial connections which has a little more friendly syntax (but I wouldn't recommend that, either) or any other language R interfaces to. I was thinking about it a bit and I may be tempted to have a dab at a tty connection, but I still would not want to mess with ioctl (the other part Matt mentioned). Cheers, Simon On Apr 21, 2010, at 6:30 AM, Philippe Grosjean wrote:
There is another option I use since a couple of years to pilot scientific devices, to program my chemostats, etc. and that is platform-independent: Tcl.
Tcl i/o design is excellent and very robust, and Tcl/Tk is integrated in R with the tcltk package.
It is really just a mather of a few lines of code in R to communicate through a serial port from R using Tcl. Something like:
require(tcltk)
.Tcl('set R_com1 [open "com1" r+]') # Works on Windows too!
# There are many config parameters available here... just an example
.Tcl('fconfigure $R_com1 -mode "9600,n,8,1" -buffering none -blocking 0')
# Send a command to your device through the serial port
.Tcl('puts -nonewline $R_com1 {my_cmd}')
# Read a line of text from the serial port
line <- tclvalue(.Tcl('gets $R_com1'))
With a little bit more code, one can program Tcl to call R code automatically everytime new data is pushed by the connected device through the serial port. This is done using something like:
.Tcl('fileevent $R_com1 readable [list Handler_com1 $R_com1]')
Here, "Handler_com1" is a Tcl function. So, some care must be taken using tcltk's .Tcl.callback() to trigger the event on the R side. One way to deal with this easily is by using tclFun() from the tcltk2 package.
In tcltk2, there is also ?tclTaskSchedule that can be of interest in the context of serial port communication to trigger a R function in the background regularly and collect data actively from the serial port.
All these tools give me a lot a flexibility to communicate through the serial port from R,... and most importantly, to write my code in a portable way (tested on Windows XP and Linux Ubuntu).
If there is some interest in this approach, I could initiate a 'tclcom' R package on R-Forge and place there the code I have.
Best,
Philippe
..............................................<?}))><........
) ) ) ) )
( ( ( ( ( Prof. Philippe Grosjean
) ) ) ) )
( ( ( ( ( Numerical Ecology of Aquatic Systems
) ) ) ) ) Mons University, Belgium
( ( ( ( (
..............................................................
On 21/04/10 03:17, shotwelm wrote:
Simon is right of course, there are plenty of sensors that would work just fine at 9600 baud (like a thermistor rigged to an ADC). There's a theorem along these lines (Nyquist sampling theorem?). I think piping the output to R is a clever solution. I added a few lines to the ttys.c program so that the baud rate is a command line option (i.e. -B9600) <http://biostatmatt.com/temp/ttys.c> and confirmed it will compile in Linux (2.6.30). Maybe it will save a step. Microcontrollers really are addictive! For an ioctl package, I was originally thinking of using file descriptors directly. However, I agree this feels like subverting what could be an extension of the connections API. Given that "everything is a file" in POSIX systems, there may be an argument for an ioctl package that is independent of the connections implementation, say to do things that connections were not designed to do. For example, interfacing with V4L2 devices usually involves many ioctl calls, an mmap call, but rarely read or write calls. But maybe it would just be better to pipe this type of output to R also... -Matt On Tue, 2010-04-20 at 16:42 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 11:51 AM, shotwelm wrote:
I've done some microcontroller work over serial also. Unfortunately, interfacing with a serial port is system dependent, and the mechanisms can be quite different, as you probably know. It appears that Simon has a solution below that will work if you are willing to accept the default baud rate (9600 is way too slow for good sensor data
[OT: define "good" ;) - good doesn't mean fast - besides it won't be any good if it is too fast to be meaningfully processed -- that's a different story, though :P - and it is trivial to change so the solution works in general]
), parity, etc.. or use external tools. On POSIX systems, you would need access to the termios.h header and the system ioctl function in order to change these settings. Although I'm not 100% sure, I don't think R has this capability ... yet. I'm new to the list, but I'd be surprised if the R developers that have been around awhile haven't already considered adding support for ioctls and the POSIX terminal interface. This makes me wonder why it's not there. If there is no good reason, I'm starting to see a series of R packages (or core extensions) developing.
Good luck ;). The issue is that connections are inherently backend-independent which implies that packages have no access to connection internals as they can change at any time. This means that you can't enhance them without putting the enhancements into R itself. This implies that you have to make a strong case since you need a volunteer in R-core to maintain that code etc.
With a package for ioctls, we could use all sorts of cool stuff, like Video4Linux2 (webcams, HAM radio, tuners)...
Ioctls are highly system-specific which is orthogonal to the design of connections. You could probably hack together a FD-based access system but it would not be compatible with connections (unless you exploit undocumented things if possible at all ...). Also ioctls can change the stream semantics entirely thus breaking anything that deals with the FD assuming some defined state ...
When I collect sensor data over serial, I do it in python or write a small C program to dump a single-column csv. Of course, R is excellent for digital signal processing after that. Check out the DSP ( http://biostatmatt.com/archives/78 ) I did in R with some ECG data I collected with an Atmel uC.
Well, we're back to calling tools to do the interfacing like the ttys (I do prefer pipe to intermediate files)... It's not that complicated and has several benefits (implicit parallelization, process separation in case things go wrong etc.) so it is not obvious that it's a bad thing ... I suspect that we're simply suck until the connection API is either exposed or re-written so packages can provide new connections types or extend existing one. Again, this is not trivial especially when you start messing with ioctl since it's easy to depart from defined behavior in that case ... That said, I agree that expanding connections is useful so some progress there would be desirable - but the "how" and "who" is not clear to me ... That's just my $0.02, though ... Cheers, Simon
On Tue, 2010-04-20 at 11:05 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 10:33 AM, Blair Christian wrote:
Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data...
Yes (I have Arduinos reporting measurements from all sensors in the house to R on my iMac which produces plots that are synchronized with my webserver). In principle you can simply use /dev/tty.usb... and read from it. In most cases the default setting is already fine (9600,n,8,1 on Mac) or you can use tools the set it up in advance (setserial on Linux etc.) so you don't have to worry about setting up the serial from R. Depending on your OS you may be able to read from the serial device directly with a regular file connection or you can use a pipe connection to a tool which pipes out from the tty to stdout (written for a Mac but may work on other unices): https://svn.rforge.net/C/trunk/tools/ttys.c and then use something like f=pipe("ttys /dev/tty.usbserial-X1234") A rather handy option -d prepends current time to each line so you can track output over time. I have some more tools for this (even allowing you to share form Arduino output with several computers or even send remote commands to your Arduino including encryption etc ...). Cheers, Simon PS: From experience I can say that Arduinos are highly addictive so beware ;).
In base::connections documentation, it's not clear if there's an easy way to do this? Any ideas on hacking it? I'm open to win/linux/mac solutions. I'm not sure how sockets work, but possibly there is a way to pipe things to a buffer and read from a buffer in bash (in my linux mind I have the thought of trying to redirect /dev/something to a file, or symlinking a file to point to the hardware, but know that there has to be some secret sauce to go from streaming in to a readable file, but don't know what the missing components are). Thoughts?
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Matt, thanks for you efforts. We cannot directly use your patch due to licensing issues (please note the licenses in the files you are modifying - I don't think anyone can currently use the patch and redistribute the resulting R; also/or we may possibly need to clarify that the work is either done on behalf of the R-foundation or the copyright is assigned to R-foundation or something similar so future licensing issues can be solved by the Foundation since that is the copyright holder of most of R). Apart from that, the overall structure looks good, but the devil is in the details. It does not quite work for me on OS X (any read/writes just hang in the same way that it would if you used a file connection on the tty device) - I don't think it is as simple (AFAIR you have to make sure you open it such that it doesn't become the controlling terminal). Also it would definitely need "blocking" argument like most other connections have. In general I don't think serial is the right name (and scope) here - what we're really looking for a is a tty connection (and serial just an application of it). What you have so far is a file connection with setserial functionality on it - and I think that may be more easily done by really using setserial on the already implemented file connection. The difference with tty is the other parts such as flow control, modes etc. It's much more work to expose those, but I think that is what the direction would be. In fact it could well be that tty is simply file connection + termio so you don't have to re-invent the wheel (under the hood). Also that would give you many aspects for free since you could inherit from file at the R level. I hope it helps. I was thinking about the above but in the end decided to back off once I was looking at the flood of parameters you have to take care of in tty. But it would be nice if someone did that ;). Cheers, Simon
On Apr 26, 2010, at 12:28 PM, Matt Shotwell wrote:
All, Our discussion of serial interfaces last week motivated me to dig into the R connections API. In short, I spent the weekend writing a patch for version 2.11.0 that adds a serial connection. I posted a blog entry that gives instructions for applying, configuring, and compiling the patched version <http://biostatmatt.com/archives/112>. The patch currently only enables a serial connection for POSIX compliant OSs. It should NOT be considered well tested, though I feel it is fairly safe. I would really like to hear from someone who can try it on Mac OS X (and Windows to ensure that it has the correct 'null' behavior). Briefly, the connection utilizes (and checks for) the termios.h header. I opted not to extend the 'file' connection in a manner similar to the 'pipe' connection because this might not be suitable under Win32. Hence, there are new read, write, flush, open, and close methods and a struct serialconn for the serial connection. Opens are non-blocking and binary only. Serial parameters can be passed when the connection is created. I overloaded the summary method in order for the user to see the serial settings. I've done my best to be consistent and conforming with the connections API and coding style. However, I would be great if someone with more experience with the R API (Simon? :)) would take a critical look at the design and coding of the new connection. If this is something that the R community would like to pursue, I'd like to work on support for Windows serial interfaces. Below is a link to the patch file and the output of diffstat. <http://biostatmatt.com/R/R-2.11.0-serial.patch> file | mod ins del ------------------------------------------------------------ configure.ac | 2 1 + 1 - src/include/Internal.h | 2 2 + 0 - src/library/base/R/connections.R | 10 10 + 0 - src/library/base/man/connections.Rd | 60 49 + 11 - src/main/connections.c | 379 379 + 0 - src/main/names.c | 2 2 + 0 - P.S. connections.c is the largest file in src/main/, before and after the patch. Making the 379 additions increased the file size by 6.9% (10.7kB). Adding Windows support should increase the file size by a lesser amount. -Matt On Wed, 2010-04-21 at 08:07 -0400, Simon Urbanek wrote:
Philippe, unfortunately that approach has one major drawback - it does not give you a connection. As I said in the previous e-mail it is fairly easy to talk to a tty directly, but the challenge is to turn it into a connection. I don't like the Tcl approach for several reasons, one of them being that it's entirely unnatural since you have to construct strings of code -- you could as well use rJava and Java serial connections which has a little more friendly syntax (but I wouldn't recommend that, either) or any other language R interfaces to. I was thinking about it a bit and I may be tempted to have a dab at a tty connection, but I still would not want to mess with ioctl (the other part Matt mentioned). Cheers, Simon On Apr 21, 2010, at 6:30 AM, Philippe Grosjean wrote:
There is another option I use since a couple of years to pilot scientific devices, to program my chemostats, etc. and that is platform-independent: Tcl.
Tcl i/o design is excellent and very robust, and Tcl/Tk is integrated in R with the tcltk package.
It is really just a mather of a few lines of code in R to communicate through a serial port from R using Tcl. Something like:
require(tcltk)
.Tcl('set R_com1 [open "com1" r+]') # Works on Windows too!
# There are many config parameters available here... just an example
.Tcl('fconfigure $R_com1 -mode "9600,n,8,1" -buffering none -blocking 0')
# Send a command to your device through the serial port
.Tcl('puts -nonewline $R_com1 {my_cmd}')
# Read a line of text from the serial port
line <- tclvalue(.Tcl('gets $R_com1'))
With a little bit more code, one can program Tcl to call R code automatically everytime new data is pushed by the connected device through the serial port. This is done using something like:
.Tcl('fileevent $R_com1 readable [list Handler_com1 $R_com1]')
Here, "Handler_com1" is a Tcl function. So, some care must be taken using tcltk's .Tcl.callback() to trigger the event on the R side. One way to deal with this easily is by using tclFun() from the tcltk2 package.
In tcltk2, there is also ?tclTaskSchedule that can be of interest in the context of serial port communication to trigger a R function in the background regularly and collect data actively from the serial port.
All these tools give me a lot a flexibility to communicate through the serial port from R,... and most importantly, to write my code in a portable way (tested on Windows XP and Linux Ubuntu).
If there is some interest in this approach, I could initiate a 'tclcom' R package on R-Forge and place there the code I have.
Best,
Philippe
..............................................<?}))><........
) ) ) ) )
( ( ( ( ( Prof. Philippe Grosjean
) ) ) ) )
( ( ( ( ( Numerical Ecology of Aquatic Systems
) ) ) ) ) Mons University, Belgium
( ( ( ( (
..............................................................
On 21/04/10 03:17, shotwelm wrote:
Simon is right of course, there are plenty of sensors that would work just fine at 9600 baud (like a thermistor rigged to an ADC). There's a theorem along these lines (Nyquist sampling theorem?). I think piping the output to R is a clever solution. I added a few lines to the ttys.c program so that the baud rate is a command line option (i.e. -B9600) <http://biostatmatt.com/temp/ttys.c> and confirmed it will compile in Linux (2.6.30). Maybe it will save a step. Microcontrollers really are addictive! For an ioctl package, I was originally thinking of using file descriptors directly. However, I agree this feels like subverting what could be an extension of the connections API. Given that "everything is a file" in POSIX systems, there may be an argument for an ioctl package that is independent of the connections implementation, say to do things that connections were not designed to do. For example, interfacing with V4L2 devices usually involves many ioctl calls, an mmap call, but rarely read or write calls. But maybe it would just be better to pipe this type of output to R also... -Matt On Tue, 2010-04-20 at 16:42 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 11:51 AM, shotwelm wrote:
I've done some microcontroller work over serial also. Unfortunately, interfacing with a serial port is system dependent, and the mechanisms can be quite different, as you probably know. It appears that Simon has a solution below that will work if you are willing to accept the default baud rate (9600 is way too slow for good sensor data
[OT: define "good" ;) - good doesn't mean fast - besides it won't be any good if it is too fast to be meaningfully processed -- that's a different story, though :P - and it is trivial to change so the solution works in general]
), parity, etc.. or use external tools. On POSIX systems, you would need access to the termios.h header and the system ioctl function in order to change these settings. Although I'm not 100% sure, I don't think R has this capability ... yet. I'm new to the list, but I'd be surprised if the R developers that have been around awhile haven't already considered adding support for ioctls and the POSIX terminal interface. This makes me wonder why it's not there. If there is no good reason, I'm starting to see a series of R packages (or core extensions) developing.
Good luck ;). The issue is that connections are inherently backend-independent which implies that packages have no access to connection internals as they can change at any time. This means that you can't enhance them without putting the enhancements into R itself. This implies that you have to make a strong case since you need a volunteer in R-core to maintain that code etc.
With a package for ioctls, we could use all sorts of cool stuff, like Video4Linux2 (webcams, HAM radio, tuners)...
Ioctls are highly system-specific which is orthogonal to the design of connections. You could probably hack together a FD-based access system but it would not be compatible with connections (unless you exploit undocumented things if possible at all ...). Also ioctls can change the stream semantics entirely thus breaking anything that deals with the FD assuming some defined state ...
When I collect sensor data over serial, I do it in python or write a small C program to dump a single-column csv. Of course, R is excellent for digital signal processing after that. Check out the DSP ( http://biostatmatt.com/archives/78 ) I did in R with some ECG data I collected with an Atmel uC.
Well, we're back to calling tools to do the interfacing like the ttys (I do prefer pipe to intermediate files)... It's not that complicated and has several benefits (implicit parallelization, process separation in case things go wrong etc.) so it is not obvious that it's a bad thing ... I suspect that we're simply suck until the connection API is either exposed or re-written so packages can provide new connections types or extend existing one. Again, this is not trivial especially when you start messing with ioctl since it's easy to depart from defined behavior in that case ... That said, I agree that expanding connections is useful so some progress there would be desirable - but the "how" and "who" is not clear to me ... That's just my $0.02, though ... Cheers, Simon
On Tue, 2010-04-20 at 11:05 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 10:33 AM, Blair Christian wrote:
Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data...
Yes (I have Arduinos reporting measurements from all sensors in the house to R on my iMac which produces plots that are synchronized with my webserver). In principle you can simply use /dev/tty.usb... and read from it. In most cases the default setting is already fine (9600,n,8,1 on Mac) or you can use tools the set it up in advance (setserial on Linux etc.) so you don't have to worry about setting up the serial from R. Depending on your OS you may be able to read from the serial device directly with a regular file connection or you can use a pipe connection to a tool which pipes out from the tty to stdout (written for a Mac but may work on other unices): https://svn.rforge.net/C/trunk/tools/ttys.c and then use something like f=pipe("ttys /dev/tty.usbserial-X1234") A rather handy option -d prepends current time to each line so you can track output over time. I have some more tools for this (even allowing you to share form Arduino output with several computers or even send remote commands to your Arduino including encryption etc ...). Cheers, Simon PS: From experience I can say that Arduinos are highly addictive so beware ;).
In base::connections documentation, it's not clear if there's an easy way to do this? Any ideas on hacking it? I'm open to win/linux/mac solutions. I'm not sure how sockets work, but possibly there is a way to pipe things to a buffer and read from a buffer in bash (in my linux mind I have the thought of trying to redirect /dev/something to a file, or symlinking a file to point to the hardware, but know that there has to be some secret sauce to go from streaming in to a readable file, but don't know what the missing components are). Thoughts?
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Simon, Thanks for reviewing it! All the modified files are under the GPL version 2 (except the configure script). According to the GPLv2, I am granted permission to modify and redistribute the code as long as I make a notice in the files of their modification and the date (which I have not done yet). As I understand (I'm not lawyer), the copyright is necessary for, and does not alter the terms of the GPLv2. Is there something specific you're thinking of that invalidates this? I updated the patch to open with O_NOCTTY to prevent opening the tty as the controlling terminal. Looks like this is default in BSD and GNU but maybe not in OS X. I'd like to hear how it goes if you try it again. I see the rationale for a tty (in scope) connection. As you observed, there are lots of options. However, some of them are designed to be used when the terminal IS the controlling terminal, for example, if you were writing a program like getty, so barring the use of R as a job control shell (which would be cool also, but probably not in the scope of R), some of the options need not be exposed. For instance, canonical mode may never be necessary/appropriate. But it would be cool to use RTS/CTS, XON/XOFF, blocking, and some of the other stuff that is not there yet. I have some ideas on how to expose these features concisely. I'll continue to work on this as time permits (i.e. probably at weekend), with consideration for your suggestions. Thanks again, -Matt
On Mon, 2010-04-26 at 14:18 -0400, Simon Urbanek wrote:
Matt, thanks for you efforts. We cannot directly use your patch due to licensing issues (please note the licenses in the files you are modifying - I don't think anyone can currently use the patch and redistribute the resulting R; also/or we may possibly need to clarify that the work is either done on behalf of the R-foundation or the copyright is assigned to R-foundation or something similar so future licensing issues can be solved by the Foundation since that is the copyright holder of most of R). Apart from that, the overall structure looks good, but the devil is in the details. It does not quite work for me on OS X (any read/writes just hang in the same way that it would if you used a file connection on the tty device) - I don't think it is as simple (AFAIR you have to make sure you open it such that it doesn't become the controlling terminal). Also it would definitely need "blocking" argument like most other connections have. In general I don't think serial is the right name (and scope) here - what we're really looking for a is a tty connection (and serial just an application of it). What you have so far is a file connection with setserial functionality on it - and I think that may be more easily done by really using setserial on the already implemented file connection. The difference with tty is the other parts such as flow control, modes etc. It's much more work to expose those, but I think that is what the direction would be. In fact it could well be that tty is simply file connection + termio so you don't have to re-invent the wheel (under the hood). Also that would give you many aspects for free since you could inherit from file at the R level. I hope it helps. I was thinking about the above but in the end decided to back off once I was looking at the flood of parameters you have to take care of in tty. But it would be nice if someone did that ;). Cheers, Simon On Apr 26, 2010, at 12:28 PM, Matt Shotwell wrote:
All, Our discussion of serial interfaces last week motivated me to dig into the R connections API. In short, I spent the weekend writing a patch for version 2.11.0 that adds a serial connection. I posted a blog entry that gives instructions for applying, configuring, and compiling the patched version <http://biostatmatt.com/archives/112>. The patch currently only enables a serial connection for POSIX compliant OSs. It should NOT be considered well tested, though I feel it is fairly safe. I would really like to hear from someone who can try it on Mac OS X (and Windows to ensure that it has the correct 'null' behavior). Briefly, the connection utilizes (and checks for) the termios.h header. I opted not to extend the 'file' connection in a manner similar to the 'pipe' connection because this might not be suitable under Win32. Hence, there are new read, write, flush, open, and close methods and a struct serialconn for the serial connection. Opens are non-blocking and binary only. Serial parameters can be passed when the connection is created. I overloaded the summary method in order for the user to see the serial settings. I've done my best to be consistent and conforming with the connections API and coding style. However, I would be great if someone with more experience with the R API (Simon? :)) would take a critical look at the design and coding of the new connection. If this is something that the R community would like to pursue, I'd like to work on support for Windows serial interfaces. Below is a link to the patch file and the output of diffstat. <http://biostatmatt.com/R/R-2.11.0-serial.patch> file | mod ins del ------------------------------------------------------------ configure.ac | 2 1 + 1 - src/include/Internal.h | 2 2 + 0 - src/library/base/R/connections.R | 10 10 + 0 - src/library/base/man/connections.Rd | 60 49 + 11 - src/main/connections.c | 379 379 + 0 - src/main/names.c | 2 2 + 0 - P.S. connections.c is the largest file in src/main/, before and after the patch. Making the 379 additions increased the file size by 6.9% (10.7kB). Adding Windows support should increase the file size by a lesser amount. -Matt On Wed, 2010-04-21 at 08:07 -0400, Simon Urbanek wrote:
Philippe, unfortunately that approach has one major drawback - it does not give you a connection. As I said in the previous e-mail it is fairly easy to talk to a tty directly, but the challenge is to turn it into a connection. I don't like the Tcl approach for several reasons, one of them being that it's entirely unnatural since you have to construct strings of code -- you could as well use rJava and Java serial connections which has a little more friendly syntax (but I wouldn't recommend that, either) or any other language R interfaces to. I was thinking about it a bit and I may be tempted to have a dab at a tty connection, but I still would not want to mess with ioctl (the other part Matt mentioned). Cheers, Simon On Apr 21, 2010, at 6:30 AM, Philippe Grosjean wrote:
There is another option I use since a couple of years to pilot scientific devices, to program my chemostats, etc. and that is platform-independent: Tcl.
Tcl i/o design is excellent and very robust, and Tcl/Tk is integrated in R with the tcltk package.
It is really just a mather of a few lines of code in R to communicate through a serial port from R using Tcl. Something like:
require(tcltk)
.Tcl('set R_com1 [open "com1" r+]') # Works on Windows too!
# There are many config parameters available here... just an example
.Tcl('fconfigure $R_com1 -mode "9600,n,8,1" -buffering none -blocking 0')
# Send a command to your device through the serial port
.Tcl('puts -nonewline $R_com1 {my_cmd}')
# Read a line of text from the serial port
line <- tclvalue(.Tcl('gets $R_com1'))
With a little bit more code, one can program Tcl to call R code automatically everytime new data is pushed by the connected device through the serial port. This is done using something like:
.Tcl('fileevent $R_com1 readable [list Handler_com1 $R_com1]')
Here, "Handler_com1" is a Tcl function. So, some care must be taken using tcltk's .Tcl.callback() to trigger the event on the R side. One way to deal with this easily is by using tclFun() from the tcltk2 package.
In tcltk2, there is also ?tclTaskSchedule that can be of interest in the context of serial port communication to trigger a R function in the background regularly and collect data actively from the serial port.
All these tools give me a lot a flexibility to communicate through the serial port from R,... and most importantly, to write my code in a portable way (tested on Windows XP and Linux Ubuntu).
If there is some interest in this approach, I could initiate a 'tclcom' R package on R-Forge and place there the code I have.
Best,
Philippe
..............................................<?}))><........
) ) ) ) )
( ( ( ( ( Prof. Philippe Grosjean
) ) ) ) )
( ( ( ( ( Numerical Ecology of Aquatic Systems
) ) ) ) ) Mons University, Belgium
( ( ( ( (
..............................................................
On 21/04/10 03:17, shotwelm wrote:
Simon is right of course, there are plenty of sensors that would work just fine at 9600 baud (like a thermistor rigged to an ADC). There's a theorem along these lines (Nyquist sampling theorem?). I think piping the output to R is a clever solution. I added a few lines to the ttys.c program so that the baud rate is a command line option (i.e. -B9600) <http://biostatmatt.com/temp/ttys.c> and confirmed it will compile in Linux (2.6.30). Maybe it will save a step. Microcontrollers really are addictive! For an ioctl package, I was originally thinking of using file descriptors directly. However, I agree this feels like subverting what could be an extension of the connections API. Given that "everything is a file" in POSIX systems, there may be an argument for an ioctl package that is independent of the connections implementation, say to do things that connections were not designed to do. For example, interfacing with V4L2 devices usually involves many ioctl calls, an mmap call, but rarely read or write calls. But maybe it would just be better to pipe this type of output to R also... -Matt On Tue, 2010-04-20 at 16:42 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 11:51 AM, shotwelm wrote:
I've done some microcontroller work over serial also. Unfortunately, interfacing with a serial port is system dependent, and the mechanisms can be quite different, as you probably know. It appears that Simon has a solution below that will work if you are willing to accept the default baud rate (9600 is way too slow for good sensor data
[OT: define "good" ;) - good doesn't mean fast - besides it won't be any good if it is too fast to be meaningfully processed -- that's a different story, though :P - and it is trivial to change so the solution works in general]
), parity, etc.. or use external tools. On POSIX systems, you would need access to the termios.h header and the system ioctl function in order to change these settings. Although I'm not 100% sure, I don't think R has this capability ... yet. I'm new to the list, but I'd be surprised if the R developers that have been around awhile haven't already considered adding support for ioctls and the POSIX terminal interface. This makes me wonder why it's not there. If there is no good reason, I'm starting to see a series of R packages (or core extensions) developing.
Good luck ;). The issue is that connections are inherently backend-independent which implies that packages have no access to connection internals as they can change at any time. This means that you can't enhance them without putting the enhancements into R itself. This implies that you have to make a strong case since you need a volunteer in R-core to maintain that code etc.
With a package for ioctls, we could use all sorts of cool stuff, like Video4Linux2 (webcams, HAM radio, tuners)...
Ioctls are highly system-specific which is orthogonal to the design of connections. You could probably hack together a FD-based access system but it would not be compatible with connections (unless you exploit undocumented things if possible at all ...). Also ioctls can change the stream semantics entirely thus breaking anything that deals with the FD assuming some defined state ...
When I collect sensor data over serial, I do it in python or write a small C program to dump a single-column csv. Of course, R is excellent for digital signal processing after that. Check out the DSP ( http://biostatmatt.com/archives/78 ) I did in R with some ECG data I collected with an Atmel uC.
Well, we're back to calling tools to do the interfacing like the ttys (I do prefer pipe to intermediate files)... It's not that complicated and has several benefits (implicit parallelization, process separation in case things go wrong etc.) so it is not obvious that it's a bad thing ... I suspect that we're simply suck until the connection API is either exposed or re-written so packages can provide new connections types or extend existing one. Again, this is not trivial especially when you start messing with ioctl since it's easy to depart from defined behavior in that case ... That said, I agree that expanding connections is useful so some progress there would be desirable - but the "how" and "who" is not clear to me ... That's just my $0.02, though ... Cheers, Simon
On Tue, 2010-04-20 at 11:05 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 10:33 AM, Blair Christian wrote:
Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data...
Yes (I have Arduinos reporting measurements from all sensors in the house to R on my iMac which produces plots that are synchronized with my webserver). In principle you can simply use /dev/tty.usb... and read from it. In most cases the default setting is already fine (9600,n,8,1 on Mac) or you can use tools the set it up in advance (setserial on Linux etc.) so you don't have to worry about setting up the serial from R. Depending on your OS you may be able to read from the serial device directly with a regular file connection or you can use a pipe connection to a tool which pipes out from the tty to stdout (written for a Mac but may work on other unices): https://svn.rforge.net/C/trunk/tools/ttys.c and then use something like f=pipe("ttys /dev/tty.usbserial-X1234") A rather handy option -d prepends current time to each line so you can track output over time. I have some more tools for this (even allowing you to share form Arduino output with several computers or even send remote commands to your Arduino including encryption etc ...). Cheers, Simon PS: From experience I can say that Arduinos are highly addictive so beware ;).
In base::connections documentation, it's not clear if there's an easy way to do this? Any ideas on hacking it? I'm open to win/linux/mac solutions. I'm not sure how sockets work, but possibly there is a way to pipe things to a buffer and read from a buffer in bash (in my linux mind I have the thought of trying to redirect /dev/something to a file, or symlinking a file to point to the hardware, but know that there has to be some secret sauce to go from streaming in to a readable file, but don't know what the missing components are). Thoughts?
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
On 27/04/2010 8:32 AM, Matt Shotwell wrote:
Simon, Thanks for reviewing it! All the modified files are under the GPL version 2 (except the configure script). According to the GPLv2, I am granted permission to modify and redistribute the code as long as I make a notice in the files of their modification and the date (which I have not done yet). As I understand (I'm not lawyer), the copyright is necessary for, and does not alter the terms of the GPLv2. Is there something specific you're thinking of that invalidates this?
I don't know what Simon noticed, but I saw that you had indicated a GPL v3 license on the web page. GPL2 is not compatible with GPL3, so that makes your contribution unusable by us. Duncan Murdoch
I updated the patch to open with O_NOCTTY to prevent opening the tty as the controlling terminal. Looks like this is default in BSD and GNU but maybe not in OS X. I'd like to hear how it goes if you try it again. I see the rationale for a tty (in scope) connection. As you observed, there are lots of options. However, some of them are designed to be used when the terminal IS the controlling terminal, for example, if you were writing a program like getty, so barring the use of R as a job control shell (which would be cool also, but probably not in the scope of R), some of the options need not be exposed. For instance, canonical mode may never be necessary/appropriate. But it would be cool to use RTS/CTS, XON/XOFF, blocking, and some of the other stuff that is not there yet. I have some ideas on how to expose these features concisely. I'll continue to work on this as time permits (i.e. probably at weekend), with consideration for your suggestions. Thanks again, -Matt On Mon, 2010-04-26 at 14:18 -0400, Simon Urbanek wrote:
Matt, thanks for you efforts. We cannot directly use your patch due to licensing issues (please note the licenses in the files you are modifying - I don't think anyone can currently use the patch and redistribute the resulting R; also/or we may possibly need to clarify that the work is either done on behalf of the R-foundation or the copyright is assigned to R-foundation or something similar so future licensing issues can be solved by the Foundation since that is the copyright holder of most of R). Apart from that, the overall structure looks good, but the devil is in the details. It does not quite work for me on OS X (any read/writes just hang in the same way that it would if you used a file connection on the tty device) - I don't think it is as simple (AFAIR you have to make sure you open it such that it doesn't become the controlling terminal). Also it would definitely need "blocking" argument like most other connections have. In general I don't think serial is the right name (and scope) here - what we're really looking for a is a tty connection (and serial just an application of it). What you have so far is a file connection with setserial functionality on it - and I think that may be more easily done by really using setserial on the already implemented file connection. The difference with tty is the other parts such as flow control, modes etc. It's much more work to expose those, but I think that is what the direction would be. In fact it could well be that tty is simply file connection + termio so you don't have to re-invent the wheel (under the hood). Also that would give you many aspects for free since you could inherit from file at the R level. I hope it helps. I was thinking about the above but in the end decided to back off once I was looking at the flood of parameters you have to take care of in tty. But it would be nice if someone did that ;). Cheers, Simon On Apr 26, 2010, at 12:28 PM, Matt Shotwell wrote:
All, Our discussion of serial interfaces last week motivated me to dig into the R connections API. In short, I spent the weekend writing a patch for version 2.11.0 that adds a serial connection. I posted a blog entry that gives instructions for applying, configuring, and compiling the patched version <http://biostatmatt.com/archives/112>. The patch currently only enables a serial connection for POSIX compliant OSs. It should NOT be considered well tested, though I feel it is fairly safe. I would really like to hear from someone who can try it on Mac OS X (and Windows to ensure that it has the correct 'null' behavior). Briefly, the connection utilizes (and checks for) the termios.h header. I opted not to extend the 'file' connection in a manner similar to the 'pipe' connection because this might not be suitable under Win32. Hence, there are new read, write, flush, open, and close methods and a struct serialconn for the serial connection. Opens are non-blocking and binary only. Serial parameters can be passed when the connection is created. I overloaded the summary method in order for the user to see the serial settings. I've done my best to be consistent and conforming with the connections API and coding style. However, I would be great if someone with more experience with the R API (Simon? :)) would take a critical look at the design and coding of the new connection. If this is something that the R community would like to pursue, I'd like to work on support for Windows serial interfaces. Below is a link to the patch file and the output of diffstat. <http://biostatmatt.com/R/R-2.11.0-serial.patch> file | mod ins del ------------------------------------------------------------ configure.ac | 2 1 + 1 - src/include/Internal.h | 2 2 + 0 - src/library/base/R/connections.R | 10 10 + 0 - src/library/base/man/connections.Rd | 60 49 + 11 - src/main/connections.c | 379 379 + 0 - src/main/names.c | 2 2 + 0 - P.S. connections.c is the largest file in src/main/, before and after the patch. Making the 379 additions increased the file size by 6.9% (10.7kB). Adding Windows support should increase the file size by a lesser amount. -Matt On Wed, 2010-04-21 at 08:07 -0400, Simon Urbanek wrote:
Philippe, unfortunately that approach has one major drawback - it does not give you a connection. As I said in the previous e-mail it is fairly easy to talk to a tty directly, but the challenge is to turn it into a connection. I don't like the Tcl approach for several reasons, one of them being that it's entirely unnatural since you have to construct strings of code -- you could as well use rJava and Java serial connections which has a little more friendly syntax (but I wouldn't recommend that, either) or any other language R interfaces to. I was thinking about it a bit and I may be tempted to have a dab at a tty connection, but I still would not want to mess with ioctl (the other part Matt mentioned). Cheers, Simon On Apr 21, 2010, at 6:30 AM, Philippe Grosjean wrote:
There is another option I use since a couple of years to pilot scientific devices, to program my chemostats, etc. and that is platform-independent: Tcl.
Tcl i/o design is excellent and very robust, and Tcl/Tk is integrated in R with the tcltk package.
It is really just a mather of a few lines of code in R to communicate through a serial port from R using Tcl. Something like:
require(tcltk)
.Tcl('set R_com1 [open "com1" r+]') # Works on Windows too!
# There are many config parameters available here... just an example
.Tcl('fconfigure $R_com1 -mode "9600,n,8,1" -buffering none -blocking 0')
# Send a command to your device through the serial port
.Tcl('puts -nonewline $R_com1 {my_cmd}')
# Read a line of text from the serial port
line <- tclvalue(.Tcl('gets $R_com1'))
With a little bit more code, one can program Tcl to call R code automatically everytime new data is pushed by the connected device through the serial port. This is done using something like:
.Tcl('fileevent $R_com1 readable [list Handler_com1 $R_com1]')
Here, "Handler_com1" is a Tcl function. So, some care must be taken using tcltk's .Tcl.callback() to trigger the event on the R side. One way to deal with this easily is by using tclFun() from the tcltk2 package.
In tcltk2, there is also ?tclTaskSchedule that can be of interest in the context of serial port communication to trigger a R function in the background regularly and collect data actively from the serial port.
All these tools give me a lot a flexibility to communicate through the serial port from R,... and most importantly, to write my code in a portable way (tested on Windows XP and Linux Ubuntu).
If there is some interest in this approach, I could initiate a 'tclcom' R package on R-Forge and place there the code I have.
Best,
Philippe
..............................................<?}))><........
) ) ) ) )
( ( ( ( ( Prof. Philippe Grosjean
) ) ) ) )
( ( ( ( ( Numerical Ecology of Aquatic Systems
) ) ) ) ) Mons University, Belgium
( ( ( ( (
..............................................................
On 21/04/10 03:17, shotwelm wrote:
Simon is right of course, there are plenty of sensors that would work just fine at 9600 baud (like a thermistor rigged to an ADC). There's a theorem along these lines (Nyquist sampling theorem?). I think piping the output to R is a clever solution. I added a few lines to the ttys.c program so that the baud rate is a command line option (i.e. -B9600) <http://biostatmatt.com/temp/ttys.c> and confirmed it will compile in Linux (2.6.30). Maybe it will save a step. Microcontrollers really are addictive! For an ioctl package, I was originally thinking of using file descriptors directly. However, I agree this feels like subverting what could be an extension of the connections API. Given that "everything is a file" in POSIX systems, there may be an argument for an ioctl package that is independent of the connections implementation, say to do things that connections were not designed to do. For example, interfacing with V4L2 devices usually involves many ioctl calls, an mmap call, but rarely read or write calls. But maybe it would just be better to pipe this type of output to R also... -Matt On Tue, 2010-04-20 at 16:42 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 11:51 AM, shotwelm wrote:
I've done some microcontroller work over serial also. Unfortunately, interfacing with a serial port is system dependent, and the mechanisms can be quite different, as you probably know. It appears that Simon has a solution below that will work if you are willing to accept the default baud rate (9600 is way too slow for good sensor data
[OT: define "good" ;) - good doesn't mean fast - besides it won't be any good if it is too fast to be meaningfully processed -- that's a different story, though :P - and it is trivial to change so the solution works in general]
), parity, etc.. or use external tools. On POSIX systems, you would need access to the termios.h header and the system ioctl function in order to change these settings. Although I'm not 100% sure, I don't think R has this capability ... yet. I'm new to the list, but I'd be surprised if the R developers that have been around awhile haven't already considered adding support for ioctls and the POSIX terminal interface. This makes me wonder why it's not there. If there is no good reason, I'm starting to see a series of R packages (or core extensions) developing.
Good luck ;). The issue is that connections are inherently backend-independent which implies that packages have no access to connection internals as they can change at any time. This means that you can't enhance them without putting the enhancements into R itself. This implies that you have to make a strong case since you need a volunteer in R-core to maintain that code etc.
With a package for ioctls, we could use all sorts of cool stuff, like Video4Linux2 (webcams, HAM radio, tuners)...
Ioctls are highly system-specific which is orthogonal to the design of connections. You could probably hack together a FD-based access system but it would not be compatible with connections (unless you exploit undocumented things if possible at all ...). Also ioctls can change the stream semantics entirely thus breaking anything that deals with the FD assuming some defined state ...
When I collect sensor data over serial, I do it in python or write a small C program to dump a single-column csv. Of course, R is excellent for digital signal processing after that. Check out the DSP ( http://biostatmatt.com/archives/78 ) I did in R with some ECG data I collected with an Atmel uC.
Well, we're back to calling tools to do the interfacing like the ttys (I do prefer pipe to intermediate files)... It's not that complicated and has several benefits (implicit parallelization, process separation in case things go wrong etc.) so it is not obvious that it's a bad thing ... I suspect that we're simply suck until the connection API is either exposed or re-written so packages can provide new connections types or extend existing one. Again, this is not trivial especially when you start messing with ioctl since it's easy to depart from defined behavior in that case ... That said, I agree that expanding connections is useful so some progress there would be desirable - but the "how" and "who" is not clear to me ... That's just my $0.02, though ... Cheers, Simon
On Tue, 2010-04-20 at 11:05 -0400, Simon Urbanek wrote:
On Apr 20, 2010, at 10:33 AM, Blair Christian wrote:
Does anybody know if there is any support to read from serial ports? I just got an arduino, and wanted to write some scripts for working with real time streaming sensor data...
Yes (I have Arduinos reporting measurements from all sensors in the house to R on my iMac which produces plots that are synchronized with my webserver). In principle you can simply use /dev/tty.usb... and read from it. In most cases the default setting is already fine (9600,n,8,1 on Mac) or you can use tools the set it up in advance (setserial on Linux etc.) so you don't have to worry about setting up the serial from R. Depending on your OS you may be able to read from the serial device directly with a regular file connection or you can use a pipe connection to a tool which pipes out from the tty to stdout (written for a Mac but may work on other unices): https://svn.rforge.net/C/trunk/tools/ttys.c and then use something like f=pipe("ttys /dev/tty.usbserial-X1234") A rather handy option -d prepends current time to each line so you can track output over time. I have some more tools for this (even allowing you to share form Arduino output with several computers or even send remote commands to your Arduino including encryption etc ...). Cheers, Simon PS: From experience I can say that Arduinos are highly addictive so beware ;).
In base::connections documentation, it's not clear if there's an easy way to do this? Any ideas on hacking it? I'm open to win/linux/mac solutions. I'm not sure how sockets work, but possibly there is a way to pipe things to a buffer and read from a buffer in bash (in my linux mind I have the thought of trying to redirect /dev/something to a file, or symlinking a file to point to the hardware, but know that there has to be some secret sauce to go from streaming in to a readable file, but don't know what the missing components are). Thoughts?
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
On Apr 27, 2010, at 2:42 PM, Duncan Murdoch wrote:
On 27/04/2010 8:32 AM, Matt Shotwell wrote:
Simon, Thanks for reviewing it! All the modified files are under the GPL version 2 (except the configure script). According to the GPLv2, I am granted permission to modify and redistribute the code as long as I make a notice in the files of their modification and the date (which I have not done yet). As I understand (I'm not lawyer), the copyright is necessary for, and does not alter the terms of the GPLv2. Is there something specific you're thinking of that invalidates this?
I don't know what Simon noticed, but I saw that you had indicated a GPL v3 license on the web page. GPL2 is not compatible with GPL3, so that makes your contribution unusable by us.
Yes, this is a bit messy, but some got sufficiently annoyed by the added restrictions of GPL3 that they insist on keeping their contributions GPL2, so adding GPL3-only stuff is off limits. I don't think we have a problem with merging user contributions that are licensed "GPL2 or later". A copyright transfer gives some legal clarification, but is only really required in case the R Foundation wants to (dual-) relicense under a GPL incompatible license, or need to be able to legally defend users' code against infringement. The former is highly unlikely, and it would require major disentanglement in other areas anyway, and I don't see contributions of this order of magnitude as a target of legal dispute either.
Duncan Murdoch
Peter Dalgaard Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Email: pd.mes at cbs.dk Priv: PDalgd at gmail.com
Oh, I just put that there for the bit about "WITHOUT ANY WARRANTY" :). I am satisfied with GPLv2 and have updated the license notice to reflect a change from GPLv3 to GPLv2 or later. -Matt
On Tue, 2010-04-27 at 09:33 -0400, Peter Dalgaard wrote:
On Apr 27, 2010, at 2:42 PM, Duncan Murdoch wrote:
On 27/04/2010 8:32 AM, Matt Shotwell wrote:
Simon, Thanks for reviewing it! All the modified files are under the GPL version 2 (except the configure script). According to the GPLv2, I am granted permission to modify and redistribute the code as long as I make a notice in the files of their modification and the date (which I have not done yet). As I understand (I'm not lawyer), the copyright is necessary for, and does not alter the terms of the GPLv2. Is there something specific you're thinking of that invalidates this?
I don't know what Simon noticed, but I saw that you had indicated a GPL v3 license on the web page. GPL2 is not compatible with GPL3, so that makes your contribution unusable by us.
Yes, this is a bit messy, but some got sufficiently annoyed by the added restrictions of GPL3 that they insist on keeping their contributions GPL2, so adding GPL3-only stuff is off limits. I don't think we have a problem with merging user contributions that are licensed "GPL2 or later". A copyright transfer gives some legal clarification, but is only really required in case the R Foundation wants to (dual-) relicense under a GPL incompatible license, or need to be able to legally defend users' code against infringement. The former is highly unlikely, and it would require major disentanglement in other areas anyway, and I don't see contributions of this order of magnitude as a target of legal dispute either.
Duncan Murdoch