Dear list, My package cda, which I was hoping to release on CRAN, fails to compile on R-forge with error, ** testing if installed package can be loaded Error in dyn.load(file, DLLpath = DLLpath, ...) : unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ It builds fine on my local machines (Mac OS 10.5, 10.6).
[Rcpp-devel] RcppArmadillo inv() depends on Lapack, zgetri_ not available on CRAN / R-forge?
16 messages · Baptiste Auguie, Douglas Bates, Dirk Eddelbuettel
Baptiste,
On 4 June 2011 at 11:45, baptiste auguie wrote:
| Dear list, | | My package cda, which I was hoping to release on CRAN, fails to | compile on R-forge with error, | | ** testing if installed package can be loaded | Error in dyn.load(file, DLLpath = DLLpath, ...) : | unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': | /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ | | It builds fine on my local machines (Mac OS 10.5, 10.6). | | >From an older discussion on this list < | http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html> | the issue seems to be that Armadillo's inv() relies on a function that | is not provided by R, only by LAPACK. I have just replaced inv() by | pinv() and solve() in my code; merely to see what happens, but chances | are they also require a full LAPACK. Sometime relatively early in the RcppArmadillo development process, Doug convinced Romain and myself to go for a pure template solution with Armadillo as all / most things found during the configure (or in this case, cmake) stage can be assumed 'found' given that we have around us by design. So no testing, no local library and full reliance and what R gives us. That was a brilliant idea, and has freed us from having to rely on building and shipping a library, having to tell users how to set PKG_LIBS etc pp and I firmly believe that this helped tremendously in getting RcppArmadillo more widely used. So I would not want to revert this. In any event, it seems that you need more LAPACK than R has for you. That is likely to be a dicey situation as you per se do not know whether R was built and linked with its own (subset) copy of LAPACK, or whether it uses system LAPACK libraries (as e.g. the Debian / Ubuntu systems do). So you may be in a spot bother and I not sure what I can recommend --- other than trying your luck at some short configure snippets that will run at package build time to determine whether the system you want to build cda on it 'rich' enough to support it. I can help you off list with some configure snippets as some of my packages have configure code; adding a test for zgetri should be feasible. | Does anybody have any experience | dealing with such issues w.r.t releasing a package on R-forge / CRAN? | Is there any chance they would consider installing LAPACK on those | servers, or is there no point in asking such things? I don't think it is a matter of fixing the R-Forge server. I think it is a matter of making your package installable on the largest number of user systems. Also try win-builder.r-project.org to see how it fares on that platform. Dirk
Gauss once played himself in a zero-sum game and won $50.
-- #11 at http://www.gaussfacts.com
Hi Dirk and all,
On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote:
Baptiste, On 4 June 2011 at 11:45, baptiste auguie wrote: | Dear list, | | My package cda, which I was hoping to release on CRAN, fails to | compile on R-forge with error, | | ** testing if installed package can be loaded | Error in dyn.load(file, DLLpath = DLLpath, ...) : | ? unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': | ? /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ | | It builds fine on my local machines (Mac OS 10.5, 10.6). | | >From an older discussion on this list < | http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html> | the issue seems to be that Armadillo's inv() relies on a function that | is not provided by R, only by LAPACK. I have just replaced inv() by | pinv() and solve() in my code; merely to see what happens, but chances | are they also require a full LAPACK.
Indeed, the error on R-forge is now with zgels_, required to solve linear systems. It seems one cannot solve Armadillo linear systems without LAPACK in the current situation.
Sometime relatively early in the RcppArmadillo development process, Doug convinced Romain and myself to go for a pure template solution with Armadillo as all / most things found during the configure (or in this case, cmake) stage can be assumed 'found' given that we have around us by design. ?So no testing, no local library and full reliance and what R gives us. That was a brilliant idea, and has freed us from having to rely on building and shipping a library, having to tell users how to set PKG_LIBS etc pp and I firmly believe that this helped tremendously in getting RcppArmadillo more widely used. So I would not want to revert this.
It sounds like a good choice, I agree. Yet solving a linear system is quite a crucial task in linear algebra; it would be nice if we could come up with a tutorial recipe for dummies like me.
In any event, it seems that you need more LAPACK than R has for you. ?That is likely to be a dicey situation as you per se do not know whether R was built and linked with its own (subset) copy of LAPACK, or whether it uses system LAPACK libraries (as e.g. the Debian / Ubuntu systems do). ?So you may be in a spot bother and I not sure what I can recommend --- other than trying your luck at some short configure snippets that will run at package build time to determine whether the system you want to build cda on it 'rich' enough to support it. ?I can help you off list with some configure snippets as some of my packages have configure code; adding a test for zgetri should be feasible. | Does anybody have any experience | dealing with such issues w.r.t releasing a package on R-forge / CRAN? | Is there any chance they would consider installing LAPACK on those | servers, or is there no point in asking such things? I don't think it is a matter of fixing the R-Forge server. I think it is a matter of making your package installable on the largest number of user systems. ?Also try win-builder.r-project.org to see how it fares on that platform.
I was under the impression that R-forge or CRAN, if it had LAPACK installed, could produce binaries for the relevant platforms, and users would not have to build the package themselves and would not be required of having LAPACK on their machine. That's probably a misconception, isn't it? Sorry for being dense, I don't know anything about linking R to external dependencies. Thanks, baptiste
On 5 June 2011 at 10:12, baptiste auguie wrote:
| Hi Dirk and all, |
| On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote:
| > | > Baptiste, | >
| > On 4 June 2011 at 11:45, baptiste auguie wrote:
| > | Dear list, | > | | > | My package cda, which I was hoping to release on CRAN, fails to | > | compile on R-forge with error, | > | | > | ** testing if installed package can be loaded | > | Error in dyn.load(file, DLLpath = DLLpath, ...) : | > | ? unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': | > | ? /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ | > | | > | It builds fine on my local machines (Mac OS 10.5, 10.6). | > | | > | >From an older discussion on this list < | > | http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html> | > | the issue seems to be that Armadillo's inv() relies on a function that | > | is not provided by R, only by LAPACK. I have just replaced inv() by | > | pinv() and solve() in my code; merely to see what happens, but chances | > | are they also require a full LAPACK. | | Indeed, the error on R-forge is now with zgels_, required to solve | linear systems. It seems one cannot solve Armadillo linear systems | without LAPACK in the current situation. Yes. Doug, Romain and myself should address that, or at least make it clear what feature of the full Armadillo are lacking in RcppArmadillo. | > Sometime relatively early in the RcppArmadillo development process, Doug | > convinced Romain and myself to go for a pure template solution with Armadillo | > as all / most things found during the configure (or in this case, cmake) | > stage can be assumed 'found' given that we have around us by design. ?So no | > testing, no local library and full reliance and what R gives us. | > | > That was a brilliant idea, and has freed us from having to rely on building | > and shipping a library, having to tell users how to set PKG_LIBS etc pp and I | > firmly believe that this helped tremendously in getting RcppArmadillo more | > widely used. So I would not want to revert this. | | It sounds like a good choice, I agree. Yet solving a linear system is | quite a crucial task in linear algebra; it would be nice if we could | come up with a tutorial recipe for dummies like me. | | > | > In any event, it seems that you need more LAPACK than R has for you. ?That is | > likely to be a dicey situation as you per se do not know whether R was built | > and linked with its own (subset) copy of LAPACK, or whether it uses system | > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). ?So you may be in | > a spot bother and I not sure what I can recommend --- other than trying your | > luck at some short configure snippets that will run at package build time to | > determine whether the system you want to build cda on it 'rich' enough to | > support it. ?I can help you off list with some configure snippets as some of | > my packages have configure code; adding a test for zgetri should be feasible. | > | > | Does anybody have any experience | > | dealing with such issues w.r.t releasing a package on R-forge / CRAN? | > | Is there any chance they would consider installing LAPACK on those | > | servers, or is there no point in asking such things? | > | > I don't think it is a matter of fixing the R-Forge server. I think it is a | > matter of making your package installable on the largest number of user | > systems. ?Also try win-builder.r-project.org to see how it fares on that | > platform. | | I was under the impression that R-forge or CRAN, if it had LAPACK | installed, could produce binaries for the relevant platforms, and | users would not have to build the package themselves and would not be | required of having LAPACK on their machine. That's probably a | misconception, isn't it? If and only statically linked binaries or libraries where produced, which is generally not the case. Many OSs (Linux incl) ship source only and otherwise link dynamically, others (Windoze) use dynamic linking and OS X is for all I know somewhere in the middle (as you can get prebuild packages with dynamic linking or build from source). | Sorry for being dense, I don't know anything about linking R to | external dependencies. It can be done; there are many examples -- for example every package using the GSL. Dirk
Gauss once played himself in a zero-sum game and won $50.
-- #11 at http://www.gaussfacts.com
Thank you for the explanations below.
On 5 June 2011 10:40, Dirk Eddelbuettel <edd at debian.org> wrote:
On 5 June 2011 at 10:12, baptiste auguie wrote: | Hi Dirk and all, | | On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote: | > | > Baptiste, | > | > On 4 June 2011 at 11:45, baptiste auguie wrote: | > | Dear list, | > | | > | My package cda, which I was hoping to release on CRAN, fails to | > | compile on R-forge with error, | > | | > | ** testing if installed package can be loaded | > | Error in dyn.load(file, DLLpath = DLLpath, ...) : | > | ? unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': | > | ? /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ | > | | > | It builds fine on my local machines (Mac OS 10.5, 10.6). | > | | > | >From an older discussion on this list < | > | http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html> | > | the issue seems to be that Armadillo's inv() relies on a function that | > | is not provided by R, only by LAPACK. I have just replaced inv() by | > | pinv() and solve() in my code; merely to see what happens, but chances | > | are they also require a full LAPACK. | | Indeed, the error on R-forge is now with zgels_, required to solve | linear systems. It seems one cannot solve Armadillo linear systems | without LAPACK in the current situation. Yes. Doug, Romain and myself should address that, or at least make it clear what feature of the full Armadillo are lacking in RcppArmadillo. | > Sometime relatively early in the RcppArmadillo development process, Doug | > convinced Romain and myself to go for a pure template solution with Armadillo | > as all / most things found during the configure (or in this case, cmake) | > stage can be assumed 'found' given that we have around us by design. ?So no | > testing, no local library and full reliance and what R gives us. | > | > That was a brilliant idea, and has freed us from having to rely on building | > and shipping a library, having to tell users how to set PKG_LIBS etc pp and I | > firmly believe that this helped tremendously in getting RcppArmadillo more | > widely used. So I would not want to revert this. | | It sounds like a good choice, I agree. Yet solving a linear system is | quite a crucial task in linear algebra; it would be nice if we could | come up with a tutorial recipe for dummies like me. | | > | > In any event, it seems that you need more LAPACK than R has for you. ?That is | > likely to be a dicey situation as you per se do not know whether R was built | > and linked with its own (subset) copy of LAPACK, or whether it uses system | > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). ?So you may be in | > a spot bother and I not sure what I can recommend --- other than trying your | > luck at some short configure snippets that will run at package build time to | > determine whether the system you want to build cda on it 'rich' enough to | > support it. ?I can help you off list with some configure snippets as some of | > my packages have configure code; adding a test for zgetri should be feasible. | > | > | Does anybody have any experience | > | dealing with such issues w.r.t releasing a package on R-forge / CRAN? | > | Is there any chance they would consider installing LAPACK on those | > | servers, or is there no point in asking such things? | > | > I don't think it is a matter of fixing the R-Forge server. I think it is a | > matter of making your package installable on the largest number of user | > systems. ?Also try win-builder.r-project.org to see how it fares on that | > platform.
Unsurprisingly, it fails, with the same complaint as R-forge.
| | I was under the impression that R-forge or CRAN, if it had LAPACK | installed, could produce binaries for the relevant platforms, and | users would not have to build the package themselves and would not be | required of having LAPACK on their machine. That's probably a | misconception, isn't it? If and only statically linked binaries or libraries where produced, which is generally not the case. Many OSs (Linux incl) ship source only and otherwise link dynamically, others (Windoze) use dynamic linking and OS X is for all I know somewhere in the middle (as you can get prebuild packages with dynamic linking or build from source).
I see; so basically the user will always need to have a full LAPACK installed. Here's one question then, if R-core didn't consider necessary to include those particular functions from LAPACK, presumably that means that R defines its own routines to solve linear systems and invert matrices. Is there any possibility to use those routines with Armadillo?
| Sorry for being dense, I don't know anything about linking R to | external dependencies. It can be done; there are many examples -- for example every package using the GSL.
I just checked how RcppGSL does it, and well, this configure magic is way above my head. Best, baptiste
On 6 June 2011 at 09:56, baptiste auguie wrote:
| > | > I don't think it is a matter of fixing the R-Forge server. I think it is a | > | > matter of making your package installable on the largest number of user | > | > systems. ?Also try win-builder.r-project.org to see how it fares on that | > | > platform. | | Unsurprisingly, it fails, with the same complaint as R-forge. Yup. | > | I was under the impression that R-forge or CRAN, if it had LAPACK | > | installed, could produce binaries for the relevant platforms, and | > | users would not have to build the package themselves and would not be | > | required of having LAPACK on their machine. That's probably a | > | misconception, isn't it? | > | > If and only statically linked binaries or libraries where produced, which is | > generally not the case. Many OSs (Linux incl) ship source only and otherwise | > link dynamically, others (Windoze) use dynamic linking and OS X is for all I | > know somewhere in the middle (as you can get prebuild packages with dynamic | > linking or build from source). | | I see; so basically the user will always need to have a full LAPACK | installed. Here's one question then, if R-core didn't consider | necessary to include those particular functions from LAPACK, | presumably that means that R defines its own routines to solve linear | systems and invert matrices. Is there any possibility to use those | routines with Armadillo? I think that *is* generally what we do. We would have at specifics for the functions you are after. | > It can be done; there are many examples -- for example every package using | > the GSL. | | I just checked how RcppGSL does it, and well, this configure magic is | way above my head. I know it looks scary but you it is reasonably well understood and there are tons of examples. I can help you off-line. Also, read configure.in / configure.ac and not the configure which is autogenerated from it, large and generally unreadable. Dirk
Gauss once played himself in a zero-sum game and won $50.
-- #11 at http://www.gaussfacts.com
1 day later
On Jun 6, 2011 4:17 AM, "baptiste auguie" <baptiste.auguie at googlemail.com> wrote:
Thank you for the explanations below. On 5 June 2011 10:40, Dirk Eddelbuettel <edd at debian.org> wrote:
On 5 June 2011 at 10:12, baptiste auguie wrote: | Hi Dirk and all, | | On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote: | > | > Baptiste, | > | > On 4 June 2011 at 11:45, baptiste auguie wrote: | > | Dear list, | > | | > | My package cda, which I was hoping to release on CRAN, fails to | > | compile on R-forge with error, | > | | > | ** testing if installed package can be loaded | > | Error in dyn.load(file, DLLpath = DLLpath, ...) : | > | unable to load shared object
'/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so':
| > | /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol:
zgetri_
| > | | > | It builds fine on my local machines (Mac OS 10.5, 10.6). | > | | > | >From an older discussion on this list < | > |
http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html
| > | the issue seems to be that Armadillo's inv() relies on a function
that
| > | is not provided by R, only by LAPACK. I have just replaced inv()
by
| > | pinv() and solve() in my code; merely to see what happens, but
chances
| > | are they also require a full LAPACK. | | Indeed, the error on R-forge is now with zgels_, required to solve | linear systems. It seems one cannot solve Armadillo linear systems | without LAPACK in the current situation. Yes. Doug, Romain and myself should address that, or at least make it
clear
what feature of the full Armadillo are lacking in RcppArmadillo. | > Sometime relatively early in the RcppArmadillo development process,
Doug
| > convinced Romain and myself to go for a pure template solution with
Armadillo
| > as all / most things found during the configure (or in this case,
cmake)
| > stage can be assumed 'found' given that we have around us by design.
So no
| > testing, no local library and full reliance and what R gives us. | > | > That was a brilliant idea, and has freed us from having to rely on
building
| > and shipping a library, having to tell users how to set PKG_LIBS etc
pp and I
| > firmly believe that this helped tremendously in getting
RcppArmadillo more
| > widely used. So I would not want to revert this. | | It sounds like a good choice, I agree. Yet solving a linear system is | quite a crucial task in linear algebra; it would be nice if we could | come up with a tutorial recipe for dummies like me. | | > | > In any event, it seems that you need more LAPACK than R has for you.
That is
| > likely to be a dicey situation as you per se do not know whether R
was built
| > and linked with its own (subset) copy of LAPACK, or whether it uses
system
| > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). So you
may be in
| > a spot bother and I not sure what I can recommend --- other than
trying your
| > luck at some short configure snippets that will run at package build
time to
| > determine whether the system you want to build cda on it 'rich'
enough to
| > support it. I can help you off list with some configure snippets as
some of
| > my packages have configure code; adding a test for zgetri should be
feasible.
| > | > | Does anybody have any experience | > | dealing with such issues w.r.t releasing a package on R-forge /
CRAN?
| > | Is there any chance they would consider installing LAPACK on those | > | servers, or is there no point in asking such things? | > | > I don't think it is a matter of fixing the R-Forge server. I think
it is a
| > matter of making your package installable on the largest number of
user
| > systems. Also try win-builder.r-project.org to see how it fares on
that
| > platform.
Unsurprisingly, it fails, with the same complaint as R-forge.
| | I was under the impression that R-forge or CRAN, if it had LAPACK | installed, could produce binaries for the relevant platforms, and | users would not have to build the package themselves and would not be | required of having LAPACK on their machine. That's probably a | misconception, isn't it? If and only statically linked binaries or libraries where produced,
which is
generally not the case. Many OSs (Linux incl) ship source only and
otherwise
link dynamically, others (Windoze) use dynamic linking and OS X is for
all I
know somewhere in the middle (as you can get prebuild packages with
dynamic
linking or build from source).
I see; so basically the user will always need to have a full LAPACK installed. Here's one question then, if R-core didn't consider necessary to include those particular functions from LAPACK, presumably that means that R defines its own routines to solve linear systems and invert matrices. Is there any possibility to use those routines with Armadillo?
One important point has been missed in this discussion, which is that the particular Lapack subroutine that is not found in the subset of Lapack shipped with R is for general, complex-valued matrices (just google the name zgetri). The way that Armadillo is structured it is either all-in or all-out with respect to complex-valued matrices If you allow for complex matrices then you must have all the supporting subroutines for whatever you are calling. If you call inv() you need to allow for all the [sdcz]getri routines to be available. So one possibility is to check the Armadillo sources to see if you can suppress the use of complex-valued matrices when compiling your package. A quick glance indicates that this might now be easy. Otherwise, remember the first rule of numerical linear algebra which is, "If your algorithm involves computing the inverse of a matrix you should rethink the algorithm because there is a better way of doing it". So why do you need inv()? If the answer is to solve a linear system of equations then you use solve(), you do not use inv(). If, for some truly unusual reason you actually need the inverse (and, remember in 99.99% of the cases where people think they need the inverse, they don't) then you use a combination of solve() and eye().
| Sorry for being dense, I don't know anything about linking R to
| external dependencies. It can be done; there are many examples -- for example every package
using
the GSL.
I just checked how RcppGSL does it, and well, this configure magic is way above my head. Best, baptiste
_______________________________________________ Rcpp-devel mailing list Rcpp-devel at lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20110607/b7797d8e/attachment-0001.htm>
Hi Doug,
On 8 June 2011 03:43, Douglas Bates <bates at stat.wisc.edu> wrote:
On Jun 6, 2011 4:17 AM, "baptiste auguie" <baptiste.auguie at googlemail.com> wrote:
Thank you for the explanations below. On 5 June 2011 10:40, Dirk Eddelbuettel <edd at debian.org> wrote:
On 5 June 2011 at 10:12, baptiste auguie wrote: | Hi Dirk and all, | | On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote: | > | > Baptiste, | > | > On 4 June 2011 at 11:45, baptiste auguie wrote: | > | Dear list, | > | | > | My package cda, which I was hoping to release on CRAN, fails to | > | compile on R-forge with error, | > | | > | ** testing if installed package can be loaded | > | Error in dyn.load(file, DLLpath = DLLpath, ...) : | > | ? unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': | > | ? /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ | > | | > | It builds fine on my local machines (Mac OS 10.5, 10.6). | > | | > | >From an older discussion on this list < | > | http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html> | > | the issue seems to be that Armadillo's inv() relies on a function that | > | is not provided by R, only by LAPACK. I have just replaced inv() by | > | pinv() and solve() in my code; merely to see what happens, but chances | > | are they also require a full LAPACK. | | Indeed, the error on R-forge is now with zgels_, required to solve | linear systems. It seems one cannot solve Armadillo linear systems | without LAPACK in the current situation. Yes. Doug, Romain and myself should address that, or at least make it clear what feature of the full Armadillo are lacking in RcppArmadillo. | > Sometime relatively early in the RcppArmadillo development process, Doug | > convinced Romain and myself to go for a pure template solution with Armadillo | > as all / most things found during the configure (or in this case, cmake) | > stage can be assumed 'found' given that we have around us by design. ?So no | > testing, no local library and full reliance and what R gives us. | > | > That was a brilliant idea, and has freed us from having to rely on building | > and shipping a library, having to tell users how to set PKG_LIBS etc pp and I | > firmly believe that this helped tremendously in getting RcppArmadillo more | > widely used. So I would not want to revert this. | | It sounds like a good choice, I agree. Yet solving a linear system is | quite a crucial task in linear algebra; it would be nice if we could | come up with a tutorial recipe for dummies like me. | | > | > In any event, it seems that you need more LAPACK than R has for you. ?That is | > likely to be a dicey situation as you per se do not know whether R was built | > and linked with its own (subset) copy of LAPACK, or whether it uses system | > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). ?So you may be in | > a spot bother and I not sure what I can recommend --- other than trying your | > luck at some short configure snippets that will run at package build time to | > determine whether the system you want to build cda on it 'rich' enough to | > support it. ?I can help you off list with some configure snippets as some of | > my packages have configure code; adding a test for zgetri should be feasible. | > | > | Does anybody have any experience | > | dealing with such issues w.r.t releasing a package on R-forge / CRAN? | > | Is there any chance they would consider installing LAPACK on those | > | servers, or is there no point in asking such things? | > | > I don't think it is a matter of fixing the R-Forge server. I think it is a | > matter of making your package installable on the largest number of user | > systems. ?Also try win-builder.r-project.org to see how it fares on that | > platform.
Unsurprisingly, it fails, with the same complaint as R-forge.
| | I was under the impression that R-forge or CRAN, if it had LAPACK | installed, could produce binaries for the relevant platforms, and | users would not have to build the package themselves and would not be | required of having LAPACK on their machine. That's probably a | misconception, isn't it? If and only statically linked binaries or libraries where produced, which is generally not the case. Many OSs (Linux incl) ship source only and otherwise link dynamically, others (Windoze) use dynamic linking and OS X is for all I know somewhere in the middle (as you can get prebuild packages with dynamic linking or build from source).
I see; so basically the user will always need to have a full LAPACK installed. Here's one question then, if R-core didn't consider necessary to include those particular functions from LAPACK, presumably that means that R defines its own routines to solve linear systems and invert matrices. Is there any possibility to use those routines with Armadillo?
One important point has been missed in this discussion, which is that the particular Lapack subroutine that is not found in the subset of Lapack shipped with R is for general, complex-valued matrices (just google the name zgetri). ?The way that Armadillo is structured it is either all-in or all-out with respect to complex-valued matrices ?If you allow for complex matrices then you must have all the supporting subroutines for whatever you are calling. ?If you call inv() you need to allow for all the [sdcz]getri routines to be available.
The matrix I need to invert is definitely always complex; in fact, convenient complex algebra is the main attraction of Armadillo for me.
So one possibility is to check the Armadillo sources to see if you can suppress the use of complex-valued matrices when compiling your package. ?A quick glance indicates that this might now be easy. Otherwise, remember the first rule of numerical linear algebra which is, "If your algorithm involves computing the inverse of a matrix you should rethink the algorithm because there is a better way of doing it". ?So why do you need inv()? If the answer is to solve a linear system of equations then you use solve(), you do not use inv(). ?If, for some truly unusual reason you actually need the inverse (and, remember in 99.99% of the cases where people think they need the inverse, they don't) then you use a combination of solve() and eye().
I tried this, and in fact I do use solve() elsewhere in my code, which calls for another LAPACK routine (zgels.f) not present in R. The reason I still need inv() is that I have to solve about 300 times a linear system AX=B with the same matrix A but varying B. A quick timing last week revealed that using solve() 300 times was typically slower by a factor of two in my use case than using inv() once (or pinv() for that matter, it makes not appreciable difference). I'm happy to be shown otherwise though. One option that I'd like to consider is whether the appropriate LAPACK routines could be wrapped and shipped in a separate package (discontinued rblas could provide a good starting point). Sadly, I know nothing about static/dynamic libraries and all this business.. Thanks, baptiste
| Sorry for being dense, I don't know anything about linking R to
| external dependencies. It can be done; there are many examples -- for example every package using the GSL.
I just checked how RcppGSL does it, and well, this configure magic is way above my head. Best, baptiste
_______________________________________________ Rcpp-devel mailing list Rcpp-devel at lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
On Tue, Jun 7, 2011 at 2:34 PM, baptiste auguie
<baptiste.auguie at googlemail.com> wrote:
Hi Doug, On 8 June 2011 03:43, Douglas Bates <bates at stat.wisc.edu> wrote:
On Jun 6, 2011 4:17 AM, "baptiste auguie" <baptiste.auguie at googlemail.com> wrote:
Thank you for the explanations below. On 5 June 2011 10:40, Dirk Eddelbuettel <edd at debian.org> wrote:
On 5 June 2011 at 10:12, baptiste auguie wrote: | Hi Dirk and all, | | On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote: | > | > Baptiste, | > | > On 4 June 2011 at 11:45, baptiste auguie wrote: | > | Dear list, | > | | > | My package cda, which I was hoping to release on CRAN, fails to | > | compile on R-forge with error, | > | | > | ** testing if installed package can be loaded | > | Error in dyn.load(file, DLLpath = DLLpath, ...) : | > | ? unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': | > | ? /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ | > | | > | It builds fine on my local machines (Mac OS 10.5, 10.6). | > | | > | >From an older discussion on this list < | > | http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html> | > | the issue seems to be that Armadillo's inv() relies on a function that | > | is not provided by R, only by LAPACK. I have just replaced inv() by | > | pinv() and solve() in my code; merely to see what happens, but chances | > | are they also require a full LAPACK. | | Indeed, the error on R-forge is now with zgels_, required to solve | linear systems. It seems one cannot solve Armadillo linear systems | without LAPACK in the current situation. Yes. Doug, Romain and myself should address that, or at least make it clear what feature of the full Armadillo are lacking in RcppArmadillo. | > Sometime relatively early in the RcppArmadillo development process, Doug | > convinced Romain and myself to go for a pure template solution with Armadillo | > as all / most things found during the configure (or in this case, cmake) | > stage can be assumed 'found' given that we have around us by design. ?So no | > testing, no local library and full reliance and what R gives us. | > | > That was a brilliant idea, and has freed us from having to rely on building | > and shipping a library, having to tell users how to set PKG_LIBS etc pp and I | > firmly believe that this helped tremendously in getting RcppArmadillo more | > widely used. So I would not want to revert this. | | It sounds like a good choice, I agree. Yet solving a linear system is | quite a crucial task in linear algebra; it would be nice if we could | come up with a tutorial recipe for dummies like me. | | > | > In any event, it seems that you need more LAPACK than R has for you. ?That is | > likely to be a dicey situation as you per se do not know whether R was built | > and linked with its own (subset) copy of LAPACK, or whether it uses system | > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). ?So you may be in | > a spot bother and I not sure what I can recommend --- other than trying your | > luck at some short configure snippets that will run at package build time to | > determine whether the system you want to build cda on it 'rich' enough to | > support it. ?I can help you off list with some configure snippets as some of | > my packages have configure code; adding a test for zgetri should be feasible. | > | > | Does anybody have any experience | > | dealing with such issues w.r.t releasing a package on R-forge / CRAN? | > | Is there any chance they would consider installing LAPACK on those | > | servers, or is there no point in asking such things? | > | > I don't think it is a matter of fixing the R-Forge server. I think it is a | > matter of making your package installable on the largest number of user | > systems. ?Also try win-builder.r-project.org to see how it fares on that | > platform.
Unsurprisingly, it fails, with the same complaint as R-forge.
| | I was under the impression that R-forge or CRAN, if it had LAPACK | installed, could produce binaries for the relevant platforms, and | users would not have to build the package themselves and would not be | required of having LAPACK on their machine. That's probably a | misconception, isn't it? If and only statically linked binaries or libraries where produced, which is generally not the case. Many OSs (Linux incl) ship source only and otherwise link dynamically, others (Windoze) use dynamic linking and OS X is for all I know somewhere in the middle (as you can get prebuild packages with dynamic linking or build from source).
I see; so basically the user will always need to have a full LAPACK installed. Here's one question then, if R-core didn't consider necessary to include those particular functions from LAPACK, presumably that means that R defines its own routines to solve linear systems and invert matrices. Is there any possibility to use those routines with Armadillo?
One important point has been missed in this discussion, which is that the particular Lapack subroutine that is not found in the subset of Lapack shipped with R is for general, complex-valued matrices (just google the name zgetri). ?The way that Armadillo is structured it is either all-in or all-out with respect to complex-valued matrices ?If you allow for complex matrices then you must have all the supporting subroutines for whatever you are calling. ?If you call inv() you need to allow for all the [sdcz]getri routines to be available.
The matrix I need to invert is definitely always complex; in fact, convenient complex algebra is the main attraction of Armadillo for me.
Argh. Of course, I was being foolish. Because Armadillo is a header-only library it does not access any Lapack subroutine until the call is instantiated. I still haven't quite gotten used to thinking only having the headers.
So one possibility is to check the Armadillo sources to see if you can suppress the use of complex-valued matrices when compiling your package. ?A quick glance indicates that this might now be easy. Otherwise, remember the first rule of numerical linear algebra which is, "If your algorithm involves computing the inverse of a matrix you should rethink the algorithm because there is a better way of doing it". ?So why do you need inv()? If the answer is to solve a linear system of equations then you use solve(), you do not use inv(). ?If, for some truly unusual reason you actually need the inverse (and, remember in 99.99% of the cases where people think they need the inverse, they don't) then you use a combination of solve() and eye().
I tried this, and in fact I do use solve() elsewhere in my code, which calls for another LAPACK routine (zgels.f) not present in R. The reason I still need inv() is that I have to solve about 300 times a linear system AX=B with the same matrix A but varying B. A quick timing last week revealed that using solve() 300 times was typically slower by a factor of two ?in my use case than using inv() once (or pinv() for that matter, it makes not appreciable difference). I'm happy to be shown otherwise though.
I forgot that Armadillo doesn't provide a convenient way of using the LU decomposition (that is one of the things that I find frustrating about Armadillo). Did you try a single solve call in which the right-hand side is an identity matrix? On looking at the Armadillo sources it seems that it should call zgesv which is included in R's subset of the Lapack routines.
One option that I'd like to consider is whether the appropriate LAPACK routines could be wrapped and shipped in a separate package (discontinued rblas could provide a good starting point). Sadly, I know nothing about static/dynamic libraries and all this business.. Thanks, baptiste
| Sorry for being dense, I don't know anything about linking R to
| external dependencies. It can be done; there are many examples -- for example every package using the GSL.
I just checked how RcppGSL does it, and well, this configure magic is way above my head. Best, baptiste
_______________________________________________ Rcpp-devel mailing list Rcpp-devel at lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
On 8 June 2011 08:03, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Jun 7, 2011 at 2:34 PM, baptiste auguie <baptiste.auguie at googlemail.com> wrote:
Hi Doug, On 8 June 2011 03:43, Douglas Bates <bates at stat.wisc.edu> wrote:
On Jun 6, 2011 4:17 AM, "baptiste auguie" <baptiste.auguie at googlemail.com> wrote:
Thank you for the explanations below. On 5 June 2011 10:40, Dirk Eddelbuettel <edd at debian.org> wrote:
On 5 June 2011 at 10:12, baptiste auguie wrote: | Hi Dirk and all, | | On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote: | > | > Baptiste, | > | > On 4 June 2011 at 11:45, baptiste auguie wrote: | > | Dear list, | > | | > | My package cda, which I was hoping to release on CRAN, fails to | > | compile on R-forge with error, | > | | > | ** testing if installed package can be loaded | > | Error in dyn.load(file, DLLpath = DLLpath, ...) : | > | ? unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': | > | ? /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ | > | | > | It builds fine on my local machines (Mac OS 10.5, 10.6). | > | | > | >From an older discussion on this list < | > | http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html> | > | the issue seems to be that Armadillo's inv() relies on a function that | > | is not provided by R, only by LAPACK. I have just replaced inv() by | > | pinv() and solve() in my code; merely to see what happens, but chances | > | are they also require a full LAPACK. | | Indeed, the error on R-forge is now with zgels_, required to solve | linear systems. It seems one cannot solve Armadillo linear systems | without LAPACK in the current situation. Yes. Doug, Romain and myself should address that, or at least make it clear what feature of the full Armadillo are lacking in RcppArmadillo. | > Sometime relatively early in the RcppArmadillo development process, Doug | > convinced Romain and myself to go for a pure template solution with Armadillo | > as all / most things found during the configure (or in this case, cmake) | > stage can be assumed 'found' given that we have around us by design. ?So no | > testing, no local library and full reliance and what R gives us. | > | > That was a brilliant idea, and has freed us from having to rely on building | > and shipping a library, having to tell users how to set PKG_LIBS etc pp and I | > firmly believe that this helped tremendously in getting RcppArmadillo more | > widely used. So I would not want to revert this. | | It sounds like a good choice, I agree. Yet solving a linear system is | quite a crucial task in linear algebra; it would be nice if we could | come up with a tutorial recipe for dummies like me. | | > | > In any event, it seems that you need more LAPACK than R has for you. ?That is | > likely to be a dicey situation as you per se do not know whether R was built | > and linked with its own (subset) copy of LAPACK, or whether it uses system | > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). ?So you may be in | > a spot bother and I not sure what I can recommend --- other than trying your | > luck at some short configure snippets that will run at package build time to | > determine whether the system you want to build cda on it 'rich' enough to | > support it. ?I can help you off list with some configure snippets as some of | > my packages have configure code; adding a test for zgetri should be feasible. | > | > | Does anybody have any experience | > | dealing with such issues w.r.t releasing a package on R-forge / CRAN? | > | Is there any chance they would consider installing LAPACK on those | > | servers, or is there no point in asking such things? | > | > I don't think it is a matter of fixing the R-Forge server. I think it is a | > matter of making your package installable on the largest number of user | > systems. ?Also try win-builder.r-project.org to see how it fares on that | > platform.
Unsurprisingly, it fails, with the same complaint as R-forge.
| | I was under the impression that R-forge or CRAN, if it had LAPACK | installed, could produce binaries for the relevant platforms, and | users would not have to build the package themselves and would not be | required of having LAPACK on their machine. That's probably a | misconception, isn't it? If and only statically linked binaries or libraries where produced, which is generally not the case. Many OSs (Linux incl) ship source only and otherwise link dynamically, others (Windoze) use dynamic linking and OS X is for all I know somewhere in the middle (as you can get prebuild packages with dynamic linking or build from source).
I see; so basically the user will always need to have a full LAPACK installed. Here's one question then, if R-core didn't consider necessary to include those particular functions from LAPACK, presumably that means that R defines its own routines to solve linear systems and invert matrices. Is there any possibility to use those routines with Armadillo?
One important point has been missed in this discussion, which is that the particular Lapack subroutine that is not found in the subset of Lapack shipped with R is for general, complex-valued matrices (just google the name zgetri). ?The way that Armadillo is structured it is either all-in or all-out with respect to complex-valued matrices ?If you allow for complex matrices then you must have all the supporting subroutines for whatever you are calling. ?If you call inv() you need to allow for all the [sdcz]getri routines to be available.
The matrix I need to invert is definitely always complex; in fact, convenient complex algebra is the main attraction of Armadillo for me.
Argh. ?Of course, I was being foolish. ?Because Armadillo is a header-only library it does not access any Lapack subroutine until the call is instantiated. ?I still haven't quite gotten used to thinking only having the headers.
So one possibility is to check the Armadillo sources to see if you can suppress the use of complex-valued matrices when compiling your package. ?A quick glance indicates that this might now be easy. Otherwise, remember the first rule of numerical linear algebra which is, "If your algorithm involves computing the inverse of a matrix you should rethink the algorithm because there is a better way of doing it". ?So why do you need inv()? If the answer is to solve a linear system of equations then you use solve(), you do not use inv(). ?If, for some truly unusual reason you actually need the inverse (and, remember in 99.99% of the cases where people think they need the inverse, they don't) then you use a combination of solve() and eye().
I tried this, and in fact I do use solve() elsewhere in my code, which calls for another LAPACK routine (zgels.f) not present in R. The reason I still need inv() is that I have to solve about 300 times a linear system AX=B with the same matrix A but varying B. A quick timing last week revealed that using solve() 300 times was typically slower by a factor of two ?in my use case than using inv() once (or pinv() for that matter, it makes not appreciable difference). I'm happy to be shown otherwise though.
I forgot that Armadillo doesn't provide a convenient way of using the LU decomposition (that is one of the things that I find frustrating about Armadillo). ?Did you try a single solve call in which the right-hand side is an identity matrix? ?On looking at the Armadillo sources it seems that it should call zgesv which is included in R's subset of the Lapack routines.
It seems curious, looking at the bestiary of LAPACK functions, that so many of them seem to perform a similar task. I wonder what are the practical downsides of using a code that solves AX=I rather than one inverting A. Also, when I replaced inv() by solve() and pinv(), R-forge still failed to build complaining that zgesv_ was not present. I assumed it was used by solve(), but perhaps it was pinv(). I'll give it a try anyway, but I'd rather hope we can figure out a less ad hoc solution. Thanks, baptiste
One option that I'd like to consider is whether the appropriate LAPACK routines could be wrapped and shipped in a separate package (discontinued rblas could provide a good starting point). Sadly, I know nothing about static/dynamic libraries and all this business.. Thanks, baptiste
| Sorry for being dense, I don't know anything about linking R to
| external dependencies. It can be done; there are many examples -- for example every package using the GSL.
I just checked how RcppGSL does it, and well, this configure magic is way above my head. Best, baptiste
_______________________________________________ Rcpp-devel mailing list Rcpp-devel at lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
On Tue, Jun 7, 2011 at 3:47 PM, baptiste auguie
<baptiste.auguie at googlemail.com> wrote:
On 8 June 2011 08:03, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Jun 7, 2011 at 2:34 PM, baptiste auguie <baptiste.auguie at googlemail.com> wrote:
Hi Doug, On 8 June 2011 03:43, Douglas Bates <bates at stat.wisc.edu> wrote:
On Jun 6, 2011 4:17 AM, "baptiste auguie" <baptiste.auguie at googlemail.com> wrote:
Thank you for the explanations below. On 5 June 2011 10:40, Dirk Eddelbuettel <edd at debian.org> wrote:
On 5 June 2011 at 10:12, baptiste auguie wrote: | Hi Dirk and all, | | On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote: | > | > Baptiste, | > | > On 4 June 2011 at 11:45, baptiste auguie wrote: | > | Dear list, | > | | > | My package cda, which I was hoping to release on CRAN, fails to | > | compile on R-forge with error, | > | | > | ** testing if installed package can be loaded | > | Error in dyn.load(file, DLLpath = DLLpath, ...) : | > | ? unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': | > | ? /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ | > | | > | It builds fine on my local machines (Mac OS 10.5, 10.6). | > | | > | >From an older discussion on this list < | > | http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html> | > | the issue seems to be that Armadillo's inv() relies on a function that | > | is not provided by R, only by LAPACK. I have just replaced inv() by | > | pinv() and solve() in my code; merely to see what happens, but chances | > | are they also require a full LAPACK. | | Indeed, the error on R-forge is now with zgels_, required to solve | linear systems. It seems one cannot solve Armadillo linear systems | without LAPACK in the current situation. Yes. Doug, Romain and myself should address that, or at least make it clear what feature of the full Armadillo are lacking in RcppArmadillo. | > Sometime relatively early in the RcppArmadillo development process, Doug | > convinced Romain and myself to go for a pure template solution with Armadillo | > as all / most things found during the configure (or in this case, cmake) | > stage can be assumed 'found' given that we have around us by design. ?So no | > testing, no local library and full reliance and what R gives us. | > | > That was a brilliant idea, and has freed us from having to rely on building | > and shipping a library, having to tell users how to set PKG_LIBS etc pp and I | > firmly believe that this helped tremendously in getting RcppArmadillo more | > widely used. So I would not want to revert this. | | It sounds like a good choice, I agree. Yet solving a linear system is | quite a crucial task in linear algebra; it would be nice if we could | come up with a tutorial recipe for dummies like me. | | > | > In any event, it seems that you need more LAPACK than R has for you. ?That is | > likely to be a dicey situation as you per se do not know whether R was built | > and linked with its own (subset) copy of LAPACK, or whether it uses system | > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). ?So you may be in | > a spot bother and I not sure what I can recommend --- other than trying your | > luck at some short configure snippets that will run at package build time to | > determine whether the system you want to build cda on it 'rich' enough to | > support it. ?I can help you off list with some configure snippets as some of | > my packages have configure code; adding a test for zgetri should be feasible. | > | > | Does anybody have any experience | > | dealing with such issues w.r.t releasing a package on R-forge / CRAN? | > | Is there any chance they would consider installing LAPACK on those | > | servers, or is there no point in asking such things? | > | > I don't think it is a matter of fixing the R-Forge server. I think it is a | > matter of making your package installable on the largest number of user | > systems. ?Also try win-builder.r-project.org to see how it fares on that | > platform.
Unsurprisingly, it fails, with the same complaint as R-forge.
| | I was under the impression that R-forge or CRAN, if it had LAPACK | installed, could produce binaries for the relevant platforms, and | users would not have to build the package themselves and would not be | required of having LAPACK on their machine. That's probably a | misconception, isn't it? If and only statically linked binaries or libraries where produced, which is generally not the case. Many OSs (Linux incl) ship source only and otherwise link dynamically, others (Windoze) use dynamic linking and OS X is for all I know somewhere in the middle (as you can get prebuild packages with dynamic linking or build from source).
I see; so basically the user will always need to have a full LAPACK installed. Here's one question then, if R-core didn't consider necessary to include those particular functions from LAPACK, presumably that means that R defines its own routines to solve linear systems and invert matrices. Is there any possibility to use those routines with Armadillo?
One important point has been missed in this discussion, which is that the particular Lapack subroutine that is not found in the subset of Lapack shipped with R is for general, complex-valued matrices (just google the name zgetri). ?The way that Armadillo is structured it is either all-in or all-out with respect to complex-valued matrices ?If you allow for complex matrices then you must have all the supporting subroutines for whatever you are calling. ?If you call inv() you need to allow for all the [sdcz]getri routines to be available.
The matrix I need to invert is definitely always complex; in fact, convenient complex algebra is the main attraction of Armadillo for me.
Argh. ?Of course, I was being foolish. ?Because Armadillo is a header-only library it does not access any Lapack subroutine until the call is instantiated. ?I still haven't quite gotten used to thinking only having the headers.
So one possibility is to check the Armadillo sources to see if you can suppress the use of complex-valued matrices when compiling your package. ?A quick glance indicates that this might now be easy. Otherwise, remember the first rule of numerical linear algebra which is, "If your algorithm involves computing the inverse of a matrix you should rethink the algorithm because there is a better way of doing it". ?So why do you need inv()? If the answer is to solve a linear system of equations then you use solve(), you do not use inv(). ?If, for some truly unusual reason you actually need the inverse (and, remember in 99.99% of the cases where people think they need the inverse, they don't) then you use a combination of solve() and eye().
I tried this, and in fact I do use solve() elsewhere in my code, which calls for another LAPACK routine (zgels.f) not present in R. The reason I still need inv() is that I have to solve about 300 times a linear system AX=B with the same matrix A but varying B. A quick timing last week revealed that using solve() 300 times was typically slower by a factor of two ?in my use case than using inv() once (or pinv() for that matter, it makes not appreciable difference). I'm happy to be shown otherwise though.
I forgot that Armadillo doesn't provide a convenient way of using the LU decomposition (that is one of the things that I find frustrating about Armadillo). ?Did you try a single solve call in which the right-hand side is an identity matrix? ?On looking at the Armadillo sources it seems that it should call zgesv which is included in R's subset of the Lapack routines.
It seems curious, looking at the bestiary of LAPACK functions, that so many of them seem to perform a similar task. I wonder what are the practical downsides of using a code that solves AX=I rather than one inverting A. Also, when I replaced inv() by solve() and pinv(), R-forge still failed to build complaining that zgesv_ was not present. I assumed it was used by solve(), but perhaps it was pinv(). I'll give it a try anyway, but I'd rather hope we can figure out a less ad hoc solution.
Once upon a time we actually counted the number of floating point operations that were needed to perform a particular calculation, which is the reason for all the special-case code in Lapack. The difference between using the LU decomposition of a matrix to solve a system in which the right hand side is an identity, versus a special-purpose piece of code like zgetri is that zgetri can take advantage of the fact that the identity matrix is diagonal, thereby saving a relatively small number of operations. I'm not sure why there should be a complaint about zgesv_ not being available. It's there in the R sources (src/modules/lapack/cmplx.f starting at line 3944). That routine is trivial because it just checks its arguments then calls zgetrf and zgetrs.
Thanks, baptiste
One option that I'd like to consider is whether the appropriate LAPACK routines could be wrapped and shipped in a separate package (discontinued rblas could provide a good starting point). Sadly, I know nothing about static/dynamic libraries and all this business.. Thanks, baptiste
| Sorry for being dense, I don't know anything about linking R to
| external dependencies. It can be done; there are many examples -- for example every package using the GSL.
I just checked how RcppGSL does it, and well, this configure magic is way above my head. Best, baptiste
_______________________________________________ Rcpp-devel mailing list Rcpp-devel at lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
On 8 June 2011 09:04, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Jun 7, 2011 at 3:47 PM, baptiste auguie <baptiste.auguie at googlemail.com> wrote:
On 8 June 2011 08:03, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Jun 7, 2011 at 2:34 PM, baptiste auguie <baptiste.auguie at googlemail.com> wrote:
Hi Doug, On 8 June 2011 03:43, Douglas Bates <bates at stat.wisc.edu> wrote:
On Jun 6, 2011 4:17 AM, "baptiste auguie" <baptiste.auguie at googlemail.com> wrote:
Thank you for the explanations below. On 5 June 2011 10:40, Dirk Eddelbuettel <edd at debian.org> wrote:
On 5 June 2011 at 10:12, baptiste auguie wrote: | Hi Dirk and all, | | On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote: | > | > Baptiste, | > | > On 4 June 2011 at 11:45, baptiste auguie wrote: | > | Dear list, | > | | > | My package cda, which I was hoping to release on CRAN, fails to | > | compile on R-forge with error, | > | | > | ** testing if installed package can be loaded | > | Error in dyn.load(file, DLLpath = DLLpath, ...) : | > | ? unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': | > | ? /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ | > | | > | It builds fine on my local machines (Mac OS 10.5, 10.6). | > | | > | >From an older discussion on this list < | > | http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html> | > | the issue seems to be that Armadillo's inv() relies on a function that | > | is not provided by R, only by LAPACK. I have just replaced inv() by | > | pinv() and solve() in my code; merely to see what happens, but chances | > | are they also require a full LAPACK. | | Indeed, the error on R-forge is now with zgels_, required to solve | linear systems. It seems one cannot solve Armadillo linear systems | without LAPACK in the current situation. Yes. Doug, Romain and myself should address that, or at least make it clear what feature of the full Armadillo are lacking in RcppArmadillo. | > Sometime relatively early in the RcppArmadillo development process, Doug | > convinced Romain and myself to go for a pure template solution with Armadillo | > as all / most things found during the configure (or in this case, cmake) | > stage can be assumed 'found' given that we have around us by design. ?So no | > testing, no local library and full reliance and what R gives us. | > | > That was a brilliant idea, and has freed us from having to rely on building | > and shipping a library, having to tell users how to set PKG_LIBS etc pp and I | > firmly believe that this helped tremendously in getting RcppArmadillo more | > widely used. So I would not want to revert this. | | It sounds like a good choice, I agree. Yet solving a linear system is | quite a crucial task in linear algebra; it would be nice if we could | come up with a tutorial recipe for dummies like me. | | > | > In any event, it seems that you need more LAPACK than R has for you. ?That is | > likely to be a dicey situation as you per se do not know whether R was built | > and linked with its own (subset) copy of LAPACK, or whether it uses system | > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). ?So you may be in | > a spot bother and I not sure what I can recommend --- other than trying your | > luck at some short configure snippets that will run at package build time to | > determine whether the system you want to build cda on it 'rich' enough to | > support it. ?I can help you off list with some configure snippets as some of | > my packages have configure code; adding a test for zgetri should be feasible. | > | > | Does anybody have any experience | > | dealing with such issues w.r.t releasing a package on R-forge / CRAN? | > | Is there any chance they would consider installing LAPACK on those | > | servers, or is there no point in asking such things? | > | > I don't think it is a matter of fixing the R-Forge server. I think it is a | > matter of making your package installable on the largest number of user | > systems. ?Also try win-builder.r-project.org to see how it fares on that | > platform.
Unsurprisingly, it fails, with the same complaint as R-forge.
| | I was under the impression that R-forge or CRAN, if it had LAPACK | installed, could produce binaries for the relevant platforms, and | users would not have to build the package themselves and would not be | required of having LAPACK on their machine. That's probably a | misconception, isn't it? If and only statically linked binaries or libraries where produced, which is generally not the case. Many OSs (Linux incl) ship source only and otherwise link dynamically, others (Windoze) use dynamic linking and OS X is for all I know somewhere in the middle (as you can get prebuild packages with dynamic linking or build from source).
I see; so basically the user will always need to have a full LAPACK installed. Here's one question then, if R-core didn't consider necessary to include those particular functions from LAPACK, presumably that means that R defines its own routines to solve linear systems and invert matrices. Is there any possibility to use those routines with Armadillo?
One important point has been missed in this discussion, which is that the particular Lapack subroutine that is not found in the subset of Lapack shipped with R is for general, complex-valued matrices (just google the name zgetri). ?The way that Armadillo is structured it is either all-in or all-out with respect to complex-valued matrices ?If you allow for complex matrices then you must have all the supporting subroutines for whatever you are calling. ?If you call inv() you need to allow for all the [sdcz]getri routines to be available.
The matrix I need to invert is definitely always complex; in fact, convenient complex algebra is the main attraction of Armadillo for me.
Argh. ?Of course, I was being foolish. ?Because Armadillo is a header-only library it does not access any Lapack subroutine until the call is instantiated. ?I still haven't quite gotten used to thinking only having the headers.
So one possibility is to check the Armadillo sources to see if you can suppress the use of complex-valued matrices when compiling your package. ?A quick glance indicates that this might now be easy. Otherwise, remember the first rule of numerical linear algebra which is, "If your algorithm involves computing the inverse of a matrix you should rethink the algorithm because there is a better way of doing it". ?So why do you need inv()? If the answer is to solve a linear system of equations then you use solve(), you do not use inv(). ?If, for some truly unusual reason you actually need the inverse (and, remember in 99.99% of the cases where people think they need the inverse, they don't) then you use a combination of solve() and eye().
I tried this, and in fact I do use solve() elsewhere in my code, which calls for another LAPACK routine (zgels.f) not present in R. The reason I still need inv() is that I have to solve about 300 times a linear system AX=B with the same matrix A but varying B. A quick timing last week revealed that using solve() 300 times was typically slower by a factor of two ?in my use case than using inv() once (or pinv() for that matter, it makes not appreciable difference). I'm happy to be shown otherwise though.
I forgot that Armadillo doesn't provide a convenient way of using the LU decomposition (that is one of the things that I find frustrating about Armadillo). ?Did you try a single solve call in which the right-hand side is an identity matrix? ?On looking at the Armadillo sources it seems that it should call zgesv which is included in R's subset of the Lapack routines.
It seems curious, looking at the bestiary of LAPACK functions, that so many of them seem to perform a similar task. I wonder what are the practical downsides of using a code that solves AX=I rather than one inverting A. Also, when I replaced inv() by solve() and pinv(), R-forge still failed to build complaining that zgesv_ was not present. I assumed it was used by solve(), but perhaps it was pinv(). I'll give it a try anyway, but I'd rather hope we can figure out a less ad hoc solution.
Once upon a time we actually counted the number of floating point operations that were needed to perform a particular calculation, which is the reason for all the special-case code in Lapack. ?The difference between using the LU decomposition of a matrix to solve a system in which the right hand side is an identity, versus a special-purpose piece of code like zgetri is that zgetri can take advantage of the fact that the identity matrix is diagonal, thereby saving a relatively small number of operations.
That makes sense, thanks.
I'm not sure why there should be a complaint about zgesv_ not being available. ?It's there in the R sources (src/modules/lapack/cmplx.f starting at line 3944). ?That routine is trivial because it just checks its arguments then calls zgetrf and zgetrs.
Sorry, I meant zgels_ not zgesv_ (who came up with those names?!). You can see the error here: https://r-forge.r-project.org/R/?group_id=160&log=build_src&pkg=cda&flavor=patched Best, baptiste
Thanks, baptiste
One option that I'd like to consider is whether the appropriate LAPACK routines could be wrapped and shipped in a separate package (discontinued rblas could provide a good starting point). Sadly, I know nothing about static/dynamic libraries and all this business.. Thanks, baptiste
| Sorry for being dense, I don't know anything about linking R to
| external dependencies. It can be done; there are many examples -- for example every package using the GSL.
I just checked how RcppGSL does it, and well, this configure magic is way above my head. Best, baptiste
_______________________________________________ Rcpp-devel mailing list Rcpp-devel at lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
On Tue, Jun 7, 2011 at 4:20 PM, baptiste auguie
<baptiste.auguie at googlemail.com> wrote:
On 8 June 2011 09:04, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Jun 7, 2011 at 3:47 PM, baptiste auguie <baptiste.auguie at googlemail.com> wrote:
On 8 June 2011 08:03, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Jun 7, 2011 at 2:34 PM, baptiste auguie <baptiste.auguie at googlemail.com> wrote:
Hi Doug, On 8 June 2011 03:43, Douglas Bates <bates at stat.wisc.edu> wrote:
On Jun 6, 2011 4:17 AM, "baptiste auguie" <baptiste.auguie at googlemail.com> wrote:
Thank you for the explanations below. On 5 June 2011 10:40, Dirk Eddelbuettel <edd at debian.org> wrote:
On 5 June 2011 at 10:12, baptiste auguie wrote: | Hi Dirk and all, | | On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote: | > | > Baptiste, | > | > On 4 June 2011 at 11:45, baptiste auguie wrote: | > | Dear list, | > | | > | My package cda, which I was hoping to release on CRAN, fails to | > | compile on R-forge with error, | > | | > | ** testing if installed package can be loaded | > | Error in dyn.load(file, DLLpath = DLLpath, ...) : | > | ? unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': | > | ? /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ | > | | > | It builds fine on my local machines (Mac OS 10.5, 10.6). | > | | > | >From an older discussion on this list < | > | http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html> | > | the issue seems to be that Armadillo's inv() relies on a function that | > | is not provided by R, only by LAPACK. I have just replaced inv() by | > | pinv() and solve() in my code; merely to see what happens, but chances | > | are they also require a full LAPACK. | | Indeed, the error on R-forge is now with zgels_, required to solve | linear systems. It seems one cannot solve Armadillo linear systems | without LAPACK in the current situation. Yes. Doug, Romain and myself should address that, or at least make it clear what feature of the full Armadillo are lacking in RcppArmadillo. | > Sometime relatively early in the RcppArmadillo development process, Doug | > convinced Romain and myself to go for a pure template solution with Armadillo | > as all / most things found during the configure (or in this case, cmake) | > stage can be assumed 'found' given that we have around us by design. ?So no | > testing, no local library and full reliance and what R gives us. | > | > That was a brilliant idea, and has freed us from having to rely on building | > and shipping a library, having to tell users how to set PKG_LIBS etc pp and I | > firmly believe that this helped tremendously in getting RcppArmadillo more | > widely used. So I would not want to revert this. | | It sounds like a good choice, I agree. Yet solving a linear system is | quite a crucial task in linear algebra; it would be nice if we could | come up with a tutorial recipe for dummies like me. | | > | > In any event, it seems that you need more LAPACK than R has for you. ?That is | > likely to be a dicey situation as you per se do not know whether R was built | > and linked with its own (subset) copy of LAPACK, or whether it uses system | > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). ?So you may be in | > a spot bother and I not sure what I can recommend --- other than trying your | > luck at some short configure snippets that will run at package build time to | > determine whether the system you want to build cda on it 'rich' enough to | > support it. ?I can help you off list with some configure snippets as some of | > my packages have configure code; adding a test for zgetri should be feasible. | > | > | Does anybody have any experience | > | dealing with such issues w.r.t releasing a package on R-forge / CRAN? | > | Is there any chance they would consider installing LAPACK on those | > | servers, or is there no point in asking such things? | > | > I don't think it is a matter of fixing the R-Forge server. I think it is a | > matter of making your package installable on the largest number of user | > systems. ?Also try win-builder.r-project.org to see how it fares on that | > platform.
Unsurprisingly, it fails, with the same complaint as R-forge.
| | I was under the impression that R-forge or CRAN, if it had LAPACK | installed, could produce binaries for the relevant platforms, and | users would not have to build the package themselves and would not be | required of having LAPACK on their machine. That's probably a | misconception, isn't it? If and only statically linked binaries or libraries where produced, which is generally not the case. Many OSs (Linux incl) ship source only and otherwise link dynamically, others (Windoze) use dynamic linking and OS X is for all I know somewhere in the middle (as you can get prebuild packages with dynamic linking or build from source).
I see; so basically the user will always need to have a full LAPACK installed. Here's one question then, if R-core didn't consider necessary to include those particular functions from LAPACK, presumably that means that R defines its own routines to solve linear systems and invert matrices. Is there any possibility to use those routines with Armadillo?
One important point has been missed in this discussion, which is that the particular Lapack subroutine that is not found in the subset of Lapack shipped with R is for general, complex-valued matrices (just google the name zgetri). ?The way that Armadillo is structured it is either all-in or all-out with respect to complex-valued matrices ?If you allow for complex matrices then you must have all the supporting subroutines for whatever you are calling. ?If you call inv() you need to allow for all the [sdcz]getri routines to be available.
The matrix I need to invert is definitely always complex; in fact, convenient complex algebra is the main attraction of Armadillo for me.
Argh. ?Of course, I was being foolish. ?Because Armadillo is a header-only library it does not access any Lapack subroutine until the call is instantiated. ?I still haven't quite gotten used to thinking only having the headers.
So one possibility is to check the Armadillo sources to see if you can suppress the use of complex-valued matrices when compiling your package. ?A quick glance indicates that this might now be easy. Otherwise, remember the first rule of numerical linear algebra which is, "If your algorithm involves computing the inverse of a matrix you should rethink the algorithm because there is a better way of doing it". ?So why do you need inv()? If the answer is to solve a linear system of equations then you use solve(), you do not use inv(). ?If, for some truly unusual reason you actually need the inverse (and, remember in 99.99% of the cases where people think they need the inverse, they don't) then you use a combination of solve() and eye().
I tried this, and in fact I do use solve() elsewhere in my code, which calls for another LAPACK routine (zgels.f) not present in R. The reason I still need inv() is that I have to solve about 300 times a linear system AX=B with the same matrix A but varying B. A quick timing last week revealed that using solve() 300 times was typically slower by a factor of two ?in my use case than using inv() once (or pinv() for that matter, it makes not appreciable difference). I'm happy to be shown otherwise though.
I forgot that Armadillo doesn't provide a convenient way of using the LU decomposition (that is one of the things that I find frustrating about Armadillo). ?Did you try a single solve call in which the right-hand side is an identity matrix? ?On looking at the Armadillo sources it seems that it should call zgesv which is included in R's subset of the Lapack routines.
It seems curious, looking at the bestiary of LAPACK functions, that so many of them seem to perform a similar task. I wonder what are the practical downsides of using a code that solves AX=I rather than one inverting A. Also, when I replaced inv() by solve() and pinv(), R-forge still failed to build complaining that zgesv_ was not present. I assumed it was used by solve(), but perhaps it was pinv(). I'll give it a try anyway, but I'd rather hope we can figure out a less ad hoc solution.
Once upon a time we actually counted the number of floating point operations that were needed to perform a particular calculation, which is the reason for all the special-case code in Lapack. ?The difference between using the LU decomposition of a matrix to solve a system in which the right hand side is an identity, versus a special-purpose piece of code like zgetri is that zgetri can take advantage of the fact that the identity matrix is diagonal, thereby saving a relatively small number of operations.
That makes sense, thanks.
I'm not sure why there should be a complaint about zgesv_ not being available. ?It's there in the R sources (src/modules/lapack/cmplx.f starting at line 3944). ?That routine is trivial because it just checks its arguments then calls zgetrf and zgetrs.
Sorry, I meant zgels_ not zgesv_ (who came up with those names?!).
Fortran programmers. Just as the founders of Microsoft decided that no one would ever have more than some amount of memory (128K, IIRC) those defining the Fortran specification in the 1950's - 60's declared that you would never need more than 6 upper-case characters in an identifier. You parse the name according to the first letter indicates the storage type (z for double precision complex), the next two for specific characteristics of the matrix (ge -> general, tr -> triangular, sy -> symmetric, po -> symmetric and positive definite, ...) and the remaining two or three for the operation (trf -> triangular factorization, sl -> solution of linear systems, ls -> least squares solution). You should only have a request for zgels if you are trying to get least squares solutions. The complex Lapack routines that you can count on being available are cmplx.f:96: SUBROUTINE ZBDSQR( UPLO, N, NCVT, NRU, NCC, D, E, VT, LDVT, U, cmplx.f:838: SUBROUTINE ZDRSCL( N, SA, SX, INCX ) cmplx.f:952: SUBROUTINE ZGEBAK( JOB, SIDE, N, ILO, IHI, SCALE, M, V, LDV, cmplx.f:1141: SUBROUTINE ZGEBAL( JOB, N, A, LDA, ILO, IHI, SCALE, INFO ) cmplx.f:1471: SUBROUTINE ZGEBD2( M, N, A, LDA, D, E, TAUQ, TAUP, WORK, INFO ) cmplx.f:1721: SUBROUTINE ZGEBRD( M, N, A, LDA, D, E, TAUQ, TAUP, WORK, LWORK, cmplx.f:1989: SUBROUTINE ZGECON( NORM, N, A, LDA, ANORM, RCOND, WORK, RWORK, cmplx.f:2182: SUBROUTINE ZGEEV( JOBVL, JOBVR, N, A, LDA, W, VL, LDVL, VR, LDVR, cmplx.f:2578: SUBROUTINE ZGEHD2( N, ILO, IHI, A, LDA, TAU, WORK, INFO ) cmplx.f:2726: SUBROUTINE ZGEHRD( N, ILO, IHI, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:2999: SUBROUTINE ZGELQ2( M, N, A, LDA, TAU, WORK, INFO ) cmplx.f:3122: SUBROUTINE ZGELQF( M, N, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:3317: SUBROUTINE ZGEQP3( M, N, A, LDA, JPVT, TAU, WORK, LWORK, RWORK, cmplx.f:3610: SUBROUTINE ZGEQR2( M, N, A, LDA, TAU, WORK, INFO ) cmplx.f:3731: SUBROUTINE ZGEQRF( M, N, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:3927: SUBROUTINE ZGESV( N, NRHS, A, LDA, IPIV, B, LDB, INFO ) cmplx.f:4034: SUBROUTINE ZGESVD( JOBU, JOBVT, M, N, A, LDA, S, U, LDU, VT, LDVT, cmplx.f:7636: SUBROUTINE ZGETF2( M, N, A, LDA, IPIV, INFO ) cmplx.f:7784: SUBROUTINE ZGETRF( M, N, A, LDA, IPIV, INFO ) cmplx.f:7943: SUBROUTINE ZGETRS( TRANS, N, NRHS, A, LDA, IPIV, B, LDB, INFO ) cmplx.f:8092: SUBROUTINE ZHEEV( JOBZ, UPLO, N, A, LDA, W, WORK, LWORK, RWORK, cmplx.f:8310: SUBROUTINE ZHETD2( UPLO, N, A, LDA, D, E, TAU, INFO ) cmplx.f:8568: SUBROUTINE ZHETRD( UPLO, N, A, LDA, D, E, TAU, WORK, LWORK, INFO ) cmplx.f:8864: SUBROUTINE ZHSEQR( JOB, COMPZ, N, ILO, IHI, H, LDH, W, Z, LDZ, cmplx.f:9259: SUBROUTINE ZLABRD( M, N, NB, A, LDA, D, E, TAUQ, TAUP, X, LDX, Y, cmplx.f:9587: SUBROUTINE ZLACGV( N, X, INCX ) cmplx.f:9647: SUBROUTINE ZLACN2( N, V, X, EST, KASE, ISAVE ) cmplx.f:9868: SUBROUTINE ZLACPY( UPLO, M, N, A, LDA, B, LDB ) cmplx.f:10004: SUBROUTINE ZLAHQR( WANTT, WANTZ, N, ILO, IHI, H, LDH, W, ILOZ, cmplx.f:10474: SUBROUTINE ZLAHR2( N, K, NB, A, LDA, TAU, T, LDT, Y, LDY ) cmplx.f:10714: SUBROUTINE ZLAHRD( N, K, NB, A, LDA, TAU, T, LDT, Y, LDY ) cmplx.f:11678: SUBROUTINE ZLAQP2( M, N, OFFSET, A, LDA, JPVT, TAU, VN1, VN2, cmplx.f:11857: SUBROUTINE ZLAQPS( M, N, OFFSET, NB, KB, A, LDA, JPVT, TAU, VN1, cmplx.f:12123: SUBROUTINE ZLAQR0( WANTT, WANTZ, N, ILO, IHI, H, LDH, W, ILOZ, cmplx.f:12724: SUBROUTINE ZLAQR1( N, H, LDH, S1, S2, V ) cmplx.f:12821: SUBROUTINE ZLAQR2( WANTT, WANTZ, N, KTOP, KBOT, NW, H, LDH, ILOZ, cmplx.f:13258: SUBROUTINE ZLAQR3( WANTT, WANTZ, N, KTOP, KBOT, NW, H, LDH, ILOZ, cmplx.f:13706: SUBROUTINE ZLAQR4( WANTT, WANTZ, N, ILO, IHI, H, LDH, W, ILOZ, cmplx.f:14308: SUBROUTINE ZLAQR5( WANTT, WANTZ, KACC22, N, KTOP, KBOT, NSHFTS, S, cmplx.f:15117: SUBROUTINE ZLARF( SIDE, M, N, V, INCV, TAU, C, LDC, WORK ) cmplx.f:15237: SUBROUTINE ZLARFB( SIDE, TRANS, DIRECT, STOREV, M, N, K, V, LDV, cmplx.f:15845: SUBROUTINE ZLARFG( N, ALPHA, X, INCX, TAU ) cmplx.f:15990: SUBROUTINE ZLARFT( DIRECT, STOREV, N, K, V, LDV, TAU, T, LDT ) cmplx.f:16214: SUBROUTINE ZLARFX( SIDE, M, N, V, TAU, C, LDC, WORK ) cmplx.f:16855: SUBROUTINE ZLARTG( F, G, CS, SN, R ) cmplx.f:17050: SUBROUTINE ZLASCL( TYPE, KL, KU, CFROM, CTO, M, N, A, LDA, INFO ) cmplx.f:17317: SUBROUTINE ZLASET( UPLO, M, N, ALPHA, BETA, A, LDA ) cmplx.f:17431: SUBROUTINE ZLASR( SIDE, PIVOT, DIRECT, M, N, C, S, A, LDA ) cmplx.f:17794: SUBROUTINE ZLASSQ( N, X, INCX, SCALE, SUMSQ ) cmplx.f:17895: SUBROUTINE ZLASWP( N, A, LDA, K1, K2, IPIV, INCX ) cmplx.f:18014: SUBROUTINE ZLATRD( UPLO, N, NB, A, LDA, E, TAU, W, LDW ) cmplx.f:18293: SUBROUTINE ZLATRS( UPLO, TRANS, DIAG, NORMIN, N, A, LDA, X, SCALE, cmplx.f:19172: SUBROUTINE ZPOTF2( UPLO, N, A, LDA, INFO ) cmplx.f:19346: SUBROUTINE ZPOTRF( UPLO, N, A, LDA, INFO ) cmplx.f:19532: SUBROUTINE ZROT( N, CX, INCX, CY, INCY, C, S ) cmplx.f:19623: SUBROUTINE ZSTEQR( COMPZ, N, D, E, Z, LDZ, WORK, INFO ) cmplx.f:20126: SUBROUTINE ZTRCON( NORM, UPLO, DIAG, N, A, LDA, RCOND, WORK, cmplx.f:20330: SUBROUTINE ZTREVC( SIDE, HOWMNY, SELECT, N, T, LDT, VL, LDVL, VR, cmplx.f:20716: SUBROUTINE ZTREXC( COMPQ, N, T, LDT, Q, LDQ, IFST, ILST, INFO ) cmplx.f:20878: SUBROUTINE ZTRTRS( UPLO, TRANS, DIAG, N, NRHS, A, LDA, B, LDB, cmplx.f:21026: SUBROUTINE ZUNG2L( M, N, K, A, LDA, TAU, WORK, INFO ) cmplx.f:21154: SUBROUTINE ZUNG2R( M, N, K, A, LDA, TAU, WORK, INFO ) cmplx.f:21284: SUBROUTINE ZUNGBR( VECT, M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:21529: SUBROUTINE ZUNGHR( N, ILO, IHI, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:21694: SUBROUTINE ZUNGL2( M, N, K, A, LDA, TAU, WORK, INFO ) cmplx.f:21830: SUBROUTINE ZUNGLQ( M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:22045: SUBROUTINE ZUNGQL( M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:22267: SUBROUTINE ZUNGQR( M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:22483: SUBROUTINE ZUNGR2( M, N, K, A, LDA, TAU, WORK, INFO ) cmplx.f:22617: SUBROUTINE ZUNGRQ( M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:22840: SUBROUTINE ZUNGTR( UPLO, N, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:23024: SUBROUTINE ZUNM2R( SIDE, TRANS, M, N, K, A, LDA, TAU, C, LDC, cmplx.f:23225: SUBROUTINE ZUNMBR( VECT, SIDE, TRANS, M, N, K, A, LDA, TAU, C, cmplx.f:23513: SUBROUTINE ZUNML2( SIDE, TRANS, M, N, K, A, LDA, TAU, C, LDC, cmplx.f:23718: SUBROUTINE ZUNMLQ( SIDE, TRANS, M, N, K, A, LDA, TAU, C, LDC, cmplx.f:23985: SUBROUTINE ZUNMQR( SIDE, TRANS, M, N, K, A, LDA, TAU, C, LDC, If you look at the definition of zgels you will see that for least squares solutions it calls ZGEQRF, ZUNMQR and ZTRTRS so you just need to set up those calls instead (although admittedly it's a royal PITA to get the Fortran calls set up properly). An alternative would be to leave a copy of the Fortran sources for ZGELS in the src directory for your package but that may croak on trying to compile the whole routine, which allows for both over-determined and under-determined systems.
You can see the error here: https://r-forge.r-project.org/R/?group_id=160&log=build_src&pkg=cda&flavor=patched Best, baptiste
Thanks, baptiste
One option that I'd like to consider is whether the appropriate LAPACK routines could be wrapped and shipped in a separate package (discontinued rblas could provide a good starting point). Sadly, I know nothing about static/dynamic libraries and all this business.. Thanks, baptiste
| Sorry for being dense, I don't know anything about linking R to
| external dependencies. It can be done; there are many examples -- for example every package using the GSL.
I just checked how RcppGSL does it, and well, this configure magic is way above my head. Best, baptiste
_______________________________________________ Rcpp-devel mailing list Rcpp-devel at lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
On 8 June 2011 10:13, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Jun 7, 2011 at 4:20 PM, baptiste auguie <baptiste.auguie at googlemail.com> wrote:
On 8 June 2011 09:04, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Jun 7, 2011 at 3:47 PM, baptiste auguie <baptiste.auguie at googlemail.com> wrote:
On 8 June 2011 08:03, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Jun 7, 2011 at 2:34 PM, baptiste auguie <baptiste.auguie at googlemail.com> wrote:
Hi Doug, On 8 June 2011 03:43, Douglas Bates <bates at stat.wisc.edu> wrote:
On Jun 6, 2011 4:17 AM, "baptiste auguie" <baptiste.auguie at googlemail.com> wrote:
Thank you for the explanations below. On 5 June 2011 10:40, Dirk Eddelbuettel <edd at debian.org> wrote:
On 5 June 2011 at 10:12, baptiste auguie wrote: | Hi Dirk and all, | | On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote: | > | > Baptiste, | > | > On 4 June 2011 at 11:45, baptiste auguie wrote: | > | Dear list, | > | | > | My package cda, which I was hoping to release on CRAN, fails to | > | compile on R-forge with error, | > | | > | ** testing if installed package can be loaded | > | Error in dyn.load(file, DLLpath = DLLpath, ...) : | > | ? unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': | > | ? /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ | > | | > | It builds fine on my local machines (Mac OS 10.5, 10.6). | > | | > | >From an older discussion on this list < | > | http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html> | > | the issue seems to be that Armadillo's inv() relies on a function that | > | is not provided by R, only by LAPACK. I have just replaced inv() by | > | pinv() and solve() in my code; merely to see what happens, but chances | > | are they also require a full LAPACK. | | Indeed, the error on R-forge is now with zgels_, required to solve | linear systems. It seems one cannot solve Armadillo linear systems | without LAPACK in the current situation. Yes. Doug, Romain and myself should address that, or at least make it clear what feature of the full Armadillo are lacking in RcppArmadillo. | > Sometime relatively early in the RcppArmadillo development process, Doug | > convinced Romain and myself to go for a pure template solution with Armadillo | > as all / most things found during the configure (or in this case, cmake) | > stage can be assumed 'found' given that we have around us by design. ?So no | > testing, no local library and full reliance and what R gives us. | > | > That was a brilliant idea, and has freed us from having to rely on building | > and shipping a library, having to tell users how to set PKG_LIBS etc pp and I | > firmly believe that this helped tremendously in getting RcppArmadillo more | > widely used. So I would not want to revert this. | | It sounds like a good choice, I agree. Yet solving a linear system is | quite a crucial task in linear algebra; it would be nice if we could | come up with a tutorial recipe for dummies like me. | | > | > In any event, it seems that you need more LAPACK than R has for you. ?That is | > likely to be a dicey situation as you per se do not know whether R was built | > and linked with its own (subset) copy of LAPACK, or whether it uses system | > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). ?So you may be in | > a spot bother and I not sure what I can recommend --- other than trying your | > luck at some short configure snippets that will run at package build time to | > determine whether the system you want to build cda on it 'rich' enough to | > support it. ?I can help you off list with some configure snippets as some of | > my packages have configure code; adding a test for zgetri should be feasible. | > | > | Does anybody have any experience | > | dealing with such issues w.r.t releasing a package on R-forge / CRAN? | > | Is there any chance they would consider installing LAPACK on those | > | servers, or is there no point in asking such things? | > | > I don't think it is a matter of fixing the R-Forge server. I think it is a | > matter of making your package installable on the largest number of user | > systems. ?Also try win-builder.r-project.org to see how it fares on that | > platform.
Unsurprisingly, it fails, with the same complaint as R-forge.
| | I was under the impression that R-forge or CRAN, if it had LAPACK | installed, could produce binaries for the relevant platforms, and | users would not have to build the package themselves and would not be | required of having LAPACK on their machine. That's probably a | misconception, isn't it? If and only statically linked binaries or libraries where produced, which is generally not the case. Many OSs (Linux incl) ship source only and otherwise link dynamically, others (Windoze) use dynamic linking and OS X is for all I know somewhere in the middle (as you can get prebuild packages with dynamic linking or build from source).
I see; so basically the user will always need to have a full LAPACK installed. Here's one question then, if R-core didn't consider necessary to include those particular functions from LAPACK, presumably that means that R defines its own routines to solve linear systems and invert matrices. Is there any possibility to use those routines with Armadillo?
One important point has been missed in this discussion, which is that the particular Lapack subroutine that is not found in the subset of Lapack shipped with R is for general, complex-valued matrices (just google the name zgetri). ?The way that Armadillo is structured it is either all-in or all-out with respect to complex-valued matrices ?If you allow for complex matrices then you must have all the supporting subroutines for whatever you are calling. ?If you call inv() you need to allow for all the [sdcz]getri routines to be available.
The matrix I need to invert is definitely always complex; in fact, convenient complex algebra is the main attraction of Armadillo for me.
Argh. ?Of course, I was being foolish. ?Because Armadillo is a header-only library it does not access any Lapack subroutine until the call is instantiated. ?I still haven't quite gotten used to thinking only having the headers.
So one possibility is to check the Armadillo sources to see if you can suppress the use of complex-valued matrices when compiling your package. ?A quick glance indicates that this might now be easy. Otherwise, remember the first rule of numerical linear algebra which is, "If your algorithm involves computing the inverse of a matrix you should rethink the algorithm because there is a better way of doing it". ?So why do you need inv()? If the answer is to solve a linear system of equations then you use solve(), you do not use inv(). ?If, for some truly unusual reason you actually need the inverse (and, remember in 99.99% of the cases where people think they need the inverse, they don't) then you use a combination of solve() and eye().
I tried this, and in fact I do use solve() elsewhere in my code, which calls for another LAPACK routine (zgels.f) not present in R. The reason I still need inv() is that I have to solve about 300 times a linear system AX=B with the same matrix A but varying B. A quick timing last week revealed that using solve() 300 times was typically slower by a factor of two ?in my use case than using inv() once (or pinv() for that matter, it makes not appreciable difference). I'm happy to be shown otherwise though.
I forgot that Armadillo doesn't provide a convenient way of using the LU decomposition (that is one of the things that I find frustrating about Armadillo). ?Did you try a single solve call in which the right-hand side is an identity matrix? ?On looking at the Armadillo sources it seems that it should call zgesv which is included in R's subset of the Lapack routines.
It seems curious, looking at the bestiary of LAPACK functions, that so many of them seem to perform a similar task. I wonder what are the practical downsides of using a code that solves AX=I rather than one inverting A. Also, when I replaced inv() by solve() and pinv(), R-forge still failed to build complaining that zgesv_ was not present. I assumed it was used by solve(), but perhaps it was pinv(). I'll give it a try anyway, but I'd rather hope we can figure out a less ad hoc solution.
Once upon a time we actually counted the number of floating point operations that were needed to perform a particular calculation, which is the reason for all the special-case code in Lapack. ?The difference between using the LU decomposition of a matrix to solve a system in which the right hand side is an identity, versus a special-purpose piece of code like zgetri is that zgetri can take advantage of the fact that the identity matrix is diagonal, thereby saving a relatively small number of operations.
That makes sense, thanks.
I'm not sure why there should be a complaint about zgesv_ not being available. ?It's there in the R sources (src/modules/lapack/cmplx.f starting at line 3944). ?That routine is trivial because it just checks its arguments then calls zgetrf and zgetrs.
Sorry, I meant zgels_ not zgesv_ (who came up with those names?!).
Fortran programmers. ?Just as the founders of Microsoft decided that no one would ever have more than some amount of memory (128K, IIRC) those defining the Fortran specification in the 1950's - 60's declared that you would never need more than 6 upper-case characters in an identifier.
Thanks for the historical background, knowing the naming convention is quite useful.
You parse the name according to the first letter indicates the storage type (z for double precision complex), the next two for specific characteristics of the matrix (ge -> general, tr -> triangular, sy -> symmetric, po -> symmetric and positive definite, ...) and the remaining two or three for the operation (trf -> triangular factorization, sl -> solution of linear systems, ls -> least squares solution). You should only have a request for zgels if you are trying to get least squares solutions. The complex Lapack routines that you can count on being available are cmplx.f:96: ? ? ?SUBROUTINE ZBDSQR( UPLO, N, NCVT, NRU, NCC, D, E, VT, LDVT, U, cmplx.f:838: ? ? ?SUBROUTINE ZDRSCL( N, SA, SX, INCX ) cmplx.f:952: ? ? ?SUBROUTINE ZGEBAK( JOB, SIDE, N, ILO, IHI, SCALE, M, V, LDV, cmplx.f:1141: ? ? ?SUBROUTINE ZGEBAL( JOB, N, A, LDA, ILO, IHI, SCALE, INFO ) cmplx.f:1471: ? ? ?SUBROUTINE ZGEBD2( M, N, A, LDA, D, E, TAUQ, TAUP, WORK, INFO ) cmplx.f:1721: ? ? ?SUBROUTINE ZGEBRD( M, N, A, LDA, D, E, TAUQ, TAUP, WORK, LWORK, cmplx.f:1989: ? ? ?SUBROUTINE ZGECON( NORM, N, A, LDA, ANORM, RCOND, WORK, RWORK, cmplx.f:2182: ? ? ?SUBROUTINE ZGEEV( JOBVL, JOBVR, N, A, LDA, W, VL, LDVL, VR, LDVR, cmplx.f:2578: ? ? ?SUBROUTINE ZGEHD2( N, ILO, IHI, A, LDA, TAU, WORK, INFO ) cmplx.f:2726: ? ? ?SUBROUTINE ZGEHRD( N, ILO, IHI, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:2999: ? ? ?SUBROUTINE ZGELQ2( M, N, A, LDA, TAU, WORK, INFO ) cmplx.f:3122: ? ? ?SUBROUTINE ZGELQF( M, N, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:3317: ? ? ?SUBROUTINE ZGEQP3( M, N, A, LDA, JPVT, TAU, WORK, LWORK, RWORK, cmplx.f:3610: ? ? ?SUBROUTINE ZGEQR2( M, N, A, LDA, TAU, WORK, INFO ) cmplx.f:3731: ? ? ?SUBROUTINE ZGEQRF( M, N, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:3927: ? ? ?SUBROUTINE ZGESV( N, NRHS, A, LDA, IPIV, B, LDB, INFO ) cmplx.f:4034: ? ? ?SUBROUTINE ZGESVD( JOBU, JOBVT, M, N, A, LDA, S, U, LDU, VT, LDVT, cmplx.f:7636: ? ? ?SUBROUTINE ZGETF2( M, N, A, LDA, IPIV, INFO ) cmplx.f:7784: ? ? ?SUBROUTINE ZGETRF( M, N, A, LDA, IPIV, INFO ) cmplx.f:7943: ? ? ?SUBROUTINE ZGETRS( TRANS, N, NRHS, A, LDA, IPIV, B, LDB, INFO ) cmplx.f:8092: ? ? ?SUBROUTINE ZHEEV( JOBZ, UPLO, N, A, LDA, W, WORK, LWORK, RWORK, cmplx.f:8310: ? ? ?SUBROUTINE ZHETD2( UPLO, N, A, LDA, D, E, TAU, INFO ) cmplx.f:8568: ? ? ?SUBROUTINE ZHETRD( UPLO, N, A, LDA, D, E, TAU, WORK, LWORK, INFO ) cmplx.f:8864: ? ? ?SUBROUTINE ZHSEQR( JOB, COMPZ, N, ILO, IHI, H, LDH, W, Z, LDZ, cmplx.f:9259: ? ? ?SUBROUTINE ZLABRD( M, N, NB, A, LDA, D, E, TAUQ, TAUP, X, LDX, Y, cmplx.f:9587: ? ? ?SUBROUTINE ZLACGV( N, X, INCX ) cmplx.f:9647: ? ? ?SUBROUTINE ZLACN2( N, V, X, EST, KASE, ISAVE ) cmplx.f:9868: ? ? ?SUBROUTINE ZLACPY( UPLO, M, N, A, LDA, B, LDB ) cmplx.f:10004: ? ? ?SUBROUTINE ZLAHQR( WANTT, WANTZ, N, ILO, IHI, H, LDH, W, ILOZ, cmplx.f:10474: ? ? ?SUBROUTINE ZLAHR2( N, K, NB, A, LDA, TAU, T, LDT, Y, LDY ) cmplx.f:10714: ? ? ?SUBROUTINE ZLAHRD( N, K, NB, A, LDA, TAU, T, LDT, Y, LDY ) cmplx.f:11678: ? ? ?SUBROUTINE ZLAQP2( M, N, OFFSET, A, LDA, JPVT, TAU, VN1, VN2, cmplx.f:11857: ? ? ?SUBROUTINE ZLAQPS( M, N, OFFSET, NB, KB, A, LDA, JPVT, TAU, VN1, cmplx.f:12123: ? ? ?SUBROUTINE ZLAQR0( WANTT, WANTZ, N, ILO, IHI, H, LDH, W, ILOZ, cmplx.f:12724: ? ? ?SUBROUTINE ZLAQR1( N, H, LDH, S1, S2, V ) cmplx.f:12821: ? ? ?SUBROUTINE ZLAQR2( WANTT, WANTZ, N, KTOP, KBOT, NW, H, LDH, ILOZ, cmplx.f:13258: ? ? ?SUBROUTINE ZLAQR3( WANTT, WANTZ, N, KTOP, KBOT, NW, H, LDH, ILOZ, cmplx.f:13706: ? ? ?SUBROUTINE ZLAQR4( WANTT, WANTZ, N, ILO, IHI, H, LDH, W, ILOZ, cmplx.f:14308: ? ? ?SUBROUTINE ZLAQR5( WANTT, WANTZ, KACC22, N, KTOP, KBOT, NSHFTS, S, cmplx.f:15117: ? ? ?SUBROUTINE ZLARF( SIDE, M, N, V, INCV, TAU, C, LDC, WORK ) cmplx.f:15237: ? ? ?SUBROUTINE ZLARFB( SIDE, TRANS, DIRECT, STOREV, M, N, K, V, LDV, cmplx.f:15845: ? ? ?SUBROUTINE ZLARFG( N, ALPHA, X, INCX, TAU ) cmplx.f:15990: ? ? ?SUBROUTINE ZLARFT( DIRECT, STOREV, N, K, V, LDV, TAU, T, LDT ) cmplx.f:16214: ? ? ?SUBROUTINE ZLARFX( SIDE, M, N, V, TAU, C, LDC, WORK ) cmplx.f:16855: ? ? ?SUBROUTINE ZLARTG( F, G, CS, SN, R ) cmplx.f:17050: ? ? ?SUBROUTINE ZLASCL( TYPE, KL, KU, CFROM, CTO, M, N, A, LDA, INFO ) cmplx.f:17317: ? ? ?SUBROUTINE ZLASET( UPLO, M, N, ALPHA, BETA, A, LDA ) cmplx.f:17431: ? ? ?SUBROUTINE ZLASR( SIDE, PIVOT, DIRECT, M, N, C, S, A, LDA ) cmplx.f:17794: ? ? ?SUBROUTINE ZLASSQ( N, X, INCX, SCALE, SUMSQ ) cmplx.f:17895: ? ? ?SUBROUTINE ZLASWP( N, A, LDA, K1, K2, IPIV, INCX ) cmplx.f:18014: ? ? ?SUBROUTINE ZLATRD( UPLO, N, NB, A, LDA, E, TAU, W, LDW ) cmplx.f:18293: ? ? ?SUBROUTINE ZLATRS( UPLO, TRANS, DIAG, NORMIN, N, A, LDA, X, SCALE, cmplx.f:19172: ? ? ?SUBROUTINE ZPOTF2( UPLO, N, A, LDA, INFO ) cmplx.f:19346: ? ? ?SUBROUTINE ZPOTRF( UPLO, N, A, LDA, INFO ) cmplx.f:19532: ? ? ?SUBROUTINE ZROT( N, CX, INCX, CY, INCY, C, S ) cmplx.f:19623: ? ? ?SUBROUTINE ZSTEQR( COMPZ, N, D, E, Z, LDZ, WORK, INFO ) cmplx.f:20126: ? ? ?SUBROUTINE ZTRCON( NORM, UPLO, DIAG, N, A, LDA, RCOND, WORK, cmplx.f:20330: ? ? ?SUBROUTINE ZTREVC( SIDE, HOWMNY, SELECT, N, T, LDT, VL, LDVL, VR, cmplx.f:20716: ? ? ?SUBROUTINE ZTREXC( COMPQ, N, T, LDT, Q, LDQ, IFST, ILST, INFO ) cmplx.f:20878: ? ? ?SUBROUTINE ZTRTRS( UPLO, TRANS, DIAG, N, NRHS, A, LDA, B, LDB, cmplx.f:21026: ? ? ?SUBROUTINE ZUNG2L( M, N, K, A, LDA, TAU, WORK, INFO ) cmplx.f:21154: ? ? ?SUBROUTINE ZUNG2R( M, N, K, A, LDA, TAU, WORK, INFO ) cmplx.f:21284: ? ? ?SUBROUTINE ZUNGBR( VECT, M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:21529: ? ? ?SUBROUTINE ZUNGHR( N, ILO, IHI, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:21694: ? ? ?SUBROUTINE ZUNGL2( M, N, K, A, LDA, TAU, WORK, INFO ) cmplx.f:21830: ? ? ?SUBROUTINE ZUNGLQ( M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:22045: ? ? ?SUBROUTINE ZUNGQL( M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:22267: ? ? ?SUBROUTINE ZUNGQR( M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:22483: ? ? ?SUBROUTINE ZUNGR2( M, N, K, A, LDA, TAU, WORK, INFO ) cmplx.f:22617: ? ? ?SUBROUTINE ZUNGRQ( M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:22840: ? ? ?SUBROUTINE ZUNGTR( UPLO, N, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:23024: ? ? ?SUBROUTINE ZUNM2R( SIDE, TRANS, M, N, K, A, LDA, TAU, C, LDC, cmplx.f:23225: ? ? ?SUBROUTINE ZUNMBR( VECT, SIDE, TRANS, M, N, K, A, LDA, TAU, C, cmplx.f:23513: ? ? ?SUBROUTINE ZUNML2( SIDE, TRANS, M, N, K, A, LDA, TAU, C, LDC, cmplx.f:23718: ? ? ?SUBROUTINE ZUNMLQ( SIDE, TRANS, M, N, K, A, LDA, TAU, C, LDC, cmplx.f:23985: ? ? ?SUBROUTINE ZUNMQR( SIDE, TRANS, M, N, K, A, LDA, TAU, C, LDC, If you look at the definition of zgels you will see that for least squares solutions it calls ZGEQRF, ZUNMQR and ZTRTRS so you just need to set up those calls instead (although admittedly it's a royal PITA to get the Fortran calls set up properly). An alternative would be to leave a copy of the Fortran sources for ZGELS in the src directory for your package but that may croak on trying to compile the whole routine, which allows for both over-determined and under-determined systems.
I like this idea, I think. What makes more sense, though, adding copies of specific routines to each package that needs them, or grouping them in a separate package dedicated to this purpose? Note that I'm fully unaware of practical difficulties in doing either of the two. Best regards, baptiste
You can see the error here: https://r-forge.r-project.org/R/?group_id=160&log=build_src&pkg=cda&flavor=patched Best, baptiste
Thanks, baptiste
One option that I'd like to consider is whether the appropriate LAPACK routines could be wrapped and shipped in a separate package (discontinued rblas could provide a good starting point). Sadly, I know nothing about static/dynamic libraries and all this business.. Thanks, baptiste
| Sorry for being dense, I don't know anything about linking R to
| external dependencies. It can be done; there are many examples -- for example every package using the GSL.
I just checked how RcppGSL does it, and well, this configure magic is way above my head. Best, baptiste
_______________________________________________ Rcpp-devel mailing list Rcpp-devel at lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
14 days later
Hi, I simply added zgels.f from Netlib in the src/ directory of my package, and to my great surprise it now seems to build fine on r-forge, and even win-builder! That seems too easy to be true. I guess I should include something related to Netlib's licence, also. Thanks, baptiste
On 8 June 2011 10:13, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Jun 7, 2011 at 4:20 PM, baptiste auguie <baptiste.auguie at googlemail.com> wrote:
On 8 June 2011 09:04, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Jun 7, 2011 at 3:47 PM, baptiste auguie <baptiste.auguie at googlemail.com> wrote:
On 8 June 2011 08:03, Douglas Bates <bates at stat.wisc.edu> wrote:
On Tue, Jun 7, 2011 at 2:34 PM, baptiste auguie <baptiste.auguie at googlemail.com> wrote:
Hi Doug, On 8 June 2011 03:43, Douglas Bates <bates at stat.wisc.edu> wrote:
On Jun 6, 2011 4:17 AM, "baptiste auguie" <baptiste.auguie at googlemail.com> wrote:
Thank you for the explanations below. On 5 June 2011 10:40, Dirk Eddelbuettel <edd at debian.org> wrote:
On 5 June 2011 at 10:12, baptiste auguie wrote: | Hi Dirk and all, | | On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote: | > | > Baptiste, | > | > On 4 June 2011 at 11:45, baptiste auguie wrote: | > | Dear list, | > | | > | My package cda, which I was hoping to release on CRAN, fails to | > | compile on R-forge with error, | > | | > | ** testing if installed package can be loaded | > | Error in dyn.load(file, DLLpath = DLLpath, ...) : | > | ? unable to load shared object '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so': | > | ? /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol: zgetri_ | > | | > | It builds fine on my local machines (Mac OS 10.5, 10.6). | > | | > | >From an older discussion on this list < | > | http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html> | > | the issue seems to be that Armadillo's inv() relies on a function that | > | is not provided by R, only by LAPACK. I have just replaced inv() by | > | pinv() and solve() in my code; merely to see what happens, but chances | > | are they also require a full LAPACK. | | Indeed, the error on R-forge is now with zgels_, required to solve | linear systems. It seems one cannot solve Armadillo linear systems | without LAPACK in the current situation. Yes. Doug, Romain and myself should address that, or at least make it clear what feature of the full Armadillo are lacking in RcppArmadillo. | > Sometime relatively early in the RcppArmadillo development process, Doug | > convinced Romain and myself to go for a pure template solution with Armadillo | > as all / most things found during the configure (or in this case, cmake) | > stage can be assumed 'found' given that we have around us by design. ?So no | > testing, no local library and full reliance and what R gives us. | > | > That was a brilliant idea, and has freed us from having to rely on building | > and shipping a library, having to tell users how to set PKG_LIBS etc pp and I | > firmly believe that this helped tremendously in getting RcppArmadillo more | > widely used. So I would not want to revert this. | | It sounds like a good choice, I agree. Yet solving a linear system is | quite a crucial task in linear algebra; it would be nice if we could | come up with a tutorial recipe for dummies like me. | | > | > In any event, it seems that you need more LAPACK than R has for you. ?That is | > likely to be a dicey situation as you per se do not know whether R was built | > and linked with its own (subset) copy of LAPACK, or whether it uses system | > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). ?So you may be in | > a spot bother and I not sure what I can recommend --- other than trying your | > luck at some short configure snippets that will run at package build time to | > determine whether the system you want to build cda on it 'rich' enough to | > support it. ?I can help you off list with some configure snippets as some of | > my packages have configure code; adding a test for zgetri should be feasible. | > | > | Does anybody have any experience | > | dealing with such issues w.r.t releasing a package on R-forge / CRAN? | > | Is there any chance they would consider installing LAPACK on those | > | servers, or is there no point in asking such things? | > | > I don't think it is a matter of fixing the R-Forge server. I think it is a | > matter of making your package installable on the largest number of user | > systems. ?Also try win-builder.r-project.org to see how it fares on that | > platform.
Unsurprisingly, it fails, with the same complaint as R-forge.
| | I was under the impression that R-forge or CRAN, if it had LAPACK | installed, could produce binaries for the relevant platforms, and | users would not have to build the package themselves and would not be | required of having LAPACK on their machine. That's probably a | misconception, isn't it? If and only statically linked binaries or libraries where produced, which is generally not the case. Many OSs (Linux incl) ship source only and otherwise link dynamically, others (Windoze) use dynamic linking and OS X is for all I know somewhere in the middle (as you can get prebuild packages with dynamic linking or build from source).
I see; so basically the user will always need to have a full LAPACK installed. Here's one question then, if R-core didn't consider necessary to include those particular functions from LAPACK, presumably that means that R defines its own routines to solve linear systems and invert matrices. Is there any possibility to use those routines with Armadillo?
One important point has been missed in this discussion, which is that the particular Lapack subroutine that is not found in the subset of Lapack shipped with R is for general, complex-valued matrices (just google the name zgetri). ?The way that Armadillo is structured it is either all-in or all-out with respect to complex-valued matrices ?If you allow for complex matrices then you must have all the supporting subroutines for whatever you are calling. ?If you call inv() you need to allow for all the [sdcz]getri routines to be available.
The matrix I need to invert is definitely always complex; in fact, convenient complex algebra is the main attraction of Armadillo for me.
Argh. ?Of course, I was being foolish. ?Because Armadillo is a header-only library it does not access any Lapack subroutine until the call is instantiated. ?I still haven't quite gotten used to thinking only having the headers.
So one possibility is to check the Armadillo sources to see if you can suppress the use of complex-valued matrices when compiling your package. ?A quick glance indicates that this might now be easy. Otherwise, remember the first rule of numerical linear algebra which is, "If your algorithm involves computing the inverse of a matrix you should rethink the algorithm because there is a better way of doing it". ?So why do you need inv()? If the answer is to solve a linear system of equations then you use solve(), you do not use inv(). ?If, for some truly unusual reason you actually need the inverse (and, remember in 99.99% of the cases where people think they need the inverse, they don't) then you use a combination of solve() and eye().
I tried this, and in fact I do use solve() elsewhere in my code, which calls for another LAPACK routine (zgels.f) not present in R. The reason I still need inv() is that I have to solve about 300 times a linear system AX=B with the same matrix A but varying B. A quick timing last week revealed that using solve() 300 times was typically slower by a factor of two ?in my use case than using inv() once (or pinv() for that matter, it makes not appreciable difference). I'm happy to be shown otherwise though.
I forgot that Armadillo doesn't provide a convenient way of using the LU decomposition (that is one of the things that I find frustrating about Armadillo). ?Did you try a single solve call in which the right-hand side is an identity matrix? ?On looking at the Armadillo sources it seems that it should call zgesv which is included in R's subset of the Lapack routines.
It seems curious, looking at the bestiary of LAPACK functions, that so many of them seem to perform a similar task. I wonder what are the practical downsides of using a code that solves AX=I rather than one inverting A. Also, when I replaced inv() by solve() and pinv(), R-forge still failed to build complaining that zgesv_ was not present. I assumed it was used by solve(), but perhaps it was pinv(). I'll give it a try anyway, but I'd rather hope we can figure out a less ad hoc solution.
Once upon a time we actually counted the number of floating point operations that were needed to perform a particular calculation, which is the reason for all the special-case code in Lapack. ?The difference between using the LU decomposition of a matrix to solve a system in which the right hand side is an identity, versus a special-purpose piece of code like zgetri is that zgetri can take advantage of the fact that the identity matrix is diagonal, thereby saving a relatively small number of operations.
That makes sense, thanks.
I'm not sure why there should be a complaint about zgesv_ not being available. ?It's there in the R sources (src/modules/lapack/cmplx.f starting at line 3944). ?That routine is trivial because it just checks its arguments then calls zgetrf and zgetrs.
Sorry, I meant zgels_ not zgesv_ (who came up with those names?!).
Fortran programmers. ?Just as the founders of Microsoft decided that no one would ever have more than some amount of memory (128K, IIRC) those defining the Fortran specification in the 1950's - 60's declared that you would never need more than 6 upper-case characters in an identifier. You parse the name according to the first letter indicates the storage type (z for double precision complex), the next two for specific characteristics of the matrix (ge -> general, tr -> triangular, sy -> symmetric, po -> symmetric and positive definite, ...) and the remaining two or three for the operation (trf -> triangular factorization, sl -> solution of linear systems, ls -> least squares solution). You should only have a request for zgels if you are trying to get least squares solutions. The complex Lapack routines that you can count on being available are cmplx.f:96: ? ? ?SUBROUTINE ZBDSQR( UPLO, N, NCVT, NRU, NCC, D, E, VT, LDVT, U, cmplx.f:838: ? ? ?SUBROUTINE ZDRSCL( N, SA, SX, INCX ) cmplx.f:952: ? ? ?SUBROUTINE ZGEBAK( JOB, SIDE, N, ILO, IHI, SCALE, M, V, LDV, cmplx.f:1141: ? ? ?SUBROUTINE ZGEBAL( JOB, N, A, LDA, ILO, IHI, SCALE, INFO ) cmplx.f:1471: ? ? ?SUBROUTINE ZGEBD2( M, N, A, LDA, D, E, TAUQ, TAUP, WORK, INFO ) cmplx.f:1721: ? ? ?SUBROUTINE ZGEBRD( M, N, A, LDA, D, E, TAUQ, TAUP, WORK, LWORK, cmplx.f:1989: ? ? ?SUBROUTINE ZGECON( NORM, N, A, LDA, ANORM, RCOND, WORK, RWORK, cmplx.f:2182: ? ? ?SUBROUTINE ZGEEV( JOBVL, JOBVR, N, A, LDA, W, VL, LDVL, VR, LDVR, cmplx.f:2578: ? ? ?SUBROUTINE ZGEHD2( N, ILO, IHI, A, LDA, TAU, WORK, INFO ) cmplx.f:2726: ? ? ?SUBROUTINE ZGEHRD( N, ILO, IHI, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:2999: ? ? ?SUBROUTINE ZGELQ2( M, N, A, LDA, TAU, WORK, INFO ) cmplx.f:3122: ? ? ?SUBROUTINE ZGELQF( M, N, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:3317: ? ? ?SUBROUTINE ZGEQP3( M, N, A, LDA, JPVT, TAU, WORK, LWORK, RWORK, cmplx.f:3610: ? ? ?SUBROUTINE ZGEQR2( M, N, A, LDA, TAU, WORK, INFO ) cmplx.f:3731: ? ? ?SUBROUTINE ZGEQRF( M, N, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:3927: ? ? ?SUBROUTINE ZGESV( N, NRHS, A, LDA, IPIV, B, LDB, INFO ) cmplx.f:4034: ? ? ?SUBROUTINE ZGESVD( JOBU, JOBVT, M, N, A, LDA, S, U, LDU, VT, LDVT, cmplx.f:7636: ? ? ?SUBROUTINE ZGETF2( M, N, A, LDA, IPIV, INFO ) cmplx.f:7784: ? ? ?SUBROUTINE ZGETRF( M, N, A, LDA, IPIV, INFO ) cmplx.f:7943: ? ? ?SUBROUTINE ZGETRS( TRANS, N, NRHS, A, LDA, IPIV, B, LDB, INFO ) cmplx.f:8092: ? ? ?SUBROUTINE ZHEEV( JOBZ, UPLO, N, A, LDA, W, WORK, LWORK, RWORK, cmplx.f:8310: ? ? ?SUBROUTINE ZHETD2( UPLO, N, A, LDA, D, E, TAU, INFO ) cmplx.f:8568: ? ? ?SUBROUTINE ZHETRD( UPLO, N, A, LDA, D, E, TAU, WORK, LWORK, INFO ) cmplx.f:8864: ? ? ?SUBROUTINE ZHSEQR( JOB, COMPZ, N, ILO, IHI, H, LDH, W, Z, LDZ, cmplx.f:9259: ? ? ?SUBROUTINE ZLABRD( M, N, NB, A, LDA, D, E, TAUQ, TAUP, X, LDX, Y, cmplx.f:9587: ? ? ?SUBROUTINE ZLACGV( N, X, INCX ) cmplx.f:9647: ? ? ?SUBROUTINE ZLACN2( N, V, X, EST, KASE, ISAVE ) cmplx.f:9868: ? ? ?SUBROUTINE ZLACPY( UPLO, M, N, A, LDA, B, LDB ) cmplx.f:10004: ? ? ?SUBROUTINE ZLAHQR( WANTT, WANTZ, N, ILO, IHI, H, LDH, W, ILOZ, cmplx.f:10474: ? ? ?SUBROUTINE ZLAHR2( N, K, NB, A, LDA, TAU, T, LDT, Y, LDY ) cmplx.f:10714: ? ? ?SUBROUTINE ZLAHRD( N, K, NB, A, LDA, TAU, T, LDT, Y, LDY ) cmplx.f:11678: ? ? ?SUBROUTINE ZLAQP2( M, N, OFFSET, A, LDA, JPVT, TAU, VN1, VN2, cmplx.f:11857: ? ? ?SUBROUTINE ZLAQPS( M, N, OFFSET, NB, KB, A, LDA, JPVT, TAU, VN1, cmplx.f:12123: ? ? ?SUBROUTINE ZLAQR0( WANTT, WANTZ, N, ILO, IHI, H, LDH, W, ILOZ, cmplx.f:12724: ? ? ?SUBROUTINE ZLAQR1( N, H, LDH, S1, S2, V ) cmplx.f:12821: ? ? ?SUBROUTINE ZLAQR2( WANTT, WANTZ, N, KTOP, KBOT, NW, H, LDH, ILOZ, cmplx.f:13258: ? ? ?SUBROUTINE ZLAQR3( WANTT, WANTZ, N, KTOP, KBOT, NW, H, LDH, ILOZ, cmplx.f:13706: ? ? ?SUBROUTINE ZLAQR4( WANTT, WANTZ, N, ILO, IHI, H, LDH, W, ILOZ, cmplx.f:14308: ? ? ?SUBROUTINE ZLAQR5( WANTT, WANTZ, KACC22, N, KTOP, KBOT, NSHFTS, S, cmplx.f:15117: ? ? ?SUBROUTINE ZLARF( SIDE, M, N, V, INCV, TAU, C, LDC, WORK ) cmplx.f:15237: ? ? ?SUBROUTINE ZLARFB( SIDE, TRANS, DIRECT, STOREV, M, N, K, V, LDV, cmplx.f:15845: ? ? ?SUBROUTINE ZLARFG( N, ALPHA, X, INCX, TAU ) cmplx.f:15990: ? ? ?SUBROUTINE ZLARFT( DIRECT, STOREV, N, K, V, LDV, TAU, T, LDT ) cmplx.f:16214: ? ? ?SUBROUTINE ZLARFX( SIDE, M, N, V, TAU, C, LDC, WORK ) cmplx.f:16855: ? ? ?SUBROUTINE ZLARTG( F, G, CS, SN, R ) cmplx.f:17050: ? ? ?SUBROUTINE ZLASCL( TYPE, KL, KU, CFROM, CTO, M, N, A, LDA, INFO ) cmplx.f:17317: ? ? ?SUBROUTINE ZLASET( UPLO, M, N, ALPHA, BETA, A, LDA ) cmplx.f:17431: ? ? ?SUBROUTINE ZLASR( SIDE, PIVOT, DIRECT, M, N, C, S, A, LDA ) cmplx.f:17794: ? ? ?SUBROUTINE ZLASSQ( N, X, INCX, SCALE, SUMSQ ) cmplx.f:17895: ? ? ?SUBROUTINE ZLASWP( N, A, LDA, K1, K2, IPIV, INCX ) cmplx.f:18014: ? ? ?SUBROUTINE ZLATRD( UPLO, N, NB, A, LDA, E, TAU, W, LDW ) cmplx.f:18293: ? ? ?SUBROUTINE ZLATRS( UPLO, TRANS, DIAG, NORMIN, N, A, LDA, X, SCALE, cmplx.f:19172: ? ? ?SUBROUTINE ZPOTF2( UPLO, N, A, LDA, INFO ) cmplx.f:19346: ? ? ?SUBROUTINE ZPOTRF( UPLO, N, A, LDA, INFO ) cmplx.f:19532: ? ? ?SUBROUTINE ZROT( N, CX, INCX, CY, INCY, C, S ) cmplx.f:19623: ? ? ?SUBROUTINE ZSTEQR( COMPZ, N, D, E, Z, LDZ, WORK, INFO ) cmplx.f:20126: ? ? ?SUBROUTINE ZTRCON( NORM, UPLO, DIAG, N, A, LDA, RCOND, WORK, cmplx.f:20330: ? ? ?SUBROUTINE ZTREVC( SIDE, HOWMNY, SELECT, N, T, LDT, VL, LDVL, VR, cmplx.f:20716: ? ? ?SUBROUTINE ZTREXC( COMPQ, N, T, LDT, Q, LDQ, IFST, ILST, INFO ) cmplx.f:20878: ? ? ?SUBROUTINE ZTRTRS( UPLO, TRANS, DIAG, N, NRHS, A, LDA, B, LDB, cmplx.f:21026: ? ? ?SUBROUTINE ZUNG2L( M, N, K, A, LDA, TAU, WORK, INFO ) cmplx.f:21154: ? ? ?SUBROUTINE ZUNG2R( M, N, K, A, LDA, TAU, WORK, INFO ) cmplx.f:21284: ? ? ?SUBROUTINE ZUNGBR( VECT, M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:21529: ? ? ?SUBROUTINE ZUNGHR( N, ILO, IHI, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:21694: ? ? ?SUBROUTINE ZUNGL2( M, N, K, A, LDA, TAU, WORK, INFO ) cmplx.f:21830: ? ? ?SUBROUTINE ZUNGLQ( M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:22045: ? ? ?SUBROUTINE ZUNGQL( M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:22267: ? ? ?SUBROUTINE ZUNGQR( M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:22483: ? ? ?SUBROUTINE ZUNGR2( M, N, K, A, LDA, TAU, WORK, INFO ) cmplx.f:22617: ? ? ?SUBROUTINE ZUNGRQ( M, N, K, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:22840: ? ? ?SUBROUTINE ZUNGTR( UPLO, N, A, LDA, TAU, WORK, LWORK, INFO ) cmplx.f:23024: ? ? ?SUBROUTINE ZUNM2R( SIDE, TRANS, M, N, K, A, LDA, TAU, C, LDC, cmplx.f:23225: ? ? ?SUBROUTINE ZUNMBR( VECT, SIDE, TRANS, M, N, K, A, LDA, TAU, C, cmplx.f:23513: ? ? ?SUBROUTINE ZUNML2( SIDE, TRANS, M, N, K, A, LDA, TAU, C, LDC, cmplx.f:23718: ? ? ?SUBROUTINE ZUNMLQ( SIDE, TRANS, M, N, K, A, LDA, TAU, C, LDC, cmplx.f:23985: ? ? ?SUBROUTINE ZUNMQR( SIDE, TRANS, M, N, K, A, LDA, TAU, C, LDC, If you look at the definition of zgels you will see that for least squares solutions it calls ZGEQRF, ZUNMQR and ZTRTRS so you just need to set up those calls instead (although admittedly it's a royal PITA to get the Fortran calls set up properly). An alternative would be to leave a copy of the Fortran sources for ZGELS in the src directory for your package but that may croak on trying to compile the whole routine, which allows for both over-determined and under-determined systems.
You can see the error here: https://r-forge.r-project.org/R/?group_id=160&log=build_src&pkg=cda&flavor=patched Best, baptiste
Thanks, baptiste
One option that I'd like to consider is whether the appropriate LAPACK routines could be wrapped and shipped in a separate package (discontinued rblas could provide a good starting point). Sadly, I know nothing about static/dynamic libraries and all this business.. Thanks, baptiste
| Sorry for being dense, I don't know anything about linking R to
| external dependencies. It can be done; there are many examples -- for example every package using the GSL.
I just checked how RcppGSL does it, and well, this configure magic is way above my head. Best, baptiste
_______________________________________________ Rcpp-devel mailing list Rcpp-devel at lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
On 23 June 2011 at 12:14, baptiste auguie wrote:
| Hi, | | I simply added zgels.f from Netlib in the src/ directory of my | package, and to my great surprise it now seems to build fine on | r-forge, and even win-builder! That seems too easy to be true. I guess | I should include something related to Netlib's licence, also. Nice one! Maybe I should add that to the Rcpp FAQ or RcppArmadillo docs. Glad you have it working now, Dirk
Gauss once played himself in a zero-sum game and won $50.
-- #11 at http://www.gaussfacts.com