[Rcpp-devel] RcppArmadillo inv() depends on Lapack, zgetri_ not available on CRAN / R-forge?

On Tue, Jun 7, 2011 at 4:20 PM, baptiste auguie
<baptiste.auguie at googlemail.com> wrote:

On 8 June 2011 09:04, Douglas Bates <bates at stat.wisc.edu> wrote:

On Tue, Jun 7, 2011 at 3:47 PM, baptiste auguie
<baptiste.auguie at googlemail.com> wrote:

On 8 June 2011 08:03, Douglas Bates <bates at stat.wisc.edu> wrote:

On Tue, Jun 7, 2011 at 2:34 PM, baptiste auguie
<baptiste.auguie at googlemail.com> wrote:

Hi Doug,

On 8 June 2011 03:43, Douglas Bates <bates at stat.wisc.edu> wrote:

On Jun 6, 2011 4:17 AM, "baptiste auguie" <baptiste.auguie at googlemail.com>
wrote:

Thank you for the explanations below.

On 5 June 2011 10:40, Dirk Eddelbuettel <edd at debian.org> wrote:

On 5 June 2011 at 10:12, baptiste auguie wrote:
| Hi Dirk and all,
|
| On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote:
| >
| > Baptiste,
| >
| > On 4 June 2011 at 11:45, baptiste auguie wrote:
| > | Dear list,
| > |
| > | My package cda, which I was hoping to release on CRAN, fails to
| > | compile on R-forge with error,
| > |
| > | ** testing if installed package can be loaded
| > | Error in dyn.load(file, DLLpath = DLLpath, ...) :
| > | ? unable to load shared object
'/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so':
| > | ? /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol:
zgetri_
| > |
| > | It builds fine on my local machines (Mac OS 10.5, 10.6).
| > |
| > | >From an older discussion on this list <
| > |
http://www.mail-archive.com/rcpp-devel at lists.r-forge.r-project.org/msg00678.html>
| > | the issue seems to be that Armadillo's inv() relies on a function
that
| > | is not provided by R, only by LAPACK. I have just replaced inv()
by
| > | pinv() and solve() in my code; merely to see what happens, but
chances
| > | are they also require a full LAPACK.
|
| Indeed, the error on R-forge is now with zgels_, required to solve
| linear systems. It seems one cannot solve Armadillo linear systems
| without LAPACK in the current situation.

Yes. Doug, Romain and myself should address that, or at least make it
clear
what feature of the full Armadillo are lacking in RcppArmadillo.

| > Sometime relatively early in the RcppArmadillo development process,
Doug
| > convinced Romain and myself to go for a pure template solution with
Armadillo
| > as all / most things found during the configure (or in this case,
cmake)
| > stage can be assumed 'found' given that we have around us by design.
?So no
| > testing, no local library and full reliance and what R gives us.
| >
| > That was a brilliant idea, and has freed us from having to rely on
building
| > and shipping a library, having to tell users how to set PKG_LIBS etc
pp and I
| > firmly believe that this helped tremendously in getting
RcppArmadillo more
| > widely used. So I would not want to revert this.
|
| It sounds like a good choice, I agree. Yet solving a linear system is
| quite a crucial task in linear algebra; it would be nice if we could
| come up with a tutorial recipe for dummies like me.
|
| >
| > In any event, it seems that you need more LAPACK than R has for you.
?That is
| > likely to be a dicey situation as you per se do not know whether R
was built
| > and linked with its own (subset) copy of LAPACK, or whether it uses
system
| > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). ?So you
may be in
| > a spot bother and I not sure what I can recommend --- other than
trying your
| > luck at some short configure snippets that will run at package build
time to
| > determine whether the system you want to build cda on it 'rich'
enough to
| > support it. ?I can help you off list with some configure snippets as
some of
| > my packages have configure code; adding a test for zgetri should be
feasible.
| >
| > | Does anybody have any experience
| > | dealing with such issues w.r.t releasing a package on R-forge /
CRAN?
| > | Is there any chance they would consider installing LAPACK on those
| > | servers, or is there no point in asking such things?
| >
| > I don't think it is a matter of fixing the R-Forge server. I think
it is a
| > matter of making your package installable on the largest number of
user
| > systems. ?Also try win-builder.r-project.org to see how it fares on
that
| > platform.

Unsurprisingly, it fails, with the same complaint as R-forge.

|
| I was under the impression that R-forge or CRAN, if it had LAPACK
| installed, could produce binaries for the relevant platforms, and
| users would not have to build the package themselves and would not be
| required of having LAPACK on their machine. That's probably a
| misconception, isn't it?

If and only statically linked binaries or libraries where produced,
which is
generally not the case. Many OSs (Linux incl) ship source only and
otherwise
link dynamically, others (Windoze) use dynamic linking and OS X is for
all I
know somewhere in the middle (as you can get prebuild packages with
dynamic
linking or build from source).

I see; so basically the user will always need to have a full LAPACK
installed. Here's one question then, if R-core didn't consider
necessary to include those particular functions from LAPACK,
presumably that means that R defines its own routines to solve linear
systems and invert matrices. Is there any possibility to use those
routines with Armadillo?

One important point has been missed in this discussion, which is that the
particular Lapack subroutine that is not found in the subset of Lapack
shipped with R is for general, complex-valued matrices (just google the name
zgetri). ?The way that Armadillo is structured it is either all-in or
all-out with respect to complex-valued matrices ?If you allow for complex
matrices then you must have all the supporting subroutines for whatever you
are calling. ?If you call inv() you need to allow for all the [sdcz]getri
routines to be available.

The matrix I need to invert is definitely always complex; in fact,
convenient complex algebra is the main attraction of Armadillo for me.

Argh. ?Of course, I was being foolish. ?Because Armadillo is a
header-only library it does not access any Lapack subroutine until the
call is instantiated. ?I still haven't quite gotten used to thinking
only having the headers.

So one possibility is to check the Armadillo sources to see if you can
suppress the use of complex-valued matrices when compiling your package. ?A
quick glance indicates that this might now be easy.

Otherwise, remember the first rule of numerical linear algebra which is, "If
your algorithm involves computing the inverse of a matrix you should rethink
the algorithm because there is a better way of doing it". ?So why do you
need inv()?
If the answer is to solve a linear system of equations then you
use solve(), you do not use inv(). ?If, for some truly unusual reason you
actually need the inverse (and, remember in 99.99% of the cases where people
think they need the inverse, they don't) then you use a combination of
solve() and eye().

I tried this, and in fact I do use solve() elsewhere in my code, which
calls for another LAPACK routine (zgels.f) not present in R. The
reason I still need inv() is that I have to solve about 300 times a
linear system AX=B with the same matrix A but varying B. A quick
timing last week revealed that using solve() 300 times was typically
slower by a factor of two ?in my use case than using inv() once (or
pinv() for that matter, it makes not appreciable difference). I'm
happy to be shown otherwise though.

I forgot that Armadillo doesn't provide a convenient way of using the
LU decomposition (that is one of the things that I find frustrating
about Armadillo). ?Did you try a single solve call in which the
right-hand side is an identity matrix? ?On looking at the Armadillo
sources it seems that it should call zgesv which is included in R's
subset of the Lapack routines.

It seems curious, looking at the bestiary of LAPACK functions, that so
many of them seem to perform a similar task. I wonder what are the
practical downsides of using a code that solves AX=I rather than one
inverting A. Also, when I replaced inv() by solve() and pinv(),
R-forge still failed to build complaining that zgesv_ was not present.
I assumed it was used by solve(), but perhaps it was pinv(). I'll give
it a try anyway, but I'd rather hope we can figure out a less ad hoc
solution.

Once upon a time we actually counted the number of floating point
operations that were needed to perform a particular calculation, which
is the reason for all the special-case code in Lapack. ?The difference
between using the LU decomposition of a matrix to solve a system in
which the right hand side is an identity, versus a special-purpose
piece of code like zgetri is that zgetri can take advantage of the
fact that the identity matrix is diagonal, thereby saving a relatively
small number of operations.

That makes sense, thanks.

I'm not sure why there should be a complaint about zgesv_ not being
available. ?It's there in the R sources (src/modules/lapack/cmplx.f
starting at line 3944). ?That routine is trivial because it just
checks its arguments then calls zgetrf and zgetrs.

Sorry, I meant zgels_ not zgesv_ (who came up with those names?!).

Fortran programmers. ?Just as the founders of Microsoft decided that
no one would ever have more than some amount of memory (128K, IIRC)
those defining the Fortran specification in the 1950's - 60's declared
that you would never need more than 6 upper-case characters in an
identifier.

You parse the name according to the first letter indicates the storage
type (z for double precision complex), the next two for specific
characteristics of the matrix (ge -> general, tr -> triangular, sy ->
symmetric, po -> symmetric and positive definite, ...) and the
remaining two or three for the operation (trf -> triangular
factorization, sl -> solution of linear systems, ls -> least squares
solution).

You should only have a request for zgels if you are trying to get
least squares solutions.

The complex Lapack routines that you can count on being available are

cmplx.f:96: ? ? ?SUBROUTINE ZBDSQR( UPLO, N, NCVT, NRU, NCC, D, E, VT, LDVT, U,
cmplx.f:838: ? ? ?SUBROUTINE ZDRSCL( N, SA, SX, INCX )
cmplx.f:952: ? ? ?SUBROUTINE ZGEBAK( JOB, SIDE, N, ILO, IHI, SCALE, M, V, LDV,
cmplx.f:1141: ? ? ?SUBROUTINE ZGEBAL( JOB, N, A, LDA, ILO, IHI, SCALE, INFO )
cmplx.f:1471: ? ? ?SUBROUTINE ZGEBD2( M, N, A, LDA, D, E, TAUQ, TAUP,
WORK, INFO )
cmplx.f:1721: ? ? ?SUBROUTINE ZGEBRD( M, N, A, LDA, D, E, TAUQ, TAUP,
WORK, LWORK,
cmplx.f:1989: ? ? ?SUBROUTINE ZGECON( NORM, N, A, LDA, ANORM, RCOND,
WORK, RWORK,
cmplx.f:2182: ? ? ?SUBROUTINE ZGEEV( JOBVL, JOBVR, N, A, LDA, W, VL,
LDVL, VR, LDVR,
cmplx.f:2578: ? ? ?SUBROUTINE ZGEHD2( N, ILO, IHI, A, LDA, TAU, WORK, INFO )
cmplx.f:2726: ? ? ?SUBROUTINE ZGEHRD( N, ILO, IHI, A, LDA, TAU, WORK,
LWORK, INFO )
cmplx.f:2999: ? ? ?SUBROUTINE ZGELQ2( M, N, A, LDA, TAU, WORK, INFO )
cmplx.f:3122: ? ? ?SUBROUTINE ZGELQF( M, N, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:3317: ? ? ?SUBROUTINE ZGEQP3( M, N, A, LDA, JPVT, TAU, WORK,
LWORK, RWORK,
cmplx.f:3610: ? ? ?SUBROUTINE ZGEQR2( M, N, A, LDA, TAU, WORK, INFO )
cmplx.f:3731: ? ? ?SUBROUTINE ZGEQRF( M, N, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:3927: ? ? ?SUBROUTINE ZGESV( N, NRHS, A, LDA, IPIV, B, LDB, INFO )
cmplx.f:4034: ? ? ?SUBROUTINE ZGESVD( JOBU, JOBVT, M, N, A, LDA, S, U,
LDU, VT, LDVT,
cmplx.f:7636: ? ? ?SUBROUTINE ZGETF2( M, N, A, LDA, IPIV, INFO )
cmplx.f:7784: ? ? ?SUBROUTINE ZGETRF( M, N, A, LDA, IPIV, INFO )
cmplx.f:7943: ? ? ?SUBROUTINE ZGETRS( TRANS, N, NRHS, A, LDA, IPIV, B,
LDB, INFO )
cmplx.f:8092: ? ? ?SUBROUTINE ZHEEV( JOBZ, UPLO, N, A, LDA, W, WORK,
LWORK, RWORK,
cmplx.f:8310: ? ? ?SUBROUTINE ZHETD2( UPLO, N, A, LDA, D, E, TAU, INFO )
cmplx.f:8568: ? ? ?SUBROUTINE ZHETRD( UPLO, N, A, LDA, D, E, TAU,
WORK, LWORK, INFO )
cmplx.f:8864: ? ? ?SUBROUTINE ZHSEQR( JOB, COMPZ, N, ILO, IHI, H, LDH,
W, Z, LDZ,
cmplx.f:9259: ? ? ?SUBROUTINE ZLABRD( M, N, NB, A, LDA, D, E, TAUQ,
TAUP, X, LDX, Y,
cmplx.f:9587: ? ? ?SUBROUTINE ZLACGV( N, X, INCX )
cmplx.f:9647: ? ? ?SUBROUTINE ZLACN2( N, V, X, EST, KASE, ISAVE )
cmplx.f:9868: ? ? ?SUBROUTINE ZLACPY( UPLO, M, N, A, LDA, B, LDB )
cmplx.f:10004: ? ? ?SUBROUTINE ZLAHQR( WANTT, WANTZ, N, ILO, IHI, H,
LDH, W, ILOZ,
cmplx.f:10474: ? ? ?SUBROUTINE ZLAHR2( N, K, NB, A, LDA, TAU, T, LDT, Y, LDY )
cmplx.f:10714: ? ? ?SUBROUTINE ZLAHRD( N, K, NB, A, LDA, TAU, T, LDT, Y, LDY )
cmplx.f:11678: ? ? ?SUBROUTINE ZLAQP2( M, N, OFFSET, A, LDA, JPVT,
TAU, VN1, VN2,
cmplx.f:11857: ? ? ?SUBROUTINE ZLAQPS( M, N, OFFSET, NB, KB, A, LDA,
JPVT, TAU, VN1,
cmplx.f:12123: ? ? ?SUBROUTINE ZLAQR0( WANTT, WANTZ, N, ILO, IHI, H,
LDH, W, ILOZ,
cmplx.f:12724: ? ? ?SUBROUTINE ZLAQR1( N, H, LDH, S1, S2, V )
cmplx.f:12821: ? ? ?SUBROUTINE ZLAQR2( WANTT, WANTZ, N, KTOP, KBOT,
NW, H, LDH, ILOZ,
cmplx.f:13258: ? ? ?SUBROUTINE ZLAQR3( WANTT, WANTZ, N, KTOP, KBOT,
NW, H, LDH, ILOZ,
cmplx.f:13706: ? ? ?SUBROUTINE ZLAQR4( WANTT, WANTZ, N, ILO, IHI, H,
LDH, W, ILOZ,
cmplx.f:14308: ? ? ?SUBROUTINE ZLAQR5( WANTT, WANTZ, KACC22, N, KTOP,
KBOT, NSHFTS, S,
cmplx.f:15117: ? ? ?SUBROUTINE ZLARF( SIDE, M, N, V, INCV, TAU, C, LDC, WORK )
cmplx.f:15237: ? ? ?SUBROUTINE ZLARFB( SIDE, TRANS, DIRECT, STOREV, M,
N, K, V, LDV,
cmplx.f:15845: ? ? ?SUBROUTINE ZLARFG( N, ALPHA, X, INCX, TAU )
cmplx.f:15990: ? ? ?SUBROUTINE ZLARFT( DIRECT, STOREV, N, K, V, LDV,
TAU, T, LDT )
cmplx.f:16214: ? ? ?SUBROUTINE ZLARFX( SIDE, M, N, V, TAU, C, LDC, WORK )
cmplx.f:16855: ? ? ?SUBROUTINE ZLARTG( F, G, CS, SN, R )
cmplx.f:17050: ? ? ?SUBROUTINE ZLASCL( TYPE, KL, KU, CFROM, CTO, M, N,
A, LDA, INFO )
cmplx.f:17317: ? ? ?SUBROUTINE ZLASET( UPLO, M, N, ALPHA, BETA, A, LDA )
cmplx.f:17431: ? ? ?SUBROUTINE ZLASR( SIDE, PIVOT, DIRECT, M, N, C, S, A, LDA )
cmplx.f:17794: ? ? ?SUBROUTINE ZLASSQ( N, X, INCX, SCALE, SUMSQ )
cmplx.f:17895: ? ? ?SUBROUTINE ZLASWP( N, A, LDA, K1, K2, IPIV, INCX )
cmplx.f:18014: ? ? ?SUBROUTINE ZLATRD( UPLO, N, NB, A, LDA, E, TAU, W, LDW )
cmplx.f:18293: ? ? ?SUBROUTINE ZLATRS( UPLO, TRANS, DIAG, NORMIN, N,
A, LDA, X, SCALE,
cmplx.f:19172: ? ? ?SUBROUTINE ZPOTF2( UPLO, N, A, LDA, INFO )
cmplx.f:19346: ? ? ?SUBROUTINE ZPOTRF( UPLO, N, A, LDA, INFO )
cmplx.f:19532: ? ? ?SUBROUTINE ZROT( N, CX, INCX, CY, INCY, C, S )
cmplx.f:19623: ? ? ?SUBROUTINE ZSTEQR( COMPZ, N, D, E, Z, LDZ, WORK, INFO )
cmplx.f:20126: ? ? ?SUBROUTINE ZTRCON( NORM, UPLO, DIAG, N, A, LDA, RCOND, WORK,
cmplx.f:20330: ? ? ?SUBROUTINE ZTREVC( SIDE, HOWMNY, SELECT, N, T,
LDT, VL, LDVL, VR,
cmplx.f:20716: ? ? ?SUBROUTINE ZTREXC( COMPQ, N, T, LDT, Q, LDQ, IFST,
ILST, INFO )
cmplx.f:20878: ? ? ?SUBROUTINE ZTRTRS( UPLO, TRANS, DIAG, N, NRHS, A,
LDA, B, LDB,
cmplx.f:21026: ? ? ?SUBROUTINE ZUNG2L( M, N, K, A, LDA, TAU, WORK, INFO )
cmplx.f:21154: ? ? ?SUBROUTINE ZUNG2R( M, N, K, A, LDA, TAU, WORK, INFO )
cmplx.f:21284: ? ? ?SUBROUTINE ZUNGBR( VECT, M, N, K, A, LDA, TAU,
WORK, LWORK, INFO )
cmplx.f:21529: ? ? ?SUBROUTINE ZUNGHR( N, ILO, IHI, A, LDA, TAU, WORK,
LWORK, INFO )
cmplx.f:21694: ? ? ?SUBROUTINE ZUNGL2( M, N, K, A, LDA, TAU, WORK, INFO )
cmplx.f:21830: ? ? ?SUBROUTINE ZUNGLQ( M, N, K, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:22045: ? ? ?SUBROUTINE ZUNGQL( M, N, K, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:22267: ? ? ?SUBROUTINE ZUNGQR( M, N, K, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:22483: ? ? ?SUBROUTINE ZUNGR2( M, N, K, A, LDA, TAU, WORK, INFO )
cmplx.f:22617: ? ? ?SUBROUTINE ZUNGRQ( M, N, K, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:22840: ? ? ?SUBROUTINE ZUNGTR( UPLO, N, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:23024: ? ? ?SUBROUTINE ZUNM2R( SIDE, TRANS, M, N, K, A, LDA,
TAU, C, LDC,
cmplx.f:23225: ? ? ?SUBROUTINE ZUNMBR( VECT, SIDE, TRANS, M, N, K, A,
LDA, TAU, C,
cmplx.f:23513: ? ? ?SUBROUTINE ZUNML2( SIDE, TRANS, M, N, K, A, LDA,
TAU, C, LDC,
cmplx.f:23718: ? ? ?SUBROUTINE ZUNMLQ( SIDE, TRANS, M, N, K, A, LDA,
TAU, C, LDC,
cmplx.f:23985: ? ? ?SUBROUTINE ZUNMQR( SIDE, TRANS, M, N, K, A, LDA,
TAU, C, LDC,

If you look at the definition of zgels you will see that for least
squares solutions it calls ZGEQRF, ZUNMQR and ZTRTRS so you just need
to set up those calls instead (although admittedly it's a royal PITA
to get the Fortran calls set up properly).

An alternative would be to leave a copy of the Fortran sources for
ZGELS in the src directory for your package but that may croak on
trying to compile the whole routine, which allows for both
over-determined and under-determined systems.

You can see the error here:
https://r-forge.r-project.org/R/?group_id=160&log=build_src&pkg=cda&flavor=patched

Best,

baptiste

Thanks,

baptiste

One option that I'd like to consider is whether the appropriate LAPACK
routines could be wrapped and shipped in a separate package
(discontinued rblas could provide a good starting point). Sadly, I
know nothing about static/dynamic libraries and all this business..

Thanks,

baptiste

| Sorry for being dense, I don't know anything about linking R to

| external dependencies.

It can be done; there are many examples -- for example every package
using
the GSL.

I just checked how RcppGSL does it, and well, this configure magic is
way above my head.

Best,

baptiste