Skip to content

gfortran bug?

8 messages · Andreas Noack Jensen, Simon Urbanek, Berend Hasselman

#
Dear list members

I have organised some code in a package including a Fortran subroutine for a
multivariate recursive filter to simulate VAR-porcesses. I have worked with
R for some time, but I am new to writing packages and coding Fortran so
first I thought I had made some mistakes in the Fortran code but I have now
tested the package on Ubuntu 10.4 and Windows XP and the problem is only
present on Mac.

On Mac I quite randomly get very weird results from the subroutine. It is
like the initial values explodes to fx 2e+290 even for identical calls
without random number generation. Most results are identical to results I
get from a pure R code but at some calls they explode in a non systematic
fashion.

I would have tried to use the gfortran 4.2.4 from r.research.att.com, to
isolate the error (right now I use their 4.2.3), but it seems like it
installs 4.2.1 instead and I get the same error still. I have tried to build
gfortran myself but was unsuccessful in doing so.

I cannot produce a small example you can run but as an example watch this:
+     cat(i)
+     tmp <- I1(civecm:::simulate.I1(fit.postWW2, res =
fit.postWW2$residuals), 2, 'drift')
+ }
1234567891011121314Error in svd(X) : infinite or missing values in 'x'
+     cat(i)
+     tmp <- I1(civecm:::simulate.I1(fit.postWW2, res =
fit.postWW2$residuals), 2, 'drift')
+ }
1234Error in svd(X) : infinite or missing values in 'x'

which is weird because there is no random number generation in play.
Completely identical calls every time. The error comes from the weird data
generated in the Fortran subroutine.

Some information about my system:
R version 2.11.0 (2010-04-22)
x86_64-apple-darwin9.8.0

locale:
[1] da_DK.UTF-8/da_DK.UTF-8/C/C/da_DK.UTF-8/da_DK.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] civecm_0.1-1   timsac_1.2.1   MASS_7.3-5     foreign_0.8-40

I use the latest prebuild version of R from r.research.att.com.

Finally the Fortran and R code for the subroute is:

      subroutine rmfilter(x, nl, coef, ndim, init)

      implicit none
      integer, intent(in) :: nl(2), ndim(3)
      double precision, intent(inout) :: x(nl(1),nl(2))
      double precision, intent(in) :: coef(ndim(1),ndim(2),ndim(3)),
     &init(ndim(3),ndim(nl(2)))
      double precision :: y(nl(1),nl(2))
      integer :: i, j

      y = 0.0d+0   
      
      do i = 1, nl(1), 1
       do j = 1, ndim(3), 1
        if ( i <= j ) then
         call dgemm('N', 'T', 1, ndim(1), nl(2), 1.0d+0,
     &init(ndim(3)-j+1,:), 1, coef(:,:,j), ndim(1), 1.0d+0, y(i,:), 1)
        else
         call dgemm('N', 'T', 1, ndim(1), nl(2), 1.0d+0, y(i-j,:), 1,
     &coef(:,:,j), ndim(1), 1.0d+0, y(i,:), 1)
        end if
       end do
       y(i,:) = y(i,:) + x(i,:)
      end do
      x = y
      return
      end subroutine rmfilter

and


rmfilter    <- function(x, coef, init)
{
    if(any(is.na(x))) stop('Series includes NAs')
    if(any(is.na(coef))) stop('Coefficients includes NAs')
    if(is.null(dim(x))) stop('Input series has no dimensions')
    nl        <- dim(x)
    if(!is.array(coef)) stop('coef must be an array')
    cdim    <- dim(coef)
    if(nl[2] != cdim[2] | cdim[1] != cdim[2]) stop('coef has wrong
dimensions')
    if(missing(init)) init <- matrix(0, cdim[3], nl[2])
    else if(any(is.na(init))) stop('Initial values includes NAs')
    else if(any(dim(init) != c(cdim[3], nl[2]))) stop('init matrix has wrong
dimensions')
    ans    <- .Fortran('rmfilter', as.double(x), as.integer(nl),
as.double(coef), as.integer(cdim), as.double(init))[[1]]
    dim(ans)    <- dim(x)
    dimnames(ans)<- dimnames(x)
    tsp(ans)    <- tsp(hasTsp(x))
    class(ans)    <- class(x)
    return(ans)
}

Best
Andreas
1 day later
#
I have been able to make an example that shows the weird behaviour. For the
Fortran code in last post run

rmfilter    <- function(x, coef)
{
    dyn.load('~/rmfilter.so')
    nl    <- dim(x)
    cdim    <- dim(coef)
    init    <- matrix(0, cdim[3], nl[2])
    ans    <- .Fortran('rmfilter', as.double(x), as.integer(nl),
as.double(coef), as.integer(cdim), as.double(init))[[1]]
    dim(ans)    <- dim(x)
    return(ans)
}

eps <- matrix(rnorm(300), ncol = 3)
tmp <- lapply(seq(5000), function(x) rmfilter(eps, array(diag(3)*0.5,
c(3,3,2))))
sum(!sapply(2:5000, function(x) all.equal(tmp[[1]], tmp[[x]])) == 'TRUE')

The three last runs I have done give

tmp <- lapply(seq(5000), function(x) rmfilter(eps, array(diag(3)*0.5,
c(3,3,2))))
[1] 639
c(3,3,2))))
[1] 0
c(3,3,2))))
[1] 29

I hope you have some ideas.

Best
Andreas


Den 25/04/10 16.22 skrev "Andreas Noack Jensen"
<andreas.noack.jensen at econ.ku.dk> f?lgende:
#
Andreas,

I can't even get your code to compile - can you send the actual files, please? Also you may want to ask on R-devel since I'm not sure there are many Fortran specialists reading this mailing list... (Although there is a possibility of miscompilations in majority the cases the cause is user code. The fact that it works on another platform is not necessarily a proof that it is not a bug in the code - it is often a coincidence).

Cheers,
Simon

PS: GNU Fortran 4.2.4 is really 4.2.4 - even if it is used in gcc 4.2.1 Xcode - that's what you see on the command line.
On Apr 27, 2010, at 8:42 AM, Andreas Noack Jensen wrote:

            
#
On 25-04-2010, at 22:22, Andreas Noack Jensen wrote:

            
As far as I can tell, your fortran code is in fixed format which is not immediately obvious.
I have put the code in free format and  have reformatted the code with more indentation (1 space of indent is really not enough).
See below after my signature.
It  now compiles with no errors.

I don't see anything immediately wrong.
It would be nice to have a small test program in fortran to check the code.

Berend

     subroutine rmfilter(x, nl, coef, ndim, init)

     implicit none
     integer, intent(in) :: nl(2), ndim(3)
     double precision, intent(inout) :: x(nl(1),nl(2))
     double precision, intent(in) :: coef(ndim(1),ndim(2),ndim(3)),   init(ndim(3),ndim(nl(2)))
     double precision :: y(nl(1),nl(2))
     integer :: i, j

     y = 0.0d+0   
         
     do i = 1, nl(1), 1
         do j = 1, ndim(3), 1
             if ( i <= j ) then
                 call dgemm('N', 'T', 1, ndim(1), nl(2), 1.0d+0,    &
                        init(ndim(3)-j+1,:), 1, coef(:,:,j), ndim(1), 1.0d+0, y(i,:), 1)
             else
                 call dgemm('N', 'T', 1, ndim(1), nl(2), 1.0d+0,    &
                        y(i-j,:), 1, coef(:,:,j), ndim(1), 1.0d+0, y(i,:), 1)
             end if
         end do
         y(i,:) = y(i,:) + x(i,:)
     end do
     x = y
     return
     end subroutine rmfilter

BTW. in       do i = 1, nl(1), 1  and    do j = 1, ndim(3), 1
the steps 1 are not necessary: 1 is the default
#
On 25-04-2010, at 22:22, Andreas Noack Jensen wrote:
...
...

I have experimented with your stuff.
Find attached the R files needed to run my test.
Below is what I have done together with the R output.
As you can see the third column of the result matrix is weird.
This result is independent from setting the seed.
Some sort of initialisation issue?
Since I haven't got a clue what rmfilter does and my memory needs some refreshing wrt dgemm,
I can't pinpoint the problem.

Berend


I am (still)  using the gfortran from gfortran-4.2.3.dmg
------------------------------------------------------------------   
I made the rmfilter.so with

MAKEFLAGS="FCFLAGS=-O2" R CMD SHLIB rmfilter.f90

NB. had to do it since this gfortran doesn't accept the -mtune=core2 option!!
----------------------------------------------------------------------
sessionInfo()

R version 2.10.1 Patched (2010-04-07 r51689) 
x86_64-apple-darwin9.8.0 

locale:
[1] en_GB/en_GB/C/C/en_GB/en_GB

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

------------------------------------------------------------------
Mac OS X 10.6.3 Intel

------------------------------------------------------------------

I ran this test job


dyn.load('rmfilter.so')
source("rmfilter.R")   

set.seed(407)

eps <- matrix(rnorm(30), ncol = 3)
eps

z <-  rmfilter(eps, array(diag(3)*0.5, c(3,3,2)))
z


with this as output
[,1]       [,2]       [,3]
 [1,]  0.841081288  1.7100891  0.9161884
 [2,]  1.072881139  1.1351244 -0.4314795
 [3,] -0.543593265 -1.0828613 -0.6333517
 [4,]  0.441951763 -1.0539372 -1.8468529
 [5,]  0.004920633 -0.9635668  0.6060301
 [6,] -0.731256023  0.4951620 -1.9168646
 [7,] -1.216400546 -1.5097126  1.1407077
 [8,] -0.210674189 -1.1110270 -0.3563495
 [9,] -0.378021236 -1.5624664 -2.5094364
[10,] -0.939183015 -0.2886860  2.1596868
[,1]       [,2]          [,3]
 [1,]  0.8410813  1.7100891 1.018209e+277
 [2,]  1.4934218  1.9901689 1.018209e+277
 [3,]  0.6236583  0.7672677 1.018209e+277
 [4,]  1.5004918  0.3247811 1.018209e+277
 [5,]  1.0669957 -0.4175424 1.018209e+277
 [6,]  0.5524877  0.4487814 1.018209e+277
 [7,] -0.4066589 -1.4940931 1.018209e+277
 [8,] -0.1377598 -1.6336829 1.018209e+277
 [9,] -0.6502306 -3.1263543 1.018209e+277
[10,] -1.3331782 -2.6687046 1.018209e+277
attr(,"tsp")
[1]  1 10  1




-------------- next part --------------
A non-text attachment was scrubbed...
Name: rmfilter.R
Type: application/octet-stream
Size: 878 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-mac/attachments/20100427/152ffbdd/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test1.R
Type: application/octet-stream
Size: 157 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-mac/attachments/20100427/152ffbdd/attachment-0001.obj>
#
Thank you so much for looking at it. Initially my code was in the "free"
F95 format but partly because of the error and partly because of portability
concerns I chose to translate it to strict F77 even thought it is ugly and
annoying with the line width limit.

The subroutine is thought as a multivariate version of filter(..., method =
'recursive') and I use it to simulate VAR processes. dgemm is the double
precision BLAS subroutine for

C = alpha*A*B + beta*C

for general matrices A, B, C and scalars alpha and beta.

Indeed I think something is going wrong with the initial values but I cannot
figure out why. Maybe because of missing Fortran skills but I am puzzled
because the code works fine on Windows and Linux.

I think the "-mtune=core2" problem has disappeared in the latest 4.2.3
gfortran at r.research because I had the error in the beginning but after
reinstall of gfortran it was gone (right now, however,  I am on gfortran
4.2.4 aka gcc 4.2.1).

Thank you once again and hope some new ideas to isolate the problem will
appear.

Best
Andreas

Den 27/04/10 13.34 skrev "Berend Hasselman" <bhh at xs4all.nl> f?lgende:
#
On 27-04-2010, at 20:28, Andreas Noack Jensen wrote:

            
Indeed, that is what I have found. See below.
The error is in the fortran line

     double precision, intent(in) :: coef(ndim(1),ndim(2),ndim(3)),   init(ndim(3),ndim(nl(2)))

The dimensions of init are wrong.
The line should read:

     double precision, intent(in) :: coef(ndim(1),ndim(2),ndim(3)),   init(ndim(3),nl(2))

I have attached the corrected rmfilter.f90, a slightly modified rmfilter.R that I used to print various stuff
the test1.R and the output.

In the test case your code declares init(2,2)
In the corrected code it is declared as init(2,3)
The actual dimension is init(2,3)

Berend


-------------- next part --------------
A non-text attachment was scrubbed...
Name: rmfilter.f90
Type: application/octet-stream
Size: 994 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-mac/attachments/20100427/8b2bf75f/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test1.R
Type: application/octet-stream
Size: 157 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-mac/attachments/20100427/8b2bf75f/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rmfilter.R
Type: application/octet-stream
Size: 984 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-mac/attachments/20100427/8b2bf75f/attachment-0002.obj>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test1.Rout.txt
URL: <https://stat.ethz.ch/pipermail/r-sig-mac/attachments/20100427/8b2bf75f/attachment.txt>
-------------- next part --------------
#
Great. There it was. Thank you so much for helping the Fortran amateur.

Andreas


Den 27/04/10 14.43 skrev "Berend Hasselman" <bhh at xs4all.nl> f?lgende: