Skip to content

"make check" fails on lapack.R and stats-Ex.R

5 messages · Pacey, Mike, Brian Ripley, Martyn Plummer

Original
Pacey, Mike
# 
Hi folks,

I suspect this is a request for a sanity check than a bug report:

I've been successfully compiling an optimised version of R for several years using the Intel compiler and MKL. I've just test-run the new Intel 15.0 compiler suite, and I'm seeing a few numeric failures that I don't see using the same build method with Intel 13.0. I've attached the output of "make check". Build details are below.

The most notable failures are in lapack.R, though I see from the comments in the output that different lapack and blas libraries may produce different signs for some outputs which can be safely ignored. My linear algebra's a bit rusty, so I'd like a sanity check: can all the sign differences be safely ignored in the attached output? (And a possible RFC: at least for the purposes of make check, can the scripts output abs() values for all cases where sign isn't an issue?)

The other failures are in stats-Ex.R. It looks like most of the problem lines are outputs from a PCA-like function, so their sign differences might due to the eigenvalue exception comment in lapack.R.

The final failures are in the "Grand Total" lines from stats-Ex.R. The values differ in the 7th sig fig, so a pretty small relative error. I think I'm using compiler flags that rule out any fast-math imprecisions - so I'm wondering if this result is actually within the acceptable variation of the IEE 754 standard?

Build details:

OS is Scientific Linux 6.4, architecture is Westmere.

"icc -v" gives: icc version 15.0.0 (gcc version 4.4.7 compatibility)

Environment variables are:

export CC=icc
export CFLAGS="-O3 -xHOST -axCORE-AVX-I -fp-model precise"
export FC=ifort
export F77=ifort
export FFLAGS="-O3 -xHOST -axCORE-AVX-I -fp-model precise"
export FCFLAGS="-O3 -xHOST -axCORE-AVX-I -fp-model precise"
export CXX=icpc
export CXXFLAGS="-O3 -xHOST -axCORE-AVX-I -fp-model precise"
export JAVA_HOME=/etc/alternatives/java_sdk_1.6.0

Configure command is:

./configure --with-blas=-mkl=sequential --with-lapack=-mkl=sequential --disable-openmp --with-tcl-config=/usr/shared_apps/packages/tcl-8.5.12/lib/tclConfig.sh --with-tk-config=/usr/shared_apps/packages/tk-8.5.12/lib/tkConfig.sh

Regards,
Mike.

-----

Dr Mike Pacey,                                          Email: M.Pacey at lancaster.ac.uk<mailto:M.Pacey at lancaster.ac.uk>
HPC Manager,                                            Phone: 01524 510659
Information Systems Services,            Fax: 01524 594459
ISS Building, Lancaster University,
Lancaster LA1 4WA
Pacey, Mike
# 
As my attachment doesn't seem to have survived transit, I'm cut'n'pasting the relevant failures here:

Testing examples for package 'stats'
  comparing 'stats-Ex.Rout' to 'stats-Ex.Rout.save' ...
6466c6466
< Grand Mean: 291.5937
---
12881c12881
< Murder   -0.536  0.418  0.341  0.649
---
12882c12882
< Assault  -0.583  0.188  0.268 -0.743
---
12883c12883
< UrbanPop -0.278 -0.873  0.378  0.134
---
12884c12884
< Rape     -0.543 -0.167 -0.818       
---
14628c14628
< Grand Mean: 291.5937
---
15777c15777
< Murder   -0.54   0.42   0.34   0.65 
---
15778c15778
< Assault  -0.58          0.27  -0.74 
---
15779c15779
< UrbanPop -0.28  -0.87   0.38        
---
15780c15780
< Rape     -0.54         -0.82        
---
running code in 'lapack.R' ... OK
  comparing 'lapack.Rout' to './lapack.Rout.save' ...23,31c23,31
<  [1,] -0.7245 -0.6266 -0.27350  0.08527 -0.02074 -0.004025
<  [2,] -0.4282  0.1299  0.64294 -0.55047  0.27253  0.092816
<  [3,] -0.3122  0.2804  0.33633  0.31418 -0.61632 -0.440904
<  [4,] -0.2479  0.3142  0.06931  0.44667 -0.02945  0.530120
<  [5,] -0.2064  0.3141 -0.10786  0.30242  0.35567  0.237038
<  [6,] -0.1771  0.3027 -0.22106  0.09042  0.38879 -0.260449
<  [7,] -0.1553  0.2877 -0.29281 -0.11551  0.19286 -0.420945
<  [8,] -0.1384  0.2722 -0.33784 -0.29313 -0.11633 -0.160790
<  [9,] -0.1249  0.2571 -0.36543 -0.43885 -0.46497  0.434600
---
35,40c35,40
< [1,] -0.7365 -0.6225 -0.2550  0.06976 -0.01328 -0.001588
< [2,] -0.4433  0.1819  0.6867 -0.50860  0.19627  0.041117
< [3,] -0.3275  0.3509  0.2611  0.50474 -0.61606 -0.259216
< [4,] -0.2626  0.3922 -0.1044  0.43748  0.40834  0.638902
< [5,] -0.2204  0.3946 -0.3510 -0.01612  0.46428 -0.675827
< [6,] -0.1904  0.3832 -0.5111 -0.53856 -0.44664  0.257249
---
-----Original Message-----
From: r-devel-bounces at r-project.org [mailto:r-devel-bounces at r-project.org] On Behalf Of Pacey, Mike
Sent: 22 October 2014 17:02
To: r-devel at r-project.org
Subject: [Rd] "make check" fails on lapack.R and stats-Ex.R

Hi folks,

I suspect this is a request for a sanity check than a bug report:

I've been successfully compiling an optimised version of R for several years using the Intel compiler and MKL. I've just test-run the new Intel 15.0 compiler suite, and I'm seeing a few numeric failures that I don't see using the same build method with Intel 13.0. I've attached the output of "make check". Build details are below.

The most notable failures are in lapack.R, though I see from the comments in the output that different lapack and blas libraries may produce different signs for some outputs which can be safely ignored. My linear algebra's a bit rusty, so I'd like a sanity check: can all the sign differences be safely ignored in the attached output? (And a possible RFC: at least for the purposes of make check, can the scripts output abs() values for all cases where sign isn't an issue?)

The other failures are in stats-Ex.R. It looks like most of the problem lines are outputs from a PCA-like function, so their sign differences might due to the eigenvalue exception comment in lapack.R.

The final failures are in the "Grand Total" lines from stats-Ex.R. The values differ in the 7th sig fig, so a pretty small relative error. I think I'm using compiler flags that rule out any fast-math imprecisions - so I'm wondering if this result is actually within the acceptable variation of the IEE 754 standard?

Build details:

OS is Scientific Linux 6.4, architecture is Westmere.

"icc -v" gives: icc version 15.0.0 (gcc version 4.4.7 compatibility)

Environment variables are:

export CC=icc
export CFLAGS="-O3 -xHOST -axCORE-AVX-I -fp-model precise"
export FC=ifort
export F77=ifort
export FFLAGS="-O3 -xHOST -axCORE-AVX-I -fp-model precise"
export FCFLAGS="-O3 -xHOST -axCORE-AVX-I -fp-model precise"
export CXX=icpc
export CXXFLAGS="-O3 -xHOST -axCORE-AVX-I -fp-model precise"
export JAVA_HOME=/etc/alternatives/java_sdk_1.6.0

Configure command is:

./configure --with-blas=-mkl=sequential --with-lapack=-mkl=sequential --disable-openmp --with-tcl-config=/usr/shared_apps/packages/tcl-8.5.12/lib/tclConfig.sh --with-tk-config=/usr/shared_apps/packages/tk-8.5.12/lib/tkConfig.sh

Regards,
Mike.

-----

Dr Mike Pacey,                                          Email: M.Pacey at lancaster.ac.uk<mailto:M.Pacey at lancaster.ac.uk>
HPC Manager,                                            Phone: 01524 510659
Information Systems Services,            Fax: 01524 594459
ISS Building, Lancaster University,
Lancaster LA1 4WA
Brian Ripley
# 
You are using the word 'failure' in a sense I do not recognize. 
Differences are not of themselves errors and not necessarily failures. 
I see 'OK', not 'ERROR' here.

For the record, other builds using MKL (but gcc not icc) show similar 
differences, and the top of lapack.R says

## NB: the signs of singular and eigenvectors are arbitrary,
## so there may be differences from the reference ouptut,
## especially when alternative BLAS are used.

and e.g. ?prcomp says

Note:

      The signs of the columns of the rotation matrix are arbitrary, and
      so may differ between different programs for PCA, and even between
      different builds of R.

So AFAICS this is all documented behaviour.
On 23/10/2014 09:19, Pacey, Mike wrote:

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Martyn Plummer
    

    

      
      #
    

  


  

      
On Thu, 2014-10-23 at 08:19 +0000, Pacey, Mike wrote:

            

      
      
      
    

        
I see the same thing, but it is not as bad as it looks. The actual value
is 291.59375 so a small amount of numerical error can make the rounding
to 4 decimal places go either way:

            

      
      
      
    

        
(Intercept) 
291.5937500000002 

Note that MKL sacrifices reproducibility (and hence precision) for
speed. See more details here:

https://software.intel.com/en-us/articles/run-reproducibility-with-intel-mkl-and-the-intel-compilers

Martyn

            

      
      
      
    

        

      
      
    

        
-----------------------------------------------------------------------
This message and its attachments are strictly confidenti...{{dropped:8}}

  
  


  


      

    

    

      
      

        


  

        

      Pacey, Mike
    

    

      
      #
    

  


  

      
Hi Martyn,

Thanks for the analysis on that, and I'll remember that test in future - I've not seen a rounding error quite so finely balanced before. I've been using the "-fp-model precise" advice referenced in that document for a while - apparently it wasn't sufficient for this case.

Thanks also to Brian for sanity checking the linear algebra results in a previous post, and apologies for my imprecise terminology.

Regards,
Mike.

-----Original Message-----
From: Martyn Plummer [mailto:plummerm at iarc.fr] 
Sent: 23 October 2014 10:46
To: Pacey, Mike
Cc: r-devel at r-project.org
Subject: Re: [Rd] "make check" fails on lapack.R and stats-Ex.R

        
On Thu, 2014-10-23 at 08:19 +0000, Pacey, Mike wrote:

            

      
      
      
    

        
I see the same thing, but it is not as bad as it looks. The actual value is 291.59375 so a small amount of numerical error can make the rounding to 4 decimal places go either way:

            

      
      
      
    

        
(Intercept)
291.5937500000002 

Note that MKL sacrifices reproducibility (and hence precision) for speed. See more details here:

https://software.intel.com/en-us/articles/run-reproducibility-with-intel-mkl-and-the-intel-compilers

Martyn