Large NetCDF file problems

My take is that it's an outdated NetCDF library being shipped with RNetCDF
and ncdf. Here's a minimal reproducible example that behaves fine in 32-bit
R, but not 64-bit. Reduce n to 2^14 and it works in either:

n <- 2^15

library(ncdf)
d1 <- dim.def.ncdf("x", "count", 1:n)
d2 <- dim.def.ncdf("y", "count", 1:n)
v <- var.def.ncdf("v", "count", list(d1, d2), -1)
nc <- create.ncdf("afile.nc", v)


You won't see this in Linux installs because they will usually be using
later versions of NetCDF. (It's a special CRAN service to build these
packages on a Windows machine with NetCDF installed.)

I am interested in learning more about compiling for myself in Windows, but
it's just quite painful.

HTH


On Thu, Jun 12, 2014 at 10:45 AM, Pablo Alfaro <palfaro at motionsoft.com.uy>
wrote:

So you think this is not an R problem then?
The executable failing the assertion is rsession.exe


2014-06-11 18:14 GMT-03:00 Roger Bivand <Roger.Bivand at nhh.no>:

I suggest that you put together a complete reproducible example with

access to the file on a server, and send this to the maintainers of ncdf
and RNetCDF. This looks either like an issue in compiler flags, or an issue
in the code of the external library used.

Roger


On Wed, 11 Jun 2014, Pablo Alfaro wrote:

 Hmm, just tried using RNetCDF (v 1.6.2-2) and it has the same problem.

When using

require('RNetCDF')
nc <- open.nc('C:/testsMCH/test/wrfout_d02_2014-06-09_00%3A00%3A00')
var.get.nc(nc, "T2", start=c(1, 1, 1), count=c(NA, NA, NA))

I get the same problem on 64bit R, no problem on 32bits.

Seems like a problem with the version of posixio.c used in 64 bit R. Just
found a version of posixio.c on the web (
http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/util/sorc/
ncdump.fd/libsrc/posixio.c)
which has the assertion on line 417.

assert(offset >= 0);

offset is a variable of type off_t which is defined in <sys/types.h> and
apparently requires that you enable Large File Support on the compiler
command line to be a 64 bit integer.
Can it be that the 32 bit version of R is using Large File Support but
the
64 bit version is not?

Best
Pablo




2014-06-11 15:07 GMT-03:00 Pablo Alfaro <palfaro at motionsoft.com.uy>:

 Hi Michael, Pascal, thank you both for your replies.

Yeah, Sorry, I'm using R 3.0.3 64 (or 32 for the working version) bit,
on
Windows 7 x64 and the NetCDF library is ncdf (not ncdf4) version 1.6.6,
the
last available version.

I need it to run on Windows since the servers I run the code on are
Windows.

I tried downloading and installing the package you mention Michael but
when loading it, it says the x64 architecture is not supported (the
author
mentions it in the site as well). The thing is that ncdf works on x86
so it
leaves me with the same options.

RNetCDF seems to use NetCDF v3, but maybe it doesn't have this problem.
I
wil try that and see if it helps.

The assertion on the error complains an offset which isn't >= 0. If I
were
to take a guess, that offset is a 32 bit signed integer and that's why
it
breaks only with large files (bigger than 2^31 - 1 bytes). But I can't
tell
for sure. Also it's weird that it works on 32 bits.

Best
Pablo


2014-06-10 23:13 GMT-03:00 Michael Sumner <mdsumner at gmail.com>:

On Wed, Jun 11, 2014 at 11:49 AM, Pascal Oettli <kridox at ymail.com>
wrote:

 Hello,

Do you get the same error if you run your code out of Rstudio?

However, another approach would be to open your NetCDF file with
'brick' function from the raster package.

I don't think either of these will help, raster simply runs ncdf (or
preferably ncdf4) to do the file read. You might find it easier to
move to
ncdf4 if you can, but potentially the stuff I write below is just an
unwelcome aside. I have seen a similar error in a file I've tried to
use,
so I'll have another look at that and see if I can isolate where the
problem lies. At minimum we really need to know the version of the
NetCDF
library in use and perhaps details of how it was compiled so at
minimum you
should provide the ncdf package version as well as R's.

It's possible to install ncdf4 on Windows, but it's not supported (no
binary .zip on CRAN) but you can do it yourself (again not supported,
and
the archive here is not updated):
http://cirrus.ucsd.edu/~pierce/ncdf/

It's much more generally available to do this on Linux, though there's
always bumps in the road.

RNetCDF is actively updated on CRAN and is another option to see if the
read is supported (I don't know how to find out what version of the
library
was used for the CRAN Windows binary, but presambly it is < 4.0).

Cheers, Mike.



 Regards,

Pascal

On Wed, Jun 11, 2014 at 9:10 AM, Pablo Alfaro <
palfaro at motionsoft.com.uy>
wrote:

Hi, I'm having trouble opening a large netcdf file (2.45 GB) with the
following code on R 3.0.3 64 bit

require('ncdf')
nc <- open.ncdf('C:/testsMCH/test/wrfout_d02_2014-06-09_00%3A00%
3A00')
get.var.ncdf(nc, 'T2', start=c(1, 1, 1), count=c(-1, -1, -1))

I get this error:
Assertion failed!

Program: C:\Program Files\RStudio\bin\x64\rsession.exe
File: posixio.c, Line 417

Expression: offset >= 0

This application has requested the Runtime to terminate it in an

unusual

way.
Please contact the application's support team for more information.


In R 32 bit there are no problems.

It is the same problem as reported here:

 http://r.789695.n4.nabble.com/Problems-with-Raster-and-clim-

pact-packages-with-large-netcdf-files-2-7G-in-x64-bit-
R-td4119638.html#a4691956

Does anyone have any clue as to why this may be?
I can upload the file on an FTP server but it'll take me a while.

Thanks in advance!
Pablo

--
Ing. Pablo Alfaro Pi??eyro

MotionSoft Consulting S.R.L.
Tel: 095 938 487

        [[alternative HTML version deleted]]

--
Pascal Oettli
Project Scientist
JAMSTEC
Yokohama, Japan

--
Michael Sumner
Software and Database Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsumner at gmail.com

--
Ing. Pablo Alfaro Pi??eyro

MotionSoft Consulting S.R.L.
Tel: 095 938 487

--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 91 00
e-mail: Roger.Bivand at nhh.no

--
Ing. Pablo Alfaro Pi?eyro

MotionSoft Consulting S.R.L.
Tel: 095 938 487


2014-06-11 18:14 GMT-03:00 Roger Bivand <Roger.Bivand at nhh.no>:

I suggest that you put together a complete reproducible example with

access to the file on a server, and send this to the maintainers of ncdf
and RNetCDF. This looks either like an issue in compiler flags, or an issue
in the code of the external library used.

Roger


On Wed, 11 Jun 2014, Pablo Alfaro wrote:

 Hmm, just tried using RNetCDF (v 1.6.2-2) and it has the same problem.

When using

require('RNetCDF')
nc <- open.nc('C:/testsMCH/test/wrfout_d02_2014-06-09_00%3A00%3A00')
var.get.nc(nc, "T2", start=c(1, 1, 1), count=c(NA, NA, NA))

I get the same problem on 64bit R, no problem on 32bits.

Seems like a problem with the version of posixio.c used in 64 bit R. Just
found a version of posixio.c on the web (
http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/util/sorc/
ncdump.fd/libsrc/posixio.c)
which has the assertion on line 417.

assert(offset >= 0);

offset is a variable of type off_t which is defined in <sys/types.h> and
apparently requires that you enable Large File Support on the compiler
command line to be a 64 bit integer.
Can it be that the 32 bit version of R is using Large File Support but
the
64 bit version is not?

Best
Pablo




2014-06-11 15:07 GMT-03:00 Pablo Alfaro <palfaro at motionsoft.com.uy>:

 Hi Michael, Pascal, thank you both for your replies.

Yeah, Sorry, I'm using R 3.0.3 64 (or 32 for the working version) bit,
on
Windows 7 x64 and the NetCDF library is ncdf (not ncdf4) version 1.6.6,
the
last available version.

I need it to run on Windows since the servers I run the code on are
Windows.

I tried downloading and installing the package you mention Michael but
when loading it, it says the x64 architecture is not supported (the
author
mentions it in the site as well). The thing is that ncdf works on x86
so it
leaves me with the same options.

RNetCDF seems to use NetCDF v3, but maybe it doesn't have this problem.
I
wil try that and see if it helps.

The assertion on the error complains an offset which isn't >= 0. If I
were
to take a guess, that offset is a 32 bit signed integer and that's why
it
breaks only with large files (bigger than 2^31 - 1 bytes). But I can't
tell
for sure. Also it's weird that it works on 32 bits.

Best
Pablo


2014-06-10 23:13 GMT-03:00 Michael Sumner <mdsumner at gmail.com>:

On Wed, Jun 11, 2014 at 11:49 AM, Pascal Oettli <kridox at ymail.com>
wrote:

 Hello,

Do you get the same error if you run your code out of Rstudio?

However, another approach would be to open your NetCDF file with
'brick' function from the raster package.

I don't think either of these will help, raster simply runs ncdf (or
preferably ncdf4) to do the file read. You might find it easier to
move to
ncdf4 if you can, but potentially the stuff I write below is just an
unwelcome aside. I have seen a similar error in a file I've tried to
use,
so I'll have another look at that and see if I can isolate where the
problem lies. At minimum we really need to know the version of the
NetCDF
library in use and perhaps details of how it was compiled so at
minimum you
should provide the ncdf package version as well as R's.

It's possible to install ncdf4 on Windows, but it's not supported (no
binary .zip on CRAN) but you can do it yourself (again not supported,
and
the archive here is not updated):
http://cirrus.ucsd.edu/~pierce/ncdf/

It's much more generally available to do this on Linux, though there's
always bumps in the road.

RNetCDF is actively updated on CRAN and is another option to see if the
read is supported (I don't know how to find out what version of the
library
was used for the CRAN Windows binary, but presambly it is < 4.0).

Cheers, Mike.



 Regards,

Pascal

On Wed, Jun 11, 2014 at 9:10 AM, Pablo Alfaro <
palfaro at motionsoft.com.uy>
wrote:

Hi, I'm having trouble opening a large netcdf file (2.45 GB) with the
following code on R 3.0.3 64 bit

require('ncdf')
nc <- open.ncdf('C:/testsMCH/test/wrfout_d02_2014-06-09_00%3A00%
3A00')
get.var.ncdf(nc, 'T2', start=c(1, 1, 1), count=c(-1, -1, -1))

I get this error:
Assertion failed!

Program: C:\Program Files\RStudio\bin\x64\rsession.exe
File: posixio.c, Line 417

Expression: offset >= 0

This application has requested the Runtime to terminate it in an

unusual

way.
Please contact the application's support team for more information.


In R 32 bit there are no problems.

It is the same problem as reported here:

 http://r.789695.n4.nabble.com/Problems-with-Raster-and-clim-

pact-packages-with-large-netcdf-files-2-7G-in-x64-bit-
R-td4119638.html#a4691956

Does anyone have any clue as to why this may be?
I can upload the file on an FTP server but it'll take me a while.

Thanks in advance!
Pablo

--
Ing. Pablo Alfaro Pi??eyro

MotionSoft Consulting S.R.L.
Tel: 095 938 487

        [[alternative HTML version deleted]]

--
Pascal Oettli
Project Scientist
JAMSTEC
Yokohama, Japan

--
Michael Sumner
Software and Database Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsumner at gmail.com

--
Ing. Pablo Alfaro Pi??eyro

MotionSoft Consulting S.R.L.
Tel: 095 938 487

--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 91 00
e-mail: Roger.Bivand at nhh.no

--
Ing. Pablo Alfaro Pi?eyro

MotionSoft Consulting S.R.L.
Tel: 095 938 487

--
Michael Sumner
Software and Database Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsumner at gmail.com

        [[alternative HTML version deleted]]