Skip to content

Issue with rgrass7 and readRAST

5 messages · Roger Bivand, Thomas Adams

Original
Roger Bivand
# 
Briefly, please try to run r.out.bin manually, to the same output or another. This is what is failing on theGRASS side.

Roger

-------
Roger Bivand
Norwegian School of Economics, Bergen, Norway
roger.bivand at nhh.no

________________________________
Fra: Thomas Adams <tea3rd at gmail.com>
Sendt: l?rdag 11. juli 2020, 17:59
Til: Roger Bivand
Kopi: r-sig-Geo at r-project.org
Emne: Re: [R-sig-Geo] Issue with rgrass7 and readRAST

Hi Roger!

Here is what was written in /media/teaiii/development/grass/greensboro_mercator/teaiii/.tmp/teaiii-Leopard-WS:

drwxr-xr-x 3 teaiii teaiii   4096 Sep 23  2019 ..
-rw-r--r-- 1 teaiii teaiii 131072 Jul  9 10:02 801.0.gpkg
-rw-r--r-- 1 teaiii teaiii 131072 Jul  9 10:04 466.0.gpkg
-rw-r--r-- 1 teaiii teaiii      0 Jul  9 20:32 801.0
-rw-r--r-- 1 teaiii teaiii      0 Jul  9 22:40 188.0
-rw-r--r-- 1 teaiii teaiii      0 Jul  9 22:51 222.0

And there is plenty of space on the disk drive...

Tom
On Sat, Jul 11, 2020 at 9:26 AM Roger Bivand <Roger.Bivand at nhh.no<mailto:Roger.Bivand at nhh.no>> wrote:
Tom
On Fri, 10 Jul 2020, Thomas Adams wrote:

            

      
      
      
    

        
I have not succeeded in reproducing this, I'm afraid. I created a location
with cell counts to match, and ran r.out.bin first to check whether there
were any issues writing the data to a temporary file. Then I ran the
equivalent of your command (readRAST()), again without problems. I'm on a
16 GB PC, so the problem is puzzling. I'm curious about the free space on
your

/media/teaiii/development/grass/greensboro_mercator/teaiii/.tmp/
teaiii-Leopard-WS

directory. Did you check whether any output arrived there, since the error
message is coming from r.out.bin?

The error is on line 53 in raster/r.out.bin/main.c, when the system
function fwrite() returns an error.

Thanks for your patience.

Roger

            

      
      
      
    

        
--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: Roger.Bivand at nhh.no<mailto:Roger.Bivand at nhh.no>
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en


--
Thomas E Adams, III
1724 Sage Lane
Blacksburg, VA 24060
tea3rd at gmail.com<mailto:tea3rd at gmail.com> (personal)
tea at terrapredictions.org<mailto:tea at terrapredictions.org> (work)

1 (513) 739-9512 (cell)

  
  


  


    

    

      
      

        


  

        

      Thomas Adams
    

    

      
      #
    

  


  

      
Hi Roger,

r.out.bin from GRASS did succeed... r.out.bin
input=mozambique_srtm_patch at teaiii output=mozambique_srtm_patch null=-999
Exporting raster as integer values (bytes=2)

A very large file!!
-rw-r--r--   1 teaiii teaiii 1045368225220 Jul 11 13:26
mozambique_srtm_patch

Tom

        
On Sat, Jul 11, 2020 at 12:12 PM Roger Bivand <Roger.Bivand at nhh.no> wrote:

        

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Thomas Adams
    

    

      
      #
    

  


  

      
Hi Roger,

I believe I may have been correct about the memory issue... it's possible,
in GRASS, that I did not restrict my REGION sufficiently, using g.region.
Additionally, I applied a MASK for my country of interest, Mozambique,
before starting R, which yielded:

srtm <- readRAST("mozambique_srtm_patch",cat=FALSE)
Creating BIL support files...
Exporting raster as integer values (bytes=4)
 100%
Warning messages:
1: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
  Discarded ellps unknown in CRS definition: +proj=merc +lat_ts=0 +lon_0=0
+x_0=0 +y_0=0 +R=6378137 +units=m +no_defs
2: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
  Discarded datum unknown in CRS definition

            

      
      
      
    

        
Error in rasterImage(m, bb[1, 1], bb[2, 1], bb[1, 2], bb[2, 2], interpolate
= interpolate,  :
  cannot allocate memory block of size 16777216 Tb

Still, a huge amount of memory needed (is 16777216 Tb correct??)

My thought was to resample the SRTM data in GRASS to 1km resolution, then
start R, and proceed, which quite honestly, for my purposes, really is
sufficient. By doing this, everything worked perfectly! I was able to read
the srtm data quickly and generate an image. I am puzzled by the "cannot
allocate memory block of size 16777216 Tb" message with my GRASS Region
settings in place.

Thank you for all your help!

Best,
Tom

        
On Sat, Jul 11, 2020 at 11:02 PM Thomas Adams <tea3rd at gmail.com> wrote:

        

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Roger Bivand
    

    

      
      #
    

  


  

      
On Sun, 12 Jul 2020, Thomas Adams wrote:

        

            

      
      
      
    

        
I'd got to an errant g.region before r.out.bin too.

graphics::rasterImage() is reporting the error. The metric is wrong, but 
the problem is real. The object is about 650M x 4 bytes, which image() 
needs to manipulate too, creating several copies (in the imported object 
it is a vector, but is cast to a matrix and transposed).

            

      
      
      
    

        
Please try r.out.gdal in 90m, then stars::read_stars(..., proxy=TRUE) 
(outside rgrass7). We need to try to get stars to work in rgrass7, but we 
had not gone into that yet. In stars and gdalcubes, the idea is to do work 
on the back end, and only bring data to (say) display in the feasible 
resolution. So I think something like:

library(stars)
big <- read_stars("big.tif", proxy=TRUE)
plot(big)

automates the downsampling. The default arguments to readRAST() assume 
that uncompressed binary is the optimal transfer strategy, and that all 
the data should be in R's workspace.

I can speculate that if GDAL used the GRASS plugin, it might be possibble 
to proxy the actual GRASS CELL or DCELL. However, someone else would need 
to take responsibility for implementing prototypes for that. The same 
applies to support for stars, alternatively for terra in addition (raster 
is used by many, but with limited resources, stars and terra should be 
preferred).

Roger

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Thomas Adams
    

    

      
      #
    

  


  

      
Hi Roger,

I did run with a 90m resolution, with the same error:

            

      
      
      
    

        
Creating BIL support files...
Exporting raster as integer values (bytes=4)
 100%
Warning messages:
1: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
  Discarded ellps unknown in CRS definition: +proj=merc +lat_ts=0 +lon_0=0
+x_0=0 +y_0=0 +R=6378137 +units=m +no_defs
2: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
  Discarded datum unknown in CRS definition

            

      
      
      
    

        
Error in rasterImage(m, bb[1, 1], bb[2, 1], bb[1, 2], bb[2, 2], interpolate
= interpolate,  :
  cannot allocate memory block of size 16777216 Tb

However, with a 250m resolution the import and visualization worked fine:

            

      
      
      
    

        
Creating BIL support files...
Exporting raster as integer values (bytes=4)
 100%
Warning messages:
1: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
  Discarded ellps unknown in CRS definition: +proj=merc +lat_ts=0 +lon_0=0
+x_0=0 +y_0=0 +R=6378137 +units=m +no_defs
2: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
  Discarded datum unknown in CRS definition

            

      
      
      
    

        
Thank you for your help!

Best regards,
Tom

        
On Sun, Jul 12, 2020 at 9:19 AM Roger Bivand <Roger.Bivand at nhh.no> wrote: