Skip to content

decimals in NA value using raster::writeRaster with .ascii format

10 messages · Jonathan Greenberg, Robert J. Hijmans, Javier Garcia-Pintado +2 more

Original
Zack Holden
# 
Dear list,

I am creating ascii files as inputs to the PCraster software (no other
gdal format is allowed). When I write my ascii file using
raster::writeRaster, the first cell of every raster that I write has a
large number of decimal values appended to it. This causes errors
later in the model workflow.

This also happens to the first NA value in the file, so that all NA
values in the file are e.g. -9999 except the first cell. which appears
as: -9999.00000000000

Can anyone suggest a method for ensuring that all of my NA values are
identical and with no trailing decimals?

A small reproducible example follows. Any help or suggestions is appreciated.

Thanks,
Zack

##################
# start code
require(raster)
r <- raster(nrows=2, ncols=2, xmn=0, xmx=2, ymn=0, ymx=2)
r <- setValues(r, seq(1:ncell(r)))
rexpand <- extend(r, c(2,2))
NAvalue(rexpand) <- -9999

temp.dir <- "/home/zholden/"
writeRaster(rexpand, file=paste(temp.dir, "testraster.asc",sep=""),
NAflag=-9999, dataType="INT2U", overwrite=T)
# end code
##################

# opening in a text editor reveals trailing decimals in the first NA value.
system(paste("gedit ", temp.dir, "testraster.asc", sep=""))

################################
 sessionInfo()
R version 3.0.2 (2013-09-25)
Platform: x86_64-redhat-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
LC_MONETARY=en_US.UTF-8
 [6] LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C
               LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] rgdal_0.8-16  raster_2.2-12 sp_1.0-14

loaded via a namespace (and not attached):
[1] grid_3.0.2      lattice_0.20-23 tools_3.0.2
Jonathan Greenberg
# 
Zack:

Are you creating the files from within R, or are you just trying to
convert an existing file?  If the latter, can you make one of those
files available (e.g. via google drive or dropbox or something like
that)?

--j
On Thu, Apr 10, 2014 at 3:06 PM, Zack Holden <zaholdenfs at gmail.com> wrote:

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Zack Holden
    

    

      
      #
    

  


  

      
Thanks for getting back to me Jonathan.

In reality, I am creating the files from an existing DEM. However, I
get the same behavior regardless of whether I generate the file in R
(as with my example code) or with actual data.
I have since tested on 4 machines. One windows machine with an older
version of R successfully generates the desired ascii file without the
decimals. The other 3 computers (Linux and windows machines) add the
decimals. As a workaround, I've tried using -Inf as my NAflag. This
eliminates the decimals, but results in a large string of white space
in front of the first row of data. These still cause problems for
PCraster (the next step in the processing chain).

I'm still stumped.

Thanks again,
Zack

        
On Thu, Apr 10, 2014 at 4:22 PM, Jonathan Greenberg <jgrn at illinois.edu> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Jonathan Greenberg
    

    

      
      #
    

  


  

      
Zack:

Hopefully Robert will get back to you (I confirmed it -- it looks like
a bug to me) -- but in the meantime, you might want to try my
gdalUtils package, which uses gdal_translate to convert files.  You
can use one of the many ASCII formats -- you'll want to use parameters
like:
gdal_translate(...,a_nodata=-9999,ot="UInt16",format="...")

You can see the available formats at: http://www.gdal.org/formats_list.html

--j



--

        
On Thu, Apr 10, 2014 at 5:29 PM, Zack Holden <zaholdenfs at gmail.com> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Robert J. Hijmans
    

    

      
      #
    

  


  

      
Zack,

You are using:     dataType="INT2U"

but it should be:     datatype="INT2U"  # (or any other INT data type)

That fixes it. If the datatype is FLT* (the default), the first value
gets decimals to assure that GDAL and others treat the values as
floats, not integers.

Robert

        
On Thu, Apr 10, 2014 at 3:36 PM, Jonathan Greenberg <jgrn at illinois.edu> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Javier Garcia-Pintado
    

    

      
      #
    

  


  

      
Note you also have a 'PCRaster' driver in GDAL.

Javier
---

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Jonathan Greenberg
    

    

      
      #
    

  


  

      
Robert:

What is the practical difference between those two? -- does dataType
get passed off to GDAL, while datatype is interpreted by your R code?

--j

        
On Thu, Apr 10, 2014 at 7:06 PM, Robert J. Hijmans <r.hijmans at gmail.com> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Zack Holden
    

    

      
      #
    

  


  

      
Robert,
Thank you for your help. I failed to notice the "datatype vs. dataType
difference. Calling 'datatype' does indeed solve my integer test case.
However, some of the raster datasets that I am processing are actually
floating point. When I call writeRaster(datatype = 'FLT4S'), I still
see the long string of decimals in the first value in my ascii grid.

Is there any way to handle this in R? From your response, it sounds
like this might be an artifact of GDAL, and I may need to manually (or
with text manipulations) edit out the decimal strings.

Thanks for any additional suggestions.

Javier, thanks for pointing out the available GDAL drivers in
PCraster, I'm just getting acquainted with that software program.

Cheers,
Zack

        
On Fri, Apr 11, 2014 at 8:48 AM, Jonathan Greenberg <jgrn at illinois.edu> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
  


  


      

    

    
2 days later

    

      
      

        


  

        

      Edzer Pebesma
    

    

      
      #
    

  


  

      
On 04/11/2014 05:46 PM, Zack Holden wrote:

            

      
      
      
    

        
Look for

https://r-forge.r-project.org/scm/viewvc.php/pkg/raster/R/writeAllAscii.R?view=markup&root=raster

and lower the 15, or else change the formatC call, then rebuild raster.

As described earlier, raster does this on purpose: the gdal asciigrid
reader makes the wrong assumption about the data type when a grid starts
with too many missing values that look like an integer (like -9999 but
unlike -9999.0). This has been reported over a decade ago, and may be
seen as an indication of gdal's liveliness. Not only OpenSSL suffers
from too few eyes to make bugs shallow.

What software are you using that can't digest the 15-digit float?

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Zack Holden
    

    

      
      #
    

  


  

      
Hi Edzer,
Thank you for your helpful comments. I am generating ascii files as
inputs to PCraster. I realize now that when producing grids for other
software, I need to be more careful about assigning data type and NA
values to my outputs when I write them. When my work is all done in R,
this hasn't ever been a problem. Using:
writeRaster(file="myraster.asc", datatype="FLT4S", NAval=-Inf )
PCraster reads them correctly.

Thanks all for the helpful suggestions.

Best,
Zack


On Mon, Apr 14, 2014 at 6:58 AM, Edzer Pebesma

        
<edzer.pebesma at uni-muenster.de> wrote: