Skip to content

Subscripting problem with is.na()

14 messages · Ivan Calandra, G.Maubach at gmx.de, PIKAL Petr +4 more

Ivan Calandra
# 
My statement "Using a single bracket '[' on a data.frame does the same 
as for matrices: you need to specify rows and columns" was not correct.


When you use a single bracket on a list with only one argument in 
between, then R extracts "elements", i.e. columns in the case of a 
data.frame. This explains your errors.

But it is possible to use a single bracket on a data.frame with 2 
arguments (rows, columns) separated by a comma, as with matrices. This 
is the solution you received.

Ivan


--
Ivan Calandra, PhD
Scientific Mediator
University of Reims Champagne-Ardenne
GEGENAA - EA 3795
CREA - 2 esplanade Roland Garros
51100 Reims, France
+33(0)3 26 77 36 89
ivan.calandra at univ-reims.fr
--
https://www.researchgate.net/profile/Ivan_Calandra
https://publons.com/author/705639/

Le 23/06/2016 ? 16:27, Ivan Calandra a ?crit :
Bert Gunter
# 
Sorry, Ivan, your statement is incorrect:

"When you use a single bracket on a list with only one argument in
between, then R extracts "elements", i.e. columns in the case of a
data.frame. This explains your errors. "

e.g.
[1] FALSE
[1] "data.frame"
[1] "integer"

Compare:
[1] TRUE

Why? Single bracket extraction on a list results in a list; double
bracket extraction results in the element of the list ( a "column" in
the case of a data frame, which is a specific kind of list). The
relevant sections of ?Extract are:

"Indexing by [ is similar to atomic vectors and selects a **list** of
the specified element(s).

Both [[ and $ select a **single element of the list**. "


Hope this clarifies this often-confused issue.


Cheers,
Bert
Bert Gunter

"The trouble with having an open mind is that people keep coming along
and sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )


On Thu, Jun 23, 2016 at 7:34 AM, Ivan Calandra
<ivan.calandra at univ-reims.fr> wrote:
Ivan Calandra
# 
Thank you Bert for this clarification. It is indeed an important point.

Ivan

--
Ivan Calandra, PhD
Scientific Mediator
University of Reims Champagne-Ardenne
GEGENAA - EA 3795
CREA - 2 esplanade Roland Garros
51100 Reims, France
+33(0)3 26 77 36 89
ivan.calandra at univ-reims.fr
--
https://www.researchgate.net/profile/Ivan_Calandra
https://publons.com/author/705639/

Le 23/06/2016 ? 17:06, Bert Gunter a ?crit :
David L Carlson
# 
The function is.na() returns a matrix when applied to a data.frame so you can easily convert all the NAs to 0's:
var1 var2
1     1    1
2     2    2
3     3    3
4    NA   NA
5     5    5
6     6    6
7     7    7
8    NA   NA
9     9    9
10   10   10
var1  var2
 [1,] FALSE FALSE
 [2,] FALSE FALSE
 [3,] FALSE FALSE
 [4,]  TRUE  TRUE
 [5,] FALSE FALSE
 [6,] FALSE FALSE
 [7,] FALSE FALSE
 [8,]  TRUE  TRUE
 [9,] FALSE FALSE
[10,] FALSE FALSE
var1 var2
1     1    1
2     2    2
3     3    3
4     0    0
5     5    5
6     6    6
7     7    7
8     0    0
9     9    9
10   10   10

-------------------------------------
David L Carlson
Department of Anthropology
Texas A&M University
College Station, TX 77840-4352

-----Original Message-----
From: R-help [mailto:r-help-bounces at r-project.org] On Behalf Of Ivan Calandra
Sent: Thursday, June 23, 2016 10:14 AM
To: R Help
Subject: Re: [R] Subscripting problem with is.na()

Thank you Bert for this clarification. It is indeed an important point.

Ivan

--
Ivan Calandra, PhD
Scientific Mediator
University of Reims Champagne-Ardenne
GEGENAA - EA 3795
CREA - 2 esplanade Roland Garros
51100 Reims, France
+33(0)3 26 77 36 89
ivan.calandra at univ-reims.fr
--
https://www.researchgate.net/profile/Ivan_Calandra
https://publons.com/author/705639/

Le 23/06/2016 ? 17:06, Bert Gunter a ?crit :
______________________________________________
R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Bert Gunter
# 
Not in general, David:

e.g.
a     b    c
[1,] FALSE FALSE TRUE
[2,]  TRUE FALSE TRUE
[3,] FALSE  TRUE TRUE
[1] NA NA NA NA NA
Warning message:
In `[<-.factor`(`*tmp*`, thisvar, value = 0) :
  invalid factor level, NA generated
a    b c
1 1    A 0
2 0    b 0
3 2 <NA> 0


The problem is the default conversion to factors and the replacement
operation for factors. So:
[1] "AsIs"  ## so NOT a factor
a b c
1 1 A 0
2 0 b 0
3 2 0 0

Of course the OP (and you) probably had a data frame of all numerics
in mind, so the problem doesn't arise. But I think one needs to make
the distinction and issue clear.

Cheers,
Bert





Bert Gunter

"The trouble with having an open mind is that people keep coming along
and sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
On Thu, Jun 23, 2016 at 8:46 AM, David L Carlson <dcarlson at tamu.edu> wrote:
David L Carlson
# 
Good point. I did not think about factors. Also your example raises another issue since column c is logical, but gets silently converted to numeric. This would seem to get the job done assuming the conversion is intended for numeric columns only:
a         b         c 
"numeric"  "factor" "logical"
a    b  c
1 1    A NA
2 0    b NA
3 2 <NA> NA

David C

-----Original Message-----
From: Bert Gunter [mailto:bgunter.4567 at gmail.com] 
Sent: Thursday, June 23, 2016 1:48 PM
To: David L Carlson
Cc: Ivan Calandra; R Help
Subject: Re: [R] Subscripting problem with is.na()

Not in general, David:

e.g.
a     b    c
[1,] FALSE FALSE TRUE
[2,]  TRUE FALSE TRUE
[3,] FALSE  TRUE TRUE
[1] NA NA NA NA NA
Warning message:
In `[<-.factor`(`*tmp*`, thisvar, value = 0) :
  invalid factor level, NA generated
a    b c
1 1    A 0
2 0    b 0
3 2 <NA> 0


The problem is the default conversion to factors and the replacement
operation for factors. So:
[1] "AsIs"  ## so NOT a factor
a b c
1 1 A 0
2 0 b 0
3 2 0 0

Of course the OP (and you) probably had a data frame of all numerics
in mind, so the problem doesn't arise. But I think one needs to make
the distinction and issue clear.

Cheers,
Bert





Bert Gunter

"The trouble with having an open mind is that people keep coming along
and sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
On Thu, Jun 23, 2016 at 8:46 AM, David L Carlson <dcarlson at tamu.edu> wrote:
Bert Gunter
# 
... actually, FWIW, I would say that this little discussion mostly
demonstrates why the OP's request is probably not a good idea in the
first place. Usually, NA's should be left as NA's to be dealt with
properly by R and packages. In biological measurements, for example,
NA's often mean "below the ability to reliably measure." Biologists
with whom I've worked over many years often want to convert these to 0
or omit the cases, both of which lead to biased estimates and/or
underestimates of variability and excess claims of "statistical
significance" (for those who belong to this religious persuasion). One
should never say never, but I suspect that there are relatively few
circumstances where the conversion the OP requested is actually wise.

Feel free to ignore/reject such extraneous comments of course.

Cheers,
Bert


Bert Gunter

"The trouble with having an open mind is that people keep coming along
and sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
On Thu, Jun 23, 2016 at 12:14 PM, David L Carlson <dcarlson at tamu.edu> wrote:
G.Maubach at gmx.de
# 
Hi Bert,

many thanks for all your help and your comments. I learn at lot this way.

My question was about is.na() at the first sight but the actual task looks like this:

I have two variables in my customer data that signal if the customer accout was closed by master data management or by sales. Say these variables are closed_mdm and closed_sls. They contain NA if the customer account is still open or a closing code from "01" to "08" if the customer account was closed and why.

For my analysis I need a variable that combines the two variables closed_mdm and closed_sls to set a filter easily on those who are closed not matter what the reason was nor who closed the account.

As I always encounter problems when dealing with ifelse statements and NA I decided to merge these two variables to one variable containing 0 = not closed and 1 = closed. In my context this seems to be - at least to me - a reasonable approach.

Replacement of missing values and merging the variables is the easiest way for me.

-- cut --

cust_id <- c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20)
closed_mdm <- c("01", NA, NA, NA, "08", "07", NA, NA, "05", NA, NA, NA, "04", NA, NA, NA, NA, NA, NA, NA)
closed_sls <- c(NA, "08", NA, NA, "08", "07", NA, NA, NA, NA, "03", NA, NA, NA, "05", NA, NA, NA, NA, NA)

# 1st try
ds_temp1 <- data.frame(cust_id, closed_mdm, closed_sls)
ds_temp1

ds_temp1$closed <- closed_mdm | closed_sls  # WRONG

# 2nd try
closed_mdm_fac1 <- as.factor(closed_mdm)
closed_sls_fac1 <- as.factor(closed_sls)

ds_temp2 <- data.frame(cust_id, closed_mdm_fac1, closed_sls_fac1)
ds_temp2

ds_temp2$closed <- ds_temp$closed_mdm_fac1 | ds_temp$closed_sls_fac1  # WRONG

# 3rd try
closed_mdm_num1 <- as.numeric(closed_mdm)  # OK
closed_sls_num1 <- as.numeric(closed_sls)  # OK

ds_temp3 <- data.frame(cust_id, closed_mdm_num1, closed_sls_num1)
ds_temp3

ds_temp3$closed <- ds_temp$closed_mdm_num1 | ds_temp$closed_sls_num1  # WRONG

# 4th try
ds_temp4 <- ds_temp3
ds_temp4

# Does not run due to not allowed NA in subscripts
ds_temp4[is.na(ds_temp4$closed_mdm_num1), ds_temp4$closed_mdm_num1] <- 0
ds_temp4[is.na(ds_temp4$closed_sls_num1), ds_temp4$closed_sls_num1] <- 0

# 5th try
ds_temp4$closed_mdm_num1 <- ifelse(is.na(ds_temp4$closed_mdm_num1), 1, 0)
ds_temp4$closed_sls_num1 <- ifelse(is.na(ds_temp4$closed_sls_num1), 1, 0)
ds_temp4

ds_temp4$closed <- ifelse(ds_temp4$closed_mdm_num1 == 1 | ds_temp4$closed_sls_num1 == 1, 1, 0)
ds_temp4

-- cut --

Is there a better way to do it?

Kind regards

Georg
PIKAL Petr
# 
Hi

I do not consider changing NA to 0 as a reasonable approach ( maybe only in some special case, which is not yours).

rowSums(is.na(ds_temp1[,2:3]))
 [1] 1 1 2 2 0 0 2 2 1 2 1 2 1 2 1 2 2 2 2 2

gives you vector of numbers which is equal 2 only if they are both NA. So

ds_temp1$open <- rowSums(is.na(ds_temp1[,2:3]))==2

gives you column which is TRUE if the account is open and FALSE in other situation.

You can use similar approach for testing the state of account closing.

Cheers
Petr
________________________________
Tento e-mail a jak?koliv k n?mu p?ipojen? dokumenty jsou d?v?rn? a jsou ur?eny pouze jeho adres?t?m.
Jestli?e jste obdr?el(a) tento e-mail omylem, informujte laskav? neprodlen? jeho odes?latele. Obsah tohoto emailu i s p??lohami a jeho kopie vyma?te ze sv?ho syst?mu.
Nejste-li zam??len?m adres?tem tohoto emailu, nejste opr?vn?ni tento email jakkoliv u??vat, roz?i?ovat, kop?rovat ?i zve?ej?ovat.
Odes?latel e-mailu neodpov?d? za eventu?ln? ?kodu zp?sobenou modifikacemi ?i zpo?d?n?m p?enosu e-mailu.

V p??pad?, ?e je tento e-mail sou??st? obchodn?ho jedn?n?:
- vyhrazuje si odes?latel pr?vo ukon?it kdykoliv jedn?n? o uzav?en? smlouvy, a to z jak?hokoliv d?vodu i bez uveden? d?vodu.
- a obsahuje-li nab?dku, je adres?t opr?vn?n nab?dku bezodkladn? p?ijmout; Odes?latel tohoto e-mailu (nab?dky) vylu?uje p?ijet? nab?dky ze strany p??jemce s dodatkem ?i odchylkou.
- trv? odes?latel na tom, ?e p??slu?n? smlouva je uzav?ena teprve v?slovn?m dosa?en?m shody na v?ech jej?ch n?le?itostech.
- odes?latel tohoto emailu informuje, ?e nen? opr?vn?n uzav?rat za spole?nost ??dn? smlouvy s v?jimkou p??pad?, kdy k tomu byl p?semn? zmocn?n nebo p?semn? pov??en a takov? pov??en? nebo pln? moc byly adres?tovi tohoto emailu p??padn? osob?, kterou adres?t zastupuje, p?edlo?eny nebo jejich existence je adres?tovi ?i osob? j?m zastoupen? zn?m?.

This e-mail and any documents attached to it may be confidential and are intended only for its intended recipients.
If you received this e-mail by mistake, please immediately inform its sender. Delete the contents of this e-mail with all attachments and its copies from your system.
If you are not the intended recipient of this e-mail, you are not authorized to use, disseminate, copy or disclose this e-mail in any manner.
The sender of this e-mail shall not be liable for any possible damage caused by modifications of the e-mail or by delay with transfer of the email.

In case that this e-mail forms part of business dealings:
- the sender reserves the right to end negotiations about entering into a contract in any time, for any reason, and without stating any reasoning.
- if the e-mail contains an offer, the recipient is entitled to immediately accept such offer; The sender of this e-mail (offer) excludes any acceptance of the offer on the part of the recipient containing any amendment or variation.
- the sender insists on that the respective contract is concluded only upon an express mutual agreement on all its aspects.
- the sender of this e-mail informs that he/she is not authorized to enter into any contracts on behalf of the company except for cases in which he/she is expressly authorized to do so in writing, and such authorization or power of attorney is submitted to the recipient or the person represented by the recipient, or the existence of such authorization is known to the recipient of the person represented by the recipient.
David L Carlson
# 
Yes, measurements below detection should be treated differently. I thought about the missing data issue, but there is another context in which spreadsheet data containing count data where 0 entries are deliberately left blank for readability or economy. In that case it is easier to import and use R to replace the missing 0s than to fill the missing cell entries in the spreadsheet before importing it.

David C

-----Original Message-----
From: Bert Gunter [mailto:bgunter.4567 at gmail.com] 
Sent: Thursday, June 23, 2016 4:56 PM
To: David L Carlson
Cc: Ivan Calandra; R Help
Subject: Re: [R] Subscripting problem with is.na()

... actually, FWIW, I would say that this little discussion mostly
demonstrates why the OP's request is probably not a good idea in the
first place. Usually, NA's should be left as NA's to be dealt with
properly by R and packages. In biological measurements, for example,
NA's often mean "below the ability to reliably measure." Biologists
with whom I've worked over many years often want to convert these to 0
or omit the cases, both of which lead to biased estimates and/or
underestimates of variability and excess claims of "statistical
significance" (for those who belong to this religious persuasion). One
should never say never, but I suspect that there are relatively few
circumstances where the conversion the OP requested is actually wise.

Feel free to ignore/reject such extraneous comments of course.

Cheers,
Bert


Bert Gunter

"The trouble with having an open mind is that people keep coming along
and sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
On Thu, Jun 23, 2016 at 12:14 PM, David L Carlson <dcarlson at tamu.edu> wrote:
MacQueen, Don
# 
See insert below.
Bert Gunter
# 
As Petr and Don have shown you, changing NA to 0 is unnecessary to get
what you want. However, recoding to 0 may be OK, as NA has a specific
meaning in this context, and you are just adding an extra code to a
factor for a different level.

But it still might cause you trouble later. One of R's strengths is
it's ability to simply deal with NA's -- most of the time anyway .For
example note that you would have to make sure these columns are
factors (*not numerics*), if you wanted to, say, investigate how
category of closing related to other covariates via e.g. multinomial
logistic regression or even just to tabulate the "closed" categories.
Keeping NA as NA allows R's built-in facilities to simply handle (e.g.
omit) the data for the "still open" cases, but you will have to do it
explicitly yourself if you code to 0. That seems to be asking for
trouble to me.

As always, contrary views welcome. This discussion still seems on
(r-help) topic to me, but if not, please say so.

Cheers,
Bert

Bert Gunter

"The trouble with having an open mind is that people keep coming along
and sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
On Fri, Jun 24, 2016 at 12:14 AM, <G.Maubach at gmx.de> wrote:
William Dunlap
# 
Is part of the issue that in common parlance "NA" or "N/A" may
mean  either "not available" or "not applicable" (e.g., isPregnant
for a male) but in R NA means only "not available"?

Bill Dunlap
TIBCO Software
wdunlap tibco.com
On Fri, Jun 24, 2016 at 8:37 AM, Bert Gunter <bgunter.4567 at gmail.com> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        

      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Bert Gunter
    

    

      
      #
    

  


  

      
I would tend to agree. But NA is still preferable for both, no?

-- Bert
Bert Gunter

"The trouble with having an open mind is that people keep coming along
and sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )

        
On Fri, Jun 24, 2016 at 8:42 AM, William Dunlap <wdunlap at tibco.com> wrote: