vctrs: a type system for the tidyverse
On Thu, Aug 9, 2018 at 3:57 AM Joris Meys <jorismeys at gmail.com> wrote:
I sent this to I?aki personally by mistake. Thank you for notifying me. On Wed, Aug 8, 2018 at 7:53 PM I?aki ?car <i.ucar86 at gmail.com> wrote:
For what it's worth, I always thought about factors as fundamentally characters, but with restrictions: a subspace of all possible strings. And I'd say that a non-negligible number of R users may think about them in a similar way.
That idea has been a common source of bugs and the most important reason why I always explain my students that factors are a special kind of numeric(integer), not character. Especially people coming from SPSS see immediately the link with categorical variables in that way, and understand that a factor is a modeling aid rather than an alternative for characters. It is a categorical variable and a more readable way of representing a set of dummy variables. I do agree that some of the factor behaviour is confusing at best, but that doesn't change the appropriate use and meaning of factors as categorical variables. Even more, I oppose the ideas that : 1) factors with different levels should be concatenated. 2) when combining factors, the union of the levels would somehow be a good choice. Factors with different levels are variables with different information, not more or less information. If one factor codes low and high and another codes low, mid and high, you can't say whether mid in one factor would be low or high in the first one. The second has a higher resolution, and that's exactly the reason why they should NOT be combined. Different levels indicate a different grouping, and hence that data should never be used as one set of dummy variables in any model. Even when combining factors, the union of levels only makes sense to me if there's no overlap between levels of both factors. In all other cases, a researcher will need to determine whether levels with the same label do mean the same thing in both factors, and that's not guaranteed. And when we're talking a factor with a higher resolution and a lower resolution, the correct thing to do modelwise is to recode one of the factors so they have the same resolution and every level the same definition before you merge that data. So imho the combination of two factors with different levels (or even levels in a different order) should give an error. Which R currently doesn't throw, so I get there's room for improvement.
I 100% agree with you, and is this the behaviour that vctrs used to have and dplyr currently has (at least in bind_rows()). But pragmatically, my experience with dplyr is that people find this behaviour confusing and unhelpful. And when I played the full expression of this behaviour in vctrs, I found that it forced me to think about the levels of factors more than I'd otherwise like to: it made me think like a programmer, not like a data analyst. So in an ideal world, yes, I think factors would have stricter behaviour, but my sense is that imposing this strictness now will be onerous to most analysts. Hadley