Skip to content

Is ALTREP "non-API"?

21 messages · hcorti@@71 m@iii@g oii gm@ii@com, Hiroaki Yutani, Gabriel Becker +6 more

Original
Hiroaki Yutani
# 
Writing R Extension[1] defines "API" as:

    Entry points which are documented in this manual and declared in an
installed header file. These can be used in distributed packages and will
only be changed after deprecation.

But, the document (WRE) doesn't have even a single mention of ALTREP, the
term "ALTREP" itself or any entry points related to ALTREP. Does this mean,
despite the widespread use of it on R packages including CRAN ones, ALTREP
is not the API and accordingly using it in distributed packages is
considered illegal?

Best,
Yutani

[1]: https://cran.r-project.org/doc/manuals/r-release/R-exts.html#The-R-API
hcorti@@71 m@iii@g oii gm@ii@com
# 
Hello, I don't believe it is illegal, as ALTREP "implements an abstraction
underneath the C API". And is "compatible with all code which uses the API".

Please see slide deck by Gabriel Becker,  with L Tierney, M Lawrence and T
Kalibera.

https://bioconductor.org/help/course-materials/2020/BiocDevelForum/16-ALTREP
.pdf

ALTREP framework implements an abstraction underneath traditional R C API
- Generalizes whats underneath the API
- Without changing how data are accessed
- Compatible with all C code which uses the API
- Compatible with R internals


I hope this helps,
Hernando


-----Original Message-----
From: R-devel <r-devel-bounces at r-project.org> On Behalf Of Hiroaki Yutani
Sent: Sunday, April 21, 2024 8:48 PM
To: r-devel <r-devel at r-project.org>
Subject: [Rd] Is ALTREP "non-API"?

Writing R Extension[1] defines "API" as:

    Entry points which are documented in this manual and declared in an
installed header file. These can be used in distributed packages and will
only be changed after deprecation.

But, the document (WRE) doesn't have even a single mention of ALTREP, the
term "ALTREP" itself or any entry points related to ALTREP. Does this mean,
despite the widespread use of it on R packages including CRAN ones, ALTREP
is not the API and accordingly using it in distributed packages is
considered illegal?

Best,
Yutani

[1]: https://cran.r-project.org/doc/manuals/r-release/R-exts.html#The-R-API


______________________________________________
R-devel at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Hiroaki Yutani
# 
Thanks, Hernando,

Sorry, "API" is a bit confusing term in this context, but what I want to
discuss is the "API" that Writing R Extension defines as quoted in my
previous email. It's probably different from an ordinary sense when we
casually say "R C API".

You might wonder why I care about such a difference. This is because
calling a "non-API" is considered a violation of CRAN repository policy,
which means CRAN will kick out the R package. I know many CRAN packages use
ALTREP, but just being accepted by CRAN at the moment doesn't mean CRAN
will keep accepting it. So, I want to clarify the current status of ALTREP.

Best,
Yutani

2024?4?22?(?) 10:17 <hcortina71 at gmail.com>:

  
  


  


      

    

    

      
      

        


  

        

      Gabriel Becker
    

    

      
      #
    

  


  

      
Hi Yutani,

ALTREP is part of the official R api, as illustrated by the presence of
src/include/R_ext/Altrep.h. Everything declared in the header files in that
directory is official API AFAIK (and I believe that is more definitive than
the manuals).

The documentation of ALTREP has lagged behind its implementation
unfortunately, which may partially my fault for not submitting doc
patches for it against the manuals. Sorry for my contribution to that, I'll
see if I can loop back around to contributing documentation for ALTREP.

Best,
~G

        
On Sun, Apr 21, 2024 at 6:36?PM Hiroaki Yutani <yutani.ini at gmail.com> wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Hiroaki Yutani
    

    

      
      #
    

  


  

      
Thanks for your convincing comment, but it seems the R core team has a
different opinion...
A few hours ago, src/include/R_ext/Altrep.h got this comment:

    /*
       Not part of the API, subject to change at any time.
    */

commit:
https://github.com/r-devel/r-svn/commit/2059bffde642f8426d1f39ab5dd995d19a575d4d

While I'm glad to see their attempt to make it clear, I'm confused. That
commit marks many other files as "not API," but I think it's a bit
inconsistent with what Writing R Extension says.

For example, src/include/R_ext/Parse.h got a comment "So not API," but the
entry point R_ParseVector is explained in Writing R Extension[1]. So, I
believe it's clearly an "API" both in the sense of WRE's dialect and in an
ordinary sense. Which should I believe? WRE? The source code?

It might be just a coincidence, but I'm sorry if my question drove the R
core team to such a too-quick clarification. I just wanted to discuss how
to fix the current inconsistencies.

I think the R core needs a proper definition of "API" first. In my opinion,
it makes little sense to call it "non-API" just to show the possibility of
future breaking changes. Whether you call it API or non-API, clever users
will still accept the breaking changes on it if it's reasonable. For
example, how about "experimental API" or "unstable API"? They sound better
to me.

Best,
Yutani

[1]:
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Parsing-R-code-from-C


2024?4?22?(?) 16:37 Gabriel Becker <gabembecker at gmail.com>:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Gabriel Becker
    

    

      
      #
    

  


  

      
Hi Yutani,

The headers have been updated by Luke Tierney: ALTREP is an *experimental*
API, in that it is an official API that is legal for packages to use, but
may it change with short notice as the framework is further developed.

Hope that helps,
~G

        
On Mon, Apr 22, 2024 at 4:46?AM Hiroaki Yutani <yutani.ini at gmail.com> wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

          

      
      
      
    

        
That is not true at all - the presence of header does not constitute declaration of something as the R API. There are cases where internal functions are in the headers for historical or other reasons since the headers are used both for the internal implementation and packages. That's why this is in R-exts under "The R API: entry points for C code":

            

      
      
      
    

        
And that's why CRAN does not allow unstable ones = those not documented in R-exts as part of the API.

Therefore Hiroaki's question is a very good one. ALTREP is declared as experimental and is not part of the API, but the development and stability of the API in some sense should get better as more packages are using it. Therefore it is currently allowed on CRAN in the hope that it will transition to stable at some point, but package authors using it must be willing to adapt to changes to the API as necessary.

Cheers,
Simon

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Hadley Wickham
    

    

      
      #
    

  


  

      
On Mon, Apr 22, 2024 at 5:14?PM Simon Urbanek <simon.urbanek at r-project.org>
wrote:

            

      
      
      
    

        
If I understand your point correctly, does this mean that Rf_allocVector()
is not part of the "official" R API? It does not appear to be documented in
the "The R API: entry points for C code" section.

Hadley

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

          

      
      
      
    

        
It does, obviously:
https://cran.r-project.org/doc/manuals/R-exts.html#Allocating-storage-1

Cheers,
Simon

  
  


  


      

    

    

      
      

        


  

        

      Hiroaki Yutani
    

    

      
      #
    

  


  

      
I just saw the recent commits about the "experimental" entry points. So, my
original question about the current status of ALTREP is now resolved. I'm
glad that ALTREP is confirmed usable on CRAN (with care). Thank you for all
your help!

I think other "non-API"s still need clarification. For example, this one I
picked in my previous email.

            

      
      
      
    

        
the entry point R_ParseVector is explained in Writing R Extension[1]. So, I
believe it's clearly an "API" both in the sense of WRE's dialect and in an
ordinary sense. Which should I believe? WRE? The source code?

But, in my understanding, R is now in the process of clearing such
ambiguities. So, I can just wait.

Lastly, I want the R core to consider marking ALTREP as stable, or the
"API". I didn't actively follow the development of ALTREP, but I think the
ALTREP entry points have been there for half a decade without any major
breaking changes. So, in my opinion, it's safe to declare it's stable.

Best,
Yutani


2024?4?23?(?) 7:14 Simon Urbanek <simon.urbanek at r-project.org>:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Hadley Wickham
    

    

      
      #
    

  


  

          

      
      
      
    

        
I'm just trying to understand the precise definition of the official API
here. So it's any function mentioned in R-exts, regardless of which section
it appears in?

Does this sentence imply that all functions starting with alloc* are part
of the official API?

            

      
      
      
    

        
those. There are quite a

            

      
      
      
    

        
them.

Generally, things in a file with "internal" in its name are internal.

Hadley

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

          

      
      
      
    

        
Again, I can only quote the R-exts (few lines below the previous "The R API" quote):


We can classify the entry points as
API
Entry points which are documented in this manual and declared in an installed header file. These can be used in distributed packages and will only be changed after deprecation.


It says "in this manual" - I don't see anywhere restriction on a particular section of the manual, so I really don't see why you would think that allocation is not part on the API.

Cheers,
Simon

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Hadley Wickham
    

    

      
      #
    

  


  

          

      
      
      
    

        
Because you mentioned that section explicitly earlier in the thread. This
obviously seems clear to you, but it's not at all clear to me and I suspect
many of the wider community. It's frustrating because we are trying
our best to do what y'all want us to do, but it feels like we keep getting
the rug pulled out from under us with very little notice, and then have to
spend a large amount of time figuring out workarounds. That is at least
feasible for my team since we have multiple talented folks who are paid
full-time to work on R, but it's a huge struggle for most people who are
generally maintaining packages in their spare time.

For the purposes of this discussion could you please "documented in the
manual" means? For example, this line mentions allocXxx functions: "There
are quite a few allocXxx functions defined in Rinternals.h?you may want to
explore them.". Does that imply that they are documented and free to use?

And in general, I'd urge R Core to make an explicit list of functions that
you consider to be part of the exported API, and then grandfather in
packages that used those functions prior to learning that we weren't
supposed to.

Hadley

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Hiroaki Yutani
    

    

      
      #
    

  


  

          

      
      
      
    

        
you consider to be part of the exported API

While I believe R Core is in the process of such clarification, I'd also
vote for this. Now that WRE has "experimental" category, and if we take the
current definition of "documented in the manual" literally, an
"experimental" entry point cannot be documented because the entry point
would promote to an "API" for the obvious reason. It would sound funny that
you cannot write precautionary statements on experimental entry points just
because of the very definition of "experimental". So, I agree R should have
the explicit list.

I'd add that R should also define a process on how to stabilize an
"experimental" or "public" entry point into an "API". For example, Rust
language has such a process [1]. After a feature is introduced as unstable,
a "tracking issue" is filed and the related problems are reported or linked
to it. Users can know what are the remaining problems before getting
stabilized, and, if they have strong will, they can contribute to resolving
such blockers. Similarly, if we can track the unresolved problems of each
non-API, we might be able to help the R core team more smoothly.

Best,
Yutani

[1]: https://rustc-dev-guide.rust-lang.org/stabilization_guide.html


2024?4?24?(?) 21:55 Hadley Wickham <h.wickham at gmail.com>:

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Hadley Wickham
    

    

      
      #
    

  


  

      
A few more thoughts based on a simple question: how do you determine the
length of a vector?

Rf_length() is used in example code in R-exts, but I don't think it's
formally documented anywhere (although it's possible I missed it). Is using
in an example sufficient to consider a function to be part of the public
API? If so, SET_TYPEOF() is used in a number of examples, and hence used by
CRAN packages, but is no longer considered part of the public API.

Rf_xlength() doesn't appear to be mentioned anywhere in R-exts. Does this
imply that long vectors are not part of the exported API? Or is there some
other way we should be determining the length of such vectors?

Are the macro variants LENGTH and XLENGTH part of the exported API? Are we
supposed to use them or avoid them?

Relatedly, I presume that LOGICAL() is the way we're supposed to extract
logical values from a vector, but it isn't documented in R-exts, suggesting
that it's not part of the public API?

---

It's also worth pointing out where R-exts does well, with the documentation
of utility functions (
https://cran.r-project.org/doc/manuals/R-exts.html#Utility-functions). I
think this is what most people would consider documentation to imply, i.e.
a list of input arguments/types, the output type, and basic notes on their
operation.
---

Finally, it's worth noting that there's some lingering ill feelings over
how the connections API was treated. It was documented in R-exts only to be
later removed, including expunging mentions of it in the news. That's
obviously water under the bridge, but I do believe that there is
the potential for the R core team to build goodwill with the community if
they are willing to engage a bit more with the users of their APIs.

Hadley

  
  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

          

      
      
      
    

        
I must be missing something here since I have no idea what you are talking about. The whole point if a stable API is that no rugs are pulled, so in fact it's exactly the opposite of what you claim - the notice is at least a year due to the release cycle, typically more. Unlike many other languages and ecosystems, R public API does not change very often - and R-core is thinking hard about making breaking changes if at all. In fact, I hear more complaints that the API does NOT change and we are too conservative, precisely because we want to avoid unnecessary breakage.

I will not further comment here - all I did was to point out the relevant text from R-exts which is the canonical source of information. If you have issues, find some parts unclear and want to improve the documentation, I would like to invite you to contribute constructively, propose changes, submit patches. The R-exts document has been around for decades, so it seem implausible that all of sudden it is being misunderstood the way you portrayed it, but it is certainly a good idea to improve documentation so contributions are welcome.

Cheers,
Simon

  
  


  


      

    

    

      
      

        


  

        

      iuke-tier@ey m@iii@g oii uiow@@edu
    

    

      
      #
    

  


  

      
On Wed, 24 Apr 2024, Hadley Wickham wrote:

        

            

      
      
      
    

        
Please try to keep this discussion non-adversarial.

            

      
      
      
    

        
As you well know, almost all R-core members are also trying to
maintain and improve R in their spare time. Good for folks to keep in
mind before demanding R-core do X, Y, or Z for you.

            

      
      
      
    

        
Where we are now in terms of what package authors can use to write R
extensions has evolved organically over many years. The current state
is certainly not ideal:

     There are entry points in installed headers that might be
     available;

     but to find out if they are in fact available requires reading
     prose text in the header files and in WRE.

Trying to fine-tune wording in WRE, or add a lot of additional entries
is not really a good or realistic way forward: WRE is both
documentation and tutorial and more legalistic language/more complete
coverage would make it less readable and still not guarantee
completeness or clarity.

We would be better off (in my view, not necessarily shared by others
in R-core) if we could get to a point where:

     all entry points listed in installed header files can be used in
     packages, at least with some caveats;

     the caveats are expressed in a standard way that is searchable,
     e.g. with a standardized comment syntax at the header file or
     individual declaration level.

In principle this is achievable, but getting there from where we are
now is a lot of work. There are some 500 entry points in the R shared
library that are in the installed headers but not mentioned in WRE.
These would need to be reviewed and adjusted. My guess is about a
third are fine and intended to be API-stable, another third are not
used in packages and don't need to be in public headers. The remainder
are things that may be used in current packages but really should not
be, for example because they expose internal data in ways that can
cause segfaults or they make it difficult to implement performance
improvements in the base engine. Sorting through these and working
with package authors to find alternate, safer options takes a lot of
time (see 'spare time' above) and energy (some package authors are
easier to work with than others). Several of us have taken cracks at
moving this forward from time to time, but it rarely gets to the top
of anyone's priority list.

            

      
      
      
    

        
Making a list and hoping that it will remain up to date is not
realistic.  The only way that would work reliably is if the list could
be programmatically generated, for example by parsing installed
headers for declarations and caveats as above. Which would be possible
with changes like the ones listed above.

Best,

luke

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Henrik Bengtsson
    

    

      
      #
    

  


  

      
On Wed, Apr 24, 2024 at 1:32?PM luke-tierney--- via R-devel

        
<r-devel at r-project.org> wrote:

            

      
      
      
    

        
The R help pages have been proven to work excellently for documenting
the R API. Maybe we could leverage that to document the C-level API?
That could help to bring more formal structure to the documentation
than the "free-text" format that is currently used in RWE.

I played around with this idea some years ago, where I created a
package 'RNativeAPI' for the sole purpose of prototyping documentation
of the C-level API of R, and to see what's possible to do with the Rd
format in this sense and to get an idea what the limits are. For
example, getAttrib() and setAttrib() are documented in
<https://github.com/HenrikBengtsson/RNativeAPI/blob/master/man/getAttrib.Rd>.
To see what it looks like, go to
<https://henrikbengtsson.github.io/RNativeAPI/reference/getAttrib.html>.
If this is feasible, I think there are enough folks out there that are
willing to contribute to such documentation. It's obviously a
long-term project, but it's doable.

/Henrik

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      iuke-tier@ey m@iii@g oii uiow@@edu
    

    

      
      #
    

  


  

      
On Wed, 24 Apr 2024, Hadley Wickham wrote:

        

            

      
      
      
    

        
My pragmatic approach to deciding if an entry point is usable in a
package is to

     grep for it in the installed headers

     grep for it in WRE

     if those are good, check the text in both places to make sure it
     doesn't tell me not to use is

The first two can be automated; the text reading can't for now.

One place this runs into trouble is when the prose in WRE doesn't
explicitly mention the entry point, but says something like 'this one
and similar ones are OK'. A couple of years ago I worked on improving
some of those by explicitly adding some of those implicit ones, which
did sometimes make the text more cumbersome. I'm pretty sure I added
LOGICAL() and RAW() at that point (but may be mis-remebering); they
are there now. In some other cases I left the text alone but added
index entries. That makes them findable with a text search. I think I
got most that can be handled that way, but there may be some others
left. Far from ideal, but at least a step forward.

            

      
      
      
    

        
As you well know R-core is not a monolith. There are several R-core
members who also are not happy about how that played out and where
that stands now. But there was and is no viable option other than to
agree to disagree. There is really no upside to re-litigating this
now.

Best,

luke

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Ivan Krylov
    

    

      
      #
    

  


  

      
On Wed, 24 Apr 2024 15:31:39 -0500 (CDT)

        
luke-tierney--- via R-devel <r-devel at r-project.org> wrote:

        

            

      
      
      
    

        
This sounds almost like Doxygen, although the exact syntax used to
denote the entry points and the necessary comments is far from the most
important detail at this point.

            

      
      
      
    

        
Is there a way for outsiders to help? For example, would it help to
produce the linking graph (package P links to entry points X, Y)? I
understand that an entry point being unpopular doesn't mean it
shouldn't be public (and the other way around), but combined with a
list of entry points that are listed in WRE, such a graph could be
useful to direct effort or estimate impact from interface changes.

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Kevin Ushey
    

    

      
      #
    

  


  

      
On Thu, Apr 25, 2024 at 4:24?AM Ivan Krylov via R-devel

        
<r-devel at r-project.org> wrote:

            

      
      
      
    

        
I'm guessing Doxygen would be overkill here? I think instead these
would just be structured comments that mark a particular function, or
set of functions, as part of the API -- and some automated tool could
then just pull those functions out into a list of API functions. Then,
we would have a single "source of truth" for what is in the API, and
could be seen at a glance by browsing / grepping the installed R
headers. I see this as a structured way of accomplishing what is
already being done to clarify whether functions are part of the API in
the R headers.

A similar approach would have macros like R_API, or with a bit more
specificity, maybe something like R_API(ALTREP), which would have no
actual definition -- they would exist in the source purely to mark
functions as part of (some subset of) the API. Or, similarly, anything
declared within a block like R_API {} would be considered part of the
API (to avoid the need to tag every declaration individually.) This
would at least make it easy to figure out what functions are part of
the R API, without requiring too much extra maintenance effort from
the R maintainers.

The other alternative I could imagine would be an installed header
like R_ext/API.h, which package authors who want to submit packages to
CRAN would be required to use, with direct usage of other headers
eventually being phased out. But that would be a larger maintenance
burden, unless its generation could be automated (e.g. from the
functions tagged above).

As a side note, it's worth stating that the set of API endpoints that
R Core wants to make available to CRAN packages, versus those that are
intended for other usages (e.g. applications embedding R), are
different sets. But I suspect this discussion is most relevant to R
package authors who wish to submit their packages to CRAN.

            

      
      
      
    

        
I'm guessing the most welcome kinds of contributions would be
documentation? IMHO, "documenting an API" and "describing how an API
can be used" are somewhat separate endeavors. I believe R-exts does an
excellent job of the latter, but may not be the right vehicle for the
former. To that end, I believe it would be helpful to have some
structured API documentation as a separate R-api document. Each
documented function would have described inputs, outputs, whether
inputs + outputs require protection from the garbage collector, and
other important usage notes. This is something that I think could be
developed and maintained by community members, with members of the R
community submitting documentation for each of the available API
functions. Such an effort could be started independently from R Core,
but some guidance would be appreciated as far as (1) would such a
document eventually be accepted as part of the official R manuals, and
(2) if so, what would be required of such a document.