Skip to content

post_processor in rmarkdown not working

10 messages · Heinz Tuechler, Thierry Onkelinx, Duncan Murdoch

Thierry Onkelinx
# 
Dear all,

I'm trying to write a post_processor() for a custom rmarkdown format. The
goal of the post_processor() is to modify the latex file before it is
compiled. For some reason the post_processor() is not run. The
post_processor() does work when I run it manually on the tex file.

Any suggestions on what I'm doing wrong? Below is the relevant snippet of
the code. The full code is available at
https://github.com/inbo/INBOmd/blob/post_processor/R/rsos_article.R
https://github.com/inbo/INBOmd/blob/post_processor/inst/rmarkdown/templates/rsos_article/skeleton/skeleton.Rmd
is an Rmd is a MWE that fails compile because the post_processor() is not
run.

Best regards,

Thierry

  post_processor <- function(
    metadata, input_file, output_file, clean, verbose
  ) {
    text <- readLines(output_file, warn = FALSE)

    # set correct text in fmtext environment
    end_first_page <- grep("\\\\EndFirstPage", text) #nolint
    if (length(end_first_page) == 1) {
      maketitle <- grep("\\\\maketitle", text) #nolint
      text <- c(
        text[1:(maketitle - 1)],
        "\\begin{fmtext}",
        text[(maketitle + 1):(end_first_page - 1)],
        "\\end{fmtext}",
        "\\maketitle",
        text[(end_first_page + 1):length(text)]
      )
      writeLines(enc2utf8(text), output_file, useBytes = TRUE)
    }
    output_file
  }

  output_format(
    knitr = knitr_options(
      opts_knit = list(
        width = 60,
        concordance = TRUE
      ),
      opts_chunk = opts_chunk,
      knit_hooks = knit_hooks
    ),
    pandoc = pandoc_options(
      to = "latex",
      latex_engine = "xelatex",
      args = args,
      keep_tex = keep_tex
    ),
    post_processor = post_processor,
    clean_supporting = !keep_tex
  )



ir. Thierry Onkelinx
Instituut voor natuur- en bosonderzoek / Research Institute for Nature and
Forest
team Biometrie & Kwaliteitszorg / team Biometrics & Quality Assurance
Kliniekstraat 25
1070 Anderlecht
Belgium

To call in the statistician after the experiment is done may be no more
than asking him to perform a post-mortem examination: he may be able to say
what the experiment died of. ~ Sir Ronald Aylmer Fisher
The plural of anecdote is not data. ~ Roger Brinner
The combination of some data and an aching desire for an answer does not
ensure that a reasonable answer can be extracted from a given body of data.
~ John Tukey
Heinz Tuechler
# 
Are you sure that you want to read in the output_file in

text <- readLines(output_file, warn = FALSE)?

best regards,

Heinz

Thierry Onkelinx wrote/hat geschrieben on/am 06.09.2017 11:41:

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Thierry Onkelinx
    

    

      
      #
    

  


  

      
Dear Heinz,

Yes. The idea of the post_processor() is that 1) pandoc converts the .md to
.tex 2) the post_processors changes the .tex 3) the .tex is compiled into
.pdf Hence the post_processors need to read, change and overwrite the tex
output file.

Best regards,

ir. Thierry Onkelinx
Instituut voor natuur- en bosonderzoek / Research Institute for Nature and
Forest
team Biometrie & Kwaliteitszorg / team Biometrics & Quality Assurance
Kliniekstraat 25
1070 Anderlecht
Belgium

To call in the statistician after the experiment is done may be no more
than asking him to perform a post-mortem examination: he may be able to say
what the experiment died of. ~ Sir Ronald Aylmer Fisher
The plural of anecdote is not data. ~ Roger Brinner
The combination of some data and an aching desire for an answer does not
ensure that a reasonable answer can be extracted from a given body of data.
~ John Tukey

2017-09-07 9:08 GMT+02:00 Heinz Tuechler <tuechler at gmx.at>:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Duncan Murdoch
    

    

      
      #
    

  


  

      
On 06/09/2017 5:41 AM, Thierry Onkelinx wrote:

            

      
      
      
    

        
I installed it and tried running it using

debug(INBOmd::rsos_article)
rmarkdown::render("skeleton.Rmd")

then after post_processor was defined, I set it to debug as well, and 
could see that the post_processor was being run.

I didn't get useful output, because the LaTeXing failed (I don't have 
the rsos.cls), but perhaps you've already fixed this problem, or perhaps 
it is intermittent?

Duncan Murdoch

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Thierry Onkelinx
    

    

      
      #
    

  


  

      
Dear Duncan,

Thanks for chiming in. Could you explain how you set debug() on
post_processor()? I've tried adding debug(post_processor) to rsos_article()
or adding debug(post_processor) when after post_processor was defined in
the debugger. Neither work for me.

All supporting files are available within the package. The code below
should be reproducible on your machine.

remove.packages("INBOmd")
devtools::install_github("inbo/INBOmd at post_processor")
setwd(system.file("rmarkdown/templates/rsos_article/skeleton", package =
"INBOmd"))
debug(INBOmd::rsos_article)
rmarkdown::render("skeleton.Rmd")

The sign that post_processor() fails when the tex file still contains
\EndFirstPage resulting in the compilation error "Undefined control
sequence. l.128 \EndFirstPage"

I still get the error with the current version of the code. Running the
post_processor manually works.

eval(parse(
  text = readLines(
    "
https://raw.githubusercontent.com/inbo/INBOmd/post_processor/R/rsos_article.R
"
  )[72:92]
))
post_processor(output_file = "skeleton.tex")
system("pdflatex skeleton.tex")

Best regards,


ir. Thierry Onkelinx
Instituut voor natuur- en bosonderzoek / Research Institute for Nature and
Forest
team Biometrie & Kwaliteitszorg / team Biometrics & Quality Assurance
Kliniekstraat 25
1070 Anderlecht
Belgium

To call in the statistician after the experiment is done may be no more
than asking him to perform a post-mortem examination: he may be able to say
what the experiment died of. ~ Sir Ronald Aylmer Fisher
The plural of anecdote is not data. ~ Roger Brinner
The combination of some data and an aching desire for an answer does not
ensure that a reasonable answer can be extracted from a given body of data.
~ John Tukey

2017-09-07 12:14 GMT+02:00 Duncan Murdoch <murdoch.duncan at gmail.com>:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Duncan Murdoch
    

    

      
      #
    

  


  

      
On 07/09/2017 10:11 AM, Thierry Onkelinx wrote:

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Duncan Murdoch
    

    

      
      #
    

  


  

      
On 07/09/2017 10:11 AM, Thierry Onkelinx wrote:

            

      
      
      
    

        
Not working for me either right now for some reason or other.  What I 
was doing was manually running debug(post_processor) in the debugger 
after single stepping past its definition.

What does show it is running is that at that same point I can execute

post_processor <- function() stop()

and it stops.

The end of the console log looks like this:


/Applications/RStudio.app/Contents/MacOS/pandoc/pandoc +RTS -K512m -RTS 
skeleton.utf8.md --to latex --from 
markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash 
--output skeleton.tex --template 
/Library/Frameworks/R.framework/Versions/3.3/Resources/library/INBOmd/rmarkdown/templates/rsos_article/resources/template.tex 
--natbib --bibliography sample.bib
Latexmk: This is Latexmk, John Collins, 19 Jan. 2017, version: 4.52c.
Error in output_format$post_processor(yaml_front_matter, utf8_input, 
output_file,  :
   unused arguments (yaml_front_matter, utf8_input, output_file, clean, 
!quiet)
Called from: output_format$post_processor(yaml_front_matter, utf8_input, 
output_file,
     clean, !quiet)

so we see pandoc being run, then Latexmk, then the post_processor call. 
It seems a little odd that Latexmk is being run.  Is that something you 
are doing, or is it pandoc asking for that?  If the latter, can you tell 
pandoc not to do so?

            

      
      
      
    

        
I'm not sure you would normally have write access in that directory, so 
it may not be typical of what you'd see in a user directory.  I 
certainly see something different when I copy the skeleton.Rmd file (and 
nothing else) to my own temp directory.

            

      
      
      
    

        
That certainly indicates it isn't doing what you want, but it might be 
running and doing something else.

Duncan Murdoch

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Duncan Murdoch
    

    

      
      #
    

  


  

      
On 07/09/2017 2:04 PM, Duncan Murdoch wrote:

            

      
      
      
    

        
I've done some debugging in rmarkdown::render.  Apparently if you need 
Bibtex (as your example does), it runs Latexmk before the post-processor.

I don't know if there's a way around this...

Duncan Murdoch

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Thierry Onkelinx
    

    

      
      #
    

  


  

      
That is strange. Another function in the same package
(INBOmd::inbo_rapport) uses the same trick. I actually started by copying
the post_processor() from that function. INBOmd::inbo_rapport() works both
with and without BibTex. Working examples are source/inbo_rapport and
source/inbo_rapport_basic from https://github.com/inbo/inbomd_examples.
Note that you need some extra work after installing INBOmd to
inbo_rapport() to run. See the README at https://github.com/inbo/INBOmd

I've created an issue https://github.com/rstudio/rmarkdown/issues/1138

ir. Thierry Onkelinx
Instituut voor natuur- en bosonderzoek / Research Institute for Nature and
Forest
team Biometrie & Kwaliteitszorg / team Biometrics & Quality Assurance
Kliniekstraat 25
1070 Anderlecht
Belgium

To call in the statistician after the experiment is done may be no more
than asking him to perform a post-mortem examination: he may be able to say
what the experiment died of. ~ Sir Ronald Aylmer Fisher
The plural of anecdote is not data. ~ Roger Brinner
The combination of some data and an aching desire for an answer does not
ensure that a reasonable answer can be extracted from a given body of data.
~ John Tukey

2017-09-07 21:18 GMT+02:00 Duncan Murdoch <murdoch.duncan at gmail.com>:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Duncan Murdoch
    

    

      
      #
    

  


  

      
On 08/09/2017 3:56 AM, Thierry Onkelinx wrote:

            

      
      
      
    

        
I think the issue there is that the LaTeX code is valid before the 
post-processor is run, it just re-orders things.  So rmarkdown::render 
runs Pandoc, then LaTeX (via Latexmk), then the post-processor, then 
LaTeX again.

This suggests a solution to the rsos_article problem:  somehow make sure 
that the LaTeX is valid from the start, e.g. by defining dummy versions 
of the missing macros.

Duncan Murdoch


I actually started by