Continuous Integration and Testing

All code for this document is located at here.

Submitting a Package

To submit a package to Neuroconductor, the author/maintainer of the package provides the GitHub URL for the package. Once the package is submitted several initial checks are conducted. These checks ensure that the package has been created correctly. After initial checks are complete, the package must be verified by email. This verification is designed to prevent spam and allow the developer to stop a package if they would like to revise the package before re-submitting.

Once the verification is complete, the package is processed. Overall, the package is copied/cloned to a remote server. Standardized Travis CI and Appveyor configuration files, specific to Neuroconductor, are added. These are to ensure that the checks performed on these services are consistent for each package. Some parameters of the package DESCRIPTION file are changed. These parameters ensure that when a package is downloaded from Neuroconductor, the correct versions of the dependent packages are used.

Next, the package is pushed to the central Neuroconductor GitHub (https://github.com/neuroconductor) and submitted to Travis CI and AppVeyor to be built and checked on multiple systems. Parameters are set to ensure that Travis CI and AppVeyor use the correct versions of Neuroconductor packages for checking and external dependencies are installed. The author of the package receives an automatic email indicating whether the package was built successfully and is integrated with Neuroconductor together with a description file containing pertinent information about the process.

Stable vs. Current Versions

We use the terminology “Stable” and “Current” to differentiate a different status of development for a Neuroconductor package. On the initial submission, after all checks are passed, the package is incorporated into Neuroconductor and deemed the Stable version. The Current version of the package is the result of nightly pulls and mirror the latest package version from the developer’s GitHub repository. This provides Neuroconductor users with a way to use the latest versions of a package and at the same time it provides the Neuroconductor platform with a safe way of checking new versions of a package against the existing set of Current Neuroconductor packages. If a Current version of a package passes all the required Neuroconductor tests, we contact the developer of the package and suggest an official re-submission to Neuroconductor. If the newly re-submitted version of the package passes the checks against the Stable Neuroconductor packages, this version is incorporated to the Stable version of Neuroconductor.

The neuroc.deps package

We have created the neuroc.deps package that perform most of the backend operations on a Neuroconductor package. It can be installed as follows:

devtools::install_github("muschellij2/neuroc.deps")

The most relevant function is use_neuroc_template, which is used to make many of the changes to the package. For a specific project, you should specify the user = "neuroconductor".

Changes to the DESCRIPTION file

In order to test packages against the relevant Neuroconductor packages, we change the DESCRIPTION file. We do this in the following ways:

  1. Modify, or add if not present, the Remotes field. Packages are installed using the install_github function, which reads this Remotes field to install dependencies if necessary. The Remotes field modifies and overrides the locations of dependencies to be installed. If a dependency for a package is present, then a newer version of the package will not be installed unless indicated by the user or indicated a newer version is necessary in the package (by the package (>= VERSION)) syntax) in the dependencies.
  2. We add the bioViews field to a package in case there are Bioconductor package in the dependencies, to ensure install_github looks in that repository, as per the issue hadley/devtools#1254.
  3. The covr package is added to the Suggests field if not already present in the dependencies (Depends, Imports, or Suggests). This is so that code coverage can be performed.

ANTsR Dependencies

If a package depends on the ANTsR workflow, a slightly modified set of continuous integration steps are performed as that build is highly technical.

Continuous Integration Services

For checking R packages, we use Continuous Integration services Travis CI, which builds on Linux and OS X operating systems, and Appveyor, which builds on Windows using MinGW.

The purpose is to ensure that the package can be built, installed, and checked on the respective systems with the appropriate dependencies.

Travis CI

For Travis CI, we delete the developer’s .travis.yml configuration script and replace it with the one located at https://github.com/muschellij2/neuroc.deps/blob/master/inst/neuroconductor_travis.yml.

Travis Helpers

before_install:
  - fname=travis_helpers.sh
  - wget -O ${fname} http://bit.ly/travis_helpers
  - cat ${fname}; source ${fname}; rm ${fname}  
  - PROJECT_NAME=neuroconductor
  - remove_neuroc_packages

which remove any packages located on Neuroconductor from the Travis machine. As caching is done, these may be present from previous builds. The travis_helpers.sh file is a set of helper bash functions that backend the ghtravis package. Most of these are changes to DESCRIPTION file, but on Travis and not the GitHub.

Installing Remotes without Dependencies

The command:

  - install_remotes_no_dep

looks at the Remotes field in the DESCRIPTION file and runs install_github(..., upgrade_dependencies = FALSE). This ensures that the Neuroconductor packages will be those with the specific commit IDs at the time of running. No old Neuroconductor packages will be present as they were removed using remove_neuroc_packages.

PACKAGE_NAME environmental variable

The environmental variable of PACKAGE_NAME is created from the DESCRIPTION file. This may be different from the repository name from the user, but will be the same repository name on Neuroconductor, as all repos are neuroconductor/PACKAGE_NAME.

  - export PACKAGE_NAME=`package_name`

Bioconductor Packages

We add the following fields to the YAML: To ensure Bioconductor packages can be installed if necessary:

bioc_required: yes
use_bioc: yes

Warnings are Errors

So that we ensure that no warnings are present in the installation (similar to CRAN):

warnings_are_errors: true

CRAN checks

That we have a similar threshold for packages similar to CRAN:

r_check_args: --as-cran

Pass or Fail

After running R CMD check, the 00install.out and 00check.log are printed for diagnostic purposes.

Deployment

When packages are being deployed, R CMD INSTALL --build is run so that they have the standardized naming conventions. The deployment information for Neuroconductor, including GitHub encrypted keys, are added. After building, the binary distribution is uploaded to the GitHub repository when tagged (from Neuroconductor’s backend not the developer).

Coverage

After deployment, we use Coveralls.io and the covr package to run code coverage. We use type = "all" so that we provide coverage of tests, vignettes, and examples:

after_deploy:
  - Rscript -e 'covr::coveralls(type = "all")'

Future work

We plan to add Neuroconductor badges to the README.md file.

Appveyor

Currently, we only formally support packages that work in *nix type of operatings systems. We will check the package for Windows as a courtesy to Windows users, but do not provide a detailed level of support.

We use the neuroconductor_appveyor.yml, which changes the PATH variable to try to replicate a Windows machine using Rtools only and not installing MinGW.

Different from the YAML from devtools::use_appveyor(), we remove the following part:

  - path: '\*_*.tar.gz'
    name: Bits

as could overwrite Linux builds depeneding on the naming convention on Deployment.

Code Coverage

Coveralls

We plan to use the covr package to check for code coverage using the Coveralls interface. We currently do not have any requirements for code coverage for our packages.

Advanced

CI and Authentication Tokens

If you need access to a secure key, such as a GitHub Personal Acccess Token (PAT), you do not to set them in your YAML files. Specifically with GitHub, if you push a secure key to a repository, GitHub will automatically deactivate that token (this may only apply to public repositories). In order to set an environment variable, such as GITHUB_PAT for GitHub authentication, you have to change the settings on the repository on the respective CI website.

Travis CI

In Travis CI you have to go to: https://travis-ci.org/USERNAME/REPO/settings, then the section labeled “Environment Variables”. Put GITHUB_PAT as the name and paste your unencrypted GitHub PAT in the Value field. When you build on Travis CI, you should see:

Setting environment variables from repository settings
$ export GITHUB_PAT=[secure]

in the build logs. Now you can use the environment variable GITHUB_PAT in your code.

Appveyor

In Appveyor you have to go to: https://ci.appveyor.com/project/USERNAME/REPO/settings, then the section labeled “Environment” and click “Add Variable”. Put GITHUB_PAT as the name and paste your unencrypted GitHub PAT in the Value field. I believe you should click the lock to encrypt it.

Session Info

devtools::session_info()
## ─ Session info ───────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 4.0.2 (2020-06-22)
##  os       macOS Catalina 10.15.7      
##  system   x86_64, darwin17.0          
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2021-02-18                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────
##  package     * version date       lib source                            
##  assertthat    0.2.1   2019-03-21 [2] CRAN (R 4.0.0)                    
##  cachem        1.0.4   2021-02-13 [1] CRAN (R 4.0.2)                    
##  callr         3.5.1   2020-10-13 [1] CRAN (R 4.0.2)                    
##  cli           2.3.0   2021-01-31 [1] CRAN (R 4.0.2)                    
##  colorout    * 1.2-2   2020-06-01 [2] Github (jalvesaq/colorout@726d681)
##  crayon        1.4.1   2021-02-08 [1] CRAN (R 4.0.2)                    
##  desc          1.2.0   2020-06-01 [2] Github (muschellij2/desc@b0c374f) 
##  devtools    * 2.3.2   2020-09-18 [1] CRAN (R 4.0.2)                    
##  digest        0.6.27  2020-10-24 [1] CRAN (R 4.0.2)                    
##  ellipsis      0.3.1   2020-05-15 [2] CRAN (R 4.0.0)                    
##  evaluate      0.14    2019-05-28 [2] CRAN (R 4.0.0)                    
##  fastmap       1.1.0   2021-01-25 [1] CRAN (R 4.0.2)                    
##  fs            1.5.0   2020-07-31 [2] CRAN (R 4.0.2)                    
##  glue          1.4.2   2020-08-27 [1] CRAN (R 4.0.2)                    
##  htmltools     0.5.1.1 2021-01-22 [1] CRAN (R 4.0.2)                    
##  knitr         1.31    2021-01-27 [1] CRAN (R 4.0.2)                    
##  lifecycle     1.0.0   2021-02-15 [1] CRAN (R 4.0.2)                    
##  magrittr      2.0.1   2020-11-17 [1] CRAN (R 4.0.2)                    
##  memoise       2.0.0   2021-01-26 [1] CRAN (R 4.0.2)                    
##  pkgbuild      1.2.0   2020-12-15 [1] CRAN (R 4.0.2)                    
##  pkgload       1.1.0   2020-05-29 [2] CRAN (R 4.0.0)                    
##  prettyunits   1.1.1   2020-01-24 [2] CRAN (R 4.0.0)                    
##  processx      3.4.5   2020-11-30 [1] CRAN (R 4.0.2)                    
##  ps            1.5.0   2020-12-05 [1] CRAN (R 4.0.2)                    
##  purrr         0.3.4   2020-04-17 [2] CRAN (R 4.0.0)                    
##  R6            2.5.0   2020-10-28 [1] CRAN (R 4.0.2)                    
##  remotes       2.2.0   2020-07-21 [2] CRAN (R 4.0.2)                    
##  rlang         0.4.10  2020-12-30 [1] CRAN (R 4.0.2)                    
##  rmarkdown   * 2.6     2020-12-14 [1] CRAN (R 4.0.2)                    
##  rprojroot     2.0.2   2020-11-15 [1] CRAN (R 4.0.2)                    
##  rstudioapi    0.13    2020-11-12 [1] CRAN (R 4.0.2)                    
##  sessioninfo   1.1.1   2018-11-05 [2] CRAN (R 4.0.0)                    
##  stringi       1.5.3   2020-09-09 [1] CRAN (R 4.0.2)                    
##  stringr     * 1.4.0   2019-02-10 [2] CRAN (R 4.0.0)                    
##  testthat      3.0.2   2021-02-14 [1] CRAN (R 4.0.2)                    
##  usethis     * 2.0.1   2021-02-10 [1] CRAN (R 4.0.2)                    
##  withr         2.4.1   2021-01-26 [1] CRAN (R 4.0.2)                    
##  xfun          0.21    2021-02-10 [1] CRAN (R 4.0.2)                    
##  yaml        * 2.2.1   2020-02-01 [2] CRAN (R 4.0.0)                    
## 
## [1] /Users/johnmuschelli/Library/R/4.0/library
## [2] /Library/Frameworks/R.framework/Versions/4.0/Resources/library