Skip to content

Commit

Permalink
[ENH][TST] Overhauls CI completely (#418)
Browse files Browse the repository at this point in the history 
* Adds integration and five echo skipping

* Style fixes

* Updates config for CircleCI

* Attempts to fix YML

* [TEST] Update Dockerfile to match new integr tests

* [TEST] Fixes integration tests in Docker image

* [FIX] Remove intermediate IO files

* Resolves merge conflict, adds output check

* Some fixes

* [TEST] Updates dev_tool testing infra

* [TEST] Fixes pytest integration path checking

* [TEST] CircleCI uses Docker image to run tests

* [FIX] Minor dev_tool issues for CircleCI

* [TEST] Use variable for integration test filename

* Attempts to fix CircleCI style check

* Revert "Attempts to fix CircleCI style check"

This reverts commit 769f4b7.

* Attempt to fix tput call

* Adds checkout to code in YML

* [TEST] Integration tests run in parallel

* [TEST] Separate data downloads from Docker build

* [TEST] Update integration test data path

* [TEST] CircleCI uses good Docker

* [TEST] No version check in circleci

* [TEST] Checkout for get_data / style check

* Attempts to fix integration test inclusion

* [TEST] Checkout for get_data / style check

* [FIX] Fix circleci config hopefully

* [FIX] No / workdir for circleci machine

* [FIX] Use ~ for coverage in circleci

* Switches integration tests to truncated length data

* [FIX] Actually merge coverage files

* [FIX] Coverage cache path circleci

* [TEST] Integration test outputs in tests/data

* [FIX] circleci config bug

* [TEST] Major testing infra overhaul

Docker image considerably slimmed down (only test python 3.6 locally),
added new dev_requirements.txt to make conda yaml files obsolete, added
Makefile to make testing easier locally (if you aren't using the Docker
image), and removed integration test data downloads from separate script
and into the integration tests themselves

* [TEST] Massive CircleCI config regression

@tsalo had it right --- moving towards a fully Dockerized implementation
was not the way forward for a simple Python package.

* [TEST] Better integration testing?

At least, more equivalent to what was happening before, where we check
that ONLY the expected output files are generated (no more, no less)

* [FIX] CircleCI workflow issue

* [MNT] No flake8-putty

* [FLK] New flake8 error detected

* [TEST] Run style check separately

@leej3 said it's not fair to stop running tests for a few minor style
errors, and he's usually right so....

* [TEST] Py37 for all non-unit test stuff
  • Loading branch information
@jbteves @rmarkello
jbteves authored and rmarkello committed Nov 8, 2019
1 parent 4a7e997 commit 4fcdb3c
Show file tree
Hide file tree
Showing 21 changed files with 451 additions and 946 deletions.
369 changes: 106 additions & 263 deletions .circleci/config.yml
View file

Large diffs are not rendered by default.

141 changes: 0 additions & 141 deletions .circleci/tedana_outputs.txt
View file

This file was deleted.

3 changes: 3 additions & 0 deletions .dockerignore
View file
Original file line number Diff line number Diff line change
@@ -1 +1,4 @@
tedana/
dist/
.git/
docs/
219 changes: 24 additions & 195 deletions Dockerfile_dev
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,4 @@
# Your version: 0.6.0 Latest version: 0.6.0
# Generated by Neurodocker version 0.6.0
# Timestamp: 2019-11-06 17:06:07 UTC
#
# Thank you for using Neurodocker. If you discover any issues
# or ways to improve this software, please submit an issue or
# pull request on our GitHub repository:
#
# https://github.com/kaczmarj/neurodocker

FROM debian:latest

ARG DEBIAN_FRONTEND="noninteractive"
FROM continuumio/miniconda3

ENV LANG="en_US.UTF-8" \
LC_ALL="en_US.UTF-8" \
Expand All @@ -24,6 +12,13 @@ RUN export ND_ENTRYPOINT="/neurodocker/startup.sh" \
curl \
locales \
unzip \
curl \
git \
wget \
gzip \
bzip2 \
sed \
make \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen \
Expand All @@ -44,196 +39,30 @@ ENTRYPOINT ["/neurodocker/startup.sh"]
ENV LANG="C.UTF-8" \
LC_ALL="C.UTF-8"

RUN apt-get update -qq \
&& apt-get install -y -q --no-install-recommends \
curl \
git \
wget \
bzip2 \
ca-certificates \
sed \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

RUN mkdir -p /dev_tools/envs /tedana

COPY ["./dev_tools/envs/py35_env.yml", "/dev_tools/envs/py35_env.yml"]

COPY ["./dev_tools/envs/py36_env.yml", "/dev_tools/envs/py36_env.yml"]
RUN git clone https://github.com/me-ica/tedana.git /tedana

COPY ["./dev_tools/envs/py37_env.yml", "/dev_tools/envs/py37_env.yml"]
COPY ["./requirements.txt", "/tedana/requirements.txt"]

ENV CONDA_DIR="/opt/conda" \
PATH="/opt/conda/bin:$PATH"
RUN export PATH="/opt/conda/bin:$PATH" \
&& echo "Downloading Miniconda installer ..." \
&& conda_installer="/tmp/miniconda.sh" \
&& curl -fsSL --retry 5 -o "$conda_installer" https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& bash "$conda_installer" -b -p /opt/conda \
&& rm -f "$conda_installer" \
&& conda update -yq -nbase conda \
&& conda config --system --prepend channels conda-forge \
&& conda config --system --set auto_update_conda false \
&& conda config --system --set show_channel_urls true \
&& sync && conda clean --all && sync \
&& conda env create -q --name tedana_py35 --file /dev_tools/envs/py35_env.yml \
&& rm -rf ~/.cache/pip/*
COPY ["./dev_requirements.txt", "/tedana/dev_requirements.txt"]

RUN conda env create -q --name tedana_py36 --file /dev_tools/envs/py36_env.yml \
&& rm -rf ~/.cache/pip/*
RUN bash -c "conda create -yq --name tedana_py36 python=3.6 pip \
&& source activate tedana_py36 \
&& pip install -r /tedana/dev_requirements.txt \
&& pip install ipython \
&& rm -rf ~/.cache/pip/* \
&& conda clean --all"

RUN conda env create -q --name tedana_py37 --file /dev_tools/envs/py37_env.yml \
&& rm -rf ~/.cache/pip/*
RUN /opt/conda/envs/tedana_py36/bin/ipython profile create \
&& sed -i 's/#c.InteractiveShellApp.extensions = \[\]/ \
c.InteractiveShellApp.extensions = \['\''autoreload'\''\]/g' \
/root/.ipython/profile_default/ipython_config.py

RUN \
mkdir -p /data/three-echo \
&& curl -L -o /data/three-echo/three_echo_Cornell_zcat.nii.gz https://osf.io/8fzse/download
RUN mkdir -p /tedana/dev_tools

RUN \
mkdir /data/five-echo \
&& curl -L -o five_echo_NIH.tar.xz https://osf.io/ea5v3/download \
&& tar xf five_echo_NIH.tar.xz -C /data/five-echo \
&& rm -f five_echo_NIH.tar.xz

RUN \
mkdir -p /data/test/three-echo \
&& curl -L -o TED.Cornell_processed_three_echo_dataset.tar.xz https://osf.io/u65sq/download \
&& tar xf TED.Cornell_processed_three_echo_dataset.tar.xz --no-same-owner -C /data/test/three-echo/ \
&& rm -f TED.Cornell_processed_three_echo_dataset.tar.xz

RUN \
mkdir -p /data/test/five-echo \
&& curl -L -o TED.p06.tar.xz https://osf.io/fr6mx/download \
&& tar xf TED.p06.tar.xz --no-same-owner -C /data/test/five-echo/ \
&& rm -f TED.p06.tar.xz

RUN \
/opt/conda/envs/tedana_py36/bin/ipython profile create \
&& sed -i 's/#c.InteractiveShellApp.extensions = \[\]/c.InteractiveShellApp.extensions = \['\''autoreload'\''\]/g' /root/.ipython/profile_default/ipython_config.py

COPY ["./dev_tools/local_testing.sh", "/dev_tools/local_testing.sh"]
COPY ["./dev_tools", "/tedana/dev_tools"]

RUN sed -i '$isource activate tedana_py36' $ND_ENTRYPOINT

RUN sed -i '$isource /dev_tools/local_testing.sh' $ND_ENTRYPOINT
RUN sed -i '$isource /tedana/dev_tools/run_tests.sh' $ND_ENTRYPOINT

WORKDIR /tedana

RUN echo '{ \
\n "pkg_manager": "apt", \
\n "instructions": [ \
\n [ \
\n "base", \
\n "debian:latest" \
\n ], \
\n [ \
\n "env", \
\n { \
\n "LANG": "C.UTF-8", \
\n "LC_ALL": "C.UTF-8" \
\n } \
\n ], \
\n [ \
\n "install", \
\n [ \
\n "curl", \
\n "git", \
\n "wget", \
\n "bzip2", \
\n "ca-certificates", \
\n "sed" \
\n ] \
\n ], \
\n [ \
\n "run", \
\n "mkdir -p /dev_tools/envs /tedana" \
\n ], \
\n [ \
\n "copy", \
\n [ \
\n "./dev_tools/envs/py35_env.yml", \
\n "/dev_tools/envs/py35_env.yml" \
\n ] \
\n ], \
\n [ \
\n "copy", \
\n [ \
\n "./dev_tools/envs/py36_env.yml", \
\n "/dev_tools/envs/py36_env.yml" \
\n ] \
\n ], \
\n [ \
\n "copy", \
\n [ \
\n "./dev_tools/envs/py37_env.yml", \
\n "/dev_tools/envs/py37_env.yml" \
\n ] \
\n ], \
\n [ \
\n "miniconda", \
\n { \
\n "create_env": "tedana_py35", \
\n "install_path": "/opt/conda", \
\n "yaml_file": "/dev_tools/envs/py35_env.yml", \
\n "activate_env": "false" \
\n } \
\n ], \
\n [ \
\n "miniconda", \
\n { \
\n "create_env": "tedana_py36", \
\n "install_path": "/opt/conda", \
\n "yaml_file": "/dev_tools/envs/py36_env.yml", \
\n "activate_env": "true" \
\n } \
\n ], \
\n [ \
\n "miniconda", \
\n { \
\n "create_env": "tedana_py37", \
\n "install_path": "/opt/conda", \
\n "yaml_file": "/dev_tools/envs/py37_env.yml", \
\n "activate_env": "false" \
\n } \
\n ], \
\n [ \
\n "run", \
\n "\\n mkdir -p /data/three-echo\\n && curl -L -o /data/three-echo/three_echo_Cornell_zcat.nii.gz https://osf.io/8fzse/download" \
\n ], \
\n [ \
\n "run", \
\n "\\n mkdir /data/five-echo\\n && curl -L -o five_echo_NIH.tar.xz https://osf.io/ea5v3/download\\n && tar xf five_echo_NIH.tar.xz -C /data/five-echo\\n && rm -f five_echo_NIH.tar.xz" \
\n ], \
\n [ \
\n "run", \
\n "\\n mkdir -p /data/test/three-echo\\n && curl -L -o TED.Cornell_processed_three_echo_dataset.tar.xz https://osf.io/u65sq/download\\n && tar xf TED.Cornell_processed_three_echo_dataset.tar.xz --no-same-owner -C /data/test/three-echo/\\n && rm -f TED.Cornell_processed_three_echo_dataset.tar.xz" \
\n ], \
\n [ \
\n "run", \
\n "\\n mkdir -p /data/test/five-echo\\n && curl -L -o TED.p06.tar.xz https://osf.io/fr6mx/download\\n && tar xf TED.p06.tar.xz --no-same-owner -C /data/test/five-echo/\\n && rm -f TED.p06.tar.xz" \
\n ], \
\n [ \
\n "run", \
\n "\\n /opt/conda/envs/tedana_py36/bin/ipython profile create\\n && sed -i '"'"'s/#c.InteractiveShellApp.extensions = \\[\\]/c.InteractiveShellApp.extensions = \\['"'"'\\'"'"''"'"'autoreload'"'"'\\'"'"''"'"'\\]/g'"'"' /root/.ipython/profile_default/ipython_config.py" \
\n ], \
\n [ \
\n "copy", \
\n [ \
\n "./dev_tools/local_testing.sh", \
\n "/dev_tools/local_testing.sh" \
\n ] \
\n ], \
\n [ \
\n "add_to_entrypoint", \
\n "source activate tedana_py36" \
\n ], \
\n [ \
\n "add_to_entrypoint", \
\n "source /dev_tools/local_testing.sh" \
\n ], \
\n [ \
\n "workdir", \
\n "/tedana" \
\n ] \
\n ] \
\n}' > /neurodocker/neurodocker_specs.json
Loading

0 comments on commit 4fcdb3c

Please sign in to comment.