DAX Command Line Tools

Table of Contents

  1. List of the Tools

List of the Tools

Each tool has a help option and some examples on how to use the tools. You can call each tool with no arguments to see the help.

XnatSetup

You can use the Xnatsetup (see below) command tool to setup your computer with the –basic options. It will do what is below automatically, but if you don’t want to do that, it can be setup manually.

This Xnat commands will use two thing :

  • install pyxnat and python packages on your computer (Check ‘Get started for Spiders’ on the wiki)

  • set your bashrc with the env variable to connect to Xnat with pyxnat :

export XNAT_HOST=http://xnat.vanderbilt.edu/xnat
export XNAT_USER=username
export XNAT_PASS=password
export PATH=/PathToMasimatlab/trunk/xnatspiders/Xnat_tools:$PATH

FYI : you can open the bashrc like :

vim (or nano or any editor you like) ~/.bashrc

and when you are done editing it, use :

. ~/.bashrc

You will after this be able to call the commands directly on your terminal.

One last thing, the Xnatupload will send you updates (errors and warnings) about the directory you are trying to upload on Xnat. If you want to receive this email, you need to set up two variables in your bashrc :

export EMAIL_ADDR=add@gmail.com
export EMAIL_PWS=passwordforthisemail

It will use this email address to send you email. It has to be a gmail address.

Xnatsetup is as you can guess a command tool to set up your computer. It will install the python package needed and ask for the variables that need to be set up. There are different kinds of setup :

  • basic to be able to use the XNAT command tools

  • advance to run spiders on your computer or ACCRE

  • redcap to use the spider to send data to redcap

  • cci package setup for ACCRE or if you need XnatUtils

  • ACCRE setup

#######################################################
#                                            XNATSETUP
# XnatSetup is a command tool to set up on your computer the variables to use
#      the tools/spiders.
# Developed by the masiLab Vanderbilt University, TN, USA.
# Operating system : Linux & Mac OS
# Shell : bash
# Requirements : python with pip & git
# Contact : benjamin.c.yvernault@vanderbilt.edu
#     No Arguments given
#     See the help bellow or Use "Xnatquery" -h
#######################################################
Usage: Xnatsetup [options]

What is the script doing : Set up your computer to use xnat.
  *Basic installation (--basic) - Needed to use the Xnat command tools or any of
the next installations : install the python package httplib2, lxml, and pyxnat if not
already install & saving your username/host/password for XNAT.
  *Advance installation (--advance ) - Needed to run the non-specific spiders :
Set up the Upload Directory, set up masimatlab path for Xnatrun, add the xnat
tools to your PATH, and add Spiders.py in your PYTHONPATH .
  *Redcap installation (--redcap ) - Needed to use redcap spiders (send data to
redcap) : Install Pycap / pandas if not install and set up the URL for redcap .
  *API installation (--api ) - Needed to use API package to run spiders on ACCRE
via jobs (Contains XnatUtils) : Install API if not install.
  *ACCRE installation (--accre) - Setup the environment to use the
package/spiders/tools on ACCRE.


Options:
 -h, --help            show this help message and exit
 --basic               Use this options to set up the env variables to use
                       the Xnat tools and have the basic set up.
 --advance             Use this options to set up the env variables to run
                       spiders in general.
 --redcap              Use this options to set up the env variables to use
                       redcap spiders.
 --api                 Use this options to set up the env variables to run
                       spiders on ACCRE via jobs.
 --Accre               Use this options if you are on Accre.
 --NoSudo              Use this options if you don't have sudo access and you
                       still want to install the package (check -d option).
 -d INSTALLDIR, --installdir=INSTALLDIR
                       Use this options to specify a directory where the
                       python package need to be install. It works only if
                       you use --NoSudo option.
 --tutorial            Give you the step for the specific setup you are
                       asking.

Contact - benjamin.c.yvernault@vanderbilt.edu

XnatQuery

Xnatquery will show you the tree on xnat. Xnatquery is a tool to query objects on XNAT for each level. You can see which projects you have access to and see the hierarchy of data on your project. It has several options (accessible with -h or –help) :

################################################################
#                            XnatQuery                         #
#                                                              #
# Developed by the MASI Lab Vanderbilt University, TN, USA.    #
# If issues, please start a thread here:                       #
# https://groups.google.com/forum/#!forum/vuiis-cci            #
# Usage:                                                       #
#     Query through XNAT at the level you want.                #
# Examples:                                                    #
#     Check the help for examples by running --help            #
################################################################

----------------------------------------------------------------
usage: XnatQuery [-h] [--host HOST] [-u USERNAME] [-p PROJECT] [-s SUBJECT]
                 [-e SESSION] [-a ASSESSOR] [-c SCAN] [--all] [--me]

What is the script doing :
   * Query on Xnat at any level.

Examples:
   *Show all the projects you have access to:
        Xnatquery --me
   *Show all projects:
        Xnatquery --all
   *Query a specific level (example scan/assessors for a session):
        Xnatquery -p PID -s 109873 -e 109873
   *Query a specific level with all objects under it :
        Xnatquery -p PID -s 109873 --all

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Host for XNAT. Default: env XNAT_HOST.
  -u USERNAME, --username USERNAME
                        Username for XNAT.
  -p PROJECT, --project PROJECT
                        project ID on Xnat or 'all' to see all the project.
  -s SUBJECT, --subject SUBJECT
                        Subject label on Xnat
  -e SESSION, --experiment SESSION
                        Session label on Xnat
  -a ASSESSOR, --assessor ASSESSOR
                        Assessor/Process label on XNAT. E.G: VUSTP-x-VUSTP1-x-VUSTP1a-x-FS
  -c SCAN, --scan SCAN  Scan ID on Xnat.
  --all                 Print all the objects on XNAT from the level you are at.
  --me                  Give the projects ID that you have access.

Extra Examples

  • To get information on the project

Xnatquery -p projectID --info
  • To get all the subjects in this project

Xnatquery -p projectID
  • To get all the experiments in this project

Xnatquery -p projectID -s subject

Contact - benjamin.c.yvernault@vanderbilt.edu

XnatCheck

Xnatcheck is a quick way to check directly on your terminal if there is the resource you just created on all your project. You can check if there is a scan type or an assessor type as well with the options -s or -a. Options available (-h or -help):

################################################################
#                            XnatCheck                         #
#                                                              #
# Developed by the MASI Lab Vanderbilt University, TN, USA.    #
# If issues, please start a thread here:                       #
# https://groups.google.com/forum/#!forum/vuiis-cci            #
# Usage:                                                       #
#     Check XNAT data (subject/session/scan/assessor/resource) #
# Examples:                                                    #
#     Check the help for examples by running --help            #
################################################################

----------------------------------------------------------------
usage: XnatCheck [-h] [--host HOST] [-u USERNAME] [-p PROJECTS]
                 [--filters FILTERS [FILTERS ...]]
                 [--delimiter DELIMITER_FILTER_RESOURSE] [--csv CSV_FILE]
                 [--format FORMAT] [--printfilters] [--printformat]

What is the script doing :
   *Check object on XNAT (subject/session/scan/assessor/resources) specify by the options.

How to write a filter string:
 - for resources filters, the string needs to follow this template:
   variable_name=value--sizeoperatorValue--nbfoperatorValue--fpathsoperatorValue
   By default, it will return the assessor that does have the resource if no other filter specify
 - for other filters, the string needs to follow this template:
   variable_name=Value
   operator can be different than =. Look at the table in --printfilters

Use --printfilters to see the different variables available

Examples:
   *See format variables:
        Xnatcheck --printformat
   *See filter variables:
        Xnatcheck --printfilters
   *Get list of T1,DTI scans that have a resource called NIFTI:
        Xnatcheck -p PID --filters type=T1,DTI assessor_res=NIFTI
   *Get list of fMRIQA assessors that have a resource called PDF:
        Xnatcheck -p PID --filters proctype=fMRIQA assessor_res=PDF
   *Get list of assessors except fMRIQA that have a resource called PDF :
        Xnatcheck -p PID --filters proctype!=fMRIQA assessor_res=PDF
   *Get list of project sessions that do not have a resource called testing:
        Xnatcheck -p PID --filters session_label=VUSTP1a,VUSTP2b,VUSTP3a session_res!=testing
   *Get list of project fMRIQA and VBMQA that used more than 45mb and less than 1hour:
        Xnatcheck -p PID1,PID2 --filters proctype=fMRIQA,VBMQA procstatus=COMPLETE "memused>45mb" "walltimeused<1:00:00" --format assessor_label,procnode,memused,walltimeused

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Host for XNAT. Default: env XNAT_HOST.
  -u USERNAME, --username USERNAME
                        Username for XNAT.
  -p PROJECTS, --project PROJECTS
                        Project(s) ID on XNAT
  --filters FILTERS [FILTERS ...]
                        List of filters separated by a space to apply to the search.
  --delimiter DELIMITER_FILTER_RESOURSE
                        Resource filters delimiter. By default: --.
  --csv CSV_FILE        File path to save the CSV output.
  --format FORMAT       Header for the csv. format: list of variables name comma-separated
  --printfilters        Print available filters.
  --printformat         Print available format for display.

Extra Examples

  • To return all the scans where there is no NIFTI on the project BLSA

Xnatcheck -p BLSA -r NIFTI
  • To return all the assessors where there is no PDF on the project BLSA

Xnatcheck -p BLSA -r PDF -l 1
  • To return all the subjects/experiments where there is no fMRIQA assessor on the project BLSA

Xnatcheck -p BLSA -a fMRIQA
  • To return all the subjects/experiments where there is no fMRIQA assessor on the project BLSA and check for the one that exists if there is a PDF resource

Xnatcheck -p BLSA -a fMRIQA -r PDF

Contact - benjamin.c.yvernault@vanderbilt.edu

XnatDownload

Xnatdownload will download all the resources that you asked for in a directory. Xnatdownload provides bulk download of data from XNAT with specific filters applied. It provides options to narrow your download to only what you need. This tool will generate a folder per project in your -d directory with two files: download_commandLine.txt and download_report.csv with the description of what you downloaded. It has several options (accessible with -h or -help) :

########################################################
#                                                XNATDOWNLOAD
#
# Developed by the masiLab Vanderbilt University, TN, USA.
# If issues, email benjamin.c.yvernault@vanderbilt.edu
# Parameters :
#     No Arguments given
#     See the help bellow or Use "Xnatdownload" -h
########################################################
usage: Xnatdownload [-h] [--host HOST] [-u USERNAME] [-p PROJECT]
                    [-d DIRECTORY] [-D] [--subj SUBJECT] [--sess SESSION]
                    [-s SCANTYPE] [-a ASSESSORTYPE] [--WOS WITHOUTS]
                    [--WOP WITHOUTA] [--quality QUALITIES] [--status STATUS]
                    [--qcstatus QCSTATUS] [-c CSVFILE] [--rs RESOURCESS]
                    [--ra RESOURCESA] [--selectionS SELECTIONSCAN]
                    [--selectionP SELECTIONASSESSOR] [--overwrite] [--update]
                    [--fullRegex] [-o OUTPUTFILE] [-i] [-b BIDS_DIR] [-xt]
                    [--bond_dir BOND_DIR]

What is the script doing :
   *Download filtered data from XNAT to your local computer using the different OPTIONS.

Examples:
   *Download all resources for all scans/assessors in a project:
        Xnatdownload -p PID -d /tmp/downloadPID -s all --rs all -a all --ra all
   *Download NIFTI for T1,fMRI:
        Xnatdownload -p PID -d /tmp/downloadPID -s T1,fMRI --rs NIFTI
   *Download only the outlogs for fMRIQA assessors that failed:
        Xnatdownload -p PID -d /tmp/downloadPID -a fMRIQA --status JOB_FAILED --ra OUTLOG
   *Download PDF for assessors that Needs QA:
        Xnatdownload -p PID -d /tmp/downloadPID -a all --qcstatus="Needs QA" --ra OUTLOG
   *Download NIFTI for T1 for some sessions :
        Xnatdownload -p PID -d /tmp/downloadPID --sess 109309,189308 -s all --rs NIFTI
   *Download same data than previous line but overwrite the data:
        Xnatdownload -p PID -d /tmp/downloadPID --sess 109309,189308 -s all --rs NIFTI --overwrite
   *Download data described by a csvfile (follow template) :
        Xnatdownload -d /tmp/downloadPID -c  upload_sheet.csv
   *Transform the XnatDownload data in BIDS format for all sessions, scantype and resources:
        Xnatdownload -p PID --sess all -d /tmp/downloadPID -s all --rs all --bids /tmp/BIDS_dataset
   *Transform the XnatDownload data in BIDS format for all sessions, scantype and resources with xnat tag:
        Xnatdownload -p PID --sess all -d /tmp/downloadPID -s all --rs all --bids /tmp/BIDS_dataset -xt
   *Transform the XnatDownload data in BIDS format for all sessions, scantype and resources with xnat tag and perform bond:
        Xnatdownload -p PID --sess all -d /tmp/downloadPID -s all --rs all --bids /tmp/BIDS_dataset -xt --bond /tmp/BOND_dir

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Host for XNAT. Default: using $XNAT_HOST.
  -u USERNAME, --username USERNAME
                        Username for XNAT. Default: using $XNAT_USER.
  -p PROJECT, --project PROJECT
                        Project(s) ID on Xnat
  -d DIRECTORY, --directory DIRECTORY
                        Directory where the data will be download
  -D, --oneDirectory    Data will be downloaded in the same directory. No sub-
                        directory.
  --subj SUBJECT        filter scans/assessors by their subject label. Format:
                        a comma separated string (E.G: --subj VUSTP2,VUSTP3).
  --sess SESSION        filter scans/assessors by their session label. Format:
                        a comma separated string (E.G: --sess VUSTP2b,VUSTP3a)
  -s SCANTYPE, --scantype SCANTYPE
                        filter scans by their types (required to download
                        scans). Format: a comma separated string (E.G : -s
                        T1,MPRAGE,REST). To download all types, set to 'all'.
  -a ASSESSORTYPE, --assessortype ASSESSORTYPE
                        filter assessors by their types (required to download
                        assessors). Format: a comma separated string (E.G : -a
                        fMRIQA,dtiQA_v2,Multi_Atlas). To download all types,
                        set to 'all'.
  --WOS WITHOUTS        filter scans by their types and removed the one with
                        the specified types. Format: a comma separated string
                        (E.G : --WOS T1,MPRAGE,REST).
  --WOP WITHOUTA        filter assessors by their types and removed the one
                        with the specified types. Format: a comma separated
                        string (E.G : --WOP fMRIQA,dtiQA).
  --quality QUALITIES   filter scans by their quality. Format: a comma
                        separated string (E.G: --quality
                        usable,questionable,unusable).
  --status STATUS       filter assessors by their job status. Format: a comma
                        separated string.
  --qcstatus QCSTATUS   filter assessors by their quality control status.
                        Format: a comma separated string.
  -c CSVFILE, --csvfile CSVFILE
                        CSV file with the following header: object_type,projec
                        t_id,subject_label,session_type,session_label,as_label
                        . object_type must be 'scan' or 'assessor' and
                        as_label the scan ID or assessor label.
  --rs RESOURCESS       Resources you want to download for scans. E.g : --rs
                        NIFTI,PAR,REC.
  --ra RESOURCESA       Resources you want to download for assessors. E.g :
                        --ra OUTLOG,PDF,PBS.
  --selectionS SELECTIONSCAN
                        Download from only one selected scan.By default : no
                        selection. E.G : project-x-subject-x-session-x-scan
  --selectionP SELECTIONASSESSOR
                        Download from only one selected processor.By default :
                        no selection. E.G : assessor_label
  --overwrite           Overwrite the previous data downloaded with the same
                        command.
  --update              Update the files from XNAT that have been downloaded
                        with the newest version if there is one (not working
                        yet).
  --fullRegex           Use full regex for filtering data.
  -o OUTPUTFILE, --output OUTPUTFILE
                        Write the display in a file giving to this OPTIONS.
  -i, --ignore          Ignore reading of the csv report file
  -b BIDS_DIR, --bids BIDS_DIR
                        Directory to store the XNAT to BIDS curated data
  -xt, --xnat_tag       Download BIDS data with XNAT subjID and sessID
  --bond_dir BOND_DIR   Download the Key groups and Param groups in BIDS data to BOND_DIR

Example

  • Downloads in /home/benjamin/temp the resources NIFTI and PDF for all the scan fMRI and the assessor fMRIQA for the project BLSA

Xnatdownload -p BLSA -d /home/benjamin/temp/ -a fMRIQA -s fMRI -r NIFTI,PDF

Contact - benjamin.c.yvernault@vanderbilt.edu

XnatUpload

Xnatupload will create subject/experiment/scan/resources for a project on XNAT and upload the data into the project from a folder. Xnatupload provides bulk upload of data to a project on XNAT. You need to provide a specific CSV file with the following header:

  • object_type,project_id,subject_label,session_type,session_label,as_label,as_type,as_description,quality,resource,fpath

where:

  • as_label corresponds to assessor or scan label

  • as_type corresponds to proctype or scantype

  • as_description corresponds to procstatus or series description for the scan

  • quality corresponds to qastatus or quality for scan

It should be similar to this (project in the example is CIBS-TEST):

object_type,project_id,subject_label,session_type,session_label,as_label,as_type,as_description,quality,resource,fpath scan,CIBS-TEST,CIBS-TEST_01,MR,CIBS-TEST_01,401,BRAIN2_3DT1,BRAIN2_3DT1,usable,NIFTI,/Users/<USER>/Downloads/Archive/DICOM_T1W_3D_TFE.nii.gz

Methods

Warning: the project must already exist on XNAT. You can add a new project via the XNAT web GUI. Follow one of the three methods to upload:

  • Number 1 : all the files are in one directory but they need to be rename like this projectID-x-subjectID-x-experimentID-x-scanID-x-scantype-x-resourcename.extention. Fastest methode but only one file can be upload in a resource.

  • Number 2 : you don’t need to rename all the files but you need to give a specific structure to your directory : folder/subjectID/experimentID/scanID-x-scantype/ResourceID/ and put the resources corresponding in it. E.G : TempDir/BLSA_0000/BLSA_0000_0/scan2-x-fMRI/NIFTI/nifti.nii.gz. It will not be as fast as the first methode but you can upload more than one file to a resources.

  • Option -o : There is a third choice. If you want to upload files to Xnat on a scan and you don’t want to create anything, you can use this options -o. It’s for only upload. It’s using something like the first methodes : put all the files into one folder with a special name : projectID-x-subjectID-x-experimentID-x-scanID-x-resourcename.extention for assessor, assessor_label-resourcename.extension for assessor (Reminder : assessor_label = projectID-x-subjectID-x-experimentID-x-scanID-x-process_name or projectID-x-subjectID-x-experimentID-x-processname).

################################################################
#                            XnatUpload                        #
#                                                              #
# Developed by the MASI Lab Vanderbilt University, TN, USA.    #
# If issues, please start a thread here:                       #
# https://groups.google.com/forum/#!forum/vuiis-cci            #
# Usage:                                                       #
#     Print a detailed report from XNAT projects.              #
# Examples:                                                    #
#     Check the help for examples by running --help            #
################################################################
IMPORTANT WARNING FOR ALL USERS ABOUT XNAT:
   session_label needs to be unique for each session.
   Two subjects can NOT have the same session_label
----------------------------------------------------------------
usage: XnatUpload [-h] [--host HOST] [-u USERNAME] -c CSV_FILE
                  [--sess SESSION_TYPE] [--report] [--force] [--delete]
                  [--deleteAll] [--noextract] [--printmodality]
                  [-o OUTPUT_FILE] [-b BIDS_DIR] [-p PROJECT]

What is the script doing :
   * Upload data to XNAT following the csv file information.
     csv header:
     object_type,project_id,subject_label,session_type,session_label,
     as_label,as_type,as_description,quality,resource,fpath

IMPORTANT: YOU NEED TO CREATE THE PROJECT ON XNAT BEFORE UPLOADING.

Examples:
   * See Session type:
        Xnatupload --printmodality
   * Simple upload:
        Xnatupload -c upload_sheet.csv
   * Upload everything with a session type:
        Xnatupload -c upload_sheet.csv --sess PET
   * Check the upload:
        Xnatupload -c upload_sheet.csv --report
   * Force upload:
        Xnatupload -c upload_sheet.csv --force
   * Upload with delete resource before uploading:
        Xnatupload -c upload_sheet.csv --delete
   * Upload with delete every resources for the object (SCAN/ASSESSOR) before uploading:
        Xnatupload -c upload_sheet.csv --deleteAll
   * Upload BIDS data to XNAT format for scan
        Xnatupload -b /tmp/bidsDataset -p PID
   * Check BIDS data to XNAT
        Xnatupload -b /tmp/bidsDataset -p PID --report
   * Force upload BIDS data to upload XNAT
        Xnatupload -b /tmp/bidsDataset -p PID --force

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Host for XNAT. Default: env XNAT_HOST.
  -u USERNAME, --username USERNAME
                        Username for XNAT.
  -c CSV_FILE, --csv CSV_FILE
                        CSV file with the information for uploading data to XNAT. Header: object_type,project_id,subject_label,session_type,session_label,as_label,as_type,as_description,as_quality,resource,fpath
  --sess SESSION_TYPE   Session type on Xnat. Use printmodality to see the options.
  --report              Print a report to verify inputs.
  --force               Force the upload and remove previous resources.
  --delete              Delete resource files prior to upload.
  --deleteAll           Delete all resources in object prior to upload.
  --noextract           Do not extract the zip files on XNAT when uploading a folder.
  --printmodality       Display the different modality available on XNAT for a session.
  -o OUTPUT_FILE, --output OUTPUT_FILE
                        File path to store the script logs.
  -b BIDS_DIR, --bids BIDS_DIR
                        BIDS Directory to convert to XNAT and then upload
  -p PROJECT, --project PROJECT
                        Project for BIDS XNAT upload

Extra Examples

  • Shows on the terminal what kind of data the command is going to upload and where with method 1

Xnatupload -d /Path/to/directory --report --up1
  • Uploads the files in the directory with the first method

Xnatupload -p projectID -d /Path/to/directory -up1 -sess MR
  • Uploads the files in the directory with the second method

Xnatupload -p projectID -d /Path/to/directory --up2 --sess CT
  • Uploads (only, no creation of subject/exp/scan) all the files from the directory into Xnat even if there is already a resources (options -force)

Xnatupload -d /Path/to/directory -o -T 1 --force

Contact - benjamin.c.yvernault@vanderbilt.edu

XnatReport

Xnatreport will give you a report on one ore more projects. It will show all the subjects/sessions/scans/assessors/resources for the projects chosen. It has several options (accessible with -h or -help) :

################################################################
#                            XnatReport                        #
#                                                              #
# Developed by the MASI Lab Vanderbilt University, TN, USA.    #
# If issues, please start a thread here:                       #
# https://groups.google.com/forum/#!forum/vuiis-cci            #
# Usage:                                                       #
#     Print a detailed report from XNAT projects.              #
# Examples:                                                    #
#     Check the help for examples by running --help            #
################################################################

----------------------------------------------------------------
usage: XnatReport [-h] [--host HOST] [-u USERNAME] [-p PROJECTS] [-c CSV_FILE]
                  [--format FORMAT] [--printformat]

What is the script doing :
   * Create a report about Xnat projects.

Examples:
   *Report of a project:
        Xnatreport -p PID
   *Report with a specific format:
        Xnatreport -p PID --format object_type,session_id,session_label,age
   *print the format available:
        Xnatreport --printformat
   *Save report in a csv:
        Xnatreport -p PID -c report.csv

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Host for XNAT. Default: env XNAT_HOST.
  -u USERNAME, --username USERNAME
                        Username for XNAT.
  -p PROJECTS, --project PROJECTS
                        List of project ID on Xnat separate by a coma
  -c CSV_FILE, --csvfile CSV_FILE
                        csv fullpath where to save the report.
  --format FORMAT       Header for the csv. format: variables name separated by comma.
  --printformat         Print available variables names for the option --format.

Extra Examples

  • Creates a report for BLSA and CTONS and will print it on the screen/terminal

Xnatreport -p BLSA,CTONS
  • Sends the report on BLSA and CTONS to your email address as a csv file. You need to set to variables gmail address and password used to sent the email in your bashrc

Xnatreport -p BLSA,CTONS --csv -e email@email.com
  • Writes the report as a “.csv” file that can be open with Excel. If not path specify, /tmp is the place where the report is save. -t will do the same but in a text file

Xnatreport -p BLSA,CTONS --csv

Contact - benjamin.c.yvernault@vanderbilt.edu

XnatSwitchProcessStatus

XnatSwitchProcessStatus is one of the most powerful and used of the Xnat_tools. It allows the user to switch/set the procstatus (job status) for a specific proctype (type of assessor) in a project. XnatSwitchProcessStatus allows the user to change the status of several type of assessors in a project that have a specific type or just for all of them.

################################################################
#                     XnatSwitchProcessStatus                  #
#                                                              #
# Developed by the MASI Lab Vanderbilt University, TN, USA.    #
# If issues, please start a thread here:                       #
# https://groups.google.com/forum/#!forum/vuiis-cci            #
# Usage:                                                       #
#     Change assessor job/quality control status.              #
# Examples:                                                    #
#     Check the help for examples by running --help            #
################################################################

----------------------------------------------------------------
usage: XnatSwitchProcessStatus [-h] [--host HOST] [-u USERNAME]
                               [--select SELECT] [-x TXT_FILE] [-p PROJECTS]
                               [--subj SUBJECTS] [--sess SESSIONS] [-s STATUS]
                               [-f FORMER_STATUS] [-t PROCTYPES]
                               [-n NEED_INPUTS] [-d] [--qc] [--printstatus]
                               [--fullRegex] [--restart] [--rerun] [--init]
                               [--rerundiskq]

What is the script doing :
    *Switch/Set the status for assessors on XNAT selected by the proctype.

Examples:
    *See status managed by DAX:
        XnatSwitchProcessStatus --printstatus
    *Set all fMRIQA to a specific status Error for a project:
        XnatSwitchProcessStatus -p PID -s Error -t fMRIQA
    *Set all Multi_Atlas that have the status JOB_FAILED to NEED_TO_RUN to have the processes run again:
        XnatSwitchProcessStatus -p PID -f JOB_FAILED -t Multi_Atlas -s NEED_TO_RUN
    *Set all VBMQA to NEED_TO_RUN for a project and delete resources:
        XnatSwitchProcessStatus -p PID -s NEED_TO_RUN -t VBMQA -d
    *Set all VBMQA to NEED_TO_RUN, delete resources, and set linked assessors fMRI_Preprocess to NEED_INPUTS:
        XnatSwitchProcessStatus -p PID -s NEED_TO_RUN -t VBMQA -d -n fMRI_Preprocess
    *Set all dtiQA_v2 qa status to Passed for a project:
        XnatSwitchProcessStatus -p PID -s Passed -t dtiQA_v2 --qc
    *Set FreeSurfer for a specific project/subject to NEED_INPUTS:
        XnatSwitchProcessStatus -p PID --subj 123 -s NEED_INPUTS -t FreeSurfer

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Host for XNAT. Default: env XNAT_HOST.
  -u USERNAME, --username USERNAME
                        Username for XNAT.
  --select SELECT       Give the assessor label that you want to change the status.
  -x TXT_FILE, --txtfile TXT_FILE
                        File txt. Each line represents the label of the assessor which need to change status.
  -p PROJECTS, --project PROJECTS
                        Project ID on XNAT or list of Project ID
  --subj SUBJECTS       Change Status for only this subject/list of subjects.
  --sess SESSIONS       Change Status for only this session/list of sessions.
  -s STATUS, --status STATUS
                        Status you want to set on the Processes. E.G: 'NEED_TO_RUN'
  -f FORMER_STATUS, --formerStatus FORMER_STATUS
                        Change assessors with this former status. E.G: 'JOB_FAILED'
  -t PROCTYPES, --type PROCTYPES
                        Assessor process type you want the status to changed.
  -n NEED_INPUTS, --Needinputs NEED_INPUTS
                        Assessor process type that need to change to NEED_INPUTS because the assessors from -t you changed are inputs to those assessors.
  -d, --deleteR         Delete the resources on the assessor.
  --qc                  Change the quality control status on XNAT.
  --printstatus         Print status used by DAX to manage assessors.
  --fullRegex           Use full regex for filtering data.
  --restart             Restart the assessors by switching the status for all assessors found to NEED_TO_RUN and delete previous resources.
  --rerun               Rerun the assessors by switching status to NEED_TO_RUN for assessors that failed and delete previous resources.
  --init                Init the assessors by switching status to NEED_INPUTS for assessors that have been set to NO_DATA.
  --rerundiskq          Rerun the assessor that have the status JOB_FAILED: switching status to NEED_INPUTS from JOB_FAILED and delete previous resources.

Extra Examples

  • Changes the status for dtiQA_v2 and Freesurfer that have a Failed status to NeedToRun in project BLSA

XnatSwitchProcessStatus -p BLSA -f Failed -s NeedToRun -t dtiQA_v2,FreeSurfer
  • Changes the status for dtiQA_v2 and Freesurfer that have a Failed status to NeedToRun in project BLSA and it will delete all the resources on the assessor

XnatSwitchProcessStatus -p BLSA -f Failed -s NeedToRun -t dtiQA_v2,FreeSurfer -d
  • Changes the status for the specific FreeSurfer assessor in BLSA_0000_00 session to NeedToRun and delete the resources

XnatSwitchProcessStatus --select BLSA-x-BLSA_0000-x-BLSA_0000_00-x-FreeSurfer -s NeedToRun -d

Contact - benjamin.c.yvernault@vanderbilt.edu

XnatProcessUpload

Xnatprocessupload allows you to upload data for an assessor (you can’t do it that with Xnatupload). You only need to give the path to the folder where the data are. If the assessor doesn’t exist, it will create one. You need to organize the data like this :

  1. One folder per assessor you want to upload, the name of the folder needs to be the name of the assessor (Remember: assessor label = projectID-x-subjectID-x-sessionID-x-(scanID if running on a only a scan)-x-processname)

  2. Put one folder for each resources you want to upload within the assessor folder with the name folder equal to the resource name.

  3. Put the file you want to upload in it.

########################################################
#                    XNATPROCESSUPLOAD
#
# Developed by the masiLab Vanderbilt University, TN, USA.
# If issues, email benjamin.c.yvernault@vanderbilt.edu
# Parameters :
#     No Arguments given
#     Use "Xnatprocessupload -h" to see the options
########################################################
Usage: Xnatprocessupload [options]
What is the script doing : Upload Data on Xnat from a Directory as an Assessor.

Options:
 -h, --help            show this help message and exit
 -d FOLDER_PATH, --directory=FOLDER_PATH
                       Directory containing the different assessors folders that you want to upload.
 --force               Force the upload.

Contact - benjamin.c.yvernault@vanderbilt.edu

XnatSubjectUpdate

Xnatsubjectupdate changes the last update date on XNAT to nothing. It will make the automatic process (in cci package when it’s setup) to run again on this subject.

########################################################
#                   XNATSUBJECTUPDATE
#
# Developed by the masiLab Vanderbilt University, TN, USA.
# If issues, email benjamin.c.yvernault@vanderbilt.edu
# Parameters :
#     No Arguments given
#     See the help bellow or Use "Xnatsubjectupdate" -h
########################################################
Usage: Xnatsubjectupdate [options]
What is the script doing : Query on Xnat at any level.

Options:
 -h, --help            show this help message and exit
 -p PROJECT_ID, --project=PROJECT_ID
                       One project ID on Xnat.
 -s SUBJECT_LABELS, --subject=SUBJECT_LABELS
                       Subject label on Xnat or list of them.

Contact - benjamin.c.yvernault@vanderbilt.edu

RedCapReport

Redcapreport is a powertool to extract data from REDCap. It will download the data and put it into a csv file. You can specify different options to have a precise download.

################################################################
#                           RedCapReport                       #
#                                                              #
# Developed by the MASI Lab Vanderbilt University, TN, USA.    #
# If issues, please start a thread here:                       #
# https://groups.google.com/forum/#!forum/vuiis-cci            #
# Usage:                                                       #
#     Create REDCap report for a redcap project.               #
# Examples:                                                    #
#     Check the help for examples by running --help            #
################################################################

----------------------------------------------------------------
usage: RedCapReport [-h] -k KEY [-c CSVFILE] [-x TXTFILE] [-p PROJECT]
                    [-s SUBJECT] [-e SESSION] [-a ASSESSOR] [-t PROCTYPE]
                    [-f PROCFILE] [-l LIBRARIES] [-F] [-L] [--all]

What is the script doing :
   *Extract data from REDCap as a csv file.

Examples:
   *Save the data in a csv file: Redcapreport -k KEY -c extract_redcap.csv
   *print the libraries name: Redcapreport -k KEY -L
   *print all fields name and label: Redcapreport -k KEY -F
   *Extract values for all record: Redcapreport -k KEY --all
   *Filter for specific project/subject/session/assessor type:
    Redcapreport -k KEY -p PID -s 109387 -e 109387_1,109387_2 -t FS,TRACULA_v1,dtiQA_v2
   *Extract for specific assessor: Redcapreport -k KEY -p PID -a PID-x-109387-x-109387_1-x-FS
   *Extract for specific libraries type: Redcapreport -k KEY -p PID -l library_name
   *Extract only the fields described in the txt file: Redcapreport -k KEY -x fields.txt

optional arguments:
  -h, --help            show this help message and exit
  -k KEY, --key KEY     API Token for REDCap project.
  -c CSVFILE, --csvfile CSVFILE
                        csv file path where the report will be save.
  -x TXTFILE, --txtfile TXTFILE
                        txt file path with per line, the name of the variable
                        on REDCap you want to extract.
  -p PROJECT, --project PROJECT
                        Extract values for processes for the projects chosen.
                        E.G: project1,project2
  -s SUBJECT, --subject SUBJECT
                        Extract values for processes for the subjects chosen.
                        E.G: subject1,subject2
  -e SESSION, --session SESSION
                        Extract values for processes for the sessions chosen.
                        E.G: session1,session2
  -a ASSESSOR, --assessor ASSESSOR
                        Extract values for processors chosen. E.G:
                        processor1,processor2
  -t PROCTYPE, --proctype PROCTYPE
                        Extract values for processes types chosen. E.G:
                        fMRIQA,dtiQA
  -f PROCFILE, --procfile PROCFILE
                        file path with each line one processor label. Extract
                        values for processes types chosen.
  -l LIBRARIES, --libraries LIBRARIES
                        Extract values for only the libraries specify. Check
                        the project for the libraries name. Switch spaces by
                        '_' and everything lower case. E.G:
                        dti_quality_assurance. By default: all libraries
  -F, --fields          Print all field names and labels
  -L, --printlib        Print all libraries names for the project.
  --all                 Extract values for all records.

Contact - benjamin.c.yvernault@vanderbilt.edu

XnatCheckLogin

XnatCheckLogin allows the user to check that environment variables are set appropriately. It will let you know in a few seconds if your logins are good or not.

usage: XnatCheckLogin [-h] [--host HOST]
Set and Check the logins for XNAT.
optional arguments:
  -h, --help   show this help message and exit
  --host HOST  Host for XNAT.

Xnatinfo

Xnatinfo is the tool to get fast statistics information on a project (number of subjects/sessions/scans/assessors and the status of the assessors). There is only one way to call Xnatinfo:

################################################################
#                             Xnatinfo                         #
#                                                              #
# Developed by the MASI Lab Vanderbilt University, TN, USA.    #
# If issues, please start a thread here:                       #
# https://groups.google.com/forum/#!forum/vuiis-cci            #
# Usage:                                                       #
#     Display information on a XNAT project.                   #
# Examples:                                                    #
#     Check the help for examples by running --help            #
################################################################

----------------------------------------------------------------
usage: Xnatinfo [-h] [--host HOST] [-u USERNAME] [-x OUTPUT_FILE] [-f] [-r]
                [--ignoreUnusable] [--ignoreScans]
                project

What is the script doing :
   * Generate a report for a XNAT project displaying scans/assessors
     information.

Examples:
    * See the information for project TEST:
        Xnatinfo TEST

positional arguments:
  project               Project ID on XNAT

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Host for XNAT. Default: env XNAT_HOST.
  -u USERNAME, --username USERNAME
                        Username for XNAT.
  -x OUTPUT_FILE, --filetxt OUTPUT_FILE
                        Path to a txt file to save the report
  -f, --failed          Add this flag to print out failed jobs
  -r, --running         Add this flag to print out running jobs
  --ignoreUnusable      Ignore print statement of unusable scans
  --ignoreScans         Ignore print statement of scans

Xnatsessionupdate

Xnatsessionupdate resets the last update date on XNAT on a session. It will force DAX update scripts to update the session. This tool is for advanced users and managers of projects on XNAT.

################################################################
#                        XnatSessionUpdate                     #
#                                                              #
# Developed by the MASI Lab Vanderbilt University, TN, USA.    #
# If issues, please start a thread here:                       #
# https://groups.google.com/forum/#!forum/vuiis-cci            #
# Usage:                                                       #
#     Reset sessions to be seen by the nex dax_update.         #
# Examples:                                                    #
#     Check the help for examples by running --help            #
################################################################

----------------------------------------------------------------
usage: XnatSessionUpdate [-h] [--host HOST] [-u USERNAME] -p PROJECTS
                         [-s SESSION] [-n] [-x TXT_FILE] [-a]

What is the script doing :
   * Reset sessions last update date to update the sessions during
     the next dax_update.

Examples:
   *Reset all sessions:
        Xnatsessionupdate -p PID --all
   *Reset some sessions :
        Xnatsessionupdate -p PID -s 109374,109348
   *Reset for the sessions that have assessors NEED_INPUTS:
        Xnatsessionupdate -p PID -n

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Host for XNAT. Default: env XNAT_HOST.
  -u USERNAME, --username USERNAME
                        Username for XNAT.
  -p PROJECTS, --project PROJECTS
                        Projects ID on Xnat.
  -s SESSION, --session SESSION
                        Session label on Xnat or list of them.
  -n, --needinputs      Change the subject last update date for all the subject with processes that have a job status equal to NEED_INPUTS.
  -x TXT_FILE, --txtfile TXT_FILE
                        File txt with at each line the label of the assessor or just the Session label where the Subject date need to be changed. E.G for label: project-x-subject-x-experiment-x-scan-x-process_name.
  -a, --all             Change for all sessions.

BIDSMapping

BIDSMapping tool allows the user to create, update or replace rules/mapping at the project level on XNAT. These rules are essential as they entail the link between scan type or series description on XNAT to the BIDS datatype, task type and repetition time. XnatToBids function uses these mapping at the project to transform XNAT data into the BIDS compliant data with BIDS filenames and folder structure.

################################################################
#                     BIDSMAPPING                              #
#                                                              #
# Developed by the MASI Lab Vanderbilt University, TN, USA.    #
# If issues, please start a thread here:                       #
# https://groups.google.com/forum/#!forum/vuiis-cci            #
# Usage:                                                       #
#     Upload rules/mapping to Project level on XNAT.           #
# Examples:                                                    #
#     Check the help for examples by running --help            #
################################################################

usage: use "BIDSMapping --help" for more information

What is the script doing :
   *Uploads BIDS datatype, tasktype and repitition time mapping to XNAT project level using the different OPTIONS.

Examples:
   *Create a new datatype mapping for scan_type of XNAT scans:
        BIDSMapping -p PID --xnatinfo scan_type --type datatype --create /tmp/projectID_datataype.csv
   *The correct format for /tmp/projectID_datataype.csv
        scan_type,datatype
        Resting State,func
   *Create a new datatype mapping for series_description of XNAT scans:
        BIDSMapping -p PID --xnatinfo series_description --type datatype --create /tmp/projectID_datataype.csv
   *Create a new tasktype mapping for scan_type of XNAT scans:
        BIDSMapping -p PID --xnatinfo scan_type --type tasktype --create /tmp/projectID_tasktype.csv
   *Replace tasktype mapping for scan_type of XNAT scans: (It removes the old mapping and upload the new mapping)
        BIDSMapping -p PID --xnatinfo scan_type --type tasktype --replace /tmp/projectID_tasktype.csv
   *Update tasktype mapping for scan_type of XNAT scans: (This is ONLY add new mapping rules, CANT remove rules use --replace to remove and add mapping rules)
        BIDSMapping -p PID --xnatinfo scan_type --type tasktype --update /tmp/projectID_tasktype.csv
   *Create default datatype mapping for scan_type of XNAT scans: (There is no default for series_description use --create)
        BIDSMapping -p PID --xnatinfo scan_type --type datatype --create_default
   *Download the current mapping on XNAT:
        BIDSMapping -p PID --xnatinfo scan_type --type datatype --download /tmp/download.csv
   *Download the scan_types on project on XNAT:
        BIDSMapping -p PID --template /tmp/scan_type_template.csv

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Host for XNAT. Default: using $XNAT_HOST.
  -u USERNAME, --username USERNAME
                        Username for XNAT. Default: using $XNAT_USER.
  -o LOGFILE, --logfile LOGFILE
                        Write the display/output in a file given to this OPTIONS.
  -p PROJECT, --project PROJECT
                        Project to create/update BIDS mapping file
  -t TYPE, --type TYPE  The type of mapping either datatype, tasktype or repetition_time_sec
  -x XNATINFO, --xnatinfo XNATINFO
                        The type of xnat info to use for mapping either scan_type or series_description
  -c CREATE, --create CREATE
                        Create the given BIDS new mapping file at project level. (EG. --create <mappingfile>.csv)
                        Default create creates the default mapping at project file. (EG. --create)
                        csvfile EG:
                        scan_type,datatype
                        T1W/3D/TFE,anat
                        Resting State,func
  -cd, --create_default
                        Default create creates the default mapping at project file. (EG. --create_default)
  -ud UPDATE, --update UPDATE
                        Update the existing BIDS mapping file at project level. (EG. --update <mappingfile>.csv)
                        This option can only add rules
  -rp REPLACE, --replace REPLACE
                        Replace the existing BIDS mapping file at project level. (EG. --replace <mappingfile>.csv)
                        This option can remove and add new rules
  -rv REVERT, --revert REVERT
                        Revert to an old mapping from a specific date/time. (EG: --revert 10-17-19-21:32:15
                        or --revert 10-17-19). Check the LOGFILE at project level for the date
  -d DOWNLOAD, --download DOWNLOAD
                        Downloads the current BIDS mapping file (EG: --download <foldername>)
  -tp TEMPLATE, --template TEMPLATE
                        Default mapping template (EG: --template <template file>)

For a walkthrough tutorial of BIDSMapping check out https://dax.readthedocs.io/en/latest/BIDS_walkthrough.html Contact - praitayini.kanakaraj@vanderbilt.edu

XnatBOND

XnatBOND takes in a BIDS directory and detects the Key and Parameter Groups. This tool can be used to Modifying Key and Parameter Group Assignment. For more details on the package used look at https://bids-bond.readthedocs.io/en/latest/readme.html

################################################################
#                             XnatBond                         #
#                                                              #
# Developed by the MASI Lab Vanderbilt University, TN, USA.    #
# If issues, please start a thread here:                       #
# https://groups.google.com/forum/#!forum/vuiis-cci            #
# Usage:                                                       #
#     Generate and alternate key params in BIDS using BOND     #
# Examples:                                                    #
#     Check the help for examples by running --help            #
################################################################

usage: XnatBOND [-h] --bids_dir BIDS_DIR [-b BOND_DIR] [-m keyparam_edited keyparam_files new_keyparam_prefix] [-o LOGFILE]

What is the script doing :
        *Generate the csv files that have the summary of key groups and param groups from the
        bidsdata and modify them in the bids data.

Examples:
        *Generate orginial key and parameter groups:
                XnatBOND --bids_dir BIDS_DIR --bond_dir BOND_DIR
        *Update the key and parameter groups:
                XnatBOND --bids_dir BIDS_DIR --modify_keyparam

optional arguments:
-h, --help            show this help message and exit
--bids_dir BIDS_DIR   BIDS data directory.
-b BOND_DIR, --bond_dir BOND_DIR
                      BOND data directory.
-m keyparam_edited keyparam_files new_keyparam_prefix, --modify_keyparam keyparam_edited keyparam_files new_keyparam_prefix
                      Values to modify the keyparam in bids.
-o LOGFILE, --logfile LOGFILE
                      Write the display/output in a file given to this OPTIONS.