Source Documentation

dax – Root package

dax.task – Task class

Task object to generate / manage assessors and cluster.

class dax.task.Task(processor, assessor, upload_dir)

Class Task to generate/manage the assessor with the cluster

check_date()
Sets the job created date if the assessor was not made through
dax_build
Returns:Returns if get_createdate() is != ‘’, sets date otherwise
check_job_usage()
The task has now finished, get the amount of memory used, the amount of
walltime used, the jobid of the process, the node the process ran on, and when it started from the scheduler. Set these values on XNAT
Returns:None
check_running(jobid=None)

Check to see if a job specified by the scheduler ID is still running

Parameters:jobid – The ID of the job in question assigned by the scheduler.
Returns:A String of JOB_RUNNING if the job is running or enqueued and JOB_FAILED if the ready flag (see read_flag_exists) does not exist in the assessor label folder in the upload directory.
commands(jobdir)

Call the get_cmds method of the class Processor.

Parameters:jobdir – Fully qualified path where the job will run on the node. Note that this is likely to start with /tmp on most grids.
Returns:A string that makes a command line call to a spider with all args.
get_createdate()

Get the date an assessor was created

Returns:String of the date the assessor was created in “%Y-%m-%d” format
get_job_status(jobid=None)

Get the status of a job given its jobid as assigned by the scheduler

Parameters:jobid – job id assigned by the scheduler
Returns:string from call to cluster.job_status or UNKNOWN.
get_job_usage()
Get the amount of memory used, the amount of walltime used, the jobid
of the process, the node the process ran on, and when it started from the scheduler.
Returns:List of strings. Memory used, walltime used, jobid, node used, and start date
get_jobid()

Get the jobid of an assessor as stored on XNAT

Returns:string of the jobid
get_jobnode()

Gets the node that a process ran on

Returns:String identifying the node that a job ran on
get_jobstartdate()

Get the date that the job started

Returns:String of the date that the job started in “%Y-%m-%d” format
get_memused()

Get the amount of memory used for a process

Returns:String of how much memory was used
get_processor_name()

Get the name of the Processor for the Task.

Returns:String of the Processor name.
get_processor_version()

Get the version of the Processor.

Returns:String of the Processor version.
get_qcstatus()

Get the qcstatus of the assessor

Returns:A string of the qcstatus for the assessor if it exists. If it does not, it returns DOES_NOT_EXIST. The else case returns an UNKNOWN xsiType with the xsiType of the assessor as stored on XNAT.
get_status()

Get the procstatus of an assessor

Returns:The string of the procstatus of the assessor. DOES_NOT_EXIST if the assessor does not exist
get_statuses()

Get the procstatus, qcstatus, and job id of an assessor

Returns:Serially ordered strings of the assessor procstatus, qcstatus, then jobid.
get_walltime()

Get the amount of walltime used for a process

Returns:String of how much walltime was used for a process
is_open()
Check to see if a task is still in “Open” status as defined in
OPEN_STATUS_LIST.
Returns:True if the Task is open. False if it is not open
launch(jobdir, job_email=None, job_email_options='a', xnat_host=None, writeonly=False, pbsdir=None, force_no_qsub=False)

Method to launch a job on the grid

Parameters:
  • jobdir – absolute path where the data will be stored on the node
  • job_email – who to email if the job fails
  • job_email_options – grid-specific job email options (e.g., fails, starts, exits etc)
  • xnat_host – set the XNAT_HOST in the PBS job
  • writeonly – write the job files without submitting them
  • pbsdir – folder to store the pbs file
  • force_no_qsub – run the job locally on the computer (serial mode)
Raises:

cluster.ClusterLaunchException if the jobid is 0 or empty as returned by pbs.submit() method

Returns:

True if the job failed

outlog_path()

Method to return the path of outlog file for the job

Returns:A string that is the absolute path to the OUTLOG file.
pbs_path(writeonly=False, pbsdir=None)

Method to return the path of the PBS file for the job

Parameters:
  • writeonly – write the job files without submitting them in TRASH
  • pbsdir – folder to store the pbs file
Returns:

A string that is the absolute path to the PBS file that will be submitted to the scheduler for execution.

ready_flag_exists()

Method to see if the flag file <UPLOAD_DIR>/<ASSESSOR_LABEL>/READY_TO_UPLOAD.txt exists

Returns:True if the file exists. False if the file does not exist.
reproc_processing()

If the procstatus of an assessor is REPROC on XNAT, rerun the assessor.

Returns:None
set_createdate(date_str)

Set the date of the assessor creation to user passed value

Parameters:date_str – String of the date in “%Y-%m-%d” format
Returns:String of today’s date in “%Y-%m-%d” format
set_createdate_today()

Set the date of the assessor creation to today

Returns:String of todays date in “%Y-%m-%d” format
set_jobid(jobid)

Set the job ID of the assessor on XNAT

Parameters:jobid – The ID of the process assigned by the grid scheduler
Returns:None
set_jobnode(jobnode)

Set the value of the the node that the process ran on on the grid

Parameters:jobnode – String identifying the node the job ran on
Returns:None
set_jobstartdate(date_str)
Set the date that the job started on the grid based on user passed
value
Parameters:date_str – Datestring in the format “%Y-%m-%d” to set the job starte date to
Returns:None
set_jobstartdate_today()

Set the date that the job started on the grid to today

Returns:call to set_jobstartdate with today’s date
set_launch(jobid)

Set the date that the job started and its associated ID on XNAT. Additionally, set the procstatus to JOB_RUNNING

Parameters:jobid – The ID of the process assigned by the grid scheduler
Returns:None
set_memused(memused)

Set the amount of memory used for a process

Parameters:memused – String denoting the amount of memory used
Returns:None
set_proc_and_qc_status(procstatus, qcstatus)

Set the procstatus and qcstatus of the assessor

Parameters:
  • procstatus – String to set the procstatus of the assessor to
  • qcstatus – String to set the qcstatus of the assessor to
Returns:

None

set_qcstatus(qcstatus)

Set the qcstatus of the assessor

Parameters:qcstatus – String to set the qcstatus to
Returns:None
set_status(status)

Set the procstatus of an assessor on XNAT

Parameters:status – String to set the procstatus of the assessor to
Returns:None
set_walltime(walltime)

Set the value of walltime used for an assessor on XNAT

Parameters:walltime – String denoting how much time was used running the process.
Returns:None
undo_processing()
Unset the job ID, memory used, walltime, and jobnode information
for the assessor on XNAT
Except:pyxnat.core.errors.DatabaseError when attempting to delete a resource
Returns:None
update_status()

Update the satus of a Task object.

Returns:the “new” status (updated) of the Task.
class dax.task.ClusterTask(assr_label, upload_dir, diskq)

Class Task to generate/manage the assessor with the cluster

batch_path()

Method to return the path of the PBS file for the job

Returns:A string that is the absolute path to the PBS file that will be submitted to the scheduler for execution.
build_commands()

Call the get_cmds method of the class Processor.

Parameters:jobdir – Fully qualified path where the job will run on the node. Note that this is likely to start with /tmp on most grids.
Returns:A string that makes a command line call to a spider with all args.
build_task()

Method to build a job

check_date()

Sets the job created date if the assessor was not made via dax_build

check_job_usage()
The task has now finished, get the amount of memory used, the amount of
walltime used, the jobid of the process, the node the process ran on, and when it started from the scheduler. Set these values locally
Returns:None
check_running()

Check to see if a job specified by the scheduler ID is still running

Parameters:jobid – The ID of the job in question assigned by the scheduler.
Returns:A String of JOB_RUNNING if the job is running or enqueued and JOB_FAILED if the ready flag (see read_flag_exists) does not exist in the assessor label folder in the upload directory.
commands(jobdir)

Call the get_cmds method of the class Processor.

Parameters:jobdir – Fully qualified path where the job will run on the node. Note that this is likely to start with /tmp on most grids.
Returns:A string that makes a command line call to a spider with all args.
get_createdate()

Get the date an assessor was created

Returns:String of the date the assessor was created in “%Y-%m-%d” format
get_job_status()

Get the status of a job given its jobid as assigned by the scheduler

Parameters:jobid – job id assigned by the scheduler
Returns:string from call to cluster.job_status or UNKNOWN.
get_job_usage()
Get the amount of memory used, the amount of walltime used, the jobid
of the process, the node the process ran on, and when it started from the scheduler.
Returns:List of strings. Memory used, walltime used, jobid, node used, and start date
get_jobid()

Get the jobid of an assessor as stored in local cache

Returns:string of the jobid
get_jobnode()

Gets the node that a process ran on

Returns:String identifying the node that a job ran on
get_jobstartdate()

Get the date that the job started

Returns:String of the date that the job started in “%Y-%m-%d” format
get_memused()

Get the amount of memory used for a process

Returns:String of how much memory was used
get_processor_name()

Get the name of the Processor for the Task.

Returns:String of the Processor name.
get_processor_version()

Get the version of the Processor.

Returns:String of the Processor version.
get_qcstatus()

Get the qcstatus

get_status()

Get the procstatus

Returns:The string of the procstatus
get_statuses()

Get the procstatus, qcstatus, and job id of an assessor

get_walltime()

Get the amount of walltime used for a process

Returns:String of how much walltime was used for a process
is_open()
Check to see if a task is still in “Open” status as defined in
OPEN_STATUS_LIST.
Returns:True if the Task is open. False if it is not open
launch(force_no_qsub=False)

Method to launch a job on the grid

Raises:cluster.ClusterLaunchException if the jobid is 0 or empty as returned by pbs.submit() method
Returns:True if the job failed
outlog_path()

Method to return the path of outlog file for the job

Returns:A string that is the absolute path to the OUTLOG file.
reproc_processing()
Raises:NotImplementedError
Returns:None
set_createdate(date_str)

Set the date of the assessor creation to user passed value

Parameters:date_str – String of the date in “%Y-%m-%d” format
Returns:String of today’s date in “%Y-%m-%d” format
set_createdate_today()

Set the date of the assessor creation to today

Returns:String of todays date in “%Y-%m-%d” format
set_jobid(jobid)

Set the job ID of the assessor

Parameters:jobid – The ID of the process assigned by the grid scheduler
Returns:None
set_jobnode(jobnode)

Set the value of the the node that the process ran on on the grid

Parameters:jobnode – String identifying the node the job ran on
Returns:None
set_jobstartdate(date_str)
Set the date that the job started on the grid based on user passed
value
Parameters:date_str – Datestring in the format “%Y-%m-%d” to set the job starte date to
Returns:None
set_launch(jobid)

Set the date that the job started and its associated ID. Additionally, set the procstatus to JOB_RUNNING

Parameters:jobid – The ID of the process assigned by the grid scheduler
Returns:None
set_memused(memused)

Set the amount of memory used for a process

Parameters:memused – String denoting the amount of memory used
Returns:None
set_proc_and_qc_status(procstatus, qcstatus)

Set the procstatus and qcstatus of the assessor

set_qcstatus(qcstatus)

Set the qcstatus of the assessor

Parameters:qcstatus – String to set the qcstatus to
Returns:None
set_status(status)

Set the procstatus of an assessor on XNAT

Parameters:status – String to set the procstatus of the assessor to
Returns:None
set_walltime(walltime)

Set the value of walltime used for an assessor

Parameters:walltime – String denoting how much time was used running the process.
Returns:None
undo_processing()
Unset the job ID, memory used, walltime, and jobnode information
for the assessor on XNAT
Except:pyxnat.core.errors.DatabaseError when attempting to delete a resource
Returns:None
update_status()

Update the status of a Cluster Task object.

Returns:the “new” status (updated) of the Task.
upload_outlog_dir()

Method to return the path of outlog file for the job

Returns:A string that is the absolute path to the OUTLOG file.
upload_pbs_dir()

Method to return the path of dir for the PBS

Returns:A string that is the directory path for the PBS dir
class dax.task.XnatTask(processor, assessor, upload_dir, diskq)

Class Task to generate/manage the assessor with the cluster

batch_path()

Method to return the path of the PBS file for the job

Returns:A string that is the absolute path to the PBS file that will be submitted to the scheduler for execution.
build_commands(assr, jobdir)

Call the build_cmds method of the class Processor.

Parameters:jobdir – Fully qualified path where the job will run on the node. Note that this is likely to start with /tmp on most grids.
Returns:A string that makes a command line call to a spider with all args.
build_task(assr, jobdir, job_email=None, job_email_options='a', xnat_host=None)

Method to build a job

check_job_usage()
The task has now finished, get the amount of memory used, the amount of
walltime used, the jobid of the process, the node the process ran on, and when it started from the scheduler. Set these values on XNAT
Returns:None
check_running()

Check to see if a job specified by the scheduler ID is still running

Parameters:jobid – The ID of the job in question assigned by the scheduler.
Returns:A String of JOB_RUNNING if the job is running or enqueued and JOB_FAILED if the ready flag (see read_flag_exists) does not exist in the assessor label folder in the upload directory.
get_job_status()

Get the status of a job given its jobid as assigned by the scheduler

Parameters:jobid – job id assigned by the scheduler
Returns:string from call to cluster.job_status or UNKNOWN.
launch()

Method to launch a job on the grid

outlog_path()

Method to return the path of outlog file for the job

Returns:A string that is the absolute path to the OUTLOG file.
set_launch(jobid)

Set the date that the job started and its associated ID on XNAT. Additionally, set the procstatus to JOB_RUNNING

Parameters:jobid – The ID of the process assigned by the grid scheduler
Returns:None
update_status()

Update the satus of an XNAT Task object.

Returns:the “new” status (updated) of the Task.

dax.spiders – Spider class

Title: spiders.py Author: Benjamin Yvernault contact: b.yvernault@ucl.ac.uk Purpose:

Spider base class and class for Scan and Session spider Spider name must be: Spider_[name]_v[version].py Utils for spiders
class dax.spiders.Spider(spider_path, jobdir, xnat_project, xnat_subject, xnat_session, xnat_host=None, xnat_user=None, xnat_pass=None, suffix='', subdir=True, skip_finish=False)

Base class for spider

check_executable(executable, name, version_opt='--version')

Method to check the executable.

Parameters:
  • executable – executable path
  • name – name of Executable
Returns:

Complete path to the executable

define_spider_process_handler()
Define the SpiderProcessHandler so the file(s) and PDF are checked for
existence and uploaded to the upload_dir accordingly.

Implemented in derived classes.

Raises:NotImplementedError() if not overridden.
Returns:None
download(obj_label, resource, folder)
Return a python list of the files downloaded for the scan’s resource
example:
download(scan_id, “DICOM”, “/Users/test”)
or
download(assessor_label, “DATA”, “/Users/test”)
Parameters:
  • obj_label – xnat object label (scan ID or assessor label)
  • resource – folder name under the xnat object
  • folder – download directory
Returns:

python list of files downloaded

download_inputs()

Download inputs data from XNAT define in self.inputs.

self.inputs = list of data dictionary with keys define below keys:

‘type’: ‘scan’ or ‘assessor’ or ‘subject’ or ‘project’ or ‘session’ ‘label’: label on XNAT (not needed for session/subject/project) ‘resource’: name of resource to download or list of resources ‘dir’: directory to download files into (optional)
  • for assessor only if not giving the label but just proctype ‘scan’: id of the scan for the assessor (if None, sessionAssessor)
self.data = list of dictionary with keys define below:
‘label’: label on XNAT ‘files’: list of files downloaded

set self.data, a python list of the data downloaded.

end()

Finish the script by sending the end of script flag and cleaning folder

Parameters:jobdir – directory for the spider
Returns:None
finish()

Method to copy the results in the Spider Results folder dax.RESULTS_DIR Implemented in derived class objects.

Raises:NotImplementedError if not overriden by user
Returns:None
static get_data_dict(otype, label, resource, directory, scan=None)

Create a data_dict for self.inputs from user need.

get_exe_version(executable, version_opt='--version')

Method to check the executable.

Parameters:
  • executable – executable to run
  • version_opt – options to get the version of the executable
Returns:

version

get_xnat_dict(data_dict, resource)

Return a OrderedDict dictionary with XNAT information.

keys:
project subject experiment scan resource assessor out/resource (for assessor)
has_spider_handler()
Check to see that the SpiderProcessHandler is defined. If it is not,
call define_spider_process_handler
Returns:None
merge_pdf_pages(pdf_pages, pdf_final)

Concatenate all pdf pages in the list into a final pdf.

See function at the end of the file.

plot_images_page(pdf_path, page_index, nii_images, title, image_labels, slices=None, cmap='gray', vmins=None, vmaxs=None, volume_ind=None, orient='ax')

Plot list of images (3D-4D) on a figure (PDF page).

See function at the end of the file.

plot_stats_page(pdf_path, page_index, stats_dict, title, tables_number=3, columns_header=['Header', 'Value'], limit_size_text_column1=30, limit_size_text_column2=10)

Generate pdf report of stats information from a csv/txt.

See function at the end of the file.

pre_run()

Pre-Run method to download and organise inputs for the pipeline Implemented in derived class objects.

Raises:NotImplementedError if not overridden.
Returns:None
print_args(argument_parse)

print arguments given to the Spider

Parameters:argument_parse – argument parser
Returns:None
print_end()

Last print statement to give the time and date at the end of the spider

Returns:None
print_err(err_message)

Print error message using time writer

Parameters:err_message – error message displayed for the user
Returns:None
print_info(author, email)

Print information on the spider using time writer

Parameters:
  • author – author of the spider
  • email – email of the author
Returns:

None

print_init(argument_parse, author, email)

Print a message to display information on the init parameters, author, email, and arguments using time writer

Parameters:
  • argument_parse – argument parser
  • author – author of the spider
  • email – email of the author
Returns:

None

print_msg(message)

Print message using time writer

Parameters:message – string displayed for the user
Returns:None
run()

Runs the “core” or “image processing process” of the pipeline Implemented in derived class objects.

Raises:NotImplementedError if not overridden.
Returns:None
run_cmd_args()

Run a command line via os.system() with arguments set in self.cmd_args

cmd_args is a dictionary:

exe: executable to use (matlab, python, sh) template: string defining the command line with argument args: dictionary with:

key = argument value = value to set

filename: name for the file if written into a file (optional)

Returns:True if succeeded, False otherwise
run_system_cmd(cmd)

Run system command line via os.system()

Parameters:cmd – command to run
Returns:True if succeeded, False otherwise
select_obj(intf, obj_label, resource)

Select scan or assessor resource

Parameters:
  • obj_label – xnat object label (scan ID or assessor label)
  • resource – folder name under the xnat object

return pyxnat object

static select_str(xnat_dict)

Return string for pyxnat to select object from python dict

Parameters:tmp_dict

python dictionary with xnat information keys = [“project”, “subject”, “experiement”, “scan”, “resource”]

or
keys = [“project”, “subject”, “experiement”, “assessor”,
”out/resource”]
Return string:string path to select pyxnat object
upload(fpath, resource)

Upload files to the queue on the cluster to be upload to XNAT by DAX E.g: spider.upload(“/Users/DATA/”, “DATA”)

spider.upload(“/Users/stats_dir/statistical_measures.txt”, “STATS”)
Parameters:
  • fpath – path to the folder/file to be uploaded
  • resource – folder name to upload to on the assessor
Raises:

ValueError if the file to upload does not exist

Returns:

None

upload_dict(files_dict)
upload files to the queue on the cluster to be upload to XNAT by DAX
following the files python dictionary: {resource_name : fpath}
E.g: fdict = {“DATA” : “/Users/DATA/”, “PDF”: “/Users/PDF/report.pdf”}
spider.upload_dict(fdict)
Parameters:files_dict – python dictionary containing the pair resource/fpath
Raises:ValueError if the filepath is not a string or a list
Returns:None
class dax.spiders.ScanSpider(spider_path, jobdir, xnat_project, xnat_subject, xnat_session, xnat_scan, xnat_host=None, xnat_user=None, xnat_pass=None, suffix='', subdir=True, skip_finish=False)

Derived class for scan-spider

define_spider_process_handler()
Define the SpiderProcessHandler for the end of scan spider
using the init attributes about XNAT
Returns:None
finish()

Method to copy the results in the Spider Results folder dax.RESULTS_DIR Implemented in derived class objects.

Raises:NotImplementedError if not overriden by user
Returns:None
pre_run()

Pre-Run method to download and organise inputs for the pipeline Implemented in derived class objects.

Raises:NotImplementedError if not overridden.
Returns:None
run()

Runs the “core” or “image processing process” of the pipeline Implemented in derived class objects.

Raises:NotImplementedError if not overridden.
Returns:None
class dax.spiders.SessionSpider(spider_path, jobdir, xnat_project, xnat_subject, xnat_session, xnat_host=None, xnat_user=None, xnat_pass=None, suffix='', subdir=True, skip_finish=False)

Derived class for session-spider

define_spider_process_handler()
Define the SpiderProcessHandler for the end of session spider
using the init attributes about XNAT
Returns:None
finish()

Method to copy the results in the Spider Results folder dax.RESULTS_DIR Implemented in derived class objects.

Raises:NotImplementedError if not overriden by user
Returns:None
pre_run()

Pre-Run method to download and organise inputs for the pipeline Implemented in derived class objects.

Raises:NotImplementedError if not overridden.
Returns:None
run()

Runs the “core” or “image processing process” of the pipeline Implemented in derived class objects.

Raises:NotImplementedError if not overridden.
Returns:None
class dax.spiders.AutoSpider(name, params, outputs, template, version=None, exe_lang=None)

Class for Autospider

copy_input(src, input_name)

Copy inputs or download from XNAT.

copy_inputs()

Copy the inputs data for AutoSpider.

copy_local_input(src, input_name)

Copy local inputs.

copy_xnat_input(src, input_name)

Copy xnat inputs.

download_xnat_file(src, dst)

Download XNAT specific file.

download_xnat_resource(src, dst)

Download XNAT complete resource.

end()

Finish the script by sending the end of script flag and cleaning folder :return: None

finish()

finish method to copy the results.

get_argparser()

Get argparser for the AutoSpider.

go()

Main method for AutoSpider.

is_xnat_uri(uri)

Check if uri is xnat or local.

pre_run()

Pre-Run method to download and organise inputs for the pipeline Implemented in derived class objects.

print_args(argument_parse)

print arguments given to the Spider

Parameters:argument_parse – argument parser
Returns:None
print_end()

Last print statement

Returns:None
run()

Run method to execute the template for AutoSpider.

class dax.spiders.TimedWriter(name=None, use_date=False)

Class to automatically write timed output message

Args:
name - Names to write with output (default=None)
Examples:
>>>a = Time_Writer() >>>a(“this is a test”) [00d 00h 00m 00s] this is a test >>>sleep(60) >>>a(“this is a test”) [00d 00h 01m 00s] this is a test

Written by Andrew Plassard (Vanderbilt)

print_stderr_message(text)

Prints a timed message to stderr

Parameters:text – The text to print
Returns:None
print_timed_message(text, pipe=<open file '<stdout>', mode 'w'>)

Prints a timed message

Parameters:
  • text – text to print
  • pipe – pipe to write to. defaults to sys.stdout
Returns:

None

dax.processors – Processor class

Processor class define for Scan and Session.

class dax.processors.Processor(walltime_str, memreq_mb, spider_path, version=None, ppn=1, env=None, suffix_proc='', xsitype='proc:genProcData', job_template=None)

Base class for processor

build_cmds(cobj, dir)

Build the commands that will go in the PBS/SLURM script :raises: NotImplementedError if not overridden from base class. :return: None

default_settings_spider(spider_path)

Get the default spider version and name

Parameters:spider_path – Fully qualified path and file of the spider
Returns:None
get_assessor_input_types()

Enumerate the assessor input types for this. The default implementation returns an empty collection; override this method if you are inheriting from a non-yaml processor. :return: a list of input assessor types

get_proctype()

Return the processor name for this processor. Override this method if you are inheriting from a non-yaml processor. :return: the name of the processor type

has_inputs()

Check to see if the spider has all the inputs necessary to run.

Raises:NotImplementedError if user does not override
Returns:None
set_spider_settings(spider_path, version)

Method to set the spider version, path, and name from filepath

Parameters:
  • spider_path – Fully qualified path and file of the spider
  • version – version of the spider
Returns:

None

should_run()

Responsible for determining if the assessor should shouw up in session.

Raises:NotImplementedError if not overridden.
Returns:None
class dax.processors.ScanProcessor(scan_types, walltime_str, memreq_mb, spider_path, version=None, ppn=1, env=None, suffix_proc='', full_regex=False, job_template=None)

Scan Processor class for processor on a scan on XNAT

get_assessor(cscan)

Returns the assessor object depending on cscan and the assessor label.

Parameters:cscan – CachedImageScan object from XnatUtils
Returns:String of the assessor label
get_assessor_name(cscan)

Returns the label of the assessor

Parameters:cscan – CachedImageScan object from XnatUtils
Returns:String of the assessor label
get_task(cscan, upload_dir)

Get the Task object

Parameters:
  • cscan – CachedImageScan object from XnatUtils
  • upload_dir – the directory to put the processed data when the process is done
Returns:

Task object

has_inputs()
Method to check and see that the process has all of the inputs
that it needs to run.
Raises:NotImplementedError if not overridden.
Returns:None
should_run(scan_dict)

Method to see if the assessor should appear in the session.

Parameters:scan_dict – Dictionary of information about the scan
Returns:True if it should run, false if it shouldn’t
class dax.processors.SessionProcessor(walltime_str, memreq_mb, spider_path, version=None, ppn=1, env=None, suffix_proc='', job_template=None)

Session Processor class for processor on a session on XNAT

get_assessor(csess)

Returns the assessor object depending on csess and the assessor label.

Parameters:csess – CachedImageSession object from XnatUtils
Returns:String of the assessor label
get_assessor_name(csess)

Returns the label of the assessor

Parameters:csess – CachedImageSession object from XnatUtils
Returns:String of the assessor label
get_task(csess, upload_dir)

Return the Task object

Parameters:
  • csess – CachedImageSession from XnatUtils
  • upload_dir – directory to put the data after run on the node
Returns:

Task object of the assessor

has_inputs()

Check to see that the session has the required inputs to run.

Raises:NotImplementedError if not overriden from base class.
Returns:None
should_run(session_dict)
By definition, this should always run, so it just returns true
with no checks
Parameters:session_dict – Dictionary of session information for XnatUtils.list_experiments()
Returns:True
class dax.processors.AutoProcessor(xnat, yaml_source, user_inputs=None)

Auto Processor class for AutoSpider using YAML files

get_assessor_input_types()

Enumerate the assessor input types for this. The default implementation returns an empty collection; override this method if you are inheriting from a non-yaml processor. :return: a list of input assessor types

get_cmds(assr, jobdir)

Method to generate the spider command for cluster job.

Parameters:
  • assessor – pyxnat assessor object
  • jobdir – jobdir where the job’s output will be generated
Returns:

command to execute the spider in the job script

get_proctype()

Return the processor name for this processor. Override this method if you are inheriting from a non-yaml processor. :return: the name of the processor type

has_inputs(cobj)

Method to check the inputs.

By definition:
status = 0 -> NEED_INPUTS, for session asr inputs and resources status = 1 -> NEED_TO_RUN status = -1 -> NO_DATA, for scan primary input isn’t usable qcstatus needs a value only when -1 or 0.

You need to set qcstatus to a short string that explain why it’s no ready to run. e.g: No NIFTI

Parameters:cobj – cached object define in dax.XnatUtils (Session or Scan) (see XnatUtils in dax for information)
Returns:status, qcstatus
parse_session(csess, sessions)

Method to run the processor parser on this session, in order to calculate the pattern matches for this processor and the sessions provided :param csess: the active session. For non-longitudinal studies, this is the session that the pattern matching is performed on. For longitudinal studies, this is the ‘current’ session from which all prior sessions are numbered for the purposes of pattern matching :param sessions: the full, time-ordered list of sessions that should be considered for longitudinal studies. :return: None

should_run(obj_dict)

Method to see if the assessor should appear in the session.

Parameters:obj_dict – Dictionary of information about the scan or sesion
Returns:True if it should run, false if it shouldn’t

dax.log – Logging utility

dax.log.setup_critical_logger(name, logfile)

Sets up the critical logger

Parameters:
  • name – Name of the logger
  • logfile – file to store the log to. sys.stdout if no file define
Returns:

logger object

dax.log.setup_debug_logger(name, logfile)

Sets up the debug logger

Parameters:
  • name – Name of the logger
  • logfile – file to store the log to. sys.stdout if no file define
Returns:

logger object

dax.log.setup_error_logger(name, logfile)

Sets up the error logger

Parameters:
  • name – Name of the logger
  • logfile – file to store the log to. sys.stdout if no file define
Returns:

logger object

dax.log.setup_info_logger(name, logfile)

Sets up the info logger

Parameters:
  • name – Name of the logger
  • logfile – file to store the log to. sys.stdout if no file define
Returns:

logger object

dax.log.setup_warning_logger(name, logfile)

Sets up the warning logger

Parameters:
  • name – Name of the logger
  • logfile – file to store the log to. sys.stdout if no file define
Returns:

logger object

dax.bin – Responsible for launching, building and updating a Task

File containing functions called by dax executables

dax.bin.build(settings_path, logfile, debug, projects=None, sessions=None, mod_delta=None, proj_lastrun=None)
Method that is responsible for running all modules and putting assessors
into the database
Parameters:
  • settings_path – Path to the project settings file
  • logfile – Full file of the file used to log to
  • debug – Should debug mode be used
  • projects – Project(s) that need to be built
  • sessions – Session(s) that need to be built
Returns:

None

dax.bin.check_default_keys(yaml_file, doc)

Static method to raise error if key not found in dictionary from yaml file. :param yaml_file: path to yaml file defining the processor :param doc: doc dictionary extracted from the yaml file

dax.bin.launch_jobs(settings_path, logfile, debug, projects=None, sessions=None, writeonly=False, pbsdir=None, force_no_qsub=False)

Method to launch jobs on the grid

Parameters:
  • settings_path – Path to the project settings file
  • logfile – Full file of the file used to log to
  • debug – Should debug mode be used
  • projects – Project(s) that need to be launched
  • sessions – Session(s) that need to be updated
  • writeonly – write the job files without submitting them
  • pbsdir – folder to store the pbs file
  • force_no_qsub – run the job locally on the computer (serial mode)
Returns:

None

dax.bin.load_from_file(filepath, args, logger, singularity_imagedir=None)

Check if a file exists and if it’s a python file :param filepath: path to the file to test :return: True the file pass the test, False otherwise

dax.bin.pi_from_project(project)

Get the last name of PI who owns the project on XNAT

Parameters:project – String of the ID of project on XNAT.
Returns:String of the PIs last name
dax.bin.raise_yaml_error_if_no_key(doc, yaml_file, key)

Method to raise an execption if the key is not in the dict :param doc: dict to check :param yaml_file: YAMLfile path :param key: key to search

dax.bin.read_yaml_settings(yaml_file, logger)

Method to read the settings yaml file and generate the launcher object.

Parameters:yaml_file – path to yaml file defining the settings
Returns:launcher object
dax.bin.set_logger(logfile, debug)

Set the logging depth

Parameters:
  • logfile – File to log output to
  • debug – Should debug depth be used?
Returns:

logger object

dax.bin.update_tasks(settings_path, logfile, debug, projects=None, sessions=None)

Method that is responsible for updating a Task.

Parameters:
  • settings_path – Path to the project settings file
  • logfile – Full file of the file used to log to
  • debug – Should debug mode be used
  • projects – Project(s) that need to be launched
  • sessions – Session(s) that need to be updated
Returns:

None

dax.XnatUtils – Collection of utilities for upload/download and general access

XnatUtils contains useful function to interface with XNAT using Pyxnat.

The functions are several categories:

1) Class Specific to XNAT and Spiders: InterfaceTemp to create an interface with XNAT using a tempfolder AssessorHandler to handle assessor label string and access object SpiderProcessHandler to handle results at the end of any spider

  1. Methods to query XNAT database and get XNAT object :
  2. Methods to access/check objects on XNAT
  3. Methods to Download / Upload data to XNAT
  4. Other Methods
  5. Cached Class for DAX
  6. Old download functions still used in some spiders
class dax.XnatUtils.InterfaceTemp(xnat_host=None, xnat_user=None, xnat_pass=None, temp_dir=None)
Extends the pyxnat.Interface class to make a temporary directory, write the
cache to it and then blow it away on the Interface.disconnect call() NOTE: This is deprecated in pyxnat 1.0.0.0

Using netrc to get username password if not given.

authenticate()

Authenticate to XNAT.

Connect to XNAT and try to Disconnect the JSESSION before reconnecting. Raise XnatAuthentificationError if it failes.

Returns:True or False
connect()

Connect to XNAT.

disconnect()

Disconnect the JSESSION and blow away the cache.

Returns:None
get_project_assessors(projectid)

List all the assessors that you have access to based on passed project.

Parameters:projectid – ID of a project on XNAT
Returns:List of all the assessors for the project
get_project_scans(project_id, include_shared=True)

List all the scans that you have access to based on passed project.

Parameters:
  • intf – pyxnat.Interface object
  • projectid – ID of a project on XNAT
  • include_shared – include the shared data in this project
Returns:

List of all the scans for the project

get_scans(projectid, subjectid, sessionid)
List all the scans that you have access to based on passed
session/subject/project.
Parameters:
  • intf – pyxnat.Interface object
  • projectid – ID of a project on XNAT
  • subjectid – ID/label of a subject
  • sessionid – ID/label of a session
Returns:

List of all the scans

get_session_resources(projectid, subjectid, sessionid)
Gets a list of all of the resources for a session associated to a
subject/project requested by the user
Parameters:
  • intf – pyxnat.Interface object
  • projectid – ID of a project on XNAT
  • subjectid – ID/label of a subject
  • sessionid – ID/label of a session to get resources for
Returns:

List of resources for the session

get_sessions(projectid=None, subjectid=None)
List all the sessions either:
  1. that you have access to
or
  1. in a single project (and single subject) based on kargs
Parameters:
  • intf – pyxnat.Interface object
  • projectid – ID of a project on XNAT
  • subjectid – ID/label of a subject
Returns:

List of sessions

class dax.XnatUtils.AssessorHandler(label)

Class to intelligently deal with the Assessor labels. Make the splitting of the strings easier.

get_proctype()

Get the proctype from the assessor label

Returns:The proctype for the assessor
get_project_id()

Get the project ID from the assessor label

Returns:The XNAT project label
get_scan_id()

Get the scan ID from teh assessor label

Returns:The scan id for the assessor label
get_session_label()

Get the session label from the assessor label

Returns:The XNAT session label
get_subject_label()

Get the subject label from the assessor label

Returns:The XNAT subject label
is_valid()

Check to see if we have a valid assessor label (aka not None)

Returns:True if valid, False if not valid
select_assessor(intf)

Run Interface.select() on the assessor label

Parameters:intf – pyxnat.Interface object
Returns:The pyxnat EObject of the assessor
class dax.XnatUtils.SpiderProcessHandler(script_name, suffix, project=None, subject=None, experiment=None, scan=None, alabel=None, assessor_handler=None, time_writer=None, host=None)

Class to handle the uploading of results for a spider.

add_file(filepath, resource)
Add a file in the assessor in the upload directory based on the
resource name as will be seen on XNAT
Parameters:
  • filepath – Full path to a file to upload
  • resource – The resource name it should appear under in XNAT
Returns:

None

add_folder(folderpath, resource_name=None)

Add a folder to the assessor in the upload directory.

Parameters:
  • folderpath – Full path to the folder to upoad
  • resource_name – Resource name chosen (if different than basename)
Raises:
Returns:

None

add_pdf(filepath)

Add the PDF and run ps2pdf on the file if it ends with .ps

Parameters:filepath – Full path to the PDF/PS file
Returns:None
add_snapshot(snapshot)

Add in the snapshots (for quick viewing on XNAT)

Parameters:snapshot – Full path to the snapshot file
Returns:None
clean(directory)

Clean directory if no error and pdf created

Parameters:directory – directory to be cleaned
done()
Create a flag file that the assessor is ready to be uploaded and set
the status as READY_TO_UPLOAD
Returns:None
file_exists(fpath)

Check to see if a file exists

Parameters:fpath – full path to a file to assert it exists
Returns:True if it exists, False if it doesn’t
folder_exists(fpath)

Check to see if a folder exists

Parameters:fpath – Full path to a folder to assert it exists
Returns:True if it exists, False if it doesn’t
print_copying_statement(label, src, dest)

Print a line that data is being copied to the upload directory

Parameters:
  • label – The XNAT resource label
  • src – Source directory or file
  • dest – Destination directory or file
Returns:

None

print_err(msg)

Print error message using time writer if set, print otherwise

Parameters:msg – Message to print
Returns:None
print_msg(msg)

Prints a message using TimedWriter or print

Parameters:msg – Message to print
Returns:None
set_assessor_status(status)

Set the status of the assessor based on passed value

Parameters:status – Value to set the procstatus to
Except:All catchable errors.
Returns:None
set_error()

Set the flag for the error to 1

Returns:None
class dax.XnatUtils.CachedImageSession(intf, proj, subj, sess)

Enumeration for assessors function, to control what assessors are returned

assessors(select=(0, ))

Get a list of CachedImageAssessor objects for the XNAT session

Returns:List of CachedImageAssessor objects for the session.
full_object()

Return a the full pyxnat Session object of this sessions

Returns:pyxnat Session object
get(name)

Get the value of a variable name in the session

Parameters:name – The variable name that you want to get the value of
Returns:The value of the variable or ‘’ if not found.
get_resources()
Return a list of dictionaries that correspond to the information
for each resource
Returns:List of dictionaries
has_shared_project()

Get the project if shared.

Returns:project_shared_id if shared, None otherwise
info()

Get a dictionary of lots of variables that correspond to the session

Returns:Dictionary of variables
label()

Get the label of the session

Returns:String of the session label
resources()

Get a list of CachedResource objects for the session

Returns:List of CachedResource objects for the session
scans()

Get a list of CachedImageScan objects for the XNAT session

Returns:List of CachedImageScan objects for the session.
session()

Get the session associated with this object :return: session asscoiated with this object

class dax.XnatUtils.CachedImageScan(intf, scan_element, parent)

Class to cache the XML information for a scan on XNAT

get(name)

Get the value of a variable associated with a scan.

Parameters:name – Name of the variable to get the value of
Returns:Value of the variable if it exists, or ‘’ otherwise.
get_resources()

Get a list of dictionaries of info for each CachedResource.

Returns:List of dictionaries of infor for each CachedResource.
info()

Get lots of variables assocaited with this scan.

Returns:Dictionary of infomation about the scan.
label()

Get the ID of the scan

Returns:String of the scan ID
parent()

Get the parent of the scan

Returns:XML String of the scan parent
resources()

Get a list of the CachedResource (s) associated with this scan.

Returns:List of the CachedResource (s) associated with this scan.
session()

Get the session associated with this object :return: session asscoiated with this object

class dax.XnatUtils.CachedImageAssessor(intf, assr_element, parent)

Class to cache the XML information for an assessor on XNAT

get(name)

Get the value of a variable associated with the assessor

Parameters:name – Variable name to get the value of
Returns:Value of the variable, otherwise ‘’.
get_in_resources()
Get a list of dictionaries of info for the CachedResource objects
for “in” type
Returns:List of dictionaries of info for the CachedResource objects for “in” type
get_out_resources()
Get a list of dictionaries of info for the CachedResource objects
for “out” type
Returns:List of dictionaries of info for the CachedResource objects for “out” type
get_resources()

Makes a call to get_out_resources.

Returns:List of dictionaries of info for the CachedResource objects for “out” type
in_resources()

Get a list of CachedResource objects for “in” type

Returns:List of CachedResource objects for “in” type
info()

Get a dictionary of information associated with the assessor

Returns:None
label()

Get the label of the assessor

Returns:String of the assessor label
out_resources()

Get a list of CachedResource objects for “out” type

Returns:List of CachedResource objects for “out” type
parent()

Get the parent element of the assessor (session)

Returns:The session element XML string
class dax.XnatUtils.CachedResource(element, parent)

Class to cache resource XML info on XNAT

get(name)

Get the value of a variable associated with the resource

Parameters:name – Variable name to get the value of
Returns:The value of the variable, ‘’ otherwise.
info()

Get a dictionary of information relating to the resource

Returns:dictionary of information about the resource.
label()

Get the label of the resource

Returns:String of the label of the resource
parent()

Get the resource parent XML string

Returns:The resource parent XML string