Source Documentation¶
dax.task
– Task class¶
Task object to generate / manage assessors and cluster.
-
class
dax.task.
Task
(processor, assessor, upload_dir)¶ Class Task to generate/manage the assessor with the cluster
-
check_date
()¶ - Sets the job created date if the assessor was not made through
- dax_build
Returns: Returns if get_createdate() is != ‘’, sets date otherwise
-
check_job_usage
()¶ - The task has now finished, get the amount of memory used, the amount of
- walltime used, the jobid of the process, the node the process ran on, and when it started from the scheduler. Set these values on XNAT
Returns: None
-
check_running
(jobid=None)¶ Check to see if a job specified by the scheduler ID is still running
Parameters: jobid – The ID of the job in question assigned by the scheduler. Returns: A String of JOB_RUNNING if the job is running or enqueued and JOB_FAILED if the ready flag (see read_flag_exists) does not exist in the assessor label folder in the upload directory.
-
commands
(jobdir)¶ Call the get_cmds method of the class Processor.
Parameters: jobdir – Fully qualified path where the job will run on the node. Note that this is likely to start with /tmp on most grids. Returns: A string that makes a command line call to a spider with all args.
-
get_createdate
()¶ Get the date an assessor was created
Returns: String of the date the assessor was created in “%Y-%m-%d” format
-
get_job_status
(jobid=None)¶ Get the status of a job given its jobid as assigned by the scheduler
Parameters: jobid – job id assigned by the scheduler Returns: string from call to cluster.job_status or UNKNOWN.
-
get_job_usage
()¶ - Get the amount of memory used, the amount of walltime used, the jobid
- of the process, the node the process ran on, and when it started from the scheduler.
Returns: List of strings. Memory used, walltime used, jobid, node used, and start date
-
get_jobid
()¶ Get the jobid of an assessor as stored on XNAT
Returns: string of the jobid
-
get_jobnode
()¶ Gets the node that a process ran on
Returns: String identifying the node that a job ran on
-
get_jobstartdate
()¶ Get the date that the job started
Returns: String of the date that the job started in “%Y-%m-%d” format
-
get_memused
()¶ Get the amount of memory used for a process
Returns: String of how much memory was used
-
get_processor_name
()¶ Get the name of the Processor for the Task.
Returns: String of the Processor name.
-
get_processor_version
()¶ Get the version of the Processor.
Returns: String of the Processor version.
-
get_qcstatus
()¶ Get the qcstatus of the assessor
Returns: A string of the qcstatus for the assessor if it exists. If it does not, it returns DOES_NOT_EXIST. The else case returns an UNKNOWN xsiType with the xsiType of the assessor as stored on XNAT.
-
get_status
()¶ Get the procstatus of an assessor
Returns: The string of the procstatus of the assessor. DOES_NOT_EXIST if the assessor does not exist
-
get_statuses
()¶ Get the procstatus, qcstatus, and job id of an assessor
Returns: Serially ordered strings of the assessor procstatus, qcstatus, then jobid.
-
get_walltime
()¶ Get the amount of walltime used for a process
Returns: String of how much walltime was used for a process
-
is_open
()¶ - Check to see if a task is still in “Open” status as defined in
- OPEN_STATUS_LIST.
Returns: True if the Task is open. False if it is not open
-
launch
(jobdir, job_email=None, job_email_options='a', xnat_host=None, writeonly=False, pbsdir=None, force_no_qsub=False)¶ Method to launch a job on the grid
Parameters: - jobdir – absolute path where the data will be stored on the node
- job_email – who to email if the job fails
- job_email_options – grid-specific job email options (e.g., fails, starts, exits etc)
- xnat_host – set the XNAT_HOST in the PBS job
- writeonly – write the job files without submitting them
- pbsdir – folder to store the pbs file
- force_no_qsub – run the job locally on the computer (serial mode)
Raises: cluster.ClusterLaunchException if the jobid is 0 or empty as returned by pbs.submit() method
Returns: True if the job failed
-
outlog_path
()¶ Method to return the path of outlog file for the job
Returns: A string that is the absolute path to the OUTLOG file.
-
pbs_path
(writeonly=False, pbsdir=None)¶ Method to return the path of the PBS file for the job
Parameters: - writeonly – write the job files without submitting them in TRASH
- pbsdir – folder to store the pbs file
Returns: A string that is the absolute path to the PBS file that will be submitted to the scheduler for execution.
-
ready_flag_exists
()¶ Method to see if the flag file <UPLOAD_DIR>/<ASSESSOR_LABEL>/READY_TO_UPLOAD.txt exists
Returns: True if the file exists. False if the file does not exist.
-
reproc_processing
()¶ If the procstatus of an assessor is REPROC on XNAT, rerun the assessor.
Returns: None
-
set_createdate
(date_str)¶ Set the date of the assessor creation to user passed value
Parameters: date_str – String of the date in “%Y-%m-%d” format Returns: String of today’s date in “%Y-%m-%d” format
-
set_createdate_today
()¶ Set the date of the assessor creation to today
Returns: String of todays date in “%Y-%m-%d” format
-
set_jobid
(jobid)¶ Set the job ID of the assessor on XNAT
Parameters: jobid – The ID of the process assigned by the grid scheduler Returns: None
-
set_jobnode
(jobnode)¶ Set the value of the the node that the process ran on on the grid
Parameters: jobnode – String identifying the node the job ran on Returns: None
-
set_jobstartdate
(date_str)¶ - Set the date that the job started on the grid based on user passed
- value
Parameters: date_str – Datestring in the format “%Y-%m-%d” to set the job starte date to Returns: None
-
set_jobstartdate_today
()¶ Set the date that the job started on the grid to today
Returns: call to set_jobstartdate with today’s date
-
set_launch
(jobid)¶ Set the date that the job started and its associated ID on XNAT. Additionally, set the procstatus to JOB_RUNNING
Parameters: jobid – The ID of the process assigned by the grid scheduler Returns: None
-
set_memused
(memused)¶ Set the amount of memory used for a process
Parameters: memused – String denoting the amount of memory used Returns: None
-
set_proc_and_qc_status
(procstatus, qcstatus)¶ Set the procstatus and qcstatus of the assessor
Parameters: - procstatus – String to set the procstatus of the assessor to
- qcstatus – String to set the qcstatus of the assessor to
Returns: None
-
set_qcstatus
(qcstatus)¶ Set the qcstatus of the assessor
Parameters: qcstatus – String to set the qcstatus to Returns: None
-
set_status
(status)¶ Set the procstatus of an assessor on XNAT
Parameters: status – String to set the procstatus of the assessor to Returns: None
-
set_walltime
(walltime)¶ Set the value of walltime used for an assessor on XNAT
Parameters: walltime – String denoting how much time was used running the process. Returns: None
-
undo_processing
()¶ - Unset the job ID, memory used, walltime, and jobnode information
- for the assessor on XNAT
Except: pyxnat.core.errors.DatabaseError when attempting to delete a resource Returns: None
-
update_status
()¶ Update the satus of a Task object.
Returns: the “new” status (updated) of the Task.
-
-
class
dax.task.
ClusterTask
(assr_label, upload_dir, diskq)¶ Class Task to generate/manage the assessor with the cluster
-
batch_path
()¶ Method to return the path of the PBS file for the job
Returns: A string that is the absolute path to the PBS file that will be submitted to the scheduler for execution.
-
build_commands
()¶ Call the get_cmds method of the class Processor.
Parameters: jobdir – Fully qualified path where the job will run on the node. Note that this is likely to start with /tmp on most grids. Returns: A string that makes a command line call to a spider with all args.
-
build_task
()¶ Method to build a job
-
check_date
()¶ Sets the job created date if the assessor was not made via dax_build
-
check_job_usage
()¶ - The task has now finished, get the amount of memory used, the amount of
- walltime used, the jobid of the process, the node the process ran on, and when it started from the scheduler. Set these values locally
Returns: None
-
check_running
()¶ Check to see if a job specified by the scheduler ID is still running
Parameters: jobid – The ID of the job in question assigned by the scheduler. Returns: A String of JOB_RUNNING if the job is running or enqueued and JOB_FAILED if the ready flag (see read_flag_exists) does not exist in the assessor label folder in the upload directory.
-
commands
(jobdir)¶ Call the get_cmds method of the class Processor.
Parameters: jobdir – Fully qualified path where the job will run on the node. Note that this is likely to start with /tmp on most grids. Returns: A string that makes a command line call to a spider with all args.
-
get_createdate
()¶ Get the date an assessor was created
Returns: String of the date the assessor was created in “%Y-%m-%d” format
-
get_job_status
()¶ Get the status of a job given its jobid as assigned by the scheduler
Parameters: jobid – job id assigned by the scheduler Returns: string from call to cluster.job_status or UNKNOWN.
-
get_job_usage
()¶ - Get the amount of memory used, the amount of walltime used, the jobid
- of the process, the node the process ran on, and when it started from the scheduler.
Returns: List of strings. Memory used, walltime used, jobid, node used, and start date
-
get_jobid
()¶ Get the jobid of an assessor as stored in local cache
Returns: string of the jobid
-
get_jobnode
()¶ Gets the node that a process ran on
Returns: String identifying the node that a job ran on
-
get_jobstartdate
()¶ Get the date that the job started
Returns: String of the date that the job started in “%Y-%m-%d” format
-
get_memused
()¶ Get the amount of memory used for a process
Returns: String of how much memory was used
-
get_processor_name
()¶ Get the name of the Processor for the Task.
Returns: String of the Processor name.
-
get_processor_version
()¶ Get the version of the Processor.
Returns: String of the Processor version.
-
get_qcstatus
()¶ Get the qcstatus
-
get_status
()¶ Get the procstatus
Returns: The string of the procstatus
-
get_statuses
()¶ Get the procstatus, qcstatus, and job id of an assessor
-
get_walltime
()¶ Get the amount of walltime used for a process
Returns: String of how much walltime was used for a process
-
is_open
()¶ - Check to see if a task is still in “Open” status as defined in
- OPEN_STATUS_LIST.
Returns: True if the Task is open. False if it is not open
-
launch
(force_no_qsub=False)¶ Method to launch a job on the grid
Raises: cluster.ClusterLaunchException if the jobid is 0 or empty as returned by pbs.submit() method Returns: True if the job failed
-
outlog_path
()¶ Method to return the path of outlog file for the job
Returns: A string that is the absolute path to the OUTLOG file.
-
reproc_processing
()¶ Raises: NotImplementedError Returns: None
-
set_createdate
(date_str)¶ Set the date of the assessor creation to user passed value
Parameters: date_str – String of the date in “%Y-%m-%d” format Returns: String of today’s date in “%Y-%m-%d” format
-
set_createdate_today
()¶ Set the date of the assessor creation to today
Returns: String of todays date in “%Y-%m-%d” format
-
set_jobid
(jobid)¶ Set the job ID of the assessor
Parameters: jobid – The ID of the process assigned by the grid scheduler Returns: None
-
set_jobnode
(jobnode)¶ Set the value of the the node that the process ran on on the grid
Parameters: jobnode – String identifying the node the job ran on Returns: None
-
set_jobstartdate
(date_str)¶ - Set the date that the job started on the grid based on user passed
- value
Parameters: date_str – Datestring in the format “%Y-%m-%d” to set the job starte date to Returns: None
-
set_launch
(jobid)¶ Set the date that the job started and its associated ID. Additionally, set the procstatus to JOB_RUNNING
Parameters: jobid – The ID of the process assigned by the grid scheduler Returns: None
-
set_memused
(memused)¶ Set the amount of memory used for a process
Parameters: memused – String denoting the amount of memory used Returns: None
-
set_proc_and_qc_status
(procstatus, qcstatus)¶ Set the procstatus and qcstatus of the assessor
-
set_qcstatus
(qcstatus)¶ Set the qcstatus of the assessor
Parameters: qcstatus – String to set the qcstatus to Returns: None
-
set_status
(status)¶ Set the procstatus of an assessor on XNAT
Parameters: status – String to set the procstatus of the assessor to Returns: None
-
set_walltime
(walltime)¶ Set the value of walltime used for an assessor
Parameters: walltime – String denoting how much time was used running the process. Returns: None
-
undo_processing
()¶ - Unset the job ID, memory used, walltime, and jobnode information
- for the assessor on XNAT
Except: pyxnat.core.errors.DatabaseError when attempting to delete a resource Returns: None
-
update_status
()¶ Update the status of a Cluster Task object.
Returns: the “new” status (updated) of the Task.
-
upload_outlog_dir
()¶ Method to return the path of outlog file for the job
Returns: A string that is the absolute path to the OUTLOG file.
-
upload_pbs_dir
()¶ Method to return the path of dir for the PBS
Returns: A string that is the directory path for the PBS dir
-
-
class
dax.task.
XnatTask
(processor, assessor, upload_dir, diskq)¶ Class Task to generate/manage the assessor with the cluster
-
batch_path
()¶ Method to return the path of the PBS file for the job
Returns: A string that is the absolute path to the PBS file that will be submitted to the scheduler for execution.
-
build_commands
(assr, jobdir)¶ Call the build_cmds method of the class Processor.
Parameters: jobdir – Fully qualified path where the job will run on the node. Note that this is likely to start with /tmp on most grids. Returns: A string that makes a command line call to a spider with all args.
-
build_task
(assr, jobdir, job_email=None, job_email_options='a', xnat_host=None)¶ Method to build a job
-
check_job_usage
()¶ - The task has now finished, get the amount of memory used, the amount of
- walltime used, the jobid of the process, the node the process ran on, and when it started from the scheduler. Set these values on XNAT
Returns: None
-
check_running
()¶ Check to see if a job specified by the scheduler ID is still running
Parameters: jobid – The ID of the job in question assigned by the scheduler. Returns: A String of JOB_RUNNING if the job is running or enqueued and JOB_FAILED if the ready flag (see read_flag_exists) does not exist in the assessor label folder in the upload directory.
-
get_job_status
()¶ Get the status of a job given its jobid as assigned by the scheduler
Parameters: jobid – job id assigned by the scheduler Returns: string from call to cluster.job_status or UNKNOWN.
-
launch
()¶ Method to launch a job on the grid
-
outlog_path
()¶ Method to return the path of outlog file for the job
Returns: A string that is the absolute path to the OUTLOG file.
-
set_launch
(jobid)¶ Set the date that the job started and its associated ID on XNAT. Additionally, set the procstatus to JOB_RUNNING
Parameters: jobid – The ID of the process assigned by the grid scheduler Returns: None
-
update_status
()¶ Update the satus of an XNAT Task object.
Returns: the “new” status (updated) of the Task.
-
dax.spiders
– Spider class¶
Title: spiders.py Author: Benjamin Yvernault contact: b.yvernault@ucl.ac.uk Purpose:
Spider base class and class for Scan and Session spider Spider name must be: Spider_[name]_v[version].py Utils for spiders
-
class
dax.spiders.
Spider
(spider_path, jobdir, xnat_project, xnat_subject, xnat_session, xnat_host=None, xnat_user=None, xnat_pass=None, suffix='', subdir=True, skip_finish=False)¶ Base class for spider
-
check_executable
(executable, name, version_opt='--version')¶ Method to check the executable.
Parameters: - executable – executable path
- name – name of Executable
Returns: Complete path to the executable
-
define_spider_process_handler
()¶ - Define the SpiderProcessHandler so the file(s) and PDF are checked for
- existence and uploaded to the upload_dir accordingly.
Implemented in derived classes.
Raises: NotImplementedError() if not overridden. Returns: None
-
download
(obj_label, resource, folder)¶ - Return a python list of the files downloaded for the scan’s resource
- example:
- download(scan_id, “DICOM”, “/Users/test”)
- or
- download(assessor_label, “DATA”, “/Users/test”)
Parameters: - obj_label – xnat object label (scan ID or assessor label)
- resource – folder name under the xnat object
- folder – download directory
Returns: python list of files downloaded
-
download_inputs
()¶ Download inputs data from XNAT define in self.inputs.
self.inputs = list of data dictionary with keys define below keys:
‘type’: ‘scan’ or ‘assessor’ or ‘subject’ or ‘project’ or ‘session’ ‘label’: label on XNAT (not needed for session/subject/project) ‘resource’: name of resource to download or list of resources ‘dir’: directory to download files into (optional)- for assessor only if not giving the label but just proctype ‘scan’: id of the scan for the assessor (if None, sessionAssessor)
- self.data = list of dictionary with keys define below:
- ‘label’: label on XNAT ‘files’: list of files downloaded
set self.data, a python list of the data downloaded.
-
end
()¶ Finish the script by sending the end of script flag and cleaning folder
Parameters: jobdir – directory for the spider Returns: None
-
finish
()¶ Method to copy the results in the Spider Results folder dax.RESULTS_DIR Implemented in derived class objects.
Raises: NotImplementedError if not overriden by user Returns: None
-
static
get_data_dict
(otype, label, resource, directory, scan=None)¶ Create a data_dict for self.inputs from user need.
-
get_exe_version
(executable, version_opt='--version')¶ Method to check the executable.
Parameters: - executable – executable to run
- version_opt – options to get the version of the executable
Returns: version
-
get_xnat_dict
(data_dict, resource)¶ Return a OrderedDict dictionary with XNAT information.
- keys:
- project subject experiment scan resource assessor out/resource (for assessor)
-
has_spider_handler
()¶ - Check to see that the SpiderProcessHandler is defined. If it is not,
- call define_spider_process_handler
Returns: None
-
merge_pdf_pages
(pdf_pages, pdf_final)¶ Concatenate all pdf pages in the list into a final pdf.
See function at the end of the file.
-
plot_images_page
(pdf_path, page_index, nii_images, title, image_labels, slices=None, cmap='gray', vmins=None, vmaxs=None, volume_ind=None, orient='ax')¶ Plot list of images (3D-4D) on a figure (PDF page).
See function at the end of the file.
-
plot_stats_page
(pdf_path, page_index, stats_dict, title, tables_number=3, columns_header=['Header', 'Value'], limit_size_text_column1=30, limit_size_text_column2=10)¶ Generate pdf report of stats information from a csv/txt.
See function at the end of the file.
-
pre_run
()¶ Pre-Run method to download and organise inputs for the pipeline Implemented in derived class objects.
Raises: NotImplementedError if not overridden. Returns: None
-
print_args
(argument_parse)¶ print arguments given to the Spider
Parameters: argument_parse – argument parser Returns: None
-
print_end
()¶ Last print statement to give the time and date at the end of the spider
Returns: None
-
print_err
(err_message)¶ Print error message using time writer
Parameters: err_message – error message displayed for the user Returns: None
-
print_info
(author, email)¶ Print information on the spider using time writer
Parameters: - author – author of the spider
- email – email of the author
Returns: None
-
print_init
(argument_parse, author, email)¶ Print a message to display information on the init parameters, author, email, and arguments using time writer
Parameters: - argument_parse – argument parser
- author – author of the spider
- email – email of the author
Returns: None
-
print_msg
(message)¶ Print message using time writer
Parameters: message – string displayed for the user Returns: None
-
run
()¶ Runs the “core” or “image processing process” of the pipeline Implemented in derived class objects.
Raises: NotImplementedError if not overridden. Returns: None
-
run_cmd_args
()¶ Run a command line via os.system() with arguments set in self.cmd_args
- cmd_args is a dictionary:
exe: executable to use (matlab, python, sh) template: string defining the command line with argument args: dictionary with:
key = argument value = value to setfilename: name for the file if written into a file (optional)
Returns: True if succeeded, False otherwise
-
run_system_cmd
(cmd)¶ Run system command line via os.system()
Parameters: cmd – command to run Returns: True if succeeded, False otherwise
-
select_obj
(intf, obj_label, resource)¶ Select scan or assessor resource
Parameters: - obj_label – xnat object label (scan ID or assessor label)
- resource – folder name under the xnat object
return pyxnat object
-
static
select_str
(xnat_dict)¶ Return string for pyxnat to select object from python dict
Parameters: tmp_dict – python dictionary with xnat information keys = [“project”, “subject”, “experiement”, “scan”, “resource”]
or- keys = [“project”, “subject”, “experiement”, “assessor”,
- ”out/resource”]
Return string: string path to select pyxnat object
-
upload
(fpath, resource)¶ Upload files to the queue on the cluster to be upload to XNAT by DAX E.g: spider.upload(“/Users/DATA/”, “DATA”)
spider.upload(“/Users/stats_dir/statistical_measures.txt”, “STATS”)Parameters: - fpath – path to the folder/file to be uploaded
- resource – folder name to upload to on the assessor
Raises: ValueError if the file to upload does not exist
Returns: None
-
upload_dict
(files_dict)¶ - upload files to the queue on the cluster to be upload to XNAT by DAX
- following the files python dictionary: {resource_name : fpath}
- E.g: fdict = {“DATA” : “/Users/DATA/”, “PDF”: “/Users/PDF/report.pdf”}
- spider.upload_dict(fdict)
Parameters: files_dict – python dictionary containing the pair resource/fpath Raises: ValueError if the filepath is not a string or a list Returns: None
-
-
class
dax.spiders.
ScanSpider
(spider_path, jobdir, xnat_project, xnat_subject, xnat_session, xnat_scan, xnat_host=None, xnat_user=None, xnat_pass=None, suffix='', subdir=True, skip_finish=False)¶ Derived class for scan-spider
-
define_spider_process_handler
()¶ - Define the SpiderProcessHandler for the end of scan spider
- using the init attributes about XNAT
Returns: None
-
finish
()¶ Method to copy the results in the Spider Results folder dax.RESULTS_DIR Implemented in derived class objects.
Raises: NotImplementedError if not overriden by user Returns: None
-
pre_run
()¶ Pre-Run method to download and organise inputs for the pipeline Implemented in derived class objects.
Raises: NotImplementedError if not overridden. Returns: None
-
run
()¶ Runs the “core” or “image processing process” of the pipeline Implemented in derived class objects.
Raises: NotImplementedError if not overridden. Returns: None
-
-
class
dax.spiders.
SessionSpider
(spider_path, jobdir, xnat_project, xnat_subject, xnat_session, xnat_host=None, xnat_user=None, xnat_pass=None, suffix='', subdir=True, skip_finish=False)¶ Derived class for session-spider
-
define_spider_process_handler
()¶ - Define the SpiderProcessHandler for the end of session spider
- using the init attributes about XNAT
Returns: None
-
finish
()¶ Method to copy the results in the Spider Results folder dax.RESULTS_DIR Implemented in derived class objects.
Raises: NotImplementedError if not overriden by user Returns: None
-
pre_run
()¶ Pre-Run method to download and organise inputs for the pipeline Implemented in derived class objects.
Raises: NotImplementedError if not overridden. Returns: None
-
run
()¶ Runs the “core” or “image processing process” of the pipeline Implemented in derived class objects.
Raises: NotImplementedError if not overridden. Returns: None
-
-
class
dax.spiders.
AutoSpider
(name, params, outputs, template, version=None, exe_lang=None)¶ Class for Autospider
-
copy_input
(src, input_name)¶ Copy inputs or download from XNAT.
-
copy_inputs
()¶ Copy the inputs data for AutoSpider.
-
copy_local_input
(src, input_name)¶ Copy local inputs.
-
copy_xnat_input
(src, input_name)¶ Copy xnat inputs.
-
download_xnat_file
(src, dst)¶ Download XNAT specific file.
-
download_xnat_resource
(src, dst)¶ Download XNAT complete resource.
-
end
()¶ Finish the script by sending the end of script flag and cleaning folder :return: None
-
finish
()¶ finish method to copy the results.
-
get_argparser
()¶ Get argparser for the AutoSpider.
-
go
()¶ Main method for AutoSpider.
-
is_xnat_uri
(uri)¶ Check if uri is xnat or local.
-
pre_run
()¶ Pre-Run method to download and organise inputs for the pipeline Implemented in derived class objects.
-
print_args
(argument_parse)¶ print arguments given to the Spider
Parameters: argument_parse – argument parser Returns: None
-
print_end
()¶ Last print statement
Returns: None
-
run
()¶ Run method to execute the template for AutoSpider.
-
-
class
dax.spiders.
TimedWriter
(name=None, use_date=False)¶ Class to automatically write timed output message
- Args:
- name - Names to write with output (default=None)
- Examples:
- >>>a = Time_Writer() >>>a(“this is a test”) [00d 00h 00m 00s] this is a test >>>sleep(60) >>>a(“this is a test”) [00d 00h 01m 00s] this is a test
Written by Andrew Plassard (Vanderbilt)
-
print_stderr_message
(text)¶ Prints a timed message to stderr
Parameters: text – The text to print Returns: None
-
print_timed_message
(text, pipe=<open file '<stdout>', mode 'w'>)¶ Prints a timed message
Parameters: - text – text to print
- pipe – pipe to write to. defaults to sys.stdout
Returns: None
dax.processors
– Processor class¶
Processor class define for Scan and Session.
-
class
dax.processors.
Processor
(walltime_str, memreq_mb, spider_path, version=None, ppn=1, env=None, suffix_proc='', xsitype='proc:genProcData', job_template=None)¶ Base class for processor
-
build_cmds
(cobj, dir)¶ Build the commands that will go in the PBS/SLURM script :raises: NotImplementedError if not overridden from base class. :return: None
-
default_settings_spider
(spider_path)¶ Get the default spider version and name
Parameters: spider_path – Fully qualified path and file of the spider Returns: None
-
get_assessor_input_types
()¶ Enumerate the assessor input types for this. The default implementation returns an empty collection; override this method if you are inheriting from a non-yaml processor. :return: a list of input assessor types
-
get_proctype
()¶ Return the processor name for this processor. Override this method if you are inheriting from a non-yaml processor. :return: the name of the processor type
-
has_inputs
()¶ Check to see if the spider has all the inputs necessary to run.
Raises: NotImplementedError if user does not override Returns: None
-
set_spider_settings
(spider_path, version)¶ Method to set the spider version, path, and name from filepath
Parameters: - spider_path – Fully qualified path and file of the spider
- version – version of the spider
Returns: None
-
should_run
()¶ Responsible for determining if the assessor should shouw up in session.
Raises: NotImplementedError if not overridden. Returns: None
-
-
class
dax.processors.
ScanProcessor
(scan_types, walltime_str, memreq_mb, spider_path, version=None, ppn=1, env=None, suffix_proc='', full_regex=False, job_template=None)¶ Scan Processor class for processor on a scan on XNAT
-
get_assessor
(cscan)¶ Returns the assessor object depending on cscan and the assessor label.
Parameters: cscan – CachedImageScan object from XnatUtils Returns: String of the assessor label
-
get_assessor_name
(cscan)¶ Returns the label of the assessor
Parameters: cscan – CachedImageScan object from XnatUtils Returns: String of the assessor label
-
get_task
(cscan, upload_dir)¶ Get the Task object
Parameters: - cscan – CachedImageScan object from XnatUtils
- upload_dir – the directory to put the processed data when the process is done
Returns: Task object
-
has_inputs
()¶ - Method to check and see that the process has all of the inputs
- that it needs to run.
Raises: NotImplementedError if not overridden. Returns: None
-
should_run
(scan_dict)¶ Method to see if the assessor should appear in the session.
Parameters: scan_dict – Dictionary of information about the scan Returns: True if it should run, false if it shouldn’t
-
-
class
dax.processors.
SessionProcessor
(walltime_str, memreq_mb, spider_path, version=None, ppn=1, env=None, suffix_proc='', job_template=None)¶ Session Processor class for processor on a session on XNAT
-
get_assessor
(csess)¶ Returns the assessor object depending on csess and the assessor label.
Parameters: csess – CachedImageSession object from XnatUtils Returns: String of the assessor label
-
get_assessor_name
(csess)¶ Returns the label of the assessor
Parameters: csess – CachedImageSession object from XnatUtils Returns: String of the assessor label
-
get_task
(csess, upload_dir)¶ Return the Task object
Parameters: - csess – CachedImageSession from XnatUtils
- upload_dir – directory to put the data after run on the node
Returns: Task object of the assessor
-
has_inputs
()¶ Check to see that the session has the required inputs to run.
Raises: NotImplementedError if not overriden from base class. Returns: None
-
should_run
(session_dict)¶ - By definition, this should always run, so it just returns true
- with no checks
Parameters: session_dict – Dictionary of session information for XnatUtils.list_experiments() Returns: True
-
-
class
dax.processors.
AutoProcessor
(xnat, yaml_source, user_inputs=None)¶ Auto Processor class for AutoSpider using YAML files
-
get_assessor_input_types
()¶ Enumerate the assessor input types for this. The default implementation returns an empty collection; override this method if you are inheriting from a non-yaml processor. :return: a list of input assessor types
-
get_cmds
(assr, jobdir)¶ Method to generate the spider command for cluster job.
Parameters: - assessor – pyxnat assessor object
- jobdir – jobdir where the job’s output will be generated
Returns: command to execute the spider in the job script
-
get_proctype
()¶ Return the processor name for this processor. Override this method if you are inheriting from a non-yaml processor. :return: the name of the processor type
-
has_inputs
(cobj)¶ Method to check the inputs.
- By definition:
- status = 0 -> NEED_INPUTS, for session asr inputs and resources status = 1 -> NEED_TO_RUN status = -1 -> NO_DATA, for scan primary input isn’t usable qcstatus needs a value only when -1 or 0.
You need to set qcstatus to a short string that explain why it’s no ready to run. e.g: No NIFTI
Parameters: cobj – cached object define in dax.XnatUtils (Session or Scan) (see XnatUtils in dax for information) Returns: status, qcstatus
-
parse_session
(csess, sessions)¶ Method to run the processor parser on this session, in order to calculate the pattern matches for this processor and the sessions provided :param csess: the active session. For non-longitudinal studies, this is the session that the pattern matching is performed on. For longitudinal studies, this is the ‘current’ session from which all prior sessions are numbered for the purposes of pattern matching :param sessions: the full, time-ordered list of sessions that should be considered for longitudinal studies. :return: None
-
should_run
(obj_dict)¶ Method to see if the assessor should appear in the session.
Parameters: obj_dict – Dictionary of information about the scan or sesion Returns: True if it should run, false if it shouldn’t
-
dax.log
– Logging utility¶
-
dax.log.
setup_critical_logger
(name, logfile)¶ Sets up the critical logger
Parameters: - name – Name of the logger
- logfile – file to store the log to. sys.stdout if no file define
Returns: logger object
-
dax.log.
setup_debug_logger
(name, logfile)¶ Sets up the debug logger
Parameters: - name – Name of the logger
- logfile – file to store the log to. sys.stdout if no file define
Returns: logger object
-
dax.log.
setup_error_logger
(name, logfile)¶ Sets up the error logger
Parameters: - name – Name of the logger
- logfile – file to store the log to. sys.stdout if no file define
Returns: logger object
-
dax.log.
setup_info_logger
(name, logfile)¶ Sets up the info logger
Parameters: - name – Name of the logger
- logfile – file to store the log to. sys.stdout if no file define
Returns: logger object
-
dax.log.
setup_warning_logger
(name, logfile)¶ Sets up the warning logger
Parameters: - name – Name of the logger
- logfile – file to store the log to. sys.stdout if no file define
Returns: logger object
dax.bin
– Responsible for launching, building and updating a Task¶
File containing functions called by dax executables
-
dax.bin.
build
(settings_path, logfile, debug, projects=None, sessions=None, mod_delta=None, proj_lastrun=None)¶ - Method that is responsible for running all modules and putting assessors
- into the database
Parameters: - settings_path – Path to the project settings file
- logfile – Full file of the file used to log to
- debug – Should debug mode be used
- projects – Project(s) that need to be built
- sessions – Session(s) that need to be built
Returns: None
-
dax.bin.
check_default_keys
(yaml_file, doc)¶ Static method to raise error if key not found in dictionary from yaml file. :param yaml_file: path to yaml file defining the processor :param doc: doc dictionary extracted from the yaml file
-
dax.bin.
launch_jobs
(settings_path, logfile, debug, projects=None, sessions=None, writeonly=False, pbsdir=None, force_no_qsub=False)¶ Method to launch jobs on the grid
Parameters: - settings_path – Path to the project settings file
- logfile – Full file of the file used to log to
- debug – Should debug mode be used
- projects – Project(s) that need to be launched
- sessions – Session(s) that need to be updated
- writeonly – write the job files without submitting them
- pbsdir – folder to store the pbs file
- force_no_qsub – run the job locally on the computer (serial mode)
Returns: None
-
dax.bin.
load_from_file
(filepath, args, logger, singularity_imagedir=None)¶ Check if a file exists and if it’s a python file :param filepath: path to the file to test :return: True the file pass the test, False otherwise
-
dax.bin.
pi_from_project
(project)¶ Get the last name of PI who owns the project on XNAT
Parameters: project – String of the ID of project on XNAT. Returns: String of the PIs last name
-
dax.bin.
raise_yaml_error_if_no_key
(doc, yaml_file, key)¶ Method to raise an execption if the key is not in the dict :param doc: dict to check :param yaml_file: YAMLfile path :param key: key to search
-
dax.bin.
read_yaml_settings
(yaml_file, logger)¶ Method to read the settings yaml file and generate the launcher object.
Parameters: yaml_file – path to yaml file defining the settings Returns: launcher object
-
dax.bin.
set_logger
(logfile, debug)¶ Set the logging depth
Parameters: - logfile – File to log output to
- debug – Should debug depth be used?
Returns: logger object
-
dax.bin.
update_tasks
(settings_path, logfile, debug, projects=None, sessions=None)¶ Method that is responsible for updating a Task.
Parameters: - settings_path – Path to the project settings file
- logfile – Full file of the file used to log to
- debug – Should debug mode be used
- projects – Project(s) that need to be launched
- sessions – Session(s) that need to be updated
Returns: None
dax.XnatUtils
– Collection of utilities for upload/download and general access¶
XnatUtils contains useful function to interface with XNAT using Pyxnat.
The functions are several categories:
1) Class Specific to XNAT and Spiders: InterfaceTemp to create an interface with XNAT using a tempfolder AssessorHandler to handle assessor label string and access object SpiderProcessHandler to handle results at the end of any spider
- Methods to query XNAT database and get XNAT object :
- Methods to access/check objects on XNAT
- Methods to Download / Upload data to XNAT
- Other Methods
- Cached Class for DAX
- Old download functions still used in some spiders
-
class
dax.XnatUtils.
InterfaceTemp
(xnat_host=None, xnat_user=None, xnat_pass=None, temp_dir=None)¶ - Extends the pyxnat.Interface class to make a temporary directory, write the
- cache to it and then blow it away on the Interface.disconnect call() NOTE: This is deprecated in pyxnat 1.0.0.0
Using netrc to get username password if not given.
-
authenticate
()¶ Authenticate to XNAT.
Connect to XNAT and try to Disconnect the JSESSION before reconnecting. Raise XnatAuthentificationError if it failes.
Returns: True or False
-
connect
()¶ Connect to XNAT.
-
disconnect
()¶ Disconnect the JSESSION and blow away the cache.
Returns: None
-
get_project_assessors
(projectid)¶ List all the assessors that you have access to based on passed project.
Parameters: projectid – ID of a project on XNAT Returns: List of all the assessors for the project
-
get_project_scans
(project_id, include_shared=True)¶ List all the scans that you have access to based on passed project.
Parameters: - intf – pyxnat.Interface object
- projectid – ID of a project on XNAT
- include_shared – include the shared data in this project
Returns: List of all the scans for the project
-
get_scans
(projectid, subjectid, sessionid)¶ - List all the scans that you have access to based on passed
- session/subject/project.
Parameters: - intf – pyxnat.Interface object
- projectid – ID of a project on XNAT
- subjectid – ID/label of a subject
- sessionid – ID/label of a session
Returns: List of all the scans
-
get_session_resources
(projectid, subjectid, sessionid)¶ - Gets a list of all of the resources for a session associated to a
- subject/project requested by the user
Parameters: - intf – pyxnat.Interface object
- projectid – ID of a project on XNAT
- subjectid – ID/label of a subject
- sessionid – ID/label of a session to get resources for
Returns: List of resources for the session
-
get_sessions
(projectid=None, subjectid=None)¶ - List all the sessions either:
- that you have access to
- or
- in a single project (and single subject) based on kargs
Parameters: - intf – pyxnat.Interface object
- projectid – ID of a project on XNAT
- subjectid – ID/label of a subject
Returns: List of sessions
-
class
dax.XnatUtils.
AssessorHandler
(label)¶ Class to intelligently deal with the Assessor labels. Make the splitting of the strings easier.
-
get_proctype
()¶ Get the proctype from the assessor label
Returns: The proctype for the assessor
-
get_project_id
()¶ Get the project ID from the assessor label
Returns: The XNAT project label
-
get_scan_id
()¶ Get the scan ID from teh assessor label
Returns: The scan id for the assessor label
-
get_session_label
()¶ Get the session label from the assessor label
Returns: The XNAT session label
-
get_subject_label
()¶ Get the subject label from the assessor label
Returns: The XNAT subject label
-
is_valid
()¶ Check to see if we have a valid assessor label (aka not None)
Returns: True if valid, False if not valid
-
select_assessor
(intf)¶ Run Interface.select() on the assessor label
Parameters: intf – pyxnat.Interface object Returns: The pyxnat EObject of the assessor
-
-
class
dax.XnatUtils.
SpiderProcessHandler
(script_name, suffix, project=None, subject=None, experiment=None, scan=None, alabel=None, assessor_handler=None, time_writer=None, host=None)¶ Class to handle the uploading of results for a spider.
-
add_file
(filepath, resource)¶ - Add a file in the assessor in the upload directory based on the
- resource name as will be seen on XNAT
Parameters: - filepath – Full path to a file to upload
- resource – The resource name it should appear under in XNAT
Returns: None
-
add_folder
(folderpath, resource_name=None)¶ Add a folder to the assessor in the upload directory.
Parameters: - folderpath – Full path to the folder to upoad
- resource_name – Resource name chosen (if different than basename)
Raises: - shutil.Error – Directories are the same
- OSError – The directory doesn’t exist
Returns: None
-
add_pdf
(filepath)¶ Add the PDF and run ps2pdf on the file if it ends with .ps
Parameters: filepath – Full path to the PDF/PS file Returns: None
-
add_snapshot
(snapshot)¶ Add in the snapshots (for quick viewing on XNAT)
Parameters: snapshot – Full path to the snapshot file Returns: None
-
clean
(directory)¶ Clean directory if no error and pdf created
Parameters: directory – directory to be cleaned
-
done
()¶ - Create a flag file that the assessor is ready to be uploaded and set
- the status as READY_TO_UPLOAD
Returns: None
-
file_exists
(fpath)¶ Check to see if a file exists
Parameters: fpath – full path to a file to assert it exists Returns: True if it exists, False if it doesn’t
-
folder_exists
(fpath)¶ Check to see if a folder exists
Parameters: fpath – Full path to a folder to assert it exists Returns: True if it exists, False if it doesn’t
-
print_copying_statement
(label, src, dest)¶ Print a line that data is being copied to the upload directory
Parameters: - label – The XNAT resource label
- src – Source directory or file
- dest – Destination directory or file
Returns: None
-
print_err
(msg)¶ Print error message using time writer if set, print otherwise
Parameters: msg – Message to print Returns: None
-
print_msg
(msg)¶ Prints a message using TimedWriter or print
Parameters: msg – Message to print Returns: None
-
set_assessor_status
(status)¶ Set the status of the assessor based on passed value
Parameters: status – Value to set the procstatus to Except: All catchable errors. Returns: None
-
set_error
()¶ Set the flag for the error to 1
Returns: None
-
-
class
dax.XnatUtils.
CachedImageSession
(intf, proj, subj, sess)¶ Enumeration for assessors function, to control what assessors are returned
-
assessors
(select=(0, ))¶ Get a list of CachedImageAssessor objects for the XNAT session
Returns: List of CachedImageAssessor objects for the session.
-
full_object
()¶ Return a the full pyxnat Session object of this sessions
Returns: pyxnat Session object
-
get
(name)¶ Get the value of a variable name in the session
Parameters: name – The variable name that you want to get the value of Returns: The value of the variable or ‘’ if not found.
-
get_resources
()¶ - Return a list of dictionaries that correspond to the information
- for each resource
Returns: List of dictionaries
Get the project if shared.
Returns: project_shared_id if shared, None otherwise
-
info
()¶ Get a dictionary of lots of variables that correspond to the session
Returns: Dictionary of variables
-
label
()¶ Get the label of the session
Returns: String of the session label
-
resources
()¶ Get a list of CachedResource objects for the session
Returns: List of CachedResource objects for the session
-
scans
()¶ Get a list of CachedImageScan objects for the XNAT session
Returns: List of CachedImageScan objects for the session.
-
session
()¶ Get the session associated with this object :return: session asscoiated with this object
-
-
class
dax.XnatUtils.
CachedImageScan
(intf, scan_element, parent)¶ Class to cache the XML information for a scan on XNAT
-
get
(name)¶ Get the value of a variable associated with a scan.
Parameters: name – Name of the variable to get the value of Returns: Value of the variable if it exists, or ‘’ otherwise.
-
get_resources
()¶ Get a list of dictionaries of info for each CachedResource.
Returns: List of dictionaries of infor for each CachedResource.
-
info
()¶ Get lots of variables assocaited with this scan.
Returns: Dictionary of infomation about the scan.
-
label
()¶ Get the ID of the scan
Returns: String of the scan ID
-
parent
()¶ Get the parent of the scan
Returns: XML String of the scan parent
-
resources
()¶ Get a list of the CachedResource (s) associated with this scan.
Returns: List of the CachedResource (s) associated with this scan.
-
session
()¶ Get the session associated with this object :return: session asscoiated with this object
-
-
class
dax.XnatUtils.
CachedImageAssessor
(intf, assr_element, parent)¶ Class to cache the XML information for an assessor on XNAT
-
get
(name)¶ Get the value of a variable associated with the assessor
Parameters: name – Variable name to get the value of Returns: Value of the variable, otherwise ‘’.
-
get_in_resources
()¶ - Get a list of dictionaries of info for the CachedResource objects
- for “in” type
Returns: List of dictionaries of info for the CachedResource objects for “in” type
-
get_out_resources
()¶ - Get a list of dictionaries of info for the CachedResource objects
- for “out” type
Returns: List of dictionaries of info for the CachedResource objects for “out” type
-
get_resources
()¶ Makes a call to get_out_resources.
Returns: List of dictionaries of info for the CachedResource objects for “out” type
-
in_resources
()¶ Get a list of CachedResource objects for “in” type
Returns: List of CachedResource objects for “in” type
-
info
()¶ Get a dictionary of information associated with the assessor
Returns: None
-
label
()¶ Get the label of the assessor
Returns: String of the assessor label
-
out_resources
()¶ Get a list of CachedResource objects for “out” type
Returns: List of CachedResource objects for “out” type
-
parent
()¶ Get the parent element of the assessor (session)
Returns: The session element XML string
-
-
class
dax.XnatUtils.
CachedResource
(element, parent)¶ Class to cache resource XML info on XNAT
-
get
(name)¶ Get the value of a variable associated with the resource
Parameters: name – Variable name to get the value of Returns: The value of the variable, ‘’ otherwise.
-
info
()¶ Get a dictionary of information relating to the resource
Returns: dictionary of information about the resource.
-
label
()¶ Get the label of the resource
Returns: String of the label of the resource
-
parent
()¶ Get the resource parent XML string
Returns: The resource parent XML string
-