Converting Files at NERSC and Making them Accessible¶

python convertToOMSI.py <infile1> /project/projectdirs/openmsi/omsi_data_private/<username>/<filename>

python convertToOMSI.py --help
USAGE: Call "convertToOMSI [options] imgBaseFile1 imgBaseFile2 ... imgBaseFileN HDF5File"

This converter script takes the basename (i.e., path+basefilename) of a single
or multiple MSI files as input and converts them to HDF5. Each MSI file is
stored as a separate experiment in the output HDF5 file. If an input file
defines multiple regions, then those regions can either be stored as separate
datasets of the same experiment and/or merged to a single MSI dataset.
Using the various paramter settings described below, one can define how the
conversion should be performed, how the data should be stored in HDF5, and
indicate which analyses should be executed.

===HELPER OPTIONS===
--suggest-chunking : Iterate over all given input files and suggest a chunking strategy.
                     No data is converted when this option is given, i.e., no name for the
                     HDF5File should be given, but only input files should be listed.

===ERROR HANDLING OPTIONS===
--error-handling <options>: Define how errors should be handled. Options are:
                   i)   terminate-and-cleanup (default) : Terminate the conversion, delete the
                           the HDF5 file and do not add the file to the database.
                   ii)  terminate-only, : Leave the generated HDF5 output file in place but  do not
                            add the file to the database.
                   iii) continue-on-error: Ignore errors if possible and continue, even if this
                            means that some data may be missing from the output.
--email <email1 email2 ...>: Send notification in case of both error or success to the email address.
--email-success <email1 email2 ...>>: Send notification in case of success to the given email address.
--email-error <email1 email2 ...>>: Send notification in case of error to the given email address.

===INPUT DATA OPTIONS===

Default input data options: --format auto --regions split+merge
--format <option>: Define which file format is used as input. By default the program tries to
           automatically determine the input format. This option can be used to indicate
           the format explicitly to in case the auto option fails. Available options are:
          {'imzml_file': <class 'omsi.dataformat.imzml_file.imzml_file'>, 'bruckerflex_file': <class 'omsi.dataformat.bruckerflex_file.bruckerflex_file'>, 'img_file': <class 'omsi.dataformat.img_file.img_file'>, 'mzml_file': <class 'omsi.dataformat.mzml_file.mzml_file'>}
--regions <option>: Some file formats (e.g., brucker) allow multiple regions to be imaged and stored
           in a single file. This option allows one to specify how these regions should be
           treated during file conversion. E.g., one may want to store i) each region as a
           separate dataset in the output file (--regions split), ii) all regions combined
           in a single dataset (--regions merge), or both (--regions split+merge)
           Available options are:
          ['split', 'merge', 'split+merge']

===FILE WRITE OPTIONS===

---FILE WRITE OPTIONS: Chunking---

Default HDF5 Chunking options: Enabled by default using --auto-chunking :
--auto-chunking : Automatically decide which chunking should be used. This option
                automatically generates two copies of the data, one with a chunking
                optimized for selection of spectra and another one optimized for
                selection of ion image slices. All --chunking, --no-chunking, and
                --optimized-chunking options are ignored if this paramter is given
--chunking <x y z> : Use chunking when writing the HDF5 file. (DEFAULT, x=4,y=4,z=2048)
--no-chunking : Disable chunking when writing the HDF5 file. Use in combination with
                --no-compression since compression depends on chunking and will enable
                it if compression is used.
--optimized-chunking <x y z> : Use this option to generate additional copies of the data
                with different chunked data layouts. Generating multiple copies of the
                data with different chunked data layouts can be help accelerate selective
                data read opeations. (DEFAULT OFF). We recommend a spectra-aligned chunking
                for the raw data, e.g., '--chunking 1 1 32768' and an image-aligned chunked
                secondary copy of the data, e.g., '--optimzied-chunking 20 20 100'.

---FILE WRITE OPTIONS: Compression---
HDF5 Compression: Default ON using (gzip, 4):
--compression: Enable compression using (gzip,4). NOTE: Compression requires the use of chunking.
--no-compression: Disable the use of compression.

===I/O OPTIONS===
--io <option>: Available options are: ['chunk', 'spectrum', 'all']
             i) all : Read the full data in memory and write it at once
             ii) spectrum : Read one spectrum at a time and write it to the file.
             iii) chunk : Read one chunk at a time and write it to the file.
             The io option applies only for the generation of subsequent chunkings
             and not the initial iteration over the file to generate the first convert.
             iv) spectrum-to-image: Default option when creating image chunk version from
             a spectrum-chunk MSI dataset. Read a block of spectra at a time to
             complete a set of images and then write the block of images at once.
--io-block-limit <MB>: When using spectrum-to-image io (default when using auto-chunking),
             what should the maximum block in MB that we load into memory. (Default=2000MB)

===DATABSE OPTIONS===

These options control whether the generated output file should be added to a server database
to manage web file access permissions
Default options are: --add-to-db --db-server http://openmsi.nersc.gov
--add-to-db : Explicitly add the output HDF5 file to the database. This option has no effect
              if --jobid is set as the file is added through the update of the job status
              in this case.
--no-add-to-db : Disable adding the file to the database.
--db-server : Specify the online server where the file should be registers. Default is
              http://openmsi.nersc.gov
--user : Name of the user that should be assigned as user. By default the user is
          determined automatically based on the file path.
--jobid : ID of the job. If set to 'auto' then the environment variable PBS_JOBID is used.
          NOTE: If job ID is set then we assume that the job has been scheduled via the
          the automated system and that the job is managed. As such the file will be added,
          to the database by updating the job status and NOT by explicitly adding the file.

===ANALYSIS OPTIONS===

NMF: Default ON: (nc=20, timeout=600, niter=2000, tolerance=0.0001, raw=False)
--nmf : Compute the nmf for all the input data files and store the results in the
        HDF5 file. NOTE: If global peak-finding (fpg) is performed, then
        nmf will be performed on the peak-cube, otherwise on the raw data
--no-nmf: Disable the execution of nmf
--nmf-nc <number>: Number of components to be computed by the NMF. (default nc=20)
--nmf-timeout <number>: Maximum time in seconds to be used for computing the NMF. (default timeout=600)
--nmf-niter <number>: Number of iterations (minimum is 2)(default niter=2000)
--nmf-tolerance <number>: Tolerance value for a relative stopping condition. (default tolerance=0.0001)
--nmf-raw <number>: Force execution of the NMF on the raw data. By default the results from
            the global peak finding (--fpg) are used to compute the NMF.

Global Peak Finding: Default ON:
--fpg : Compute the global peak finding for all input data files and save results
           in the HDF5 file (DEFAULT)
--no-fpg: Disable the global peak finding

Local Peak Finding: Default OFF:
--fpl : Compute the local peak finding for all input data files and save results
        in the HDF5 file
--no-fpl: Disable the local peak finding (DEFAULT)


TIC normalization:
--ticnorm : Compute tic normalization
--no-ticnorm : Disable computation of tic normaltization (DEFAULT)

---OTHER OPTIONS---

Generate Thumbnail image: Default OFF:
--thumbnail: Generate thumbnail image for the file based on, in order of availability:
             * The first three components of the NMF
             * The three most intense peaks from the global peak finding (fpg)
             * The three most intense peaks in the raw data that are at least 1 percent
               of the total m/z range apart.
--no-thumbnail: Do not generate a thumbnail image.

Generate XDMF header file for output file: Default OFF:
--xdmf: Write XDMF XML-based header-file for the output HDF5 file.
--no-xdmf: Do not generate a XDMF XML-based header for the HDF5 file.

===Metadata Options===

NOTE: Input datasets are numbers starting from 0 based on there order on the command line.

--methods : JSON describing the experimental methods
--methods# : JSON describing the experimental methods for input file number #
--instrument : JSON dictionary describing the instrument
--instrument# : JSON dictionary describing the instrument for input file number #
--notes : JSON dictionary with additional user notes about the data
--notes# : JSON dictionary with additional notes for input file number #