PYME.config module

Support for configuration / settings in PYME. This will eventually replace a motley collection of different ways of storing the configuration in use in different parts of PYME. Configuration is read from the following locations (if they exist) which are searched in order, with later entries over-riding prior ones:

  • an etc/PYME directory under the python sys.prefix. This would usually be /usr/local/ under standard unix python and the distribution base directory for a distribution such as anaconda. This allows installation wide configuration and is also a place that should be writable when installing conda packages and the like (useful for writing a package with plugins which register themselves - see plugin functions below.
  • a machine wide /etc/PYME directory. This is included for compatibility, but it is not envisaged that this will be used often, as it will only work on posix systems while the the above {{sys.prefix}}/etc/PYME option will also work under windows.
  • a .PYME directory under the users home directory. This should be used for user specific settings.

Within each configuration directory there can be a config.yaml file which stores configuration settings as key-value pairs. These are accessed using the get() function.

The directories may also contain a plugins folder, which in turn can contain subfolders for visgui, dsviewer, and recipes. PYME will also detect custom acquisition protocols saved in the .PYME/protocols directory, similarly init scripts will be detected in .PYME/init_scripts directory. The overall template for a configuration directory is as follows:

.PYME
  |- config.yaml
  |- plugins
  |     |- visgui
  |     |     |- somemodule.txt
  |     |     |- anothermodule.txt
  |     |
  |     |- dsviewer
  |     |     |- somemodule.txt
  |     |
  |     |- recipes
  |           |- anothermodule.txt
  |
  |- protocols
  |     |- a_protocol.py
  |     |- another_protocol.py
  |
  |- customrecipes
  |     |- myrecipe.yaml
  |     |- myOtherRecipe.yaml
  |
  |- init_scripts
        |- init_mymachine.py
        |- init_my_other_config.py

If you want to verify which directories will be searched on your particular python installation, running the following on the command line will tell you:

python -c "import PYME.config;print PYME.config.config_dirs"

with anaconda on OSX this gives me:

['/Users/david/.PYME', '/etc/PYME', '/Users/david/anaconda/etc/PYME']

Examples

config.yaml

The config.yaml file is essentially nothing more than a set of keys and values separated by colons. List and dictionary parameter values are supported using standard yaml notation.

dataserver-root: "/Users/david/srvtest/test1"
h5f-flush_interval: 1
PYMEAcquire-extra_init_dir: "C:/pyme-init-scripts"

plugins/visgui/PYME.txt

If we were to use the plugin architecture to register some of the native plugins (rather than using explicit knowledge of their locations), the registration file would look something like this. Each line is a fully qualified module import path.

PYME.LMVis.Extras.photophysics
PYME.LMVis.Extras.particleTracking

In addition to the configuration derived from config.yaml, a few legacy environment variables are recognized. Subpackages are also permitted to save configuration files in the .PYME directory.

Functions

Known Config Keys

This is a non-exhaustive list of configuration keys - if adding a new key, please add it here.

PYMEAcquire-extra_init_dir : default=None, a path to an extra directory to search for PYMEAcquire init files.

SplitterRatioDatabase : default=None, path to a .json formatted file containing information about ratiometric splitting ratios

VisGUI-console-startup-file : default=None, path to a script to run within the VisGUI interactive console on startup,
used to populate the console namespace with additional functions. Note that this script should not manipulate the pipeline as this cannot be assumed to be loaded when the script runs.
dh5View-console-startup-file : default=None, path to a script to run within the dh5View interactive console on startup,
used to populate the console namespace with additional functions. Note that this script should not manipulate the data as this cannot be assumed to be loaded when the script runs.
TaskServer.process_queues_in_order : default = True, an optimisation for old style task distribution so that it processes
series in the order that they were added to the queue (rather than the previous approach of choosing from a series at random). Means that caches behave sensibly, rather than potentially getting scrambled when multiple analyses run at once, but also means that you need to wait for previous series to finish before you get any results from the current series if you are doing real time analysis and not keeping up.
dataserver-root : what directory should PYMEDataSever serve. Overridden by the –root command line option. If undefined,
the current directory is served. Note that this is also used in clusterIO to allow short-circuit access to data on the same node.
dataserver-filter : default = ‘’, multi-cluster support. Specifies a cluster name / clusterfilter which identifies which
cluster PYMEDataServer should identify with. First part of the PYME-CLUSTER url. Overridden by the –server-filter command line option.

dataserver-port : default=8080, what port to run the PYMEDataServer on. Overridden by the –port command line option (e.g. if you want to run multiple servers on one machine).

cluster-listing-no-countdir : default=False, hack to disable the loading of the low-level countdir module which allows rapid
directory statistics on posix systems. Needed on OSX if dataserver-root is a mapped network drive rather than a physical disk
h5r-flush_interval : default=1, how often (in s) should we call the .flush() method and write records from the HDF/pytables
caches to disk when writing h5r files.
nodeserver-chunksize : default=50, how many frames should we give a worker process at once (larger numbers = better
background cache performance, but potentially not distributing as widely). Should be larger than the number of background frames when doing running average / percentile background subtraction [new style distribution].
nodeserver-worker-get-timeout : default=60, a timeout (in s). When the worker asks for tasks, the nodeserver tries to
get nodeserver-chunksize tasks off its internal queue (which is filled by a separate thread which communicates with the ruleserver). This timeout specifies how long should the nodeserver wait when accessing this queue in the hope of finding a full chunk. If it times out, a partial chunk will be given to the worker. In practice, this timeout behaviour is responsible for clearing the small tail of tasks at the end of a series. [new-style distribution].

nodeserver-num_workers : default= CPU count. Number of workers to launch on an individual node.

ruleserver-retries : default = 3. [new-style task distribution]. The number of times to retry a given task before it is deemed to have failed.

Deprecated config options

distributor-* : parameters for a previous implementation of cluster-based task distribution. Largely irrelevant now we use PYMERuleServer

VisGUI-new_layers : default = True, use the new-style layers view in VisGUI. Same as the –new_layers command line option.
Largely a remnant of when I was running layers and old style VisGUI in parallel.
PYME.config.get(key, default=None)

Gets a configuration parameter, by name

Parameters:
key : basestring

The parameter name

default : unspecified, optional

The default value you want to assume if the parameter is undefined.

Returns:
The parameter value, or the default value if undefined.
PYME.config.get_custom_protocols()

Get a dictionary recording the locations of any custom protocols.

Returns:
A dictionary of {basename : full path} for any protocols found. In the current implementation
custom protocols overwrite protocols of the same name in the PYME distribution.
PYME.config.get_custom_recipes()

Get a dictionary recording the locations of any custom recipes.

Returns:
A dictionary of {basename : full path} for any recipes found.
PYME.config.get_init_filename(filename, legacy_scripts_directory=None)

Look for an init file in the various locations. If the given filename exists (i.e. is a fully resolved path) it will be used. Otherwise ‘init_scripts’ subdirectory of the configuration directories will be searched, in order of precedence user - site - dist. It also checks for files in a provided directory (to support legacy usage with the PYMEAcquire/scripts directory) and the config option PYMEAcquire-extra_init_dir.

Parameters:
filename: init file name to locate in script dirs
Returns:
If found returns first match as full path to init file
returns None if not found.
PYME.config.get_plugins(application)

Get a list of plugins for a given application

Modules are registered by adding fully resolved module paths (one per line) to a text file in the relevant directory. The code searches all files in the relevant directories, and the intention is that there is one registration file for each standalone package that provides modules and can e.g. be conda or pip-installed which contains a list of all the plugins that package provides. The registration filename should ideally be the same as the package name, although further subdivision for large packages is fine. registration filenames should however be unique - e.g. by prefixing with the package name. By structuring it this way, a package can add this file to the anaconda/etc/PYME/plugins/XXX/ folder through the standard conda packaging tools and it will be automatically discovered without conflicts

Parameters:
application : basestring

One of ‘visgui’, ‘dsviewer’, or ‘recipes’

Returns:
list of fully resolved module paths