PYME.Analysis.distributed_pyramid module

class PYME.Analysis.distributed_pyramid.DistributedImagePyramid(storage_directory, pyramid_tile_size=256, mdh=None, n_tiles_x=0, n_tiles_y=0, depth=0, x0=0, y0=0, pixel_size=1, backend=<class 'PYME.Analysis.tile_pyramid.ClusterPZFTileIO'>, servers=None, chunk_shape=[8, 8, 1], timeout=10, repeats=3)

Bases: ImagePyramid

Subclass of ImagePyramid which supports distribution of pyramid files over a PYME cluster. Implementation from the microscope side.

create_remote_part(server_idx, backend)

Initializes PartialPyramid a on the specified server.

finish_base_tiles()

Notifies all cluster servers that all frames have been processed on the microscope end. Blocks until the servers have finished constructing their partial pyramids

property partial_pyramid_depth

update_base_tile(tile_x, tile_y, data, weights, tile_offset=(0, 0), frame_offset=(0, 0), frame_shape=None)

Over-ridden version of update_base_tile which causes this to be called on the server rather than the client

In practice it simply adds each chunk to a queue of chunks that get pushed asynchronously in multiple threads (one for each server).

update_pyramid()

Builds the final layers once the lower levels have been built on the cluster. The topmost layers on the cluster are aggregated on the microscope and then used to make the final layers.

class PYME.Analysis.distributed_pyramid.PartialPyramid(storage_directory, pyramid_tile_size=256, backend=<class 'PYME.Analysis.tile_pyramid.PZFTileIO'>, chunk_shape=[8, 8, 1])

Bases: ImagePyramid

Subclass of ImagePyramid which supports distribution of pyramid files over a PYME cluster.

Implementation on the server side

Note: we don’t want to write metadata for the partial pyramids, meaning that they don’t need all of the parameters

finalise()

classmethod from_request(filepath, body)

queue_base_tile_update(data, query)

update_pyramid()

Updates this PartialPyramid as far as its chunk_shape allows.

TODO - refactor base update_pyramid to allow max_depth as a parameter.

PYME.Analysis.distributed_pyramid.create_distributed_pyramid_from_dataset(filename, outdir, tile_size=128, **kwargs)

PYME.Analysis.distributed_pyramid.distributed_pyramid(out_folder, ds, xm, ym, mdh, split=False, skipMoveFrames=False, shiftfield=None, mixmatrix=[[1.0, 0.0], [0.0, 1.0]], correlate=False, dark=None, flat=None, pyramid_tile_size=256)

Create a distributed pyramid through PYMECluster.

Parameters

out_folderstr

directory to save pyramid tiles(/directories). The same folder will be created on the cluster servers.

dsPYME.IO.DataSources.BaseDataSource, np.ndarray

array-like image

xmnp.ndarray or PYME.Analysis.piecewiseMapping.piecewiseMap

x positions of frames in ds. Raw stage positions in [um]. ImagePyramid origin will be at at minimum x, and offset to camera chip origin will be handled in SupertileDatasource tile_coords_um method. to the camera chip origin.

ymnp.ndarray or PYME.Analysis.piecewiseMapping.piecewiseMap

y positions of frames in ds. Raw stage positions in [um]. ImagePyramid origin will be at at minimum y, and offset to camera chip origin will be handled in SupertileDatasource tile_coords_um method.

mdhPYME.IO.MetaDataHandler.MDataHandlerBase

metadata for ds

splitbool, optional

whether this is a splitter datasource and should be treated like one, by default False

skipMoveFramesbool, optional

flag to drop frames which are the first frame acquired at a given position, by default False

shiftfield[type], optional

required for splitter data, see PYME.Acquire.Hardware.splitter, by default None

mixmatrixlist, optional

for splitter data, see PYME.Acquire.Hardware.splitter, by default [[1., 0.], [0., 1.]]

correlatebool, optional

whether to add a 300 pixel padding to the edges, by default False

darkndarray, float, optional

(appropriately-cropped or scalar) dark frame (analog-digital offset) calibration to subtract when adding frames to the pyramid, by default None, in which case Camera.ADOffset from metadata will be used, if available

flatndarray, optional

(appropriately-cropped or scalar) flatfield calibration to apply to frames when adding them to the pyramid, by default None

pyramid_tile_sizeint, optional

base tile size, by default 256 pixels

Returns

DistributedImagePyramid

coalesced/averaged/etc multilevel DistributedImagePyramid instance

Notes

Code is currently somewhat alpha in that the splitter functionality is more or less untested, and we only get tile orientations right for primary cameras (i.e. when the stage is registered with multipliers to match the camera, rather than camera registered with orientation metadata to match it to the stage).

TODO - this largely duplicates the corresponding function in tile_pyramid => refactor

PYME.Analysis.distributed_pyramid.server_for_chunk(x, y, z=0, chunk_shape=[8, 8, 1], n_servers=1)

Returns the server responsible for the chunk of tiles at given (x, y, z).