PYME.IO.testClusterSpooling module

This file provides a minimal test for spooling into the cluster with fake data, and with a multi-threaded spooler

Setup and Usage:

  • Run PYMEDataServer instances (on either local or remote host) with a -t option. This activates a testing mode where data is dumped on arrival rather than being written to disk.
  • Optionally use -p portNum and -v HTTP_version options on server
  • Adjust the frame size (below) as appropriate. 2k by 2k should be the standard initial benchmark as at this size per-frame overhead should not be excessive. We can optimize for per-frame overhead later. TODO - make this a command line option.
  • Run this file

Performance on Macbook to 10 local servers (as of 19/7/2016):

| | HTTP/1.0 | HTTP/1.1 with keep-alive |

2kx2k frames | 420 MB/s | 455 MB/s |
200x800 frames | 133 MB/s | 372 MB/s |

New benchmark on macbook to 10 local servers (25/7/2016) using raw sockets (HTTP/1.1 only)

| | HTTP/1.0 | HTTP/1.1 with keep-alive |

2kx2k frames | N/A | 533 MB/s (69 FPS) |
200x800 frames | N/A | 723 MB/s (2260 FPS) |

Target: 800 MB/s


  • Occasional connection timeout bug. Seems to be worse with HTTP/1.1
class PYME.IO.testClusterSpooling.TestSpooler(testFrameSize=[2000, 2000], serverfilter='')


run([filename, nFrames, interval, compression])
run(filename=None, nFrames=2000, interval=0, compression=False)