LSR runtime database pickling is unsafe

[akrherz]

I pickle a dictionary of current runtime LSRs for dedup purposes. This appears to sometimes fail like so:

Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] cleandb() init_size: 3640 final_size: 3624
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] Unhandled Error
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011Traceback (most recent call last):
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011  File "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011    self.__bootstrap_inner()
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011    self.run()
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011  File "/usr/lib64/python2.7/threading.py", line 764, in run
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011    self.__target(*self.__args, **self.__kwargs)
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011--- <exception caught here> ---
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011  File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 196, in _worker
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011    result = context.call(ctx, function, *args, **kwargs)
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011  File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011    return self.currentContext().callWithContext(ctx, func, *args, **kw)
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011  File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011    return func(*args,**kw)
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011  File "pyWWA/parsers/lsr_parser.py", line 60, in pickledb
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011    pickle.dump(LSRDB, open('lsrdb.p', 'w'))
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011  File "/usr/lib64/python2.7/pickle.py", line 1370, in dump
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011    Pickler(file, protocol).dump(obj)
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011  File "/usr/lib64/python2.7/pickle.py", line 224, in dump
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011    self.save(obj)
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011  File "/usr/lib64/python2.7/pickle.py", line 286, in save
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011    f(self, obj) # Call unbound method with explicit self
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011  File "/usr/lib64/python2.7/pickle.py", line 649, in save_dict
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011    self._batch_setitems(obj.iteritems())
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011  File "/usr/lib64/python2.7/pickle.py", line 661, in _batch_setitems
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011    for k, v in items:
Jul  3 23:39:23 iem12 pyWWA/lsr_parser: [-] #011exceptions.RuntimeError: dictionary changed size during iteration

Maybe I should do something more elegant.