pyscenic grn fails

[mgalardini]

Hi,

I'm trying to run the pyscenic grn command using pyscenic 0.9.3.

This is the command I'm using:

pyscenic grn input.loom --cell_id_attribute CellID --gene_attribute Gene tf_list.txt --num_workers 40 -o out.csv

and this is the output I'm getting (paths shortened):

2019-01-18 12:12:44,011 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
2019-01-18 12:14:06,165 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
tornado.application - ERROR - Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x2aeb2b3d0ae8>, <tornado.concurrent.Future object at 0x2aeb0b75c780>)
Traceback (most recent call last):
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/ioloop.py", line 605, in _run_callback
    ret = callback()
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/ioloop.py", line 626, in _discard_future_result
    future.result()
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/distributed/worker.py", line 640, in heartbeat
    metrics=self.get_metrics()
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/distributed/core.py", line 642, in send_recv_from_rpc
    result = yield send_recv(comm=comm, op=key, **kwargs)
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/distributed/core.py", line 455, in send_recv
    yield comm.write(msg, serializers=serializers, on_error='raise')
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/distributed/comm/tcp.py", line 232, in write
    stream.write(b)
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/iostream.py", line 406, in write
    self._handle_write()
  File "/[...]/20190116_pyscenic/venv/lib/python3.6/site-packages/tornado/iostream.py", line 872, in _handle_write
    del self._write_buffer[:self._write_buffer_pos]
BufferError: Existing exports of data: object cannot be re-sized
tornado.application - ERROR - Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x2aeb2ab1e598>, <tornado.concurrent.Future object at 0x2aeb2adad080>)
[...]

This error shows up multiple times and then the job eventually stops after considerable time.

This is the output of my pip freeze (I'm inside a virtualenv):

arboreto==0.1.5
attrs==18.2.0
boltons==18.0.1
Click==7.0
cloudpickle==0.6.1
cytoolz==0.9.0.1
dask==0.18.1
decorator==4.3.0
dill==0.2.8.2
distributed==1.25.2
frozendict==1.2
h5py==2.9.0
HeapDict==1.0.0
interlap==0.2.6
llvmlite==0.27.0
loompy==2.0.2
msgpack==0.6.0
multiprocessing-on-dill==3.5.0a4
networkx==2.2
numba==0.42.0
numpy==1.16.0
pandas==0.23.4
psutil==5.4.8
pyarrow==0.11.1
pyscenic==0.9.3
python-dateutil==2.7.5
pytz==2018.9
PyYAML==3.13
scikit-learn==0.20.2
scipy==1.2.0
six==1.12.0
sortedcontainers==2.1.0
tblib==1.3.2
toolz==0.9.0
tornado==4.5
tqdm==4.29.1
typing==3.6.6
umap-learn==0.3.7
zict==0.1.3

As suggested in #30 I have downgraded tornado and dask, but it doesn't seem to be helping unfortunately. Should pyscenic also be downgraded?

Many thanks,
M

p.s. I downgraded tornado and dask in a different way:

virtualenv venv
source activate venv/bin/activate
pip install pyscenic
pip install tornado==4.5
pip install dask==0.18.1

[frenkiboy]

I would like to bump this question - same problem, downgrading didn't help.

Any suggestion would be appreciated

[ccshao]

Similar issue here,

tornado.application - ERROR - Exception in callback <function Worker.init.. at 0x7fe5c3413048>

but I got an adj.tsv file. Did the pyscenic finished successfully? 0.10.2 pyscenic, 2.19.0 dask.

2020-07-01 08:59:13,607 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.

2020-07-01 09:00:18,967 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
tornado.application - ERROR - Exception in callback <function Worker.init.. at 0x7fe5c3413048>
Traceback (most recent call last):
File "/home/shao/.local/lib/python3.6/site-packages/tornado/ioloop.py", line 907, in _run
return self.callback()
File "/home/shao/miniconda3/envs/env_scenic/lib/python3.6/site-packages/distributed/worker.py", line 648, in
lambda: self.batched_stream.send({"op": "keep-alive"}), 60000,
File "/home/shao/miniconda3/envs/env_scenic/lib/python3.6/site-packages/distributed/batched.py", line 117, in send
raise CommClosedError
distributed.comm.core.CommClosedError
tornado.application - ERROR - Exception in callback <function Worker.init.. at 0x7fe5c3413048>
Traceback (most recent call last):
File "/home/shao/.local/lib/python3.6/site-packages/tornado/ioloop.py", line 907, in _run
return self.callback()
File "/home/shao/miniconda3/envs/env_scenic/lib/python3.6/site-packages/distributed/worker.py", line 648, in
lambda: self.batched_stream.send({"op": "keep-alive"}), 60000,
File "/home/shao/miniconda3/envs/env_scenic/lib/python3.6/site-packages/distributed/batched.py", line 117, in send
raise CommClosedError
distributed.comm.core.CommClosedError
tornado.application - ERROR - Exception in callback <function Worker.init.. at 0x7fe5c3413048>
Traceback (most recent call last):
File "/home/shao/.local/lib/python3.6/site-packages/tornado/ioloop.py", line 907, in _run
return self.callback()
File "/home/shao/miniconda3/envs/env_scenic/lib/python3.6/site-packages/distributed/worker.py", line 648, in
lambda: self.batched_stream.send({"op": "keep-alive"}), 60000,
File "/home/shao/miniconda3/envs/env_scenic/lib/python3.6/site-packages/distributed/batched.py", line 117, in send
raise CommClosedError
distributed.comm.core.CommClosedError
/home/shao/.local/lib/python3.6/site-packages/arboreto/algo.py:214: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
expression_matrix = expression_data.as_matrix()

2020-07-01 14:26:14,678 - pyscenic.cli.pyscenic - INFO - Writing results to file.
preparing dask client
parsing input
creating dask graph
15 partitions
computing dask graph
not shutting down client, client was created externally
finished

[cflerin]

Hi @ccshao ,

If you got an output (pyscenic.cli.pyscenic - INFO - Writing results to file. seems to suggest that you did), then it does appear that the grn step completed, although you do get a number of these tornado comm errors.

But please see also #163 for other suggestion on how to solve this.