cannot run scenic protocol

[rdesantis-git]

Hi, I am trying to run scenic on google colab but I get this error can you help me out?

!pyscenic grn {'Opto_d14_filtered_scenic.loom'} {f_tfs} -o adj.csv --num_workers 8

2020-12-09 18:27:17,987 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.

2020-12-09 18:27:21,650 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
preparing dask client
parsing input
/usr/local/lib/python3.6/dist-packages/arboreto/algo.py:214: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
expression_matrix = expression_data.as_matrix()
creating dask graph
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/distributed/protocol/core.py", line 114, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
distributed.core - ERROR - tuple is not allowed for map key
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/distributed/core.py", line 297, in handle_comm
msg = yield comm.read()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.6/dist-packages/distributed/comm/tcp.py", line 207, in read
deserializers=deserializers)
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 326, in wrapper
yielded = next(result)
File "/usr/local/lib/python3.6/dist-packages/distributed/comm/utils.py", line 82, in from_frames
res = _from_frames()
File "/usr/local/lib/python3.6/dist-packages/distributed/comm/utils.py", line 68, in _from_frames
deserializers=deserializers)
File "/usr/local/lib/python3.6/dist-packages/distributed/protocol/core.py", line 114, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/distributed/protocol/core.py", line 114, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
not shutting down client, client was created externally
finished
Traceback (most recent call last):
File "/usr/local/bin/pyscenic", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/pyscenic/cli/pyscenic.py", line 470, in main
args.func(args)
File "/usr/local/lib/python3.6/dist-packages/pyscenic/cli/pyscenic.py", line 74, in find_adjacencies_command
network = method(expression_data=ex_mtx, tf_names=tf_names, verbose=True, client_or_address=client, seed=args.seed)
File "/usr/local/lib/python3.6/dist-packages/arboreto/algo.py", line 41, in grnboost2
early_stop_window_length=early_stop_window_length, limit=limit, seed=seed, verbose=verbose)
File "/usr/local/lib/python3.6/dist-packages/arboreto/algo.py", line 128, in diy
seed=seed)
File "/usr/local/lib/python3.6/dist-packages/arboreto/core.py", line 403, in create_graph
future_tf_matrix = client.scatter(tf_matrix, broadcast=True)
File "/usr/local/lib/python3.6/dist-packages/distributed/client.py", line 1874, in scatter
asynchronous=asynchronous, hash=hash)
File "/usr/local/lib/python3.6/dist-packages/distributed/client.py", line 675, in sync
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/distributed/utils.py", line 277, in sync
six.reraise(*error[0])
File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/distributed/utils.py", line 262, in f
result[0] = yield future
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.6/dist-packages/distributed/client.py", line 1736, in _scatter
timeout=timeout)
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.6/dist-packages/distributed/core.py", line 580, in send_recv_from_rpc
result = yield send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.6/dist-packages/distributed/core.py", line 457, in send_recv
response = yield comm.read(deserializers=deserializers)
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.6/dist-packages/distributed/comm/tcp.py", line 207, in read
deserializers=deserializers)
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 326, in wrapper
yielded = next(result)
File "/usr/local/lib/python3.6/dist-packages/distributed/comm/utils.py", line 82, in from_frames
res = _from_frames()
File "/usr/local/lib/python3.6/dist-packages/distributed/comm/utils.py", line 68, in _from_frames
deserializers=deserializers)
File "/usr/local/lib/python3.6/dist-packages/distributed/protocol/core.py", line 114, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key

Requirements:
Requirement already satisfied: pyscenic in /usr/local/lib/python3.6/dist-packages (0.10.4)
Requirement already satisfied: frozendict in /usr/local/lib/python3.6/dist-packages (from pyscenic) (1.2)
Requirement already satisfied: umap-learn in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.4.6)
Requirement already satisfied: arboreto in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.1.5)
Requirement already satisfied: boltons in /usr/local/lib/python3.6/dist-packages (from pyscenic) (20.2.1)
Requirement already satisfied: llvmlite in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.31.0)
Requirement already satisfied: numba in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.48.0)
Requirement already satisfied: pandas<1.0.0,>=0.20.1 in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.25.3)
Requirement already satisfied: cloudpickle in /usr/local/lib/python3.6/dist-packages (from pyscenic) (1.3.0)
Requirement already satisfied: loompy in /usr/local/lib/python3.6/dist-packages (from pyscenic) (3.0.6)
Requirement already satisfied: distributed<2.0.0,>=1.21.6 in /usr/local/lib/python3.6/dist-packages (from pyscenic) (1.25.3)
Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from pyscenic) (4.41.1)
Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from pyscenic) (1.4.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from pyscenic) (50.3.2)
Requirement already satisfied: multiprocessing-on-dill in /usr/local/lib/python3.6/dist-packages (from pyscenic) (3.5.0a4)
Requirement already satisfied: cytoolz in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.11.0)
Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (from pyscenic) (2.5)
Requirement already satisfied: interlap in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.2.7)
Requirement already satisfied: dask==1.0.0 in /usr/local/lib/python3.6/dist-packages (from pyscenic) (1.0.0)
Requirement already satisfied: attrs in /usr/local/lib/python3.6/dist-packages (from pyscenic) (20.3.0)
Requirement already satisfied: pyarrow<0.17.0,>=0.11.1 in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.14.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pyscenic) (1.18.5)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from pyscenic) (3.13)
Requirement already satisfied: scikit-learn>=0.20 in /usr/local/lib/python3.6/dist-packages (from umap-learn->pyscenic) (0.22.2.post1)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas<1.0.0,>=0.20.1->pyscenic) (2018.9)
Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas<1.0.0,>=0.20.1->pyscenic) (2.8.1)
Requirement already satisfied: numpy-groupies in /usr/local/lib/python3.6/dist-packages (from loompy->pyscenic) (0.9.13)
Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from loompy->pyscenic) (7.1.2)
Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from loompy->pyscenic) (2.10.0)
Requirement already satisfied: msgpack in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (1.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (2.3.0)
Requirement already satisfied: tblib in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (1.7.0)
Requirement already satisfied: tornado>=4.5.1 in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (5.1.1)
Requirement already satisfied: psutil>=5.0 in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (5.4.8)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (1.15.0)
Requirement already satisfied: toolz>=0.7.4 in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (0.11.1)
Requirement already satisfied: zict>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (2.0.0)
Requirement already satisfied: dill in /usr/local/lib/python3.6/dist-packages (from multiprocessing-on-dill->pyscenic) (0.3.3)
Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx->pyscenic) (4.4.2)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.20->umap-learn->pyscenic) (0.17.0)
Requirement already satisfied: heapdict in /usr/local/lib/python3.6/dist-packages (from zict>=0.1.3->distributed<2.0.0,>=1.21.6->pyscenic) (1.0.1)

[cflerin]

Hi @rdesantis-git ,

I don't have any experience running this in Google Colab, but the error you get is due to Dask issues in Arboreto. See aertslab/pySCENIC/issues/163 for some solutions. The main advice I'd give is to use the multiprocessing script included in the later pySCENIC versions to work around the Dask problems.