Dogfooding by dashanji #2680

dashanji · 2023-05-09T07:08:28Z

dashanji
May 9, 2023
Collaborator

Task 1: 在本地试用 GraphScope

安装 GraphScope
载入 ogbn_mag 内置图
Project ogbn_mag到一个简单图，选择点标签为 paper, 边标签为 cites, 属性都为空。
在简单图上运行一个 Kcore 算法, 并打印结果为 pandas.DataFrame
在 ogbn_mag图上启动交互引擎（Interactive）
通过 gremlin 计算图有多少点，边
对于 ogbn_mag图，使用 learning 定义并执行一个 GCN 训练过程，可参考 CI 或文档中定义的过程
退出 GraphScope。

Task 2: 在本地开发调试 GraphScope

Task 3: 运行本地测试

找到 CI 定义的位置和文件
找到单元测试的位置，参考在 CI 中是如何运行这些单元测试的步骤，并在本地运行CI上的本地测试（如果没有，可开 Issue 加入运行）
GAE 的单元测试
GIE 的单元测试
GLE 的单元测试
运行一组 Python 中的单元测试 test_min.py 。（Python 中的单元测试更像是集成测试，因为往往由多个引擎参与）
在 python/graphscope/tests/minitest下新增一个test 文件，在里面添加一个涉及多个引擎的测试，其中要包括载入一张 ogbn_mag 图，打印 Kcore 算法结果，通过 Gremlin 查询图的点边数量，执行 GCN 训练。

Task 4: 在 Kubernetes 上使用 GraphScope （Python SDK）

通过客户端在 K8S 环境下安装 Graphscope，建立一个 Session，通过 kubectl 查看拉起来的 Pod 和 Service （可通过 minikube/kind 在单机搭建 K8S 环境）
通过载图 API 载一个图 ogbn_mag
在图上运行一个 LPA 算法，并打印结果为 pandas.DataFrame
在图上启动交互引擎（Interactive）
通过 gremlin 计算图有多少点，边
使用 learning 定义并执行一个 GCN 训练过程
关闭 Session

Task 5: 在 Kubernetes 上使用 GraphScope （Helm）

通过 Helm 安装 GraphScope，并载图，运行算法，Gremlin 和 Learning

Task 6: 独立部署

独立部署 GAE，并载图 modern_graph，运行 LPA 算法
独立部署 GIE，发送 gremlin 查询点边数量

Task 7: 打包镜像与了解 Kubernetes 上运行的基本概念

构建 GAE 的镜像
构建 GIE 的镜像（包括 Frontend 和 Executor）
构建 GLE 的镜像
构建 Coordinator 的镜像
为 GAE 添加一条日志 ( analytical_engine/core/server/analytical_server.cc) 并重新构建镜像，在不使用 docker push的情况下，使用新的镜像启动并在日志中可以找到新添加的日志（使用 registry 和 tag 来选择使用哪个镜像）
为 GIE 的 Frontend (interactive_engine/frontend/src/main/java/com/alibaba/graphscope/frontend/Frontend.java) 和 executor (interactive_engine/executor/assembly/v6d/src/bin/gaia_executor.rs) 各添加一条日志并重新构建 Frontend 和 Executor 镜像，在不使用 docker push的情况下，使用新的镜像启动并可以找到新添加的日志
为 Coordinator 添加一个异常 (coordinator/gscoordinator/coordinator.py)，并重新构建 Coordinator 镜像。启动并找到异常的报错。（日志在控制台中或者在 Coordinator Pod 的日志中）
清除所有 GraphScope 的资源，包括镜像，容器。

Task 8: 运行 K8S 上的测试

找到在 Kubernetes 上运行的测试，找到在 CI 中是如何运行这些单元测试的
参考 CI 的步骤，运行 test_demo_scripts.py 中的 test_demo_distribute 测试
尝试添加一个在 Kubernetes 上运行的测试，并运行。其中要包括载入一张 ogbn_mag 图，打印 Kcore 算法结果，通过 Gremlin 查询图的点边数量，执行 GCN 训练。

Advance 部分

Task 9 : 打包为 Wheel 包

从源码构建 Python graphscope-client Wheels
从源码构建 Python graphscope wheels
安装构建出的 Wheel

Environment:

OS:

$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	AlibabaCloud
Description:	Alibaba Cloud Linux release 3 (Soaring Falcon)
Release:	3
Codename:	SoaringFalcon

dashanji · 2023-05-09T16:02:07Z

dashanji
May 9, 2023
Collaborator Author

Q1: the python version of Alibaba Cloud Linux release 3 is too low. (v3.6)

Python3 version: 3.6.8
Pip version: pip 21.3.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)

Run python3 -m pip install graphscope -U and get the following error.

ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bhhvjkct/treelib_48e693c6f0fc4e4bace45f86038cd126/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bhhvjkct/treelib_48e693c6f0fc4e4bace45f86038cd126/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-8q4v8y_i
       cwd: /tmp/pip-install-bhhvjkct/treelib_48e693c6f0fc4e4bace45f86038cd126/
  Complete output (7 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-bhhvjkct/treelib_48e693c6f0fc4e4bace45f86038cd126/setup.py", line 4, in <module>
      from treelib import __version__
    File "/tmp/pip-install-bhhvjkct/treelib_48e693c6f0fc4e4bace45f86038cd126/treelib/__init__.py", line 1, in <module>
      from tree import Tree
  ModuleNotFoundError: No module named 'tree'
  ----------------------------------------
WARNING: Discarding http://mirrors.cloud.aliyuncs.com/pypi/packages/7c/6d/f47cb8993f31c06f882c9f2ed3041fabcce3024e899d9599a317e560d7a2/treelib-1.2.4.tar.gz#sha256=fbeddbb03eb34a614cb926674b23135edb4774f4a6c3d1f37db4db8b9ab8e4a6 (from http://mirrors.cloud.aliyuncs.com/pypi/simple/treelib/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached http://mirrors.cloud.aliyuncs.com/pypi/packages/1d/b8/7ed7e9c96224ce2403d1587a54c773dbe6be9bedc98489320219b6919328/treelib-1.2.3.tar.gz (10 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bhhvjkct/treelib_224ea85bb8174f72b5cecfa9ace1df85/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bhhvjkct/treelib_224ea85bb8174f72b5cecfa9ace1df85/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-_0kteoi6
       cwd: /tmp/pip-install-bhhvjkct/treelib_224ea85bb8174f72b5cecfa9ace1df85/
  Complete output (7 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-bhhvjkct/treelib_224ea85bb8174f72b5cecfa9ace1df85/setup.py", line 4, in <module>
      from treelib import __version__
    File "/tmp/pip-install-bhhvjkct/treelib_224ea85bb8174f72b5cecfa9ace1df85/treelib/__init__.py", line 1, in <module>
      from tree import Tree
  ModuleNotFoundError: No module named 'tree'
  ----------------------------------------
WARNING: Discarding http://mirrors.cloud.aliyuncs.com/pypi/packages/1d/b8/7ed7e9c96224ce2403d1587a54c773dbe6be9bedc98489320219b6919328/treelib-1.2.3.tar.gz#sha256=39d76cde41cc8ed98d586c136c12ceb97766dc436c11e75b190fb4315e8a19cd (from http://mirrors.cloud.aliyuncs.com/pypi/simple/treelib/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached http://mirrors.cloud.aliyuncs.com/pypi/packages/33/08/6179be3e9b0cd597d50d23c84b0efbca404011f251ad28a11b47a94f7c1e/treelib-1.2.2.tar.gz (9.4 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bhhvjkct/treelib_fdaec9a710d74d1abedf9a3c7f71f809/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bhhvjkct/treelib_fdaec9a710d74d1abedf9a3c7f71f809/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-d4um46oj
       cwd: /tmp/pip-install-bhhvjkct/treelib_fdaec9a710d74d1abedf9a3c7f71f809/
  Complete output (7 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-bhhvjkct/treelib_fdaec9a710d74d1abedf9a3c7f71f809/setup.py", line 4, in <module>
      from treelib import __version__
    File "/tmp/pip-install-bhhvjkct/treelib_fdaec9a710d74d1abedf9a3c7f71f809/treelib/__init__.py", line 1, in <module>
      from tree import Tree
  ModuleNotFoundError: No module named 'tree'
  ----------------------------------------
WARNING: Discarding http://mirrors.cloud.aliyuncs.com/pypi/packages/33/08/6179be3e9b0cd597d50d23c84b0efbca404011f251ad28a11b47a94f7c1e/treelib-1.2.2.tar.gz#sha256=f479f660a2c4cc8d7fb9ca3a0fa16f0d0459580c6ea816a6fc2b924060b971eb (from http://mirrors.cloud.aliyuncs.com/pypi/simple/treelib/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached http://mirrors.cloud.aliyuncs.com/pypi/packages/e3/43/8d5192b8bddd2e832fdf5769094c7573a79749ab5fb8bfbd01a43e08fd41/treelib-1.2.tar.gz (8.6 kB)
  Preparing metadata (setup.py) ... done
INFO: pip is looking at multiple versions of tqdm to determine which version is compatible with other requirements. This could take a while.
Collecting tqdm
  Using cached http://mirrors.cloud.aliyuncs.com/pypi/packages/8a/c4/d15f1e627fff25443ded77ea70a7b5532d6371498f9285d44d62587e209c/tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
pip install --upgrade pip^CERROR: Operation cancelled by user

A1: upgrade the python version to a high one(e,g. v3.9)
Install some libraries:

sudo yum install -y make automake gcc gcc-c++ kernel-devel libffi-devel openssl-devel bzip2-devel java

Upgrade the python version to v3.9.

$ wget https://www.python.org/ftp/python/3.9.2/Python-3.9.2.tgz && tar -xzvf Python-3.9.2.tgz && cd Python-3.9.2/ && ./configure --enable-optimizations --with-ensurepip=install --with-bz2 && sudo make && sudo make install &&  rm -rf /usr/local/bin/python3 && ln -s /usr/local/bin/python3.9 /usr/local/bin/python3 && rm /usr/bin/python && ln -s /usr/local/bin/python3.9 /usr/bin/python

Uprage pip3.

python3 -m pip install --upgrade pip

Install Graphscope

python3 -m pip install graphscope -U

0 replies

dashanji · 2023-05-25T04:57:12Z

dashanji
May 25, 2023
Collaborator Author

Install the deps for gae and get some errors.

$ ./gs install-deps dev
Setting the environment for development.

[WARNING] GRAPHSCOPE_HOME will set to source root (/root/GraphScope) for development.
[WARNING] To use you assigned GRAPHSCOPE_HOME, export GRAPHSCOPE_ENV=prod.

Read the env: GRAPHSCOPE_HOME=/root/GraphScope
args:
- ${args[--from-local]} = /tmp/gs-local-deps
- ${args[--install-prefix]} = /opt/graphscope
- ${args[type]} = dev
- ${args[--v6d-version]} = main
[WARNING] Please note that I am running as root.
Installing dev dependencies for GraphScope on Alibaba Cloud Linux-3...
[ERROR] The script is only support platforms of Ubuntu/CentOS/macOS

0 replies

dashanji · 2023-06-05T11:56:43Z

dashanji
Jun 5, 2023
Collaborator Author

Great thanks to @sighingnow

If you do the test on the Alibaba-Cloud-Linux, update the locale as follows.

export LC_ALL=en_US.UTF-8

1 reply

sighingnow Jun 5, 2023
Maintainer

Fixed by v6d-io/v6d#1415

dashanji · 2023-06-05T12:31:44Z

dashanji
Jun 5, 2023
Collaborator Author

Install deps for Alibaba Cloud linux

./gs install-deps dev
source /root/.graphscope_env

install the engines.

make install

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dogfooding by dashanji #2680

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Dogfooding by dashanji #2680

dashanji May 9, 2023 Collaborator

Task 1: 在本地试用 GraphScope

Task 2: 在本地开发调试 GraphScope

Task 3: 运行本地测试

Task 4: 在 Kubernetes 上使用 GraphScope （Python SDK）

Task 5: 在 Kubernetes 上使用 GraphScope （Helm）

Task 6: 独立部署

Task 7: 打包镜像与了解 Kubernetes 上运行的基本概念

Task 8: 运行 K8S 上的测试

Advance 部分

Task 9 : 打包为 Wheel 包

Replies: 4 comments · 1 reply

dashanji May 9, 2023 Collaborator Author

dashanji May 25, 2023 Collaborator Author

dashanji Jun 5, 2023 Collaborator Author

sighingnow Jun 5, 2023 Maintainer

dashanji Jun 5, 2023 Collaborator Author

dashanji
May 9, 2023
Collaborator

Replies: 4 comments 1 reply

dashanji
May 9, 2023
Collaborator Author

dashanji
May 25, 2023
Collaborator Author

dashanji
Jun 5, 2023
Collaborator Author

sighingnow Jun 5, 2023
Maintainer

dashanji
Jun 5, 2023
Collaborator Author