Dogfooding by Longbin

[longbinlai]

Task 1: 在本地试用 GraphScope

• 安装 GraphScope
• 载入 ogbn_mag 内置图
• Project ogbn_mag到一个简单图，选择点标签为 paper, 边标签为 cites, 属性都为空。
• 在简单图上运行一个 Kcore 算法, 并打印结果为 pandas.DataFrame
• 在 ogbn_mag图上启动交互引擎 （Interactive）
• 通过 gremlin 计算图有多少点，边
• 对于 ogbn_mag图，使用 learning 定义并执行一个 GCN 训练过程，可参考 CI 或文档中定义的过程
• 退出 GraphScope。

Task 2: 在本地开发调试 GraphScope

• 通过 vineyardd直接启动 vineyard
• 从源代码编译 Analytical Engine，找到编译出的binary grape_engine， 通过 grape_engine直接运行 analytical_engine，然后退出 grape_engine 进程
• 为 GAE 添加一条日志 ( analytical_engine/core/server/analytical_server.cc)，并重新编译，启动并找到新添加的日志
• 从源代码编译 Interactive Engine，找到编译出的 frontend-x-SNAPSHOT.jar和 gaia_executor,直接运行这个 jar 包和这个 gaia_executor，使 frontend 能够连接到 executor。
• 了解 Frontend 和 Executor 的代码位置和组织结构，为 GIE 的 Frontend (interactive_engine/frontend/src/main/java/com/alibaba/graphscope/frontend/Frontend.java) 和 executor (interactive_engine/executor/assembly/v6d/src/bin/gaia_executor.rs) 各添加一条日志，并重新编译，启动并找到新添加的日志
• 从源代码编译 Learning Engine, 找到 libgraphlearn_shared.so
• 使用 setup.py 安装 coordinator 和 client。 注意因依赖关系， client 安装要在 coordinator 之前
• 继续安装 GAE，GIE，和 GLE 到预设安装目录。（上一步只有编译。相当于 make 和 make install 的区别）
• 找到 Client， Coordinator， GAE，GIE 和 GLE 的安装位置目录
• 若一切正常，这时你能通过 python 拉起 graphscope。启动python，import graphscope 并载入一个内置 modern_graph 图，在图上启动图交互引擎，发送一个查所有点的语句。
• 退出 graphscope

[longbinlai]

Hardware & System Info

20.5.0 Darwin Kernel Version 20.5.0: Sat May  8 05:10:31 PDT 2021; root:xnu-7195.121.3~9/RELEASE_ARM64_T8101 arm64 arm

Install GraphScope via:

pip3 install graphscope

Error encountered:

[siyuan0322]

Please use python3.10 or python3.11 to install instead. The M1 wheel for 3.8, 3.9 is not avaiable yet. 😭

[longbinlai]

 python3 --version 
Python 3.11.2

FYI.

[siyuan0322]

3.10 is okay

[siyuan0322]

in 3.11 the aiohttp 3.8.1 (which is a dependency of gremlin 3.6.2) failed to compile. Next release we could pin the gremlin version to 3.6.3rc1, which bumps the aiohttp to 3.8.4, thus resolve the issue.

[longbinlai]

I am referring to this doc: https://graphscope.io/docs/latest/overview/getting_started.html.
Now using Python 3.10, the installation has some progress, but still buggy.

[siyuan0322]

Weird... What's your pip version? Seems it's trying to compile some meson... which should not happen. Could you paste some more detailed logs? or just send to me personally.

[longbinlai]

Weird... What's your pip version? Seems it's trying to compile some meson... which should not happen. Could you paste some more detailed logs? or just send to me personally.

pip3 --version
pip 23.0.1 from /opt/homebrew/lib/python3.10/site-packages/pip (python 3.10)

[siyuan0322]

Is there more logs before that error? It should install scipy from wheels, but somehow it decides to compile from source, thus causing errors (openblas not found).

[longbinlai]

And, I have a question: why the installation still continues, even there have encountered some errors?

[longbinlai]

Potentially the installation error is caused by mac-OS-11. Turn out that vineyard did not include a package in pip for mac-OS-11 (now included thanks to @sighingnow ). In addition, scipy also does not have a package for mac-OS-11. According to the suggestions by @sighingnow, the following code may resolve the issue:

export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$(brew --prefix openblas)

pip3 install -U scipy

[sighingnow]

Added to FAQ in #2666.

[longbinlai]

While doing:

import graphscope
from graphscope.dataset import load_ogbn_mag
g = load_ogbn_mag()

The following error encountered:

GRPCError: RPC failed: rpc _connect_session_impl failed: status <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.ABORTED
        details = "Create GraphScope cluster failed"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:127.0.0.1:55577
{created_time:"2023-04-27T14:30:46.110779+08:00", grpc_status:10, grpc_message:"Create GraphScope cluster failed"}"

[longbinlai]

According to a previous PR, this may be related to: #2650. The program is corrupted by an old version of vineyardd. Remove the old vineyardd, and the other error prompts:

2023-04-27 15:54:47,412 [INFO][local_launcher:423]: Server is initializing vineyardd.
dyld: Symbol not found: __ZNKSt3__115basic_stringbufIcNS_11char_traitsIcEENS_9allocatorIcEEE3strEv
  Referenced from: /Users/robin/tutorial-env/lib/python3.10/site-packages/vineyard/bdist/../../vineyard_bdist.dylibs/libprotobuf.3.21.12.0.dylib (which was built for Mac OS X 12.0)
  Expected in: /usr/lib/libc++.1.dylib

2023-04-27 15:54:49,435 [ERROR][local_launcher:528]: Error when launching GraphScope on local
Traceback (most recent call last):
  File "/Users/robin/tutorial-env/lib/python3.10/site-packages/gscoordinator/local_launcher.py", line 525, in start
    self.launch_vineyard()
  File "/Users/robin/tutorial-env/lib/python3.10/site-packages/gscoordinator/local_launcher.py", line 458, in launch_vineyard
    raise RuntimeError(msg)
RuntimeError: Launch vineyardd failed.
Rerun with `graphscope.set_option(log_level='debug')`, to get verbosed vineyardd logs.
2023-04-27 15:54:49,435 [ERROR][local_launcher:528]: Error when launching GraphScope on local
Traceback (most recent call last):
  File "/Users/robin/tutorial-env/lib/python3.10/site-packages/gscoordinator/local_launcher.py", line 525, in start
    self.launch_vineyard()
  File "/Users/robin/tutorial-env/lib/python3.10/site-packages/gscoordinator/local_launcher.py", line 458, in launch_vineyard
    raise RuntimeError(msg)
RuntimeError: Launch vineyardd failed.

[sighingnow]

Fixed by v6d-io/v6d#1352

[longbinlai]

Thanks to @sighingnow, I can now connect to vineyardd without complaining the above dynamic library not found. However, loading load_ogbn_mag() is still in issue (yet another dynamic library not found)

dyld: Symbol not found: _kIOMainPortDefault
  Referenced from: /Users/robin/tutorial-env/lib/python3.10/site-packages/graphscope.runtime/open-mpi/bin/../../../graphscope_runtime/.dylibs/libhwloc.15.dylib (which was built for Mac OS X 13.0)
  Expected in: /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit

2023-04-28 10:43:28,177 [INFO][coordinator:544]: Clean up resources, cleanup_instance: True, is_dangling: False
2023-04-28 10:43:28,178 [INFO][coordinator:544]: Clean up resources, cleanup_instance: True, is_dangling: False
2023-04-28 10:43:28,178 [INFO][coordinator:544]: Clean up resources, cleanup_instance: True, is_dangling: False

[sighingnow]

Tracked by #2668