vineyard_read_orc faild on k8s with GraphScope

[JackyYangPassion]

Describe your problem

A clear and concise description of what your problem is. It might be a bug,
a feature request, or just a problem that need support from the vineyard team.

---

If is is a bug report, to help us reproducing this bug, please provide information below:

1. Your Operation System version (uname -a):
   k8s v1.23.6

2. The version of vineyard you use (vineyard.__version__):
   0.18.1

3. Versions of crucial packages, such as gcc, numpy, pandas, etc.:

4. Full stack of the error (if there are a crash):

2023-11-17 06:41:04,779 [DEBUG][script:82]: command is: /home/graphscope/.local/lib/python3.10/site-packages/vineyard/drivers/io/kube_ssh.sh gs-new-orc-jacky:gs-engine-saddem-0 vineyard_read_orc /tmp/vineyard_workspace/vineyard.sock "hdfs:///vertex/graphscope_uid_vertex/dt=2023-10-29/000000_0" eyJob3N0IjogImh3eS1objEtcmlza21ndC1oZHAtbm4tcHJkLTAyLm15aGxsLmNuIiwgInBvcnQiOiA4MDIwLCAiZmlsZXR5cGUiOiAiT1JDIn0= eyJkZWxpbWl0ZXIiOiAiLCIsICJoZWFkZXJfcm93IjogdHJ1ZSwgInNjaGVtYSI6ICJ1X2lkIiwgImNvbHVtbl90eXBlcyI6ICJzdGQ6OnN0cmluZyIsICJpbmNsdWRlX2FsbF9jb2x1bW5zIjogdHJ1ZSwgImFjY3VtdWxhdGUiOiBmYWxzZX0= False 1 0
2023-11-17 06:41:07,193 [DEBUG][launcher:155]: driver: [return] o04952ddf24ad78dc
2023-11-17 06:41:07,193 [DEBUG][stream:184]: [wait] partial ids = [o04952ddf24ad78dc]
Exception ignored in:
<function Launcher.__del__ at 0x7fd0bb505360>
Traceback (most recent call last):
  File "/home/graphscope/.local/lib/python3.10/site-packages/vineyard/launcher/launcher.py", line 63, in __del__
self.join()
  File "/home/graphscope/.local/lib/python3.10/site-packages/vineyard/drivers/io/stream.py", line 156, in join
raise RuntimeError(
RuntimeError
:
Subprocesses failed with the following error: 
Failed to launch job [/home/graphscope/.local/lib/python3.10/site-packages/vineyard/drivers/io/kube_ssh.sh gs-new-orc-jacky:gs-engine-saddem-0 vineyard_read_orc /tmp/vineyard_workspace/vineyard.sock "hdfs:///vertex/graphscope_uid_vertex/dt=2023-10-29/000000_0" ...], exited with 134: 
2023-11-17 14:41:06,409 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
terminate called after throwing an instance of 'orc::TimezoneError'
  what():  Can't open /etc/localtime
command terminated with exit code 134

extra diagnostics are as follows:
2023-11-17 06:41:08,228 [DEBUG][op_executor:921]: new_protocol = vineyard, new_source = o04952ddf6e732724

5. Minimized code to reproduce the error:

# GraphScope Load Vertex and Edge
graph = (
        graph.add_vertices(Loader("hdfs:////vertex/graphscope_uid_vertex/dt=2023-10-29/000000_0",filetype="ORC",host="ip", port=8020)
                           ,label="u_id"
                           ,vid_field='u_id'
                          )
            .add_edges(
                   Loader("hdfs:///edge/graphscope_uid_edge/dt=2023-10-30/000000_0",filetype="ORC",host="ip", port=8020)
                   ,label="login"
                   ,src_label="u_id"
                   ,dst_label="u_id"
                   ,src_field='src_u_id'
                   ,dst_field='dst_u_id'
                 )
             
          )

---

[JackyYangPassion]

CMD ERROR

Failed to launch job 
/home/graphscope/.local/lib/python3.10/site-packages/vineyard/drivers/io/kube_ssh.sh \
gs-new-orc-jacky:gs-engine-saddem-0 \
vineyard_read_orc \
/tmp/vineyard_workspace/vineyard.sock \
"hdfs:///vertex/graphscope_uid_vertex/dt=2023-10-29/000000_0" \

LOG

exited with 134: 
2023-11-17 14:41:06,409 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
terminate called after throwing an instance of 'orc::TimezoneError'
  what():  Can't open /etc/localtime
command terminated with exit code 134

[JackyYangPassion]

when change read csv file, it's running success!

[dashanji]

Hi @JackyYangPassion, thanks for reporting the issue.

Based on the log info, it looks like caused by the timezone error. Could you please check whether the time file /etc/localtime exist in the pod? Also, you could try to add the following env to workaround it.

env:
- name: TZ
  value: "Asia/Shanghai"

[sighingnow]

See also: https://issues.apache.org/jira/browse/ARROW-16386

[JackyYangPassion]

Hi @JackyYangPassion, thanks for reporting the issue.

Based on the log info, it looks like caused by the timezone error. Could you please check whether the time file /etc/localtime exist in the pod? Also, you could try to add the following env to workaround it.

env:
- name: TZ
  value: "Asia/Shanghai"

Not exist in the pod
kubectl exec -it gs-engine-kggmoi-0 bash -n gs-new-orc-jacky2

in the pod the

localtime -> /usr/share/zoneinfo/Asia/Shanghai

do you mean add /etc/localtime file in each pod?

@dashanji

[JackyYangPassion]

See also: issues.apache.org/jira/browse/ARROW-16386

thanks for your reply! @sighingnow

the pod linux version is

PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

but follow the issue
i don't know which dockerfile modify?
can you help me?

[sighingnow]

Fixed in alibaba/GraphScope#3363