Proposal: Add ppc64le support

[aklyachkin]

I would like to see support for Linux/ppc64le architecture in OpenFaaS.

What I saw till now it should be quite easy task. I didn't test it, but compilation of almost everything went without any problems. I plan to try out what I compiled till now this or next week.

Expected Behaviour

OpenFaaS works on Linux/ppc64le

Current Behaviour

OpenFaaS doesn't work on Linux/ppc64le

Your Environment

My test environments are IBM Cloud Private 3.2.0 Community Edition on one OpenPOWER server and IBM Cloud Private 3.2.0 Enterprise Edition on IBM Power.

• Docker version docker version (e.g. Docker 17.0.05 ):
  18.06

• Are you using Docker Swarm or Kubernetes (FaaS-netes)?
  Kubernetes

• Operating System and version (e.g. Linux, Windows, MacOS):
  RHEL 7/ppc64le

Next steps

PR

[alexellis]

Thanks for raising this issue. I would think that complication should be relatively straightforward. I'm not aware of any contributors with access to equipment, so testing, support and debugging would be a challenge.

Here are some of the component parts:

https://docs.openfaas.com/contributing/get-started/#openfaas

What kind of involvement or support are you looking for from the project? I noticed a few PRs and new Dockerfiles.

Are you hoping to upstream support to each component and then test everything together, or would it also work to prove it out in your pipeline and then report back on the issue?

Perhaps any issues or PRs could be linked back to this issue for tracking?

[aklyachkin]

As for now I would like just to push necessary changes to support the platform a "clean" build as far as I can and as near to the original as it possible. That's why I made first PRs to some components used in OpenFaaS:

• ppc64le support license-check#9
• https://github.com/openfaas-incubator/faas-idler/pull/29
• ppc64le support nats-queue-worker#63

In my own repositories I have OpenFaaS and Faas-Netes for ppc64le:

• https://github.com/aklyachkin/faas
• https://github.com/aklyachkin/faas-netes

I also compiled several docker images for ppc64le today which are available at https://hub.docker.com/u/powerlinux

The next steps:

• get new official release of license-check compiled for ppc64le - here I need your support ;-)
• then I can update Dockerfiles in faas repo and they're almost identical to amd64 Dockerfiles
• I try to start faas-netes on ICp/ppc64le

If it works, it means for me that OpenFaaS can work on ppc64le.

The most difficult question is testing/support/debugging afterwards. I can do it only in my free time and as long as the project is interesting to me.

[alexellis]

I think there are potentially up to 20 +/- projects (excluding templates) that you may need to rebuild for your architecture. See a complete list at: https://docs.openfaas.com/contributing/get-started/#openfaas

What the difference between s390x and ppc64le? I've had one of your colleagues send a PR to add a binary for s390x to the license-check component. Is this the same thing in other words, or are there two IBM platforms which we need to support upstream in each component?

The most difficult question is testing/support/debugging afterwards. I can do it only in my free time and as long as the project is interesting to me.

Are you with IBM? I see that there are patches / questions about these two IBM platforms coming in from around 3-4 people at the moment. I'm finding it confusing to track all these efforts and would like to know if they are co-ordinated, or if OpenFaaS is just gaining random traction inside a new community.

Durgadas Kamath @imdurgadas has also requested both of these IBM-specific architectures for license-check.

[alexellis]

Related PRs:

openfaas/license-check#6
openfaas/license-check#9

[alexellis]

I would suggest that you may also need faas-cli / watchdog / of-watchdog in addition to the server-side components.

How far do you want to go with the experiment before reviewing whether to submit upstream patches to every component? Is there a way or place where the OpenFaaS community can test the patches or test new, breaking changes against the ppc64le platform?

[alexellis]

openfaas/nats-queue-worker#62
https://github.com/openfaas-incubator/faas-idler/pull/29

[alexellis]

@aklyachkin see the binary release here that you requested: https://github.com/teamserverless/license-check/releases/tag/0.3.2

[aklyachkin]

1. I'm not from IBM and I don't know anybody from IBM interesting in this project.

2. s390x and ppc64le are two different platforms. s390x is IBM Z mainframe hardware and ppc64le is little-endian Linux running on IBM Power Systems. I don't have access to s390x, so it is out of scope for me. I recently learned some people who are running Kubernetes on s390x, but they didn't show interest on OpenFaaS.

3. As for me I'm interested in bringing OpenFaaS server side on ppc64le. I understand I have to port a little bit more thing as I did before, but as for now it works:

 [root@k8s yaml_ppc64le]# kubectl get nodes -o wide
 NAME             STATUS   ROLES                                 AGE     VERSION       INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                   KERNEL-VERSION                CONTAINER-RUNTIME
 172.16.200.139   Ready    etcd,management,master,proxy,worker   4d11h   v1.13.5+icp   172.16.200.139   <none>        Red Hat Enterprise Linux   3.10.0-957.21.3.el7.ppc64le   docker://18.6.2

 [root@k8s yaml_ppc64le]# kubectl get pod -n openfaas
 NAME                                 READY   STATUS    RESTARTS   AGE
 alertmanager-554f5fdc78-hh2l5        1/1     Running   0          19m
 basic-auth-plugin-6f7df46d79-tg5rl   1/1     Running   0          20m
 faas-idler-858f9cb89b-fn2bq          1/1     Running   0          4m40s
 gateway-d95cb88f7-fl2nv              2/2     Running   0          5m32s
 nats-bbfdfbcc8-hzfjh                 1/1     Running   0          20m
 prometheus-7d4cfd954f-lj455          1/1     Running   0          11m
 queue-worker-7df45bfb8d-95crg        1/1     Running   0          13m

All patches I did to get it working I will commit back to the project. It is not so much work.

4. I would also like to try get working OpenFaaS functions on IBM AIX. It is not Linux and I don't expect here easy going. But AIX can work as Kubernetes working node (https://www.ibm.com/developerworks/community/blogs/cgaix/entry/Kubernetes_on_AIX_Did_you_say_AIX?lang=en), so why not to try?

[alexellis]

1. I hope they will comment on this thread and introduce themselves. You all appear to be interested in the same result.
2. Thanks for clearing that up. @imdurgadas requested both and Rishi requested just ppc64le
3. For ppc64le to run server-side, you may also need to patch the watchdog and of-watchdog. You can use the faas-cli for x86_64. You should also look at the templates and how they are constructed. A separate repo can be maintained for that i.e. a fork of any of the templates seen via faas-cli template store list

Congratulations on getting it working!

• Can you now build and deploy a function to show that it works end to end?
• Can you port a template like node10-express? https://github.com/openfaas-incubator/node10-express-template/
• Does the UI work?
• Is the auto-scaling functional? https://github.com/openfaas/workshop/blob/master/lab9.md

Once your patches are contributed back, who will be fixing, updating and maintaining support for ppc64le? And for how long?

4. IBM AIX is a kind of UNIX unless I'm mistaken. That's certainly interesting, but I think the focus should be kept to doing ppc64le properly rather than spreading out what seems to be a relatively small amount of your time.

[alexellis]

I believe Rishi from the IBM enablement team was trying to get [OpenFaaS Cloud](https://github.com/openfaas/openfaas-cloud working) - which would of course also include OpenFaaS (this issue)

[aklyachkin]

Pull request #1258

As you can see Dockerfile for gateway is absolute identical for amd64 and ppc64le.

[aklyachkin]

I checked also faas-cli. There is no single change needed to compile it on ppc64le.

[aklyachkin]

At least the most easiest function works:

[root@k8s faas-ppc64le]# faas-cli new hello-ppc64le --lang python-ppc64le
Folder: hello-ppc64le created.
  ___                   _____           ____
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
      |_|


Function created in folder: hello-ppc64le
Stack file written: hello-ppc64le.yml

[root@k8s faas-ppc64le]# faas-cli build -f ./hello-ppc64le.yml
[0] > Building hello-ppc64le.
Clearing temporary build folder: ./build/hello-ppc64le/
Preparing ./hello-ppc64le/ ./build/hello-ppc64le//function
Building: k8s.enfence.net:8500/openfaas-fn/hello-ppc64le:latest with python-ppc64le template. Please wait..
Sending build context to Docker daemon  8.192kB
Step 1/29 : FROM powerlinux/classic-watchdog:latest-dev-ppc64le as watchdog
 ---> b66ef1b27615
Step 2/29 : FROM ppc64le/python:2.7-alpine
 ---> 4e90e5cc7f1b
Step 3/29 : ARG ADDITIONAL_PACKAGE
 ---> Using cache
 ---> 64cd296363f3
Step 4/29 : COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
 ---> Using cache
 ---> ec95dc31b623
Step 5/29 : RUN chmod +x /usr/bin/fwatchdog
 ---> Using cache
 ---> 9ccc8d988ce1
Step 6/29 : RUN apk --no-cache add ca-certificates ${ADDITIONAL_PACKAGE}
 ---> Using cache
 ---> 6a40fb26f47e
Step 7/29 : RUN addgroup -S app && adduser app -S -G app
 ---> Using cache
 ---> c1c5a926ffc3
Step 8/29 : WORKDIR /home/app/
 ---> Using cache
 ---> 74823a8a0a07
Step 9/29 : COPY index.py           .
 ---> Using cache
 ---> bf89a5a2faca
Step 10/29 : COPY requirements.txt   .
 ---> Using cache
 ---> 9d767e38817b
Step 11/29 : RUN chown -R app /home/app &&   mkdir -p /home/app/python && chown -R app /home/app
 ---> Using cache
 ---> 2708aefcf526
Step 12/29 : USER app
 ---> Using cache
 ---> 37d2733ebe95
Step 13/29 : ENV PATH=$PATH:/home/app/.local/bin:/home/app/python/bin/
 ---> Using cache
 ---> 3255279c1e7c
Step 14/29 : ENV PYTHONPATH=$PYTHONPATH:/home/app/python
 ---> Using cache
 ---> e9d6901b8e1c
Step 15/29 : RUN pip install -r requirements.txt --target=/home/app/python
 ---> Using cache
 ---> 098aa3bc4736
Step 16/29 : RUN mkdir -p function
 ---> Using cache
 ---> 43ced72ed227
Step 17/29 : RUN touch ./function/__init__.py
 ---> Using cache
 ---> 4715f9604dd9
Step 18/29 : WORKDIR /home/app/function/
 ---> Using cache
 ---> ca004600b735
Step 19/29 : COPY function/requirements.txt	.
 ---> Using cache
 ---> f485f5a48d77
Step 20/29 : RUN pip install -r requirements.txt --target=/home/app/python
 ---> Using cache
 ---> 74b4e29d13ed
Step 21/29 : WORKDIR /home/app/
 ---> Using cache
 ---> 3484604d606a
Step 22/29 : USER root
 ---> Using cache
 ---> f64bef9e3593
Step 23/29 : COPY function           function
 ---> Using cache
 ---> 409e2055e1bc
Step 24/29 : RUN chown -R app:app ./ &&   chmod -R 777 /home/app/python
 ---> Using cache
 ---> d11b57289573
Step 25/29 : USER app
 ---> Using cache
 ---> 358dfb0099b7
Step 26/29 : ENV fprocess="python index.py"
 ---> Using cache
 ---> 6ae3f1fadb62
Step 27/29 : EXPOSE 8080
 ---> Using cache
 ---> 2d1fecd36bb1
Step 28/29 : HEALTHCHECK --interval=3s CMD [ -e /tmp/.lock ] || exit 1
 ---> Using cache
 ---> 10791fbccc6d
Step 29/29 : CMD ["fwatchdog"]
 ---> Using cache
 ---> 5ff3a8f97d4c
Successfully built 5ff3a8f97d4c
Successfully tagged k8s.enfence.net:8500/openfaas-fn/hello-ppc64le:latest
Image: k8s.enfence.net:8500/openfaas-fn/hello-ppc64le:latest built.
[0] < Building hello-ppc64le done.
[0] worker done.

[root@k8s faas-ppc64le]# docker push k8s.enfence.net:8500/openfaas-fn/hello-ppc64le:latest
The push refers to repository [k8s.enfence.net:8500/openfaas-fn/hello-ppc64le]
d6e3a21c5437: Pushed
4258813f7d04: Pushed
aa10c958c3c7: Pushed
9eabaf692b12: Pushed
614c5e3ae3ea: Pushed
6f0f83f8a964: Pushed
addbc69d004c: Pushed
9a518ec71d66: Pushed
fdd6c21b969c: Pushed
8851091ea059: Pushed
1b6665f8f6f8: Pushed
82f2ffdb0adc: Pushed
58cfb351121c: Pushed
b2f2b401a975: Pushed
5e2687363386: Pushed
a131a19cf682: Pushed
3533830e9fa9: Pushed
latest: digest: sha256:670429ea822bd578615f0d261a1f897154b5eae1cfb3fb3c8d3b50708861dca8 size: 4074

[root@k8s faas-ppc64le]# faas-cli deploy -f ./hello-ppc64le.yml
Deploying: hello-ppc64le.
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.

Deployed. 202 Accepted.
URL: http://k8s.enfence.net:31112/function/hello-ppc64le

[root@k8s faas-ppc64le]# faas-cli list -g $OPENFAAS_URL
Function                      	Invocations    	Replicas
hello-ppc64le                 	0              	1

[root@k8s faas-ppc64le]# echo hi | faas-cli invoke hello-ppc64le -g $OPENFAAS_URL
Hello from ppc64le! You said: hi

hi