Knative on Raspberry Pi

Recently I set up Knative on Raspberry Pi, this post is on the set up, from purchases to the first knative service and beyond :-).

Rationale

I had lot of reasons to set this cluster up, listing some below

1. ARM64 experience
2. Home lab
3. Hacking with k8s/knative
4. Reduce some of my cloud expenses :-)

Purchases

3 X Raspberry Pi 4b (8gb) — $75 each

3 X Power Supply — $8 each

3 X MicroSD (64gb) — $10 each

Cluster Case — $25

Flashing the OS on Micro SD

I installed ubuntu 23.04 LTS with the steps provided in this link . When you set up pi as headless, it is very important to set up the wifi, so please complete the wifi step in the above link. Repeated the steps in all PI’s.

Kubernetes

Time for kubernetes. I used k3s to set up kubernetes. Inserted the microsd and powered up the pi’s.

Preparation

Did the below steps on all nodes.

1. Edit the file -> /boot/firmware/cmdline.txt , add the below to end

cgroup_enable=memory cgroup_memory=1

2. Change the host name under /etc/hostname and reboot

sudo reboot

Kubernetes

Time to install kubernetes. K3s by default installs klipper load balancer and traefik gateway on the cluster. Klipper lb assigns one of the available node IP’s to lb type kube services. Main caveat is, if the ports collide between two load balancer type kubernetes services, then one would not start. In my case I would install contour gateway later, which will collide with the traefik gateway, so I disabled traefik while installing.

$ curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik" sh -s -
# test if master node is up
$ sudo k3s kubectl get node
# Get the node token for workers to join
$ sudo cat /var/lib/rancher/k3s/server/node-token

On each node run the below commands by replacing the master node ip and node token to join the cluster

$ curl -sfL https://get.k3s.io | K3S_URL=https://<masternode_ip>:6443 K3S_TOKEN=$NODE_TOKEN sh -
# On the master to check if all nodes are up and running
$ sudo k3s kubectl get node

kubeconfig (optional)

Optionally, if you want to set up the kubeconfig on your workstation. Get the kubeconfig from the cluster

$ sudo k3s kubectl config view

I did this step, so I can run the rest of the steps from my workstation.

Contour

Knative needs a gateway for networking, I used contour for this cluster. Latest knative and contour nightly images ship with arm64 support.

$ kubectl apply -f https://storage.googleapis.com/knative-nightly/net-contour/latest/contour.yaml 

Knative Serving

Installed the nightly.

$ kubectl apply -f https://storage.googleapis.com/knative-nightly/serving/latest/serving-crds.yaml
$ kubectl apply -f https://storage.googleapis.com/knative-nightly/serving/latest/serving-core.yaml
$ kubectl apply -f https://storage.googleapis.com/knative-nightly/net-contour/latest/net-contour.yaml
$ kubectl patch configmap/config-network \
  --namespace knative-serving \
  --type merge \
  --patch '{"data":{"ingress.class":"contour.ingress.networking.knative.dev"}}'

Now you should have fully functional knative cluster.

Testing

Time to create the first Knative service

kubectl apply -f - <<EOF
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hw
spec:
  template:
    spec:
      containers:
      - image: murugappans/hw
EOF

Invoke the service

$ curl -H "Host: hw.default.example.com" "http://$(sudo k3s kubectl get svc envoy -n contour-external  -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')"
{"status":"Status OK"}

This completes the basic set up of Knative on Raspberry Pi

Advanced Steps

Now that you have basic working cluster, let us take it forward

ko multi arch build

ko 0.6 onwards, multi arch build is supported, which makes it very easy to deploy Go services onto to arm64 cluster. It us important to have the correct base image in .ko.yaml to take advantage of this. “gcr.io/distroless/static:nonroot” is what I use and it is the default image, supports amd64, arm64, power and z.

Assuming you have set up kubeconfig of the rpi cluster, kn, and ko in your machine, if not follow the hyper links.

# set docker registry
$ export KO_DOCKER_REPO=your-docker-repo
# build, publish image and create knative service
$ kn service create hw-with-ko --image=$(ko publish -B github.com/itsmurugappan/knative-examples/ksvc-ksvc/cmd/hw --platform=all)
.
.
.
.
Creating service 'hw-with-ko' in namespace 'default':
0.330s The Configuration is still working to reflect the latest desired specification.
  0.723s The Route is still working to reflect the latest desired specification.
  1.133s Configuration "hw-with-ko" is waiting for a Revision to become ready.
 11.483s ...
 12.199s Ingress has not yet been reconciled.
 14.196s Waiting for Envoys to receive Endpoints data.
 15.991s Waiting for load balancer to be ready
 16.847s Ready to serve.
Service 'hw-with-ko' created to latest revision 'hw-with-ko-hpmfh-1' is available at URL:
http://hw-with-ko.default.example.com
# Invoke service
$ curl -H "Host: hw-with-ko.default.example.com" "http://$(sudo k3s kubectl get svc envoy -n contour-external  -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')"
Hello World

DNS set up

My intention was to set up a home lab (no external access). Initially wanted to set up metal lb and a dns server. Setting up metal lb in ARP mode threw some weird challenges (like it would stop working after ~ 8 min.). So for now I have put metal lb and dns server in the back burner. As a work around using klipper load balancer, seems to solve the purpose , will have issues if I have another lb kube service with port 80/443.

Now we have an ip for contour gateway, next choose a domain, update the knative config and make the domain resolvable in the home network.

Follow the step here to change the domain in knative config.

To make the domain resolvable I used dnsmasq. In my work station followed the below steps to set up dnsmasq

$ brew install dnsmasq
# Open /usr/local/etc/dnsmasq.conf and add the address
$ address=/.yourdomain.fun/<contour gateway ip>
# please note if your domain is .dev it can only be accessed securely over https on browsers.
# create a file called /etc/resolver/yourdomain.fun with the following content:
nameserver 127.0.0.1
$ sudo brew services start dnsmasq
$ dig test.yourdomain.fun @127.0.0.1
;; ANSWER SECTION:
test.yourdomain.fun.  0 IN A <contour gateway ip>

Invoke the knative service from the browser/curl

curl http://hw-with-ko.default.yourdomain.fun/
Hello World

Cluster Case Assembly

Final Thoughts

Am sure this set up is going to be dated soon, but am planning to update the post as and when things change.