Testing out Alibaba’s RISC-V CPU’s for AI inference

How far can I take RISC-V with my test workloads in a cloud based development environment.

Scaleway recently announced they were launching in their Labs Alibaba’s T-Head CPU (C910) RV64GCV running 4 cores @ 1.85 GHz with 16GB LPDDR4 RAM and 128 GB eMMC storage. It’s one of the most powerful RISC-V chips available.

My first thoughts was can I run a website on it, it turns out getting Caddy up and running was simple for a basic site and I even got Tailscale working right away with no problems. More on Caddy later though as I needed to dive deeper into this to get the Cloudflare extension enabled, requiring I compile Go for the system.

The next challenge was figuring out if I could get any AI workloads working on it, I thought to try out Tinygrad a small project but super powerful. I was surprised how easy it was to get things running, only minor modifications and I had the example inference tool running for EfficientNet.

This led me to combine it with a frontend built in a simple Flask app and use Caddy as a reverse proxy for gunicorn. I also wanted to keep it secure for now behind a private subdomain using Tailscale which Caddy makes easy. Getting the Cloudflare plugin installed proved a bit more challenging as I had to compile Go for the first time (this takes a while) to use xbuild for Caddy to get the extension. Once this was done I had a test environment running a simple web app that ran inferencing.

I had made a dropdown to test various datasets and see how things perform. This is still work in progress at the moment but something I’ll get back to once I’ve tested out video encoding / transcoding performance. I also want to try run Android on them for giggles.

So far I’m impressed with these little machines that seem to sip energy and look forward to seeing the price come down so I can build out a bigger cluster perhaps with Kubernetes in the near future!

What do you want me to try? Have you ran into issues or know of any cool RISC-V projects let me know in a comment.

Looking for something else interesting to read: https://www.hpcwire.com/2023/11/08/china-deploys-massive-risc-v-server-in-commercial-cloud/