Setting up IBM Watsonx.ai for API-based text inference

I went though my worst developer-onboarding experience with watsonx, and I want to help sparing a few braincells for anyone, who’s interested in using IBM’s foundational models through an API.

Prelude

Currently, I’m working on ways to test various LLMs for Hungarian text generation. I’ve developed a versatile tool for interacting with various models on various platforms, and I was advised to take a look at the IBM foundation models as well, as they might be “good”. As I had already integrated with Groq, OpenAI, and Antrhopic, it shouldn’t be a problem, right? It should be just a few clicks to sign up here, and I’m off to go, right?

WRONG. I won’t bore you with the details, it involves an old (sleeping) account, non-functioning service-desk pages, and generally lack of information. I got fed up, so after 2 days of waiting, I used my work email to create a new account. That’s where my real agony started.

Expectations

So, I’m at the watsonx dashboard, and searching for what to do to set up a “hello world” example.

Let’s make a dashboard like it’s 1999

I’ve got lot’s of tutorials, except for a specific one for API access. No problem, I have some experience, let’s fiddle around. The dashboard invites me to start up with a sandbox project:

Just go with the defaults, whatever they mean

Give it a name, click on Create. If you can’t find it, just select the Project / view all projects menu item:

And select the “New Project” in the top right corner. After you’ve created your sandbox project, the list would look like this:

Click on the name of the project, it’s a link. You’ll get to the projects overview page:

Project overview page. Resource usage would be 0 for newly created projects. Hopefully.

I thought, I need to create an API key or anything that resembles it, so I checked out the Manage tab, and clicked on Access token

Then I’d use the New access token. You give it a name, and and a role, and that’s it:

Access roles are viewer or editor — at this point, I had no idea what this meant, so I went with viewer. When you get back to the list, the access token can be shown. It’s a pretty long text.

Ok, since we have the things we usually need, let’s head for the API documentation, specifically here.

Ok, now I have the endpoint, and a bit of information about authentication: I have to include an access token. Seems fair, though in the other platforms, I can straight up use an API key.

I checked the API for inferencing text, which is under the Text generation / Infer text menu:

Let’s see, what we need:

• so the version query parameter is required. Let’s go what’s given in the example.
• authorization: I’ve got the access token, fine
• model_id: there’s an endpoint to list the models, but I chose to select one from the list, which I found after some Googling.
• project_id: this is new, none of the other platforms asked for one

The project id must be somewhere around the sandbox project I created. So let’s head back in watsonx’s dashboard, and select our project, and navigate to the Manage / General section:

The only thing worth mentioning is the Project-id at this point

There you go, there’s your project id. Now you can set up the example curl instruction, and you’ll see that…

It will fail

You’ll get a 401 error. The error message is not really helpful (authentication error), so let’s go back to the authentication section of the documentation. It states, that the API we’d like to call uses IBM IAM for authentication, and you’ll need an IAM access token. You can find the link in the menu under Access (IAM), which takes to the IBM Cloud site. For some strange reason, this move always requires me to re-authenticate.

Don’t be tempted to create API keys, just go with the Service IDs, since we’d like to authenticate a machine-to-machine communication, and click on Create at the top right corner.

Give it a name, and click Create. It will return you to the list of service ids. From the list, select the service id, you’ve just created:

Click on API keys tab, then click Create. Give it a name, and click Create. It will pop up an API key:

It has been redacted already, don’t even try it ;-)

Copy it, and save it somewhere safe, since you won’t be able to retrieve it anymore.

Cool, we have an API key, let’s use it in our curl, it should be good, aaaaand…

It will fail again

With the same 401 error message. Argh. Let’s read through the authentication section again. In an utter desperation, follow the link to the IBM access token link, which points to the Generating IBM Cloud IAM token by using an API key document. Copy the example curl include your API key, you’ve just generated, and execute it. You’ll see something like this (at least in a Visual Studio Code terminal):

If you dig through the documentation, you’ll find that this is actually a JWT token, so you can even check it’s validity on jwt.io:

For some mysterious reason, the terminal inserts a new line character before the M character (in black), that should be removed, and now we have an access token. Let’s go!!!!! Try the inference curl, and…

It will fail yet again

But this time, the error code is not 401, but 403, so something is definitively changing! The error message mentions something like this: “Failed to find the iam-ServiceId_your_service_id member in project_id whaterver_your_project_id_is”

I’ll spare you my 2 days of agony and the detailed story of me trying everything I could. Here’s some of my moves:

• Creating a project in IBM Cloud, and use that project_id — that won’t do.
• Adding a configuration to the project, that gives it access to watsonx. Note, in IBM Cloud resources, there’s no such thing as watsonx, rather, you need to look for Watson Machine Learning. Nothing changed.
• Trying to give access rights to my Service ID for IBM Cloud project. Didn’t work either.
• Deleted, recreated everything at least 3 times, to no avail.

My last attempt was searching for the error message, maybe somebody solved the issue. And yes, I’ve stumbled upon a nice solution in Japanese(?) here:

watsonxのAPIを呼び出せるようになるまで - Qiita

The solution

It turns out, I did everything right, except for the last step: giving access to my service id. It is not done from IAM’s side (choosing the resource), but you have to do it in watsonx’s project access control:

A the begining, you’ll see only yourself as a collaborator. The trick is, that you have to add the created service id as a collaborator. This is done by clicking Add collaborators, and selecting Add service ids. First it will pop up an empty list:

Don’t be afraid, type the name of your service id, you’ve created, and it will pop up. Select it, give it rights, and there you go.

Note: you can only add the selected service ids, if you have selected a role for each of them.

Let’s try curl again. Thank god, it works. Text is inferenced, it works pretty fast, I’m delighted. Let’s pop up a bottle of champagne.

Outro

Let’s sum up what would have been the case, if there was a good tutorial:

• Sign up to IBM watsonx, if you haven’t done already.
• Create a project in watsonx — check the project_id, it will be needed. Don’t create an access token, that’s not what you are looking for.
• Go to IAM, and create a Service Id.
• In the API keys section of the service id, create an API key. Save it.
• Go back to the project in watsonx, and add the service id in the projects’ access controls section as collaborator.
• Generate an access token with the help of the Service Id’s API key with IAM’s endpoint.
• Use the access token and project id in the API call to invoke watsonx’ REST API.

That’s it. It would have spared me a lot of headackes. But then again, it gave me a perfect opportunity, to celebrate the success with opening a bottle of wine, and giving a silent toast toast to my Japanese hero, katahiro.