Access Anthropic Claude 3.5 Sonnet on Google Vertex AI via python SDK

Claude Sonnet on Vertex

Let’s get right into it. In this explorer video I want to check out Anthropic Claude Sonnet on Vertex AI. It’s super easy to access AI models on Google Cloud since it supports third party easily via model garden.

p.s. This lab uses a VM and Private Service Connect googleapis endpoint for private connectivity to the model.

It’s always fun to lab up things and so to do this test I utilise:

# 1 — Anthropic Claude 3.5 from Vertex AI model garden.
# 2 — Custom VPC with NAT gateway
# 3 — A VM with Anthropic python SDK installed
# 4 — PSC, DNS

Prerequisites:

• You have a project created
• You have permissions to deploy compute, firewall, and use vertex in your environment.
• Note this will incur a cost

diagram

Set up — Create project

# 1 — Create a project. Note you can also use an existing project. (Optional skip if you already have your test project and network.)

Open Cloud Shell and configure as follows.
p.s change YOUR-PROJECT-ID for the name of your project.

gcloud config list project
gcloud config set project YOUR-PROJECT-ID
projectid=YOUR-PROJECT-ID
networkid=nvidia-network
echo $projectid
echo $networkid

Enable API

gcloud services enable dns.googleapis.com
gcloud services enable aiplatform.googleapis.com
gcloud services enable servicedirectory.googleapis.com

Enable Anthropic in model garden

• Go to Vertex AI and select Model Garden
• Search for Anthropic and select Claude 3.5 Sonnet

claude

• Select Enable, you will be required to fill out some info. Fill out form an select Next
• On the final page select Agree to enable Claude 3.5 Sonnet

enable

p.s the process should be the same for any model.

Create VPC

gcloud compute networks create $networkid \
--project=$projectid \
--subnet-mode=custom \
--mtu=1460 \
--bgp-routing-mode=global

Create custom subnet

gcloud compute networks subnets create vm-subnet \
--project=$projectid --range=10.0.88.0/24 \
--stack-type=IPV4_ONLY --network=$networkid \
--region=us-east1

Default firewall rules

gcloud compute firewall-rules create $networkid-allow-icmp --project=$projectid \
--network=$networkid \
--description="Allows ICMP connections from any source to any instance on the network." \
--direction=INGRESS \
--priority=65534 \
--source-ranges=0.0.0.0/0 \
--action=ALLOW \
--rules=icmp

gcloud compute firewall-rules create $networkid-allow-ssh \
--project=$projectid \
--network=$networkid \
--description="Allows TCP connections from any source to any instance on the network using port 22." \
--direction=INGRESS --priority=65534 \
--source-ranges=0.0.0.0/0 --action=ALLOW \
--rules=tcp:22

Create NAT gateway

gcloud compute routers create anthro-out-nat \
    --network $networkid \
    --region us-east1

gcloud compute routers nats create anthro-out-nat-gw \
    --router-region us-east1 \
    --router anthro-out-nat \
    --nat-all-subnet-ip-ranges \
    --auto-allocate-nat-external-ips

Create one VM to access Anthropic on Vertex AI via python SDK.

gcloud compute instances create anthro-vm \
--project=$projectid \
--zone=us-east1-b \
--network-interface=stack-type=IPV4_ONLY,subnet=vm-subnet,no-address,network=$networkid \
--metadata startup-script="#! /bin/bash    
      sudo apt-get update
      sudo apt-get install python3 python3-dev python3-venv -y
      sudo apt-get install tcpdump dnsutils -y
      sudo -i
      sudo mkdir -p ~/py-anthro-env
      cd ~/py-anthro-env
      python3 -m venv env
      source env/bin/activate
      pip install -U ipython google-cloud-aiplatform 'anthropic[vertex]'"

Configure VM and test 1

• Navigate to VM instances. Select the vm starting with anthro-vm. Choose SSH.
• Once you SSH into anthro-vm, enable root by typing sudo -i
• Activate your venv environment:

cd py-anthro-env
source env/bin/activate

• Now let’s authenticate this to do some testing later on. Run the following command in the VM, press y when prompted.

gcloud auth application-default login

• Next copy the url which appears in the starting with https:// open a new tab in your lab browser window and paste the url. Accept the prompts.
• When you see the following select copy, switch back to the vm anthro-vm session and for Enter authorization code: paste the code you copied and press enter to authenticate
• Now let’s do a quick test to see if we can connect to the Vertex API, Claude Sonnet is available in us-east5. We use the dig command with us-east5-aiplatform.googleapis.com to see how the traffic routes.

dig us-east5-aiplatform.googleapis.com

• You should see something similar (the address will differ). Note the path is via public IP addresses since the API is a public API.
• Now let’s use python. Type ipython to activate the ipython interface.

ipython

• Now copy and paste the following code. This asks Claude “Where is the olympics being held in 2024!”
  Replace PUT-YOUR-PROJECT-ID-HERE with your project ID.

from anthropic import AnthropicVertex

project_id = "PUT-YOUR-PROJECT-ID-HERE"
region = "us-east5"

client = AnthropicVertex(project_id=project_id, region=region)

message = client.messages.create(
    model="claude-3-5-sonnet@20240620",
    max_tokens=100,
    messages=[
        {
            "role": "user",
            "content": "Where is the olympics being held in 2024!",
        }
    ],
)
print(message)

• Press enter to run and see the result.
• This request accessed Anthropic via the Vertex public API.
• Close off SSH session let’s continue.

Create PSC googleapis endpoint

To enable private connectivity to our Vertex API endpoint we will create a Private Service Connect endpoint for googleapis. This will allow us to use a private IP address we assign to route traffic to the googleapis we need, in this case Vertex.

• Open Cloud Shell if not open already. Create an IP for the PSC endpoint. We will use 192.168.255.230 in this case.

gcloud compute addresses create anthro-ip \
    --global \
    --purpose=PRIVATE_SERVICE_CONNECT \
    --addresses=192.168.255.230 \
    --network=$networkid


gcloud compute addresses list --filter="name=( 'anthro-ip' ...)"

• Next create the PSC endpoint for googleapis

gcloud compute forwarding-rules create pscanthrovertex \
    --global \
    --network=$networkid \
    --address=anthro-ip \
    --target-google-apis-bundle=all-apis

• Verify it exist

gcloud compute forwarding-rules describe pscanthrovertex --global

Create manual private DNS zone (optional)

You can create a manual DNS entry to point to the PSC endpoint using private DNS. This would affect all the networks you assign to it.

• Navigate to Network services and select Cloud DNS.
• In zones you should see an automatically created zone for the Private Service Connect for Google APIs, with zone type service directory. This can be used to connect to the PSC endpoint with the format SERVICE-ENDPOINT.p.googleapis.com
  Example aiplatform-pscanthrovertex.p.googleapis.com
• In this case we want to manually create a private DNS entry. Go to Cloud DNS and select Create Zone
• We want to manually create a private DNS entry. Go to Cloud DNS and select Create Zone and configure

private dns

• In the Zone details area select Add standard to add an A record

A record

• In the Zone detail area select Add standard to add a CNAME record

cname

• Next we verify connectivity with these changes on anthro-vm

Verification using PSC endpoint

• Go to VM Instance anthro-vm. Select SSH and SSH into the VM
• Gain root access by typing sudo -i
• Check the connectivity path to us-east5-aiplatform.googleapis.com using the ping command. This will ping the IP address in the private DNS, A record for googleapis. This IP a PSC endpoint and your pings will be unsuccessful.

ping -c 2 us-east5-aiplatform.googleapis.com

• Check the connectivity path to us-east5-aiplatform.googleapis.com using the dig command. This should be the IP address of the PSC endpoint.

dig us-east5-aiplatform.googleapis.com

• Check the connectivity path to aiplatform-pscanthrovertex.p.googleapis.com using the dig command. You should see the IP of the PSC endpoint

dig aiplatform-pscanthrovertex.p.googleapis.com

Look at a TCP DUMP

• Open another SSH session into anthro-vm, type sudo -i
• In the vm use the follow command

sudo tcpdump -i any port 53 -n or host us-east5-aiplatform.googleapis.com

• Now switch back to the first SSH instance of VM Instance anthro-vm
• Activate the env using

cd py-gem-env
source env/bin/activate

• Now let’s test python. Type ipython to activate the ipython interface.

ipython

• Now copy and paste the following. This ask Claude “What color are roses”
  Replace PUT-YOUR-PROJECT-ID-HERE with your project ID.

from anthropic import AnthropicVertex

project_id = "PUT-YOUR-PROJECT-ID-HERE"
region = "us-east5"

client = AnthropicVertex(project_id=project_id, region=region)

message = client.messages.create(
    model="claude-3-5-sonnet@20240620",
    max_tokens=100,
    messages=[
        {
            "role": "user",
            "content": "What color are roses?",
        }
    ],
)
print(message)

• Press enter to run and see the result.
• Switch back over to the second instance of VM Instance anthro-vm. You should see the result of the TCPDUMP. You will notice in and out, and the IP address of the VM and the PSC endpoint IP address to connect to us-east5-aiplatform.googleapis.com
• Close all SSH sessions to VM Instance anthro-vm

Clean up

Delete all created elements
Elements to delete
- VM
- Cloud NAT, Cloud Router
- You can also delete the VPC

I’ll be in touch