How to Make Weaviate Calls for Efficient Data Queries

Let’s talk about something that we all face during development: API Testing with Postman for your Development Team.

Yeah, I’ve heard of it as well, Postman is getting worse year by year, but, you are working as a team and you need some collaboration tools for your development process, right? So you paid Postman Enterprise for…. $49/month.

Now I am telling you: You Don’t Have to:

That’s right, APIDog gives you all the features that comes with Postman paid version, at a fraction of the cost. Migration has been so easily that you only need to click a few buttons, and APIDog will do everything for you.

How to Migrate Postman Collections/Environments to Apidog

APIDog has a comprehensive, easy to use GUI that makes you spend no time to get started working (If you have migrated from Postman). It’s elegant, collaborate, easy to use, with Dark Mode too!

Want a Good Alternative to Postman? APIDog is definitely worth a shot. But if you are the Tech Lead of a Dev Team that really want to dump Postman for something Better, and Cheaper, Check out APIDog!

Apidog An integrated platform for API design, debugging, development, mock, and testing

Weaviate Calls: An In-Depth Examination

Understanding Weaviate and Its Purpose

Weaviate is an open-source vector search engine designed for managing and retrieving high-dimensional data using a machine learning model. It employs a graph-like structure, allowing businesses and researchers to perform searches that are not solely based on keyword matching but on the contextual understanding of data. This capability becomes increasingly essential in today’s data-rich environment, as conventional search techniques often fall short of meeting user demands for relevance and specificity.

Weaviate calls refer to the interactions made with the Weaviate backend to execute various tasks such as data ingestion, search queries, and cluster management. Users interact with Weaviate through its API, which exposes many functions that allow users to perform complex queries and store vector embeddings efficiently. This paper will explore the different types of Weaviate calls, provide detailed mechanisms of how these calls work, and guide users in employing Weaviate effectively in diverse scenarios.

Types of Weaviate Calls

The core types of Weaviate calls can be categorized into four main categories: creating and managing schema, data ingestion, searching, and managing clusters. Each of these categories serves a distinct purpose and employs unique endpoints in the Weaviate API.

Schema Management Calls

Creating and managing schemas is an essential first step when working with Weaviate. Schemas dictate how data is structured within the database. Before data can be ingested, you must define the classes, properties, and relationships between assets.

Steps to Create a Schema:

1. Define Your Class: Begin by identifying what entities you want to store in Weaviate. For example, you may want to store “Books” as a class.
2. Structure Properties: Within the “Books” class, you might define properties such as “title”, “author”, “publishedYear”, and “genre”.
3. Using the API Endpoint: You can create a schema by making a POST request to the following endpoint:

• POST /schemas

1. Example Payload:

• { "classes": [ { "class": "Books", "properties": [ { "name": "title", "dataType": ["string"] }, { "name": "author", "dataType": ["string"] }, { "name": "publishedYear", "dataType": ["int"] }, { "name": "genre", "dataType": ["string"] } ] } ] }

1. Execute the Call: Use tools like Postman or cURL to execute the API call. After execution, you will receive a response that confirms the successful creation of the schema.

Data Ingestion Calls

After defining your schema, the next step involves feeding data into the Weaviate vector database. This process includes creating instances of your defined classes and populating them with relevant attributes.

Steps for Data Ingestion:

1. Prepare Your Data: Collect the data you want to add to Weaviate. For example, if you are adding a book, you might gather the title, author, genre, and published year.
2. Formulate the Instances: Each instance contains the class you previously defined and its associated properties.
3. Using the API Endpoint: To ingest data, you will make POST calls directed to:

• POST /<your-class-name>

1. Example Payload:

• { "title": "The Great Gatsby", "author": "F. Scott Fitzgerald", "publishedYear": 1925, "genre": "Fiction" }

1. Execute the Call: Similar to the schema creation, use Postman or cURL to send the data to Weaviate. Ensure all required properties are included according to your schema.
2. Validation: After the ingestion call, verify the success of the operation through the API response, which should confirm the addition of data.

Search Calls

The primary functionality of Weaviate lies in its ability to perform powerful search queries. These queries leverage vector embeddings to return contextually relevant results, rather than relying on simple keyword searches.

Steps to Perform a Search:

1. Identify the Search Parameters: You need to decide what data you want to search for. For example, if you wish to find books related to “American Dream.”
2. Formulate the Search Query: Weaviate uses a GraphQL-based search syntax for querying:

• GET /v1/graphql

1. Example Query:

• { Get { Books (where: { path: ["genre"] operator: Equal valueString: "Fiction" }) { title author publishedYear } } }

1. Execute the Call: Send your request using tools like Postman, ensuring to set the request type to GET.
2. Analyze Results: Once executed, the response will contain the filtered list of books matching your search criteria. Examine the data to ensure it meets your expectations.

Managing Clusters

In a production environment, managing clusters effectively becomes crucial for performance and resilience. Weaviate supports various cluster management calls that enable users to monitor and control their instances.

Steps for Cluster Management Calls:

1. Monitor Cluster Health: For maintaining visibility into your cluster’s health, issue a GET request to:

• GET /v1/cluster

1. This provides vital information such as active nodes and their statuses.
2. Scaling the Cluster: If you anticipate an increase in data or queries, a scaling operation would be prudent. You might execute a POST call to add nodes:

• POST /v1/cluster/add

1. Example Payload:

• { "nodes": 5 }

1. Analyze Resource Utilization: Investigate the resource utilization and performance by accessing metrics from:

• GET /v1/metrics

1. This can help in optimizing performance and understanding bottlenecks.

Advanced Weaviate Calls

Beyond basic CRUD operations and search functionalities, Weaviate provides advanced features such as contextual search, filtering by vector similarity, multi-tenancy support, and more. These capabilities enable organizations to leverage more complex queries and data manipulations.

Implementing Contextual Searches

Contextual searches utilize natural language processing to infer intent from user queries. This is especially useful in cases where exact keyword matches may not yield desired results.

1. Encapsulate User Intent: Consider a user who searches for “love stories,” which may refer to a broader set of themes.
2. Use the Vector API: You can make a POST request to the vector endpoint:

• POST /v1/vector

1. Example Payload for Generating Vector:

• { "text": "love stories" }

1. Execute Search Based on Vector: After generating the vector, perform a follow-up search using it to return relevant results:

• { Get { Books (nearVector: { vector: <result_from_prior_step> certainty: 0.7 }) { title author } } }

1. Fine-tuning Results: The use of certainty helps refine results further by filtering based on how closely related the results are to the query vector.

Best Practices in Using Weaviate Calls

To optimize your interactions with Weaviate, adherence to best practices is advisable:

1. Schema Design: Structure schemas thoughtfully to reflect expected query patterns and keep them flexible to accommodate future adjustments.
2. Batch Ingestion: Minimize the number of API calls by batching data ingestion whenever possible to improve performance.
3. Use Vectorization: Always look into converting textual data into vectors for better search outcomes.
4. Monitor Performance: Regularly check cluster health and performance metrics to preemptively identify issues.
5. Security and Access Control: If your Weaviate instance handles sensitive data, implementing robust access controls is essential for compliance and safety.
6. Testing and Validation: Make use of environment setups where you can test schemas, ingestion methods, and search queries without affecting production data.

By following these practices, users can make the most of Weaviate’s capabilities and streamline their data management processes, enabling more effective searching and retrieval of high-dimensional data.

Let’s talk about something that we all face during development: API Testing with Postman for your Development Team.

Yeah, I’ve heard of it as well, Postman is getting worse year by year, but, you are working as a team and you need some collaboration tools for your development process, right? So you paid Postman Enterprise for…. $49/month.

Now I am telling you: You Don’t Have to:

That’s right, APIDog gives you all the features that comes with Postman paid version, at a fraction of the cost. Migration has been so easily that you only need to click a few buttons, and APIDog will do everything for you.

How to Migrate Postman Collections/Environments to Apidog

APIDog has a comprehensive, easy to use GUI that makes you spend no time to get started working (If you have migrated from Postman). It’s elegant, collaborate, easy to use, with Dark Mode too!

Want a Good Alternative to Postman? APIDog is definitely worth a shot. But if you are the Tech Lead of a Dev Team that really want to dump Postman for something Better, and Cheaper, Check out APIDog!

Apidog An integrated platform for API design, debugging, development, mock, and testing