Automating AI Research and Content Creation: Building a Smart Pipeline with CrewAI and Google’s Gemini Pro

DhanushKumar

6 min readJun 26, 2024

To know more about detail : https://medium.com/@danushidk507/crewai-ai-agent-9a1684064094

To integrate CrewAI with other AI frameworks, you can follow these steps:

Step 1: Choose the AI Framework

Select the AI Framework: Identify the AI framework you want to integrate with CrewAI. This could be a language model, a chatbot platform, or any other AI framework that supports integration with CrewAI.

Step 2: Understand the Integration Requirements

Understand the Integration Requirements: Familiarize yourself with the integration requirements for both CrewAI and the chosen AI framework. This includes understanding the APIs, data formats, and any specific requirements for the integration.

Step 3: Set Up the Integration

Set Up the Integration: Configure the integration by setting up the necessary APIs, data formats, and other requirements. This may involve creating custom tools or modifying existing tools to work with the chosen AI framework.

Step 4: Test the Integration

Test the Integration: Test the integration by running a simple test case to ensure that the integration is working correctly. This includes verifying that the data is being passed correctly and that the AI framework is responding as expected.

Step 5: Refine the Integration

Refine the Integration: Refine the integration by optimizing the performance, handling errors, and improving the overall user experience.

Example: Integrating CrewAI with Ollama

Integrate CrewAI with Ollama: To integrate CrewAI with Ollama, you can use the Ollama tool to run local LLMs and connect them to CrewAI. This allows you to utilize the power of LLMs within your CrewAI workflows.

Example: Integrating CrewAI with OpenAI

Integrate CrewAI with OpenAI: To integrate CrewAI with OpenAI, you can use the OpenAI API to connect your CrewAI agents to OpenAI models. This allows you to leverage the capabilities of OpenAI models within your CrewAI workflows.

Example: Integrating CrewAI with LangChain

Integrate CrewAI with LangChain: To integrate CrewAI with LangChain, you can use the LangChain API to connect your CrewAI agents to LangChain models. This allows you to leverage the capabilities of LangChain models within your CrewAI workflows.

In the rapidly evolving world of artificial intelligence, new frameworks are emerging that allow for more sophisticated and collaborative AI systems. One such framework is CrewAI, which enables the creation of AI agent teams that can work together on complex tasks. In this article, we’ll explore a practical implementation of CrewAI for automating the process of researching and writing about AI advancements.

Understanding CrewAI: CrewAI is a framework that allows developers to create and manage teams of AI agents, each with specific roles and capabilities. It’s built on top of language models and provides a structured way to assign tasks, manage workflows, and coordinate between different AI entities.

Key Components of Our Implementation:

The Language Model: Our system uses Google’s Gemini Pro, a state-of-the-art language model, as the brain behind our AI agents. We configure it with a temperature of 0.5, striking a balance between creativity and focus in the generated content.
Agents: We define two specialized agents: a) Senior Research Agent: Tasked with uncovering and analyzing the latest AI advancements. b) Tech Content Writer: Responsible for creating engaging content based on the research. Each agent is given a specific role, goal, and backstory, which helps shape their behavior and outputs.
Tools: The research agent is equipped with a DuckDuckGo search tool, allowing it to gather up-to-date information from the web.
Tasks: We define two main tasks: a) Analyzing 2024’s AI advancements, trends, and implications. b) Creating a blog post that communicates these findings clearly and engagingly.
Crew: The Crew object brings together our agents and tasks, orchestrating their collaboration in a sequential process.

Workflow Breakdown:

Research Phase: The Senior Research Agent starts by using its search tool to gather information on recent AI advancements. It analyzes this data, identifying key trends and technologies that have emerged in 2024.
Content Creation Phase: Once the research is complete, the Tech Content Writer takes over. This agent uses the research findings to craft a compelling blog post, translating complex technical information into an engaging and accessible format.
Sequential Processing: By setting the process to sequential, we ensure that the writing task only begins after the research is complete, maintaining a logical flow of information.

Benefits of This Approach:

Automation: This system automates the entire process from research to content creation, saving time and resources.
Specialization: By using specialized agents, we ensure that each part of the process is handled by an entity optimized for that task.
Up-to-date Information: The use of a search tool allows the system to access current information, ensuring the content is relevant and timely.
Scalability: This approach can be easily scaled or modified for different topics or content types.
Consistency: The defined roles and goals for each agent help maintain consistency in the output quality.

Challenges and Considerations:

Quality Control: While AI-generated content can be impressive, human oversight is still crucial for ensuring accuracy and quality.
Ethical Use: It’s important to use such systems responsibly, especially when publishing content under human authorship.
Customization: Fine-tuning the agents and tasks for specific needs may require iteration and experimentation.

import os
from langchain_google_genai import ChatGoogleGenerativeAI
from crewai import Agent, Task, Crew, Process
from langchain.tools import DuckDuckGoSearchRun

ChatGoogleGenerativeAI: To interact with Google's Generative AI (Gemini Pro).
Agent, Task, Crew, Process: Core components from CrewAI for building the AI team.
DuckDuckGoSearchRun: A tool from LangChain for web searches.

llm = ChatGoogleGenerativeAI(
    model="gemini-pro",
    verbose=True,
    temperature=0.5,
    google_api_key=""
)

This sets up the Language Model:

We’re using Google’s Gemini Pro model.
verbose=True enables detailed logging of the model's operations.
temperature=0.5 sets a balance between creativity and determinism in the model's outputs.

search_tool = DuckDuckGoSearchRun()

This creates a search tool using DuckDuckGo. This tool will allow our AI agents to perform web searches for up-to-date information.

researcher_agent = Agent(
    role="Senior Research Agent",
    goal="Uncover cutting-edge developments in AI and Data Science",
    backstory="""
    You are an expert at a technology research group,
    skilled in identifying trends and analyzing complex data""",
    verbose=True,
    allow_delegation=False,
    tools=[search_tool],
    llm=llm
)

This creates our first AI agent, the researcher:

role and goal define the agent's purpose.
backstory provides context that shapes the agent's responses.
verbose=True enables detailed logging of the agent's actions.
allow_delegation=False prevents this agent from delegating its tasks to others.
tools=[search_tool] equips the agent with the ability to perform web searches.
llm=llm assigns the Gemini Pro model to this agent.

writer_agent = Agent(
    role="Tech Content Writer",
    goal="Create compelling content on tech advancements",
    backstory="""
    You are a content strategist known for making complex tech topics interesting and easy to 
    understand and analyze""",
    verbose=True,
    allow_delegation=True,
    llm=llm
)

This creates our second AI agent, the writer:

Similar structure to the researcher agent, but with a different role and goal.
allow_delegation=True allows this agent to delegate tasks if needed (though in this setup, it won't have anyone to delegate to).
No additional tools are provided to this agent.

task1 = Task(
    description="""
    Analyze 2024's AI advancements. Find major trends, new technologies, and their implications.
    """,
    expected_output="A detailed analysis of 2024's AI advancements including trends and new technologies.",
    agent=researcher_agent
)

task2 = Task(
    description="""
    Create a blog post about major AI advancements using insights. Make it interesting and clear.
    """,
    expected_output="A compelling blog post that is interesting, clear, and easy to understand.",
    agent=writer_agent
)

Each task has a description of what needs to be done.
expected_output describes what the task should produce.
Each task is assigned to a specific agent.

crew = Crew(
    agents=[researcher_agent, writer_agent],
    tasks=[task1, task2],
    verbose=True,
    process=Process.sequential
)

It includes both our agents and their tasks.
verbose=True enables detailed logging of the crew's operations.
process=Process.sequential means the tasks will be executed in order, one after the other.

print("Crew: Working on AI Advancements Task")
result = crew.kickoff()

print("**********")
print(result)

It prints a message indicating the crew is starting its work.
crew.kickoff() starts the execution of the tasks.
The result is then printed.

In practice, this code will:

Have the researcher agent search for and analyze recent AI advancements.
Once that’s complete, the writer agent will take the research and create a blog post about it.
The final output will be a blog post summarizing the latest AI advancements in an engaging and understandable way.

This implementation showcases how CrewAI can be used to create a pipeline of AI agents working together on a complex task. It demonstrates the power of specialized AI roles, the integration of external tools (like web search), and the orchestration of multiple AI agents to produce a cohesive output. This approach can be particularly useful for automating research and content creation processes, ensuring that the content is both up-to-date (thanks to the search capability) and well-crafted (thanks to the specialized writing agent).