«`html
A Coding Implementation to Build an Advanced Web Intelligence Agent with Tavily and Gemini AI
In this tutorial, we introduce an advanced, interactive web intelligence agent powered by Tavily and Google’s Gemini AI. We will learn how to configure and use this smart agent to seamlessly extract structured content from web pages, perform sophisticated AI-driven analyses, and present insightful results. With user-friendly, interactive prompts, robust error handling, and a visually appealing terminal interface, this tool offers an intuitive and powerful environment for exploring web content extraction and AI-based content analysis.
Essential Libraries Setup
We import and set up essential libraries for handling data structures, asynchronous programming, and type annotations, alongside a rich library that enables visually appealing terminal outputs. These modules collectively facilitate efficient, structured, and interactive execution of web intelligence tasks within the notebook.
Initializing LangChain Components
We initialize essential LangChain components: TavilyExtract enables advanced web content retrieval, init_chat_model sets up the Gemini AI-powered chat model, and create_react_agent builds a dynamic, reasoning-based agent capable of intelligent decision-making during web analysis tasks. Together, these tools form the core engine for sophisticated AI-driven web intelligence workflows.
Web Intelligence Configuration
@dataclass
class WebIntelligence:
"""Web Intelligence Configuration"""
tavily_key: str = os.getenv("TAVILY_API_KEY", "")
google_key: str = os.getenv("GOOGLE_API_KEY", "")
extract_depth: str = "advanced"
max_urls: int = 10
The WebIntelligence dataclass serves as a structured configuration container, holding API keys for Tavily and Google Gemini, and setting extraction parameters like extract_depth and the maximum number of URLs (max_urls). It simplifies the management and access of crucial settings, ensuring seamless integration and customization of web content extraction tasks within the intelligence agent.
Smart Web Agent Class
class SmartWebAgent:
"""Intelligent Web Content Extraction & Analysis Agent"""
def __init__(self, config: WebIntelligence):
self.config = config
self.console = Console()
self._setup_environment()
self._initialize_tools()
The SmartWebAgent class encapsulates an intelligent web content extraction and analysis system, utilizing APIs from Tavily and Google’s Gemini AI. It interactively sets up essential tools, securely handles API keys, extracts structured data from provided URLs, and leverages an AI-driven agent to perform insightful content analyses. It also utilizes rich visual outputs to communicate results, thereby enhancing readability and user experience during interactive tasks.
Content Extraction and Analysis
def extract_content(self, urls: List[str]) -> Dict[str, Any]:
"""Extract and structure content from URLs"""
This method extracts and structures content from the provided URLs. It tracks the extraction process and handles errors appropriately to ensure reliable results.
def analyze_with_ai(self, query: str, urls: List[str] = None) -> str:
"""Intelligent analysis using AI agent"""
This method performs intelligent analysis using the AI agent. It constructs messages for the AI based on the user query and the URLs provided, managing the response to deliver insightful analysis.
Displaying Results
def display_results(self, results: Dict[str, Any]):
"""Beautiful result display"""
This method formats and displays the results of content extraction, providing clear feedback for successful and failed operations.
Running the Agent
def main():
"""Interactive Web Intelligence Demo"""
The main function provides an interactive command-line demonstration of the Smart Web Intelligence Agent. It presents users with an intuitive menu that allows them to extract web content from custom URLs, perform sophisticated AI-driven analyses on selected topics, or explore predefined demos involving AI, machine learning, and quantum computing. Rich visual formatting enhances user engagement, making complex web analysis tasks straightforward and user-friendly.
Conclusion
By following this comprehensive tutorial, we have built an enhanced Tavily Web Intelligence Agent capable of sophisticated web content extraction and intelligent analysis using Google’s Gemini AI. Through structured data extraction, dynamic AI queries, and visually appealing results, this powerful agent streamlines research tasks, enriches data analytics workflows, and fosters deeper insights from web content. This foundation equips users to extend the agent further, customize it for specific use cases, and harness the combined power of AI and web intelligence to enhance productivity and decision-making in projects.
All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and join our ML SubReddit community.
«`