🤖 CrawlBot - Transform Any Website into an Intelligent Knowledge Assistant - n8n workflow
by Victor
$0+
Transform any website content into intelligent, queryable knowledge with this powerful n8n workflow.
CrawlBot automatically crawls all URLs from XML sitemaps, processes web content, converts them into vector embeddings for advanced retrieval augmented generation (RAG) applications, and enables intelligent chat-based querying of your website data.
🚀 Key Features
Website Crawling: Automatically crawls all URLs found in XML sitemaps using the powerful Crawl4AI service
Smart Content Extraction: Removes HTML tags and extracts clean, meaningful text content using AI-powered processing
Vector Embeddings: Converts cleaned content into high-dimensional vector embeddings for semantic search capabilities
Intelligent RAG System: Implements an agentic RAG system that searches through crawled content to provide accurate, context-aware responses
Conversation Memory: Maintains chat history across sessions using PostgreSQL for contextual conversations
💡 Use Cases
Documentation Sites: Transform technical documentation into an AI-powered knowledge assistant that can answer complex questions about your products or services
Company Websites: Create intelligent chatbots that can answer questions about your company, products, and services based on your website content
Knowledge Bases: Convert help centers and FAQ sections into conversational AI assistants for better customer support
Content Archives: Make large content libraries searchable through natural language queries with semantic understanding
Research Websites: Enable researchers to query academic content and publications through conversational interfaces
News and Media Sites: Create AI assistants that can answer questions about archived articles and current content
⚙️ Requirements
n8n instance (cloud or self-hosted)
Crawl4AI service(free) running on locally
OpenAI API key for embeddings and chat models
Google Gemini API key for content cleaning
Supabase account(free)
🔧 Easy Setup
The workflow comes with comprehensive documentation that guides you through:
Setting up n8n (both local and cloud deployment options)
Installing and configuring Crawl4AI service
Configuring required credentials (OpenAI, Google Gemini, PostgreSQL/Supabase)
Setting up the PostgreSQL database with pgvector extension
Creating necessary tables and functions for vector storage
Testing the crawling and chat functionality
💬 FAQ
Q: How does CrawlBot handle large websites?
A: CrawlBot processes URLs sequentially from the sitemap, with built-in retry logic and status monitoring to ensure reliable crawling of all pages.
Q: Can it crawl dynamic JavaScript websites?
A: Yes! Crawl4AI is designed to handle modern JavaScript-rendered websites and extract content after the page fully loads.
Q: How often is the content updated?
A: You can manually trigger the crawling process whenever needed. For automatic updates, you can schedule the workflow to run periodically.
Q: Can I limit crawling to specific sections of a website?
A: Yes! You can provide a sitemap that only includes the URLs you want to crawl, or modify the workflow to filter URLs based on patterns.
Q: How does the chat interface work?
A: The AI agent uses semantic search to find relevant content from the vector database and provides responses based on the crawled website data.
📞 Support
Need assistance? Connect with me on Twitter/X @victor_explore and send a DM for:
Personalized setup assistance with this workflow
Custom workflow development for your specific analytical needs
Transform website content into intelligent, queryable knowledge with CrawlBot today!