EthosPower.ai
The Problem: Legacy Web Tools Don't Understand LLMs
I spent three months last year helping a West Texas transmission operator build an AI system that monitors equipment vendor documentation. They needed to track firmware updates, security bulletins, and compatibility matrices across 47 different vendor portals. Beautiful use case for AI — except none of their existing web scraping tools could feed clean data to their local Ollama deployment.
Traditional scrapers give you HTML soup. Selenium scripts break every time a vendor redesigns their portal. And the commercial "AI-ready web data" platforms? All cloud-only, all sending your vendor credentials and equipment configurations to third-party servers. That's a NERC CIP violation waiting to happen.
The AI-native web stack solves this: tools purpose-built to extract, structure, and vectorize web content for LLM consumption, all running on your own infrastructure. At EthosPower, we deploy five open-source platforms that cover the entire pipeline from browser automation to content publishing. Here's what actually works in regulated energy environments.
Firecrawl: Web Pages to LLM-Ready Markdown
Firecrawl does one thing exceptionally well: it turns any website into clean, structured markdown that LLMs can actually process. You point it at a URL, it handles JavaScript rendering, strips navigation chrome, and returns semantically chunked content ready for embedding.
I deployed Firecrawl for a Southeast utility tracking state regulatory filings. Their previous approach used BeautifulSoup scripts that choked on PDF-embedded viewer widgets and required constant maintenance. Firecrawl's JavaScript execution engine handles the rendering, its content extraction algorithm identifies the actual document text, and the semantic chunking splits 200-page rate case filings into coherent sections.
Key capabilities: JavaScript rendering for single-page applications, automatic content cleaning that removes navigation and ads, semantic chunking that respects document structure, and metadata extraction for titles and publication dates. The self-hosted version runs entirely on your network — no external API calls, no data leaving your environment.
Where it falls short: Firecrawl doesn't handle authentication flows or complex user interactions. If you need to log into a vendor portal and navigate through multi-step wizards, you need browser automation. That's where Playwright enters.
Playwright: Browser Automation That Actually Works
Playwright is Microsoft's answer to Selenium, and it's dramatically better. Cross-browser support for Chromium, Firefox, and WebKit. Auto-waiting that eliminates flaky timeouts. And now, AI-powered test generation through their experimental Codegen tool and Model Context Protocol integration.
I use Playwright for any workflow that requires authentication or multi-step navigation. That Texas transmission operator? Their vendor portals required certificate-based login, multi-factor authentication, and navigation through equipment selection menus before reaching the firmware download pages. Playwright handles all of it with headed or headless browser instances, full network interception for credential injection, and screenshot capture for debugging.
The AI integration is legitimately useful. Playwright's Codegen watches you interact with a website and generates the automation code. Their MCP server lets Claude or other AI assistants write and execute Playwright scripts directly. For one-off data extraction tasks, I often describe what I need to an LLM and let it generate the Playwright script rather than writing it myself.
Critical for energy deployments: Playwright runs completely offline once installed. No telemetry, no cloud dependencies. You can lock it down in air-gapped OT environments. Pair it with Firecrawl — Playwright navigates to the content, Firecrawl extracts the markdown.
ChromaDB: Vector Storage Without the Complexity
Once you've extracted clean content with Firecrawl and Playwright, you need somewhere to store the embeddings for semantic search. ChromaDB is the simplest vector database that actually works at production scale.
I deployed ChromaDB for that regulatory filing project. Every filing extracted by Firecrawl gets embedded using Ollama's nomic-embed-text model and stored in ChromaDB collections organized by docket number. When analysts query the system, ChromaDB returns the most relevant filing sections in under 50 milliseconds for collections with 500K+ vectors.
What makes ChromaDB work: single-file SQLite backend for development and testing, persistent storage with automatic index optimization, built-in embedding function support for Ollama and other local models, and collection-based multitenancy. The Python API is four lines of code to get started. No complex schema design, no cluster orchestration, no JVM tuning.
Limitations you need to know: ChromaDB doesn't do distributed deployments yet. Maximum recommended dataset is about 10 million vectors on a single node. If you're building enterprise-scale knowledge graphs with hundreds of millions of entities, you need Qdrant or Weaviate. But for 90% of energy sector AI projects — equipment manuals, regulatory filings, operational procedures — ChromaDB's simplicity wins.
Ollama: The LLM Runtime That Changed Everything
Every tool I've described feeds data to LLMs. Ollama is how you run those models on your own hardware without sending anything to OpenAI or Anthropic. At EthosPower, it's the foundation of every sovereign AI deployment.
Ollama packages open-source models — Llama 3.1, Mistral, Qwen, Gemma — into a single-command runtime. Download a model with ollama pull, run inference with ollama run, expose an OpenAI-compatible API for application integration. No Docker containers to configure, no Python environment management, no CUDA driver debugging. It just works.
I run Ollama on an NVIDIA RTX 4090 workstation for development and on datacenter A100 instances for production. The same commands work on both. For that vendor documentation system, we deployed Ollama with Llama 3.1 70B for question-answering and nomic-embed-text for generating embeddings from Firecrawl's markdown output. Total inference cost: electricity and hardware depreciation. No per-token API charges.
Critical detail: Ollama's model library includes quantized versions optimized for different hardware profiles. The 70B model runs acceptably on 48GB of VRAM with 4-bit quantization. The 8B models run on consumer GPUs. This matters for energy companies deploying AI at substations or field offices where you can't rack multi-GPU servers.
Integration points: Ollama exposes an OpenAI-compatible API, so any tool designed for GPT-4 works with local models. ChromaDB can use Ollama for embedding generation. Firecrawl can send extracted content to Ollama for summarization. The entire pipeline runs air-gapped. Try the AI Readiness Assessment to see if your infrastructure can support local LLM deployments.
Postiz: Self-Hosted Social Publishing
Postiz seems like an outlier in this stack until you consider the full workflow: extract web content, process it with AI, publish insights back to stakeholders. Energy companies increasingly use LinkedIn, internal social platforms, and stakeholder portals to distribute operational updates and regulatory summaries.
Postiz is the open-source alternative to Buffer and Hootsuite. Schedule posts across LinkedIn, Twitter, Facebook, and Mastodon from a single interface. AI-assisted content generation using local Ollama models. Multi-account management for teams. Self-hosted deployment means your draft regulatory commentary never touches third-party servers.
I deployed Postiz for a renewable developer that publishes weekly construction updates to community stakeholder groups. Their communications team drafts updates in Postiz, uses the AI assistant (connected to Ollama) to generate platform-specific variations, and schedules coordinated publication across three social channels and their community portal. The AI assistant understands each platform's character limits and tone conventions.
Where Postiz fits the stack: it's the publication endpoint. Firecrawl extracts regulatory updates, Ollama summarizes them, Postiz distributes the summaries to stakeholders. Close the loop from web monitoring to stakeholder communication without leaving your infrastructure.
How These Tools Work Together
Here's the actual workflow we deployed for that transmission operator:
- Scheduled Monitoring: Playwright scripts run nightly, authenticating to vendor portals and navigating to firmware update pages
- Content Extraction: Firecrawl processes each page, extracting clean markdown and metadata
- Embedding and Storage: Ollama generates embeddings using nomic-embed-text, ChromaDB stores them in vendor-specific collections
- Change Detection: Daily comparison queries identify new firmware versions or security bulletins
- AI Summarization: Ollama generates plain-language summaries of technical release notes using Llama 3.1
- Stakeholder Distribution: Postiz publishes summaries to internal Slack channels and the operations team portal
Total infrastructure: one Ubuntu 22.04 server with an RTX 4090, 128GB RAM, 4TB NVMe storage. Total external dependencies: none. The entire pipeline runs air-gapped.
For different starting points: if you're monitoring public regulatory filings, start with Firecrawl and ChromaDB. If you're extracting data from authenticated vendor portals, add Playwright. If you're publishing insights to stakeholders, add Postiz. Ollama is non-negotiable — it's the engine that makes everything AI-native.
The Verdict
The minimum AI-native web stack for energy sector practitioners: Firecrawl for content extraction, Ollama for embeddings and inference, ChromaDB for vector storage. That's 90% of use cases covered. Add Playwright when you need authentication or complex navigation. Add Postiz when you need to publish insights back to stakeholders.
What makes this stack work in regulated environments: complete data sovereignty, air-gapped deployment capability, no recurring API costs, no telemetry or phone-home behavior. Every component runs on your infrastructure under your control. That's the difference between a proof-of-concept that legal kills and a production system that passes NERC CIP audits.
The learning curve is real but manageable. Firecrawl and Ollama are genuinely simple — you'll be productive in an afternoon. Playwright requires understanding browser automation concepts but pays off immediately in reliability versus traditional scrapers. ChromaDB is straightforward if you've used any database before. Postiz is the easiest of the five.
Start with a single use case — vendor documentation monitoring, regulatory filing tracking, equipment manual search — and deploy the minimal stack. Add components as requirements expand. This is infrastructure that grows with your AI capabilities rather than forcing you into a vendor's architecture vision. Ask EthosAI Chat for help sizing the right hardware configuration for your specific deployment.