rs-trafilatura: Page-Type-Aware Web Content Extraction in Rust

Web content extraction is the task of isolating the main content of a web page from its surrounding boilerplate — navigation menus, cookie banners, ads, sidebars, footers, and the other 80% of a page that isn't the actual content. If you process web pages at scale, you need it. Search engines use it for indexing. RAG pipelines use it to feed clean context to LLMs. SEO practitioners use it to approximate what Google sees when it evaluates a page. The open-source ecosystem for this is strong. Trafilatura (Python), Readability (JavaScript), jusText, BoilerPy3 — all solid tools that work well on news articles and blog posts. On articles, the top systems all converge above F1 = 0.90. The problem is largely solved. But the web is not just articles. The Problem: Everything That Isn't an Article When I was running SEO audits across thousands of competitor pages from search results, I kept hitting the same issue. The extraction tools worked on articles but fell apart on everything else: Product