trafilatura
Trafilatura is a Python library and command line tool developed to extract text and metadata from web pages. HTML cleans its content and converts it to structured data and can scan through RSS feeds with sitemap. Without a database requirement, CSV, JSON, XML and Markdown produce outputs in various formats.
adbar · Web Scraping & Automation
Common commercial products in this category: Bright Data, Apify, ScrapingBee
- You can extract the main texts and metadata from the HTML structure on the web pages.
- You can scan websites using the Sitemap and RSS feeds.
- You can convert the data you have collected into JSON, CSV, or Markdown format.
- • NeutralHacker News
Trafilatura: Python tool to gather text on the Web
“Thanks! Here is what I put together in the docs, you could basically preprocess/render/filter the webpages with the software of your choice and then pass the result to trafilatura:”
- ↓ NegativeGitHub Issue
List of smaller extraction bugs (text & metadata)
“I have mostly tested `trafilatura` on a set of English, German and French web pages I had run into by surfing or during web crawls. There are definitely further web pages and cases in other languages for which the extraction doesn't work so far. Corresponding bug reports can…”