Ask HN: How Spam are removed from large datasets of web crawls for AI training

https://news.ycombinator.com/item?id=40237641
by arromatic • 2 years ago
1 0 2 years ago

Large crawls for example commoncrawl or companies that have in house web scrapers/crawlers to collect data / scrape from the web to train ai . 1. How are blogspams / regular spams are cleaned from such huge data ? 2. Is it possible to teach ai to clean datasets with small human annotated datasets to clean large datasets for ai training ? 3. How misinfo/disinfo in cleaned ?

Related Stories

Loading related stories...

Source preview

news.ycombinator.com