TLDR
- Perplexity AI is scraping the web without respecting the Robots Exclusion Protocol
- The company is also facing legal threats for copyright infringement
Perplexity, an AI search startup backed by Jeff Bezos, is under fire for breaking a basic rule of the internet – the Robots Exclusion Protocol. The company is apparently ignoring the protocol and scraping parts of the web that operators have blocked from bots. Despite claiming to respect the protocol, reports confirm that Perplexity uses an unlisted IP address to circumvent the block and scrape websites anyway.
Additionally, Perplexity is facing legal threats for copyright infringement after allegedly ripping off Forbes reporting without proper attribution. The AI startup created AI-generated articles, podcasts, and videos using Forbes’ content, leading to a potential lawsuit.
Perplexity’s actions have raised concerns about the impact on digital media business models. While Google indexes pages and provides AI summaries that lead users to the original websites, Perplexity’s detailed AI articles may discourage users from clicking through to the actual sites. The company is reportedly working on content partnerships to address this issue, but for now, it seems to be scraping websites and bypassing paywalls to gather information for its AI answers.