Web scraping
Source reports covering Web scraping. Stories will appear when analysis and community reactions are ready.
8 reports · page 1 of 1
Connected topics are temporarily unavailable. You can still browse stories below.
Source reports
DeepSeek search and fetch – $0.75 per 1000 successful requests
tool
7 min read
DeepSeek’s Search and Fetch APIs provide agents and retrieval-augmented generation applications with live web access through raw JSON endpoints at api.deepseek.com. Search uses DeepSeek’s in-house search engine to return structured results for queries, while Fetch retrieves one HTTP or HTTPS page and returns its title and content as Markdown. Both endpoints use bearer authentication with a DeepSeek API key. Search accepts queries from 1 to 128 Unicode characters and can return 1 to 50 results, with optional localization, the originating user question, and page content. Queries support site, language, exact-phrase, exclusion, and date operators. Results include relevance scores, snippets, model-oriented summaries, optional content, publication dates, and authority tiers. A rerank score of 1.5 or higher is considered high quality, while authority tiers of 3 or above indicate highly authoritative sources. Search and Fetch cost $0.75 per 1,000 successful requests. Search is billed only for non-empty results, and Fetch only when page content is returned. Server-side calls through DeepSeek Harness, Claude Code, or Codex are free until per-request billing begins at 00:00 Beijing time on November 1, 2026. Each API has a separate account-level limit of 5 requests per second; Fetch also has per-domain limits.
Business accounts will consent to Meta scraping their websites Starting Oct. 30
news
6 min read
Meta’s updated commercial terms, scheduled to take effect October 30, authorize the company and entities acting on its behalf to use automated tools to access, index, cache, analyze, collect, retrieve, and crawl websites associated with business accounts on Meta platforms. The terms cover content on domains, pages, and URLs associated with a business’s products or services and say Meta may use collected content to provide, develop, improve, research, train, and maintain its products and services. Meta defines “AI at Meta” as one of those products elsewhere in its terms. The change affects businesses using accounts to manage Facebook pages, Messenger accounts, Instagram profiles, WhatsApp channels, advertising, marketing messages, groups, or Meta’s business dashboard. Legal experts who reviewed the language said it could enable Meta to argue that it has permission to scrape news publishers’ websites and train AI systems on the material. They also said the terms’ authorization to create “test credentials” could potentially let Meta access content behind registration walls, though Meta did not say whether such credentials would be used to bypass paywalls. Business users cannot opt out while retaining commercial access; the terms instruct users who disagree to stop using Meta products for business or commercial purposes. Meta said the language consolidates previously scattered crawling provisions, will not change its future crawling practices, and leaves content owners able to manage crawler access through robots.txt.
LibreOffice 26.8 Released with Many Nice Improvements
tool
1 min read
The Document Foundation’s wiki is using Anubis, a proof-of-work challenge system, to protect its server from aggressive web scraping by AI companies. The site says large-scale scraping can cause downtime and make its resources inaccessible to other users. Anubis uses a proof-of-work scheme modeled on Hashcash, which was proposed to reduce email spam. The system is intended to impose negligible additional computing work on individual visitors while making high-volume scraping more expensive. The site describes Anubis as a temporary measure while work continues on identifying headless browsers, including through signals such as font rendering, so that likely legitimate users may avoid proof-of-work challenges. Anubis requires modern JavaScript features. The site says browser extensions such as JShelter can disable required features and instructs visitors to disable JShelter or similar plugins for the domain.
99% of My Website Traffic Is Bots
opinion
17 min read
99% of traffic on the PatronView website consists of bots. The site owner has implemented various firewall rules to combat scrapers and has shared insights from a year of battling automated traffic.
Launch HN: Context.dev (YC S26) – API to get structured data from any website
tool
5 min read
Context.dev provides a web scraping and crawl API that enables AI agents to extract structured data from any website using JSON schema. Mintlify utilized Context.dev to transform GitHub repository URLs into fully branded documentation sites in under 10 minutes.
Launch HN: Intuned (YC S22) – Build and run reliable browser automations as code
tool
10 min read
Intuned Agent generates production-ready Playwright code for web scraping, deploying it and adapting to site changes automatically. It allows users to extract data from various websites with built-in features for stealth, authentication, scheduling, and scalability.
Miasma: A tool to trap AI web scrapers in an endless poison pit
tool
3 min read
Miasma is a tool designed to trap AI web scrapers by redirecting malicious traffic to a server that delivers poisoned training data and self-referential links. It aims to protect public websites from unauthorized data scraping by overwhelming scrapers with irrelevant information.
Amazon wins court order to block Perplexity's AI shopping agent
news
2 min read
A federal judge granted Amazon a temporary order blocking Perplexity's Comet AI browser from accessing its site. Amazon's lawsuit claims Perplexity concealed its AI agents to scrape the website without permission.