# agdata

> Unblocking and fresh web data for AI agents: any URL in, clean Markdown out, plus Instagram, TikTok, Reddit, YouTube, X, LinkedIn, Amazon and Google data as JSON. Also screenshots and PDFs of pages, YouTube transcripts, Google Maps reviews, job listings, Google News, Amazon reviews and web search with page content. Each call is paid in stablecoins: x402 on Base and Polygon (plus Solana on some routes) or MPP on Base. No API keys, accounts or subscriptions. A scrape that fails is never charged; otherwise you pay a small fixed per-call fee plus the results you get.

## How to call it

- POST a small JSON body to an endpoint below. Without payment you get HTTP 402 with the price.
- x402: the 402 carries a `PAYMENT-REQUIRED` header. Sign it (`exact` scheme) and retry the same request with `PAYMENT-SIGNATURE`.
- MPP: the 402 carries `WWW-Authenticate: Payment`. Sign it and retry with `Authorization: Payment ...`.
- Every endpoint is also served under `/mpp/<name>`; either path accepts either protocol.
- MPP is Base only. Solana is taken on every route except unblock, twitter, reddit, youtube-transcript, screenshot, linkedin-jobs, amazon-reviews, web-search, x-replies, youtube-comments, tiktok-comments and instagram-comments: pay those on Base or Polygon (their 402 has no Solana offer).
- The 402 quotes the price for the `limit` you ask for: a fixed per-call fee plus a per-result price for each result. Endpoints with no `limit` field return one result.
- That quote is what you sign (x402 `exact` and MPP `charge` are fixed-amount). If fewer results come back, the per-result price of the missing ones is refunded on-chain to the paying address, usually within minutes of the response; refunds too small to be worth a transfer are not sent. If the scrape finds nothing, you pay only the fixed per-call fee: every result is refunded (except on unblock, google-maps-reviews, youtube-transcript, screenshot and amazon-reviews, which charge nothing when they deliver nothing). If the scrape fails, the payment is not settled and nothing is charged.
- Response headers `X-Agdata-Results-Requested`, `X-Agdata-Results-Delivered` and `X-Agdata-Refund` (USDC base units, when a refund is due) report what happened. Ask for the `limit` you need.
- `limit` is a whole number. A value above an endpoint's maximum is lowered to the maximum, 0 or less means the default, and the 402 quotes the value used. Free-text fields are at most 500 characters and take no control characters; a body over 64 KiB is refused (413).
- A field an endpoint does not know is ignored on instagram, google-maps, tiktok, linkedin, youtube, twitter, reddit, amazon and google-search and refused on unblock, google-maps-reviews, youtube-transcript, screenshot, linkedin-jobs, indeed-jobs, google-news, amazon-reviews, web-search, x-replies, youtube-comments, tiktok-comments and instagram-comments.
- Errors the service itself produces are JSON: `{"error": code, "message": text}`. An invalid request body gets the 400 `invalid_request` once a payment is attached; without a payment it gets the 402 quote for 1 result with the header `X-Agdata-Invalid-Request: 1`, which is not worth signing: read `error` in it.
- instagram, reddit, youtube, linkedin and amazon records carry a `normalized` object with the same keys whichever upstream provider served the request (null where a provider has no value), plus a `source` field. The provider's own fields are kept alongside it.

## What to call

- Who is saying what (people & social): Posts, profiles and search results from the big social platforms. Replies under a post on X. Comments under a YouTube video. Comments under a TikTok video. Comments under an Instagram post. Endpoints: `instagram`, `tiktok`, `reddit`, `twitter`, `youtube`, `linkedin`, `x-replies`, `youtube-comments`, `tiktok-comments`, `instagram-comments`.
- What customers say (places & reviews): Google Maps places by search and location. Reviews of a place. Reviews of an Amazon product. Endpoints: `google-maps`, `google-maps-reviews`, `amazon-reviews`.
- What is out there right now (search & news): Google Search results for a query. Current news articles for a query, in the language and country edition you pick. Web search with each result's page read as Markdown, in one call. Endpoints: `google-search`, `google-news`, `web-search`.
- What is hiring, what is selling (jobs & products): LinkedIn job listings. Indeed job listings with salary and employer details. Amazon product search. Endpoints: `linkedin-jobs`, `indeed-jobs`, `amazon`.
- Everything else: any URL in, clean Markdown out (unblocking · any page): When there is no dedicated endpoint, unblocking reads the page for you: it renders JavaScript, copes with common bot protection, and returns Markdown, HTML or text with title, language and final URL. Need to see it instead? Capture a screenshot or PDF. Need what a video says? Get its transcript. Endpoints: `unblock`, `screenshot`, `youtube-transcript`.

## Unblocker

- POST `/x402/unblock` (or `/mpp/unblock`) with `{"url":"https://example.com/","format":"markdown"}`. `format` is `markdown` (the default), `html` or `text`. Only these two fields are accepted: headers, cookies and POSTs to the target are not supported.
- It gets through Cloudflare and other common bot protection and can render JavaScript-heavy pages (best effort: some sites still block it). Use it when your own fetch got a 403 or a challenge page; to read product, listing and review pages from sites that block plain HTTP clients; to turn JavaScript-rendered pages into Markdown for RAG; or as one paid fallback instead of running headless browsers and proxies.
- The answer is a JSON array with one record: `url`, `finalUrl`, `statusCode`, `contentType`, `fetchedAt`, `title`, `description`, `language`, `canonicalUrl`, `jsonLd`, `format`, `content`, `contentBytes`, `truncated` (null where the page has no value). `content` is cut at 512 KiB (`truncated` is then true); a page with no Markdown comes back as `text`.
- A page that could not be fetched (blocked, an error status or empty) is never charged: the answer is `502 {"error":"not_fetched","reason":...,"charged":false}` and the payment is not settled. There is no `limit` field: one page per call, one flat quote.
- A call can take up to about 90 s: keep the connection open. Retrying with the same payment returns the stored result. It is not offered on Solana: pay on Base or Polygon.
- Fetch only pages you may access; hosts are blocked on complaint (agdata@shveik.dev).

## Screenshot

- POST `/x402/screenshot` (or `/mpp/screenshot`) with `{"url":"https://example.com/","format":"jpeg"}`. `format` is `jpeg` (the default), `png`, `webp` or `pdf`. Optional: `fullPage`, `device` (`desktop`, `desktop_hd`, `mobile` or `tablet`), `width`, `height`, `delayMs`, `colorScheme` (`light` or `dark`), `selector` (capture only that element) and `waitForSelector`, both plain CSS selectors. Cookies, headers and proxies are not accepted.
- The answer is a JSON array with one record: `finalUrl`, `format`, `contentType`, `width`, `height`, `fullPage`, `bytes`, `capturedAt` and `imageBase64` (standard base64: decode it to get the file). A capture over 5 MiB is not delivered: ask for `jpeg` or leave `fullPage` off.
- It shows what a normal browser sees. It does not get past bot protection: a protected page may come back as its challenge screen. Use the unblock endpoint to read such a page's text.
- A page that cannot be captured, renders blank or gives an image over 5 MiB is never charged: the answer is `502 {"error":"not_fetched","reason":...,"charged":false}` and the payment is not settled. One capture per call, one flat quote, no `limit` field.
- A call can take up to about 60 s: keep the connection open. Retrying with the same payment returns the stored capture for 24 hours. It is not offered on Solana: pay on Base or Polygon.

## YouTube transcript

- POST `/x402/youtube-transcript` (or `/mpp/youtube-transcript`) with `{"video":"dQw4w9WgXcQ","format":"json","lang":"en"}`. `video` is an 11-character video id or the URL of one video (watch, shorts, live, embed or youtu.be; not a channel, playlist or search). `format` is `json` (the default), `text`, `srt` or `vtt`. `lang` is `en` (the default), `any`, `de`, `es`, `fr`, `it`, `ja`, `ko`, `nl`, `pt` or `ru`.
- The answer is a JSON array with one record: `videoId`, `url`, `title`, `channelName`, `channelUrl`, `durationSeconds`, `publishedAt`, `viewCount`, `language`, `availableLanguages`, `subtitleType`, `format`, `transcript`, `segmentCount` and `truncated`. With `json`, `transcript` is a list of `{start, duration, text}` (seconds); with `text`, `srt` or `vtt` it is one string. Only published subtitles are returned, there is no speech-to-text. `any` takes the English track when the video has one, else the first; `language` says which came back.
- A video with no subtitles in the language asked for (or an unavailable one) is never charged: `502 {"error":"not_fetched",...,"charged":false}`; try `lang` `any`. One video per call, one flat quote, no `limit` field.
- A call can take up to about 60 s: keep the connection open. Retrying with the same payment returns the stored first answer, in the format of the first call. It is not offered on Solana: pay on Base or Polygon.

## Google Maps reviews

- POST `/x402/google-maps-reviews` (or `/mpp/google-maps-reviews`) with `{"place":"ChIJN1t_tDeuEmsRUsoyG83frY4","limit":10}`. `place` is a Google place_id (27 characters starting with `ChIJ` or `GhIJ`) or a Google Maps place URL (`https://www.google.com/maps/place/...` or `https://www.google.com/maps?cid=...`; a URL with only the place's name and no place id is best effort and can end as not fetched, which is not charged). `limit` is 1 to 50 (default 10), `sort` is `newest` (the default), `relevant`, `highest` or `lowest`, `language` is a Google Maps language code (default `en`).
- The answer is a JSON array with one record per review: `reviewId`, `stars`, `text`, `textTranslated`, `originalLanguage`, `translatedLanguage`, `publishedAt`, `publishedAgo`, `likesCount`, `ownerResponse` (`text`, `publishedAt` or null), `imageCount`, `origin`, `context`, `detailedRating` and `place` (`placeId`, `name`, `address`, `category`, `rating`, `reviewsCount`, `url`). Reviewer names, ids, profile links and photos are never returned.
- The quote is for the `limit` you ask for; if fewer reviews exist, the missing ones are refunded like any other results. A place with no reviews, or an unknown place, is never charged: `502 {"error":"not_fetched",...,"charged":false}`.

## Job listings

- POST `/x402/linkedin-jobs` (or `/mpp/linkedin-jobs`) with `{"query":"data engineer","location":"Prague","limit":10,"posted":"week"}`. `query` is a job title or keywords; `location`, `limit` (1 to 50, default 10) and `posted` (`24h`, `week` or `month`) are optional.
- The answer is a JSON array with one record per job: `id`, `url`, `title`, `companyName`, `companyUrl`, `location`, `postedDate`, `postedTimeAgo`, `contractType`, `experienceLevel`, `workType`, `sector`, `salary`, `applicants`, `applyType`, `applyUrl`, `description` and `descriptionTruncated` (null where a listing has no value). Recruiter names are never returned. A search that finds nothing returns `[]` and pays only the fixed per-call fee. It is not offered on Solana: pay on Base or Polygon.
- POST `/x402/indeed-jobs` (or `/mpp/indeed-jobs`) with `{"query":"nurse","location":"Austin, TX","country":"us","limit":10,"posted":"week"}`. `query` is a job title or keywords; `location`, `country` (a two-letter code, default `us`), `limit` (1 to 50, default 10) and `posted` (`24h`, `3d`, `week` or `14d`) are optional.
- The answer is a JSON array with one record per job: `id`, `url`, `applyUrl`, `title`, `publishedAt`, `expired`, `language`, `location` (`city`, `state`, `country`, `postalCode`), `employer` (`name`, `url`, `website`, `industry`, `employeeCount`, `rating`, `ratingsCount`), `salary` (`min`, `max`, `unit`, `currency`, or null), `jobTypes`, `benefits`, `description` and `descriptionTruncated`. A search that finds nothing returns `[]` and pays only the fixed per-call fee.

## Google News

- POST `/x402/google-news` (or `/mpp/google-news`) with `{"query":"openai","limit":10}`. `query` is a keyword or phrase; `limit` (1 to 30, default 10), `language` (two letters, default `en`) and `country` (two letters, default `US`) are optional.
- The answer is a JSON array with one record per article: `title`, `url`, `source`, `publishedAt` and `description`. `url` is the article's Google News link, which opens the publisher's page; the article's text is not included. A search that finds nothing returns `[]` and pays only the fixed per-call fee.

## Amazon reviews

- POST `/x402/amazon-reviews` (or `/mpp/amazon-reviews`) with `{"product":"B079JLY5M5","limit":10}`. `product` is an ASIN or an Amazon product URL (a URL sets the marketplace). `limit` is 1 to 50 (default 10), `sort` is `recent` (the default) or `helpful`, `stars` is `all` (the default), `5` to `1`, `positive` or `critical`, `country` is `us` (the default), `uk`, `de`, `fr`, `it`, `es`, `ca` or `jp`.
- The answer is a JSON array with one record per review: `asin`, `rating`, `title`, `text`, `date`, `country`, `verifiedPurchase`, `vine`, `helpfulCount` and `imageCount`. Reviewer names and photos are never returned.
- The quote is for the `limit` you ask for; if fewer reviews exist, the missing ones are refunded like any other results. A product with no reviews, or an unknown one, is never charged: `502 {"error":"not_fetched",...,"charged":false}`. A call can take up to about 15 s. It is not offered on Solana: pay on Base or Polygon.

## Replies to a post on X

- POST `/x402/x-replies` (or `/mpp/x-replies`) with `{"url":"https://x.com/NASA/status/1234567890","limit":10}`. `url` is a post on x.com or twitter.com; `limit` is 1 to 50 (default 10).
- The answer is a JSON array with one record per reply: `id`, `url`, `text` (cut at 4,000 characters, with `textTruncated`), `createdAt`, `lang`, `likeCount`, `replyCount`, `repostCount`, `quoteCount`, `viewCount`, `inReplyToId` and `author` (`username`, `name`, `verified`). The post itself is not in the array. A post with no replies returns `[]` and pays only the fixed per-call fee; fewer replies than asked for refunds the missing ones. A call takes about 20 to 50 s. It is not offered on Solana: pay on Base or Polygon.

## Comments under a video or post

- POST `/x402/youtube-comments` (or `/mpp/youtube-comments`) with `{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ","limit":20,"sort":"top"}`. `url` is a YouTube video; `limit` is 1 to 100 (default 20); `sort` is `top` (the default) or `newest`.
- POST `/x402/tiktok-comments` (or `/mpp/tiktok-comments`) with `{"url":"https://www.tiktok.com/@nasa/video/7665075736742530317","limit":20}`. `url` is a TikTok video; `limit` is 1 to 100 (default 20).
- POST `/x402/instagram-comments` (or `/mpp/instagram-comments`) with `{"url":"https://www.instagram.com/p/DdG4RIxIPyf/","limit":20}`. `url` is an Instagram post or reel; `limit` is 1 to 100 (default 20).
- The answer is a JSON array with one record per comment: `id`, `text` (cut at 2,000 characters, with `textTruncated`), `postedAt`, `likeCount`, `replyCount`, `isReply`, `byOwner`, `ownerLiked` and `pinned` (null where the platform does not say). The commenter's name, handle and photo are never returned. A post with no comments, or one that cannot be read, returns `[]` and pays only the fixed per-call fee; fewer comments than asked for refunds the missing ones. A call takes up to about a minute. These routes are not offered on Solana: pay on Base or Polygon.

## Web search with page content

- POST `/x402/web-search` (or `/mpp/web-search`) with `{"query":"lithium prices","limit":3}`. `query` is search keywords (not a URL; `site:` and quoted phrases work); `limit` is 1 to 5 (default 3).
- The answer is a JSON array with one record per page that was read: `url`, `title`, `description`, `language`, `content` (the page as Markdown, cut at 20,000 characters) and `contentTruncated`. A result whose page could not be fetched is dropped and its price refunded; a search that finds nothing returns `[]` and pays only the fixed per-call fee.
- A call takes about 20 to 30 s. It is not offered on Solana: pay on Base or Polygon.

## MCP

- The same routes are also an MCP server (streamable HTTP, stateless, latest and earlier protocol versions): one tool per route, paid with x402 inside the tool call. Endpoint: `https://agdata.shveik.dev/mcp` (POST only, no session). `tools/list` shows the tools; call a tool without a payment to get the quote in the result, sign it, and call the tool again with the signed payment in `_meta["x402/payment"]`.

## Endpoints

- [POST /x402/unblock](https://agdata.shveik.dev/x402/unblock): Unblocker: any URL to clean Markdown. Example body: `{"format":"markdown","url":"https://example.com/"}`
- [POST /x402/instagram](https://agdata.shveik.dev/x402/instagram): Instagram posts by profile, tag, search. Example body: `{"limit":5,"mode":"profile","query":"natgeo"}`
- [POST /x402/google-maps](https://agdata.shveik.dev/x402/google-maps): Google Maps places and businesses. Example body: `{"limit":5,"location":"Prague","query":"coffee shops"}`
- [POST /x402/tiktok](https://agdata.shveik.dev/x402/tiktok): TikTok videos by user, hashtag, search. Example body: `{"limit":5,"mode":"profile","query":"nasa"}`
- [POST /x402/linkedin](https://agdata.shveik.dev/x402/linkedin): LinkedIn profile lookup. Example body: `{"query":"jane-example"}`
- [POST /x402/youtube](https://agdata.shveik.dev/x402/youtube): YouTube video search and channel videos. Example body: `{"limit":5,"mode":"search","query":"web scraping tutorials"}`
- [POST /x402/twitter](https://agdata.shveik.dev/x402/twitter): X (Twitter) tweets by handle or search. Example body: `{"limit":5,"mode":"profile","query":"elonmusk"}`
- [POST /x402/reddit](https://agdata.shveik.dev/x402/reddit): Reddit posts by subreddit, user, search. Example body: `{"limit":5,"mode":"hashtag","query":"programming"}`
- [POST /x402/amazon](https://agdata.shveik.dev/x402/amazon): Amazon product search. Example body: `{"limit":5,"query":"wireless keyboard"}`
- [POST /x402/google-search](https://agdata.shveik.dev/x402/google-search): Google Search results (SERP). Example body: `{"limit":5,"query":"best CRM tools"}`
- [POST /x402/google-maps-reviews](https://agdata.shveik.dev/x402/google-maps-reviews): Google Maps reviews of a place. Example body: `{"limit":10,"place":"ChIJN1t_tDeuEmsRUsoyG83frY4"}`
- [POST /x402/youtube-transcript](https://agdata.shveik.dev/x402/youtube-transcript): YouTube transcript of a video. Example body: `{"format":"json","lang":"en","video":"dQw4w9WgXcQ"}`
- [POST /x402/screenshot](https://agdata.shveik.dev/x402/screenshot): Website screenshot or PDF of any URL. Example body: `{"format":"jpeg","url":"https://example.com/"}`
- [POST /x402/linkedin-jobs](https://agdata.shveik.dev/x402/linkedin-jobs): LinkedIn job listings for a search. Example body: `{"limit":10,"location":"Prague","query":"data engineer"}`
- [POST /x402/indeed-jobs](https://agdata.shveik.dev/x402/indeed-jobs): Indeed job listings for a search. Example body: `{"country":"us","limit":10,"location":"Austin, TX","query":"nurse"}`
- [POST /x402/google-news](https://agdata.shveik.dev/x402/google-news): Google News articles for a search. Example body: `{"limit":10,"query":"openai"}`
- [POST /x402/amazon-reviews](https://agdata.shveik.dev/x402/amazon-reviews): Amazon customer reviews of a product. Example body: `{"limit":10,"product":"B079JLY5M5"}`
- [POST /x402/web-search](https://agdata.shveik.dev/x402/web-search): Web search with page content. Example body: `{"limit":3,"query":"lithium prices"}`
- [POST /x402/x-replies](https://agdata.shveik.dev/x402/x-replies): Replies to a post on X. Example body: `{"limit":10,"url":"https://x.com/NASA/status/2106751099063484493"}`
- [POST /x402/youtube-comments](https://agdata.shveik.dev/x402/youtube-comments): Comments under a YouTube video. Example body: `{"limit":20,"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}`
- [POST /x402/tiktok-comments](https://agdata.shveik.dev/x402/tiktok-comments): Comments under a TikTok video. Example body: `{"limit":20,"url":"https://www.tiktok.com/@nasa/video/7665075736742530317"}`
- [POST /x402/instagram-comments](https://agdata.shveik.dev/x402/instagram-comments): Comments under an Instagram post. Example body: `{"limit":20,"url":"https://www.instagram.com/p/DdG4RIxIPyf/"}`

## Machine-readable

- [OpenAPI 3.1](https://agdata.shveik.dev/openapi.json): full request schemas and x-payment-info
- [x402 manifest](https://agdata.shveik.dev/.well-known/x402)
- [Health](https://agdata.shveik.dev/health)