The identity our services present when they fetch web content, and how to control that access. This is the page our fetch user-agent points to.
When GDELT Cloud fetches web content, our requests carry the user-agent string: Mozilla/5.0 (compatible; GDELTCloudBot/1.0; +https://gdeltcloud.com/bot). We identify ourselves honestly and do not disguise our requests as an ordinary browser to bypass access controls.
Fetches originate from our cloud infrastructure. If you need a stable network identifier to allow-list or block, contact us and we will share current details.
We fetch two kinds of content. During ingest, we retrieve news article pages that have been surfaced by our discovery sources, and extract their text so we can code the underlying events into structured records. Separately, our research agent retrieves pages relevant to a specific user question at the time that question is asked.
Our product is derived structured data — coded events, entities, and scores — not a re-hosting of article text. We do not republish full article bodies.
We fetch at a modest rate and cache aggressively so we do not re-request the same content unnecessarily. We honor a source's declared content policy: sources that decline automated access are not fetched for their content.
Machine-readable crawl controls — robots.txt directives for our user-agent, crawl-delay, and text-and-data-mining reservations (TDMRep / ai.txt) — are being rolled out across our fetch paths. If you have set one of these and believe we are not yet honoring it, tell us and we will prioritize your domain.
The fastest way to opt a domain out is to email us; we maintain a per-domain exclusion list and will add you promptly. You can also disallow GDELTCloudBot in your robots.txt.
Opting out removes your content from our fetch paths going forward. It does not by itself remove derived records that were previously coded; ask us if you would like those addressed as well.
