Web Infrastructure News

Web infrastructure changes constantly. Certificate authorities shorten lifetimes. DNS providers get hijacked. AI crawlers ignore robots.txt. Domain registrars change policies. Miss an update and your site goes down, drops out of search results, or gets impersonated.

This page tracks the changes that matter across SSL/TLS, DNS, domain registration, crawlability, and monitoring. Updated regularly with policy changes, security incidents, industry developments, and standards updates that affect anyone running a website.

Bookmark this page. When something breaks, check here first.

Don't wait for the news to find you

Monitor your entire web infrastructure — uptime, SSL, DNS, domains, and vendor status — from one dashboard.

April 2026

Apple and Google push for 47-day certificate lifetimes

The CA/Browser Forum voted in March 2026 to reduce maximum TLS certificate lifetimes from 398 days to 47 days by March 2029, with intermediate steps: 200 days by March 2027, 100 days by March 2028. Apple proposed the ballot; Google, Mozilla, and the major CAs backed it.

This is the biggest change to SSL/TLS operations since Let's Encrypt made free certificates mainstream. At 47 days, manual certificate renewal becomes operationally impossible for any site with more than a handful of certificates. Automation via ACME (Let's Encrypt, ZeroSSL, Google Trust Services) becomes mandatory.

What this means now:

If you renew certificates manually, start planning your migration to automated renewal
If you use Let's Encrypt with certbot, you're already on 90-day cycles — the tooling works, but your renewal window shrinks from 30 days to roughly 15 days
Monitor certificate expiry dates closely during the transition. A missed renewal at 47 days gives you very little buffer before browsers show errors

See sslcertificateexpiry.com for SSL monitoring and expiry tracking.

Google removes cached page links from search results

Google has fully removed the "Cached" link from search results, completing a phased removal that started in late 2024. The Google Cache served as an informal uptime check for decades — you could see the last version Google crawled, verify indexing, and check for content issues.

With the cache gone:

Use site:yourdomain.com to verify indexing (but you can't see the cached version)
The URL Inspection tool in Google Search Console is now the only way to see how Google renders a page
Third-party tools like the Wayback Machine remain for historical snapshots

This doesn't affect monitoring directly, but it removes a free diagnostic tool that many site operators relied on.

Cloudflare reports record DDoS attack at 5.8 Tbps

Cloudflare mitigated the largest publicly reported DDoS attack in history — 5.8 terabits per second — in Q1 2026. The attack used a Mirai-variant botnet targeting infrastructure in Eastern Asia. Attack duration was under a minute, suggesting automated detection and mitigation.

DDoS attack volume continues to grow quarter over quarter. If your site doesn't use a CDN with DDoS protection, even modest attacks can overwhelm origin servers. Uptime monitoring with low check intervals (60 seconds or less) is the fastest way to detect an attack in progress.

robots.txt legal enforcement: first EU court ruling

A German court issued the first significant ruling on robots.txt enforceability in the context of AI training data, finding that systematic scraping of a news publisher's content in violation of robots.txt directives constituted an infringement under EU copyright law. The ruling is limited in jurisdiction but signals a broader trend.

This matters because robots.txt has historically been a voluntary protocol — crawlers honor it by convention, not by law. If courts start treating robots.txt as a legally enforceable access control mechanism, the stakes for correct configuration rise significantly.

Review your robots.txt to ensure it explicitly addresses AI crawlers. See robotstxttest.com for testing and validation.

March 2026

Let's Encrypt hits 500 million active certificates

Let's Encrypt crossed 500 million active certificates in March 2026, securing approximately 65% of all HTTPS websites globally. The milestone underscores how central free, automated certificate issuance has become to the web's security infrastructure.

The flip side: because Let's Encrypt certificates expire every 90 days (and will need to shorten further under the new CA/Browser Forum rules), the volume of renewals creates a constant background risk. A single ACME client misconfiguration, a DNS provider outage during validation, or an expired domain can silently break renewal — and you won't know until the certificate expires.

Automated certificate monitoring catches these failures before browsers do.

ICANN proposes mandatory DNSSEC for new gTLDs

ICANN's Security and Stability Advisory Committee (SSAC) published recommendations urging mandatory DNSSEC signing for all new generic top-level domains (gTLDs) going forward. Existing gTLDs would have a multi-year compliance window.

DNSSEC adoption remains low (~30% of domains globally) despite being available for over a decade. The proposal wouldn't force individual domain owners to sign their records, but it would require the registry operators to support it, removing one barrier to adoption.

If you manage DNS for domains on newer gTLDs, watch for registry-level DNSSEC changes that may affect your resolution chain. DNS monitoring tools can alert you to signing changes.

Google tightens crawl budget allocation

Google's John Mueller confirmed in a March 2026 Search Central blog post that Googlebot now more aggressively deprioritizes crawling of low-value pages. Sites with large numbers of thin, duplicate, or parameter-heavy URLs will see their crawl budget concentrated on fewer pages.

This makes robots.txt and sitemap configuration more important than ever:

Use robots.txt to explicitly block parameter URLs, staging environments, and internal search result pages
Keep your sitemap lean — only include canonical, indexable pages
Monitor your crawl stats in Google Search Console for drops in coverage

Early 2026

Domain registrar consolidation continues

GoDaddy completed its acquisition of a mid-tier registrar in Q1 2026, continuing a trend of domain registrar consolidation. The top 5 registrars now control over 70% of all registered domains. Consolidation isn't inherently bad, but it concentrates risk — a policy change or outage at a major registrar affects a larger portion of the web.

For domain portfolio management:

Don't put all domains at one registrar
Enable registry lock for critical domains
Monitor domain expiry dates independently of registrar notifications — auto-renewal failures happen

See domainexpirywatcher.com for domain expiry monitoring.

Monitoring industry shifts toward unified platforms

The website monitoring market continues to fragment between deep single-purpose tools and broad unified platforms. The trend in 2026 is toward consolidation: teams want fewer dashboards, not more. Uptime, SSL, DNS, domain, and vendor status checks that were once separate tools are increasingly bundled.

This is exactly why Site Watcher exists — one dashboard for all five monitoring types, with consolidated alerts so you don't get paged by five different tools for the same incident.

Late 2025

AI crawlers overwhelm robots.txt

By the end of 2025, over 40 distinct AI crawlers were documented in the wild — GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Bytespider (ByteDance), CCBot, Meta-ExternalAgent, and dozens more. Each requires its own User-agent line in robots.txt, and new ones appear faster than site operators can block them.

The core problem: robots.txt was designed for a handful of search engine crawlers. It was never intended to be an access control mechanism for an entire industry of AI training pipelines. By late 2025, some sites had robots.txt files with 30+ User-agent blocks — fragile, hard to maintain, and ineffective against crawlers that don't identify themselves honestly.

Cloudflare responded by adding an "AI Bot" toggle in their dashboard, automatically blocking known AI crawlers at the edge without relying on robots.txt. Other CDN providers followed. But for sites without a CDN-level solution, robots.txt remains the primary (and often inadequate) defense.

If you haven't reviewed your robots.txt since early 2025, it's almost certainly out of date. See robotstxttest.com for a current list of AI crawler user-agents and testing tools.

Google deprecates sitemaps ping endpoint

In November 2025, Google officially shut down the https://www.google.com/ping?sitemap= endpoint that site operators used to notify Google of sitemap updates. The endpoint had been deprecated since 2023 but continued working until Google finally removed it.

Going forward, the only ways to notify Google of sitemap changes are:

Google Search Console — manual submission or via API
robots.txt — include your Sitemap: directive (Google periodically checks this)
Wait for Googlebot to discover the sitemap during regular crawling

For most sites this changes nothing in practice — Google was already ignoring pings from high-frequency senders. But automated systems that relied on the ping endpoint (CI/CD pipelines, CMS plugins) need updating.

Cloudflare outage takes down millions of sites

A Cloudflare configuration error in October 2025 caused widespread outages affecting an estimated 3–4 million websites for approximately 45 minutes. The incident highlighted the concentration risk of modern web infrastructure: when a single provider proxies such a large percentage of internet traffic, a misconfiguration becomes a global event.

The outage was notable for what monitoring tools caught — and what they didn't. Uptime monitors that checked from multiple geographic locations detected the issue immediately. Monitors checking from a single location, especially one behind Cloudflare's network, sometimes reported the site as "up" because they were hitting a cached edge node.

Takeaway: monitor from multiple locations, and don't rely solely on your CDN provider's status page to tell you when they're down.

DNSSEC validation failures spike after .com key rotation

Verisign performed a routine KSK (Key Signing Key) rotation for the .com zone in September 2025. While the rotation itself was technically correct, a number of resolvers with stale trust anchors failed to validate the new key, causing transient DNSSEC validation failures for .com domains.

Affected sites appeared to be down from the perspective of users behind strict DNSSEC-validating resolvers (including some enterprise networks and ISPs). The issue resolved within hours as resolver caches refreshed, but it underscored a risk: DNSSEC failures look identical to "site is down" from the user's perspective, and most uptime monitors don't distinguish between a DNSSEC failure and a genuine outage.

DNS monitoring that checks DNSSEC chain validity is the only way to catch these issues proactively.

Mid-2025

Microsoft enforces bulk sender authentication

On May 5, 2025, Microsoft began enforcing sender authentication requirements for Outlook.com, Hotmail.com, and Live.com. Senders transmitting 5,000+ emails per day must now have SPF, DKIM, and DMARC properly configured. Non-compliant messages go to Junk; persistent non-compliance results in outright blocks.

Microsoft was the last major email provider to enforce these requirements. With Google, Yahoo, and Microsoft all requiring authentication, properly configured DNS records (SPF TXT records, DKIM CNAME/TXT records, DMARC TXT records) are no longer optional for any domain that sends email at scale.

This matters for website monitoring because email alerting depends on deliverability. If your monitoring tool sends alerts from a domain with broken authentication, those alerts may never reach your inbox. Monitor your DNS records to ensure authentication stays configured.

Certificate transparency log requirements tighten

Google Chrome began enforcing stricter Certificate Transparency (CT) requirements in mid-2025, requiring all publicly trusted certificates to appear in at least three independent CT logs (up from two). Certificates not meeting this threshold display warnings in Chrome.

For site operators, this is invisible in normal operation — certificate authorities handle CT logging automatically. The risk surfaces when using certificates from smaller or regional CAs, private PKI that accidentally gets used for public-facing sites, or during certificate migrations where the new CA's logging pipeline has issues.

SSL monitoring that verifies CT log presence provides early warning if your certificate isn't meeting browser requirements.

Google Search Console adds Core Web Vitals threshold changes

Google updated the Core Web Vitals thresholds in June 2025, replacing FID (First Input Delay) with INP (Interaction to Next Paint) as the responsiveness metric. The "good" threshold for INP is 200ms, considerably harder to meet than FID's 100ms, because INP measures the worst interaction during the entire page lifecycle rather than just the first one.

Many sites that had "good" FID scores found themselves with "needs improvement" INP scores overnight. JavaScript-heavy pages, complex form interactions, and client-side rendering frameworks were most affected.

Monitor your Core Web Vitals continuously — not just after each deployment, but on an ongoing basis as real-user data accumulates.

Early 2025

Let's Encrypt announces intent to shorten certificate lifetimes

In January 2025, Let's Encrypt announced it was exploring issuing certificates with lifetimes shorter than 90 days — down to as low as 6 days for specific use cases. This preceded the CA/Browser Forum ballot that would later formalize the 47-day maximum.

The announcement signaled that the entire industry was moving toward much shorter certificate lifetimes, making automated renewal not just a best practice but a survival requirement. Any site still renewing certificates manually got its first clear warning that the runway was finite.

Major DNS provider migration causes widespread propagation delays

A major hosting provider migrated its DNS infrastructure to a new anycast network in February 2025, causing 12–24 hours of propagation inconsistency for millions of domains. The provider's own monitoring showed the migration as successful, but end users experienced intermittent resolution failures depending on their geographic location and resolver cache state.

The incident was a reminder that DNS propagation is not instantaneous and is not under your control. When a DNS provider makes infrastructure changes — even routine ones — the effects ripple through resolver caches worldwide. DNS monitoring from multiple geographic locations is the only way to detect propagation inconsistencies before your users report them.

Google continues deprioritizing low-quality pages

Throughout early 2025, Google's helpful content system continued to deprioritize thin, duplicate, and AI-generated content that didn't demonstrate expertise or original value. Sites with large volumes of template-generated pages saw significant indexing drops.

The SEO impact was directly visible in crawl statistics: Googlebot reduced crawl frequency for affected sites, sometimes by 60–80%. Pages that were previously indexed weekly dropped to monthly or less frequent crawling.

For site operators, the takeaway is to keep sitemaps lean (only include pages worth indexing), use robots.txt to block low-value URLs from crawling, and monitor your Google Search Console crawl stats for sudden drops in pages discovered or crawled.

Late 2024

Google and Yahoo escalate bulk sender enforcement

From November 2024 onwards, Google escalated enforcement of its February 2024 bulk sender requirements from warnings to active rejections. Non-compliant messages that previously landed in spam now bounce with permanent failures. Yahoo mirrored the escalation.

Key thresholds now strictly enforced:

Spam complaint rate below 0.3% (Google recommends below 0.1%)
SPF, DKIM, and DMARC all required for 5,000+ daily senders
One-click unsubscribe in marketing emails
TLS encryption for message transmission

For website infrastructure, this made DNS record correctness critical. A missing SPF include, a rotated DKIM key that wasn't updated in DNS, or a DMARC policy of p=none without plans to strengthen — any of these could mean your transactional emails (password resets, order confirmations, monitoring alerts) silently stop arriving.

Apple Mail Privacy Protection reshapes email metrics

By late 2024, Apple Mail Privacy Protection (MPP) affected over 95% of Apple Mail users, with Apple Mail representing roughly half of all email opens globally. Open rates became fundamentally unreliable as a deliverability metric.

This is relevant to monitoring because many monitoring tools send alert emails and track whether alerts were "read" using open tracking pixels. With MPP, all Apple Mail alerts appear as "opened" whether the recipient saw them or not. If you're relying on email alert read receipts to verify your monitoring pipeline is working, those receipts are meaningless for Apple Mail users.

What to watch for the rest of 2026

47-day certificate timeline. The CA/Browser Forum ballot passed. The first reduction (200-day max) lands March 2027. Start automating now.
AI crawler proliferation. New AI crawlers appear monthly. Your robots.txt needs regular review — blocking GPTBot today doesn't block the crawler that launches next month.
DNSSEC push. ICANN's SSAC recommendations may lead to registry-level requirements. Monitor your DNS signing status.
Google crawl budget tightening. Sites with bloated URL spaces will see indexing drop. Clean up your robots.txt and sitemap.
EU AI Act enforcement. AI training data regulations take effect in phases through 2026. Expect more legal scrutiny of crawling practices.
IPv4 exhaustion effects. Shared hosting providers increasingly stack sites on fewer IPv4 addresses, affecting reverse DNS, SSL SNI edge cases, and IP-based reputation. IPv6 adoption in monitoring tools becomes relevant.

This page is updated regularly. Last updated April 13, 2026.

Web Infrastructure News: SSL, DNS, Domain & Monitoring Updates