How my app fetches links it doesn’t trust

Network Growth Assistant shows job postings and checks that each Apply link still works. Those links come from other websites, so a stranger decides where they point. That is harmless when a person clicks a link in a browser. It is risky when my server follows the link, because my server can reach places a stranger cannot.

The trick this protects against

Every cloud server has a private side: other machines on the same internal network, and a special address, 169.254.169.254, where many cloud providers hand out information about the server itself. Someone on the internet cannot reach any of that. My server can. So if a “job link” points at an internal address and my server fetches it, the stranger has borrowed my server’s access. This kind of attack is usually called server-side request forgery.

Why checking the link’s text is not enough

• A name can hide an address. jobs.example.com looks public, but its DNS record can point at a private address such as 10.0.0.5.
• A link can bounce. A harmless-looking page can answer with a redirect to an internal address, and a plain fetch follows it without asking.
• The answer can change. A site can give one answer when I look the name up and a different one a moment later when I connect. This is called DNS rebinding.

So the check has to happen at the moment of connecting, on the real address, at every step of the journey. safe-fetch.ts does that in four steps.

Check 1: is the link the right shape?

export function assertFetchableUrl(raw: string | URL): URL {
  let url: URL;
  try {
    url = new URL(raw.toString());
  } catch {
    throw new UnsafeUrlError("not a valid URL");
  }
  if (url.protocol !== "https:" && url.protocol !== "http:") throw new UnsafeUrlError("only http and https are allowed");
  if (url.username || url.password) throw new UnsafeUrlError("credentials in the URL");
  if (url.port && url.port !== (url.protocol === "https:" ? "443" : "80")) throw new UnsafeUrlError("non-standard port");
  const host = url.hostname.replace(/^\[|\]$/g, "").toLowerCase();
  if (isIP(host)) throw new UnsafeUrlError("IP address hosts aren't allowed");
  if (!host.includes(".") || METADATA_HOSTS.has(host) || /\.(local|localhost|internal|lan|home|corp)$/.test(host) || host === "localhost") throw new UnsafeUrlError("not a public hostname");
  return url;
}

This catches the obvious things before any network is touched: only http and https, no username and password hidden in the link, no unusual ports, no raw IP addresses, no internal-sounding names such as .local or .internal, and no known cloud metadata hostnames.

Check 2: where does the name really point?

Next the app looks the name up itself and requires every address that comes back to be public. One private address in the list is enough to refuse. The blocked ranges cover private networks, the machine talking to itself (loopback), link-local addresses like 169.254.169.254, and the IPv6 versions of all of these, including IPv4 addresses wrapped inside IPv6 ones.

Check 3: connect to exactly the address I checked

If the app looked up the name and then connected by name, the second lookup could give a different answer. So the connection is forced to use only the address that was just checked.

// Every lookup for this request returns only the address we checked.
        lookup: ((_host: string, options: { all?: boolean }, cb: (...args: unknown[]) => void) =>
          options?.all ? cb(null, [{ address: address.address, family: address.family }]) : cb(null, address.address, address.family)) as unknown as LookupFunction,
        servername: url.hostname,

The servername line keeps the website’s certificate checked against its real name, so HTTPS still works normally.

Check 4: every redirect starts again from Check 1

  for (let hop = 0; hop <= maxRedirects; hop++) {
    const host = url.hostname.replace(/^\[|\]$/g, "");
    const addresses = await resolvePublic(host, resolve);
    const left = deadline - Date.now();
    if (left <= 0) throw new Error("timed out");
    const res = await transport(url, addresses[0], { timeoutMs, deadlineMs: left, maxBytes: opts.maxBytes ?? 64 * 1024, headers });
    const location = res.headers.location;
    if (res.status >= 300 && res.status < 400 && location && opts.followRedirects !== false) {
      // Every hop is checked again before it's followed.
      url = assertFetchableUrl(new URL(location, url));
      redirects.push(url.toString());
      continue;
    }
    return { ...res, url: url.toString(), redirects };
  }
  throw new UnsafeUrlError("too many redirects");

Redirects are followed by hand, at most five. Each new address goes through the same checks before the app follows it, so a safe page cannot pass me along to an unsafe one.

Limits, so a bad link cannot waste my server either

• Only the first 64 KB of a page is read by default.
• A connection that goes quiet is dropped after 8 seconds, and the whole fetch, redirects included, has a deadline of three times that.
• No cookies are sent, so nothing about a signed-in user can reach the other website.

How I tested it

The name lookup and the connection can be swapped out in tests, so the checks run without touching the real internet. The test file covers a public-looking name that resolves to a private address, a list where only one address is private, IPv6 loopback, a redirect to a private address, and a check that the connection really goes only to the address that was checked.

What I would still be careful about

The point of this file is that it is the one gate for outside links. One gate is much easier to trust than checks scattered around the app. The risk is someone, including me later, adding a plain fetch somewhere else. The rule has to stay that every outside link goes through safeFetch.