Deploying to GCP Compute Engine: Two Failures and What They Taught Us

We recently deployed a backend service to a GCP Compute Engine VM: built the Docker image, pushed it to Artifact Registry, and deployed with gcloud compute instances update-container. The container came up clean. The moment we pointed a Cloudflare subdomain at it, two separate issues surfaced that are worth documenting for anyone doing this for the first time.

Failure 1: the IP changed under us

The domain worked, then stopped resolving to anything real.

update-container doesn't just restart the container in place. It stops and starts the whole VM to apply the new metadata, and Compute Engine assigns a new ephemeral external IP on every restart. Our Cloudflare DNS record was still pointing at the old IP, which by then belonged to nobody.

gcloud compute instances update-container INSTANCE_NAME \
  --container-image asia-south2-docker.pkg.dev/PROJECT/REPO/IMAGE:latest

Any command that updates the container, whether for a new image or new env vars, can trigger this. The external IP shifts silently unless you've planned for it.

Fix: reserve a static external IP for the VM. It's a single command, and it removes this entire class of failure permanently.

Failure 2: Cloudflare 521, two root causes at once

With the IP fixed, Cloudflare started returning 521, "web server is down." The server wasn't down. It just wasn't reachable the way Cloudflare expected, for two independent reasons that both needed fixing.

The port. The app was listening on a non-standard port. Cloudflare's proxy only forwards to a fixed set of origin ports, so traffic to anything outside that list fails to connect, silently, with nothing useful in the server logs since the request never arrives.

The SSL mode. Cloudflare's SSL/TLS setting was Full, which means it connects to the origin over HTTPS on 443. Nothing was listening there.

Fix: bind the app to 127.0.0.1 so it's unreachable from outside the VM, run nginx as a reverse proxy on 80 and 443, and close the firewall on the app's original port. A free Cloudflare Origin CA certificate let nginx terminate TLS on 443 correctly.

server {
    listen 443 ssl;
    server_name your-subdomain.example.com;

    ssl_certificate     /etc/nginx/certs/cert.pem;
    ssl_certificate_key /etc/nginx/certs/key.pem;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

After this, nginx is the only thing facing the internet, and every request to the app has already passed through the proxy.

Takeaways

• Reserve a static IP before pointing any DNS record at a Compute Engine VM. Do it before the first deploy, not after DNS breaks.
• Check which ports Cloudflare actually proxies before choosing a port for anything behind it.
• A Cloudflare 521 doesn't mean the app crashed. Check the SSL/TLS mode and what the origin is actually listening on first.

Both issues came from small, reasonable assumptions about how Compute Engine and Cloudflare behave, and both cost more time to diagnose than to prevent.