DEV Community

Cover image for Apigee X Eval Org Stuck at 502? Check These 2 Things First
Sunny JayaRaju
Sunny JayaRaju

Posted on

Apigee X Eval Org Stuck at 502? Check These 2 Things First

#gcp #apigee #cloud #devops

Quick-fix reference. For the full debugging story behind this, see: The 502 That Wouldn't Die

Symptom

Your Apigee X evaluation org shows fully provisioned:

  • Console wizard: all 4 setup steps green ✅
  • organizations.get API: "state": "ACTIVE" ✅
  • instances.list API: "state": "ACTIVE", real host/port ✅
  • Environment attached, proxy deployed, envgroup hostname bound — all clean ✅

And yet every request — even to a brand-new proxy — returns:

HTTP/2 502
Error: Server Error
The server encountered a temporary error and could not complete your request.
Enter fullscreen mode Exit fullscreen mode

Waiting longer doesn't fix it. Deploying a different proxy doesn't fix it.

First: confirm this is actually your bug 

gcloud compute backend-services list
gcloud compute backend-services get-health apigee-proxy-backend --global
Enter fullscreen mode Exit fullscreen mode

If this shows healthState: UNHEALTHY on the apigee-proxy-* instances, keep reading — this is the load-balancer layer that sits between the external HTTPS LB and your actual Apigee runtime, and it's separate from Apigee's own control plane. That's why everything above reports "ACTIVE" while requests still 502: Apigee's config is correct, but the forwarding instances behind the LB aren't actually serving traffic.

Note before you start: the fixes below both work on the instance template, not just the live running instances. These forwarding VMs are preemptible — Google can reclaim and silently replace them at any time, and a replacement rebuilds straight from the template. If you patch a live instance directly instead of the template, the fix will quietly disappear the next time that VM gets swapped, and you'll be back here wondering why it "randomly broke again."

Cause #1: Missing service account on the instance template

Check the boot log of one of the unhealthy instances:

gcloud compute instances get-serial-port-output <INSTANCE_NAME> --zone=<ZONE> | tail -60
Enter fullscreen mode Exit fullscreen mode

Look for:

Instance has service account: false, ...
Failed to download from GCS: ... credentials: cannot fetch token ...
Trying unauthenticated download
Enter fullscreen mode Exit fullscreen mode

Confirm it:

gcloud compute instance-templates describe apigee-proxy-<REGION> \
  --format="yaml(properties.serviceAccounts)"
Enter fullscreen mode Exit fullscreen mode

If this prints null, the template has no service account attached, so the VM can never authenticate to Cloud Storage to pull its real startup script.

Fix — clone the template with a service account attached, then roll the MIG onto it:

# Get every field from your existing template first so you replicate it exactly:
gcloud compute instance-templates describe apigee-proxy-<REGION> --format=yaml

gcloud compute instance-templates create apigee-proxy-<REGION>-fixed \
  --machine-type=e2-micro \
  --image-project=debian-cloud --image-family=debian-12 \
  --boot-disk-size=20GB \
  --network=default --subnet=default --region=<REGION> \
  --tags=https-server,apigee-proxy,gke-apigee-proxy \
  --metadata=startup-script-url=gs://apigee-5g-saas/apigee-envoy-proxy-release/latest/conf/startup-script.sh,ENDPOINT= \
  --service-account=<PROJECT_NUMBER>[email protected] \
  --scopes=cloud-platform \
  --preemptible --no-restart-on-failure --maintenance-policy=TERMINATE

gcloud compute instance-groups managed set-instance-template apigee-proxy-<REGION> \
  --template=apigee-proxy-<REGION>-fixed --region=<REGION>

gcloud compute instance-groups managed rolling-action replace apigee-proxy-<REGION> \
  --region=<REGION>
Enter fullscreen mode Exit fullscreen mode

Match every field from your describe --format=yaml output — machine type, disk, network, tags, and especially the scheduling block. --preemptible, --no-restart-on-failure, and --maintenance-policy=TERMINATE must be specified together or gcloud rejects the combination.

Cause #2: Blank ENDPOINT metadata

Even after fixing the service account, health checks can still fail. These forwarding VMs don't run a proxy application themselves — they install an iptables DNAT rule redirecting incoming port-443 traffic to your real Apigee runtime instance's internal IP. That IP comes from an instance metadata key called ENDPOINT.

Check it from inside an instance:

gcloud compute ssh <INSTANCE_NAME> --zone=<ZONE>
curl -H "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/attributes/ENDPOINT"
Enter fullscreen mode Exit fullscreen mode

If this returns nothing, that's the second bug. Get your runtime instance's real internal IP:

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://apigee.googleapis.com/v1/organizations/<ORG>/instances"
Enter fullscreen mode Exit fullscreen mode

Look for the "host" field in the response (e.g. 10.51.204.98).

⚠️ Don't just patch the running instances — patch the template. These forwarding VMs are typically --preemptible, meaning Google can reclaim them at any time (often within 24h). When that happens, the managed instance group silently rebuilds a replacement straight from the template — and if you only patched the live instance's metadata, the new one comes back with ENDPOINT blank again, and you're right back to a 502 with no idea why it "randomly broke." Fix the template, not just the instance, or this will relapse on you.

Fix — bake ENDPOINT into a corrected template, then roll the instance group onto it (this is the fix that actually survives instance replacement):

gcloud compute instance-templates create apigee-proxy-<REGION>-fixed-v2 \
  --machine-type=e2-micro \
  --image-project=debian-cloud --image-family=debian-12 \
  --boot-disk-size=20GB \
  --network=default --subnet=default --region=<REGION> \
  --tags=https-server,apigee-proxy,gke-apigee-proxy \
  --metadata=startup-script-url=gs://apigee-5g-saas/apigee-envoy-proxy-release/latest/conf/startup-script.sh,ENDPOINT=<RUNTIME_INTERNAL_IP> \
  --service-account=<PROJECT_NUMBER>[email protected] \
  --scopes=cloud-platform \
  --preemptible --no-restart-on-failure --maintenance-policy=TERMINATE

gcloud compute instance-groups managed set-instance-template apigee-proxy-<REGION> \
  --template=apigee-proxy-<REGION>-fixed-v2 --region=<REGION>

gcloud compute instance-groups managed rolling-action replace apigee-proxy-<REGION> \
  --region=<REGION>
Enter fullscreen mode Exit fullscreen mode

This is the same pattern as the Cause #1 fix — clone the template with the correction included, point the MIG at it, roll. Once ENDPOINT lives in the template itself, any future automatic replacement (preemption, autohealing, whatever) inherits the correct value with zero manual intervention.

(If you only need a same-minute workaround while the rolling-replace above is in flight, you can patch the live instance directly — gcloud compute instances add-metadata <INSTANCE_NAME> --zone=<ZONE> --metadata=ENDPOINT=<RUNTIME_INTERNAL_IP> followed by sudo google_metadata_script_runner startup over SSH — but treat that as temporary only, not the real fix.)

Verify the NAT rule landed on the new instances:

sudo iptables -t nat -L -n -v
# Look for: DNAT tcp dpt:443 to:<RUNTIME_INTERNAL_IP>
Enter fullscreen mode Exit fullscreen mode

Confirm it's fixed 

gcloud compute backend-services get-health apigee-proxy-backend --global
Enter fullscreen mode Exit fullscreen mode

Both instances should now show healthState: HEALTHY. Then:

curl "https://<YOUR_HOSTNAME>.nip.io/hello-world"
Enter fullscreen mode Exit fullscreen mode

You should get a real response instead of the 502 page.

This is one specific failure mode out of many possible causes of a 502 on Apigee X — always confirm the backend health check first before assuming this applies to you. Full context and the debugging process that led here: The 502 That Wouldn't Die.

Top comments (0)