Isolating East-West Traffic: Default-Deny NetworkPolicy and Observability with Cilium Hubble

Most Kubernetes clusters I audit are flat networks wearing a service mesh costume. Ingress is locked down with a WAF, mTLS terminates at the edge, secrets live in Vault — and any compromised pod in namespace-a can open a TCP connection to the PostgreSQL primary in namespace-b, or to the cloud metadata endpoint at 169.254.169.254, without a single log line being written.

The fix is well known: default-deny NetworkPolicy. The reason it is rarely applied to an existing cluster is also well known: nobody knows what actually talks to what. Applying a blanket deny to a production namespace and waiting for pager duty to tell you what broke is not a migration strategy.

This article describes the approach that works: observe first, generate policies from real traffic, enforce in audit mode, then flip to deny. Cilium Hubble gives you the flow data; CiliumNetworkPolicy gives you the enforcement primitives that standard NetworkPolicy lacks (DNS-aware egress, L7 rules, cluster-wide policies).

Context: why east-west is the blind spot

Kubernetes has no native network isolation. The default model is "every pod can reach every pod, in every namespace, on every port". NetworkPolicy is an opt-in allowlist mechanism: as soon as one policy selects a pod, that pod becomes default-deny for the direction(s) covered by that policy. That nuance is where most teams get it wrong — a policy with only an ingress block leaves egress wide open.

The attack path that matters:

1. An RCE in a Java app pod (Log4Shell-style, or a vulnerable dependency).
2. The attacker enumerates the service CIDR and the cluster DNS.
3. Lateral movement to the Redis cache with no auth, to an internal admin API, or to the Kubernetes API server with the pod's ServiceAccount token.
4. Egress to a C2 server, or to the cloud provider IMDS to steal node credentials.

Steps 2 to 4 are all east-west or egress. None of them touch your Ingress controller. Without network policies, none of them produce a denial you can alert on.

Three additional reasons this matters now: PCI-DSS 4.0 and SOC2 both explicitly ask for network segmentation between workloads of different trust levels; the CIS Kubernetes Benchmark flags namespaces without NetworkPolicy; and egress filtering is the cheapest reliable detection point for data exfiltration.

Step 0: verify your CNI actually enforces policies

Before anything else, confirm that your cluster's CNI enforces NetworkPolicy. On OpenShift, OVN-Kubernetes does. On a self-managed cluster running flannel, it does not — the API objects are accepted silently and nothing happens. This silent no-op has burned more than one team into thinking they were segmented.

# Which CNI?
kubectl get pods -n kube-system -l k8s-app=cilium -o wide
kubectl -n openshift-ovn-kubernetes get pods 2>/dev/null | head

# Cilium status and whether Hubble is on
kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose | grep -iE 'hubble|policy'

# Install the CLI tools if needed
cilium status --wait
cilium hubble enable --ui

A quick functional smoke test is worth the two minutes:

kubectl create ns np-test
kubectl -n np-test run victim --image=registry.access.redhat.com/ubi9/ubi-minimal --command -- sleep 3600
kubectl -n np-test expose pod victim --port=8080
kubectl apply -n np-test -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
spec:
  podSelector:
    matchLabels:
      run: victim
  policyTypes:
    - Ingress
EOF
kubectl -n np-test run probe --rm -it --image=registry.access.redhat.com/ubi9/ubi-minimal \
  --restart=Never -- curl -m 3 -s -o /dev/null -w '%{http_code}\n' http://victim:8080
# Expect a timeout / exit 28, not a connection refused.

Step 1: observe before you enforce

Hubble is Cilium's flow observability layer. It reads flow events from the eBPF datapath, so it sees every packet decision including the ones a sidecar-based mesh never sees (raw TCP, UDP, DNS, ICMP).

Enable the Hubble Relay port-forward and start looking:

cilium hubble port-forward &

# Every flow in a namespace, live
hubble observe --namespace payments --follow

# Only flows leaving the namespace (the ones that need egress rules)
hubble observe --namespace payments --not --to-namespace payments --follow

# DNS resolution performed by a given workload — critical for egress design
hubble observe --namespace payments --pod checkout -t l7 --protocol dns

# Anything already being dropped (misconfigured policy, or legitimate noise)
hubble observe --verdict DROPPED --last 500

Collecting a representative sample

A ten-minute capture during a quiet afternoon is useless. You need a window that covers your batch jobs, CronJobs, certificate renewals, nightly reconciliation, and your monthly billing run. In practice: capture for at least a full business cycle, 7 days minimum, 30 if you have monthly jobs.

Hubble's in-memory ring buffer is bounded (default 4095 flows per node). For a real capture, export to a file and aggregate:

# Long-running capture to JSONL
hubble observe --namespace payments --follow -o jsonl >> /tmp/payments-flows.jsonl

# Reduce to a unique (source identity -> destination identity : port) tuple set
jq -r 'select(.flow.verdict=="FORWARDED")
  | [ .flow.source.namespace, (.flow.source.labels | join(",")),
      .flow.destination.namespace, (.flow.destination.labels | join(",")),
      .flow.l4.TCP.destination_port // .flow.l4.UDP.destination_port ]
  | @tsv' /tmp/payments-flows.jsonl | sort -u

For production, do not rely on a port-forward and a shell loop. Enable the Hubble metrics exporter and the file-based flow log on each node:

# Helm values for cilium (hubble section)
hubble:
  enabled: true
  relay:
    enabled: true
  metrics:
    enableOpenMetrics: true
    enabled:
      - "dns:query;ignoreAAAA"
      - "drop:sourceContext=workload-name;destinationContext=workload-name"
      - "flow:sourceContext=workload-name;destinationContext=workload-name"
      - "policy:sourceContext=workload-name;destinationContext=workload-name"
  export:
    static:
      enabled: true
      filePath: /var/run/cilium/hubble/events.log
      allowList:
        - '{"verdict":["DROPPED","ERROR"]}'

Ship events.log to Loki or your SIEM with your existing log agent. You now have a durable, queryable flow archive instead of a ring buffer.

Step 2: derive policies from observed flows

Cilium can do a first pass for you. hubble observe combined with the policy generation in Cilium Enterprise, or the open-source cilium policy tooling, produces a draft — but treat the output as a starting point, never as the final artifact. Generated policies encode accidents (a developer's curl from a debug pod) as permanent allow rules.

The discipline that works: generate, then rewrite by hand using label selectors that express intent, not the labels that happen to exist.

The baseline: cluster-wide default deny

Start with a CiliumClusterwideNetworkPolicy that denies everything except the non-negotiable primitives (DNS, health checks). Apply it in audit mode first — see Step 3.

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: default-deny-all
spec:
  description: "Baseline default-deny for all namespaces except control plane"
  endpointSelector:
    matchExpressions:
      - key: k8s:io.kubernetes.pod.namespace
        operator: NotIn
        values:
          - kube-system
          - cilium-system
          - openshift-dns
          - openshift-ingress
          - openshift-monitoring
  ingress:
    - fromEntities:
        - health
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP
          rules:
            dns:
              - matchPattern: "*"

Two things to note. First, the rules.dns block makes Cilium proxy DNS at L7 — this is what enables toFQDNs later. Second, excluding the platform namespaces is pragmatic: isolating openshift-monitoring is a separate project with its own blast radius.

Per-workload allowlists

Now express the real flows. Example for a checkout service that talks to Postgres, Redis, and one external payment provider:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: checkout-allow
  namespace: payments
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: checkout
  ingress:
    - fromEndpoints:
        - matchLabels:
            app.kubernetes.io/name: api-gateway
            k8s:io.kubernetes.pod.namespace: edge
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "POST"
                path: "/v1/checkout.*"
              - method: "GET"
                path: "/healthz"
  egress:
    - toEndpoints:
        - matchLabels:
            app.kubernetes.io/name: postgres
            k8s:io.kubernetes.pod.namespace: data
      toPorts:
        - ports: [{ port: "5432", protocol: TCP }]
    - toEndpoints:
        - matchLabels:
            app.kubernetes.io/name: redis
            k8s:io.kubernetes.pod.namespace: data
      toPorts:
        - ports: [{ port: "6379", protocol: TCP }]
    - toFQDNs:
        - matchName: "api.stripe.com"
      toPorts:
        - ports: [{ port: "443", protocol: TCP }]

toFQDNs is the feature that makes egress filtering practical. Cilium's DNS proxy observes the resolution, caches the returned IPs, and programs them into the eBPF policy map for the TTL duration. You write api.stripe.com, not a CIDR list that rotates weekly. It requires the DNS L7 rule from the baseline policy to be in effect for that endpoint.

If you prefer to stay on upstream NetworkPolicy for portability, the same intent in vanilla form — note that FQDN egress is simply not expressible:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: checkout-allow
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: checkout
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: edge
          podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - port: 8080
          protocol: TCP
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: data
          podSelector:
            matchLabels:
              app.kubernetes.io/name: postgres
      ports:
        - port: 5432
          protocol: TCP
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - port: 53
          protocol: UDP

Remember the kubernetes.io/metadata.name label is set automatically by the API server on every namespace since 1.21 — use it instead of hand-maintained namespace labels.

Step 3: audit mode, the no-downtime switch

This is the part that lets you ship to production without an outage. Cilium supports a per-endpoint policy audit mode: policy is evaluated, verdicts are recorded as AUDIT in Hubble, but packets are still forwarded.

# Find the endpoint IDs for a workload on a given node
NODE_POD=$(kubectl -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker-02 -o name | head -1)

kubectl -n kube-system exec $NODE_POD -- cilium-dbg endpoint list | grep checkout

# Enable audit mode on endpoint 1234
kubectl -n kube-system exec $NODE_POD -- cilium-dbg endpoint config 1234 PolicyAuditMode=Enabled

# Watch what *would* be denied
hubble observe --namespace payments --verdict AUDIT --follow

Run in audit mode for a full business cycle. Every AUDIT verdict is a flow your policy does not cover. Triage each one into two buckets: legitimate (add a rule) or unexpected (you just found an undocumented dependency, or something worse).

For a cluster-wide rollout, set policyAuditMode: true in the Cilium Helm values on a canary node pool first, rather than per-endpoint by hand.

# Count the AUDIT flows per source/destination pair to prioritise
hubble observe --verdict AUDIT --last 5000 -o jsonl \
  | jq -r '[.flow.source.pod_name, .flow.destination.pod_name,
            (.flow.l4.TCP.destination_port // .flow.l4.UDP.destination_port)] | @tsv' \
  | sort | uniq -c | sort -rn | head -30

When the audit stream is empty for 24 hours, disable audit mode and the policy becomes enforcing. Nothing changed in the datapath configuration except one flag.

Step 4: validate and keep it validated

Enforcement without a regression test decays. Add negative tests to CI:

# Should FAIL (blocked by policy)
kubectl -n payments run redteam --rm -i --restart=Never \
  --image=registry.access.redhat.com/ubi9/ubi-minimal -- \
  timeout 5 bash -c 'curl -s -o /dev/null http://169.254.169.254/latest/meta-data/' \
  && echo "FAIL: IMDS reachable" || echo "OK: IMDS blocked"

# Should SUCCEED (allowed path)
kubectl -n payments exec deploy/checkout -- \
  timeout 5 bash -c 'echo > /dev/tcp/postgres.data.svc.cluster.local/5432' \
  && echo "OK: DB reachable" || echo "FAIL: DB unreachable"

# Cilium's own connectivity test suite
cilium connectivity test --test-namespace cilium-test

Alert on drops. A sustained increase in hubble_drop_total for a workload means either a deployment introduced a new dependency or something is probing:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: hubble-policy-drops
  namespace: kube-system
spec:
  groups:
    - name: network-policy
      rules:
        - alert: PolicyDropSpike
          expr: |
            sum by (source_workload, destination_workload) (
              rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])
            ) > 1
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Policy drops between {{ $labels.source_workload }} and {{ $labels.destination_workload }}"

Security best practices

• Always set policyTypes explicitly. A NetworkPolicy with only an ingress: key does not restrict egress at all. Half-isolated workloads are the most common finding in cluster audits.
• Never allow 0.0.0.0/0 egress. If you must allow broad internet access, use toCIDRSet with except: for RFC1918 ranges, link-local (169.254.0.0/16), and your cloud metadata endpoint. Metadata endpoint access from a pod is an instance-credential theft primitive.
• Protect the DNS path deliberately. Allowing port 53 to 0.0.0.0/0 recreates a covert channel. Restrict DNS to the cluster resolver and enable the L7 DNS proxy so you get query visibility.
• Use CiliumClusterwideNetworkPolicy for the baseline, namespaced policies for exceptions. A namespaced default-deny can be deleted by anyone with namespace edit rights; a cluster-wide one cannot.
• Select on stable, semantic labels (app.kubernetes.io/name, app.kubernetes.io/part-of), never on pod-template-hash or Helm release labels that change on upgrade.
• Keep policies next to the workload in the same Helm chart or Kustomize overlay, so a new dependency and its allow rule land in the same pull request.
• Do not treat NetworkPolicy as authentication. Labels are mutable by anyone who can patch pods; identity is derived from labels. Combine with mTLS and proper authn/authz for anything that matters.
• Use enableDefaultDeny: false sparingly. Cilium 1.14+ lets a policy add allowances without triggering default-deny — useful for shared add-on policies, dangerous if you forget which policy actually establishes the deny.
• Audit egress rules on a schedule. toFQDNs entries accumulate. An allow rule to a SaaS vendor you stopped using three quarters ago is dead weight and attack surface.

Conclusion

Zero-trust networking in an existing cluster is not a big-bang migration; it is a measurement problem followed by a configuration problem. The sequence that works on live production:

1. Verify your CNI enforces policy at all — flannel and some managed CNIs silently ignore NetworkPolicy objects.
2. Observe with Hubble for a full business cycle, exported to durable storage, not a ring buffer. Pay special attention to DNS and to CronJobs that only run monthly.
3. Generate a draft from real flows, then rewrite it by hand with semantic label selectors. Generated policy is evidence, not an artifact.
4. Enforce in audit mode with PolicyAuditMode=Enabled — this is the step that removes the downtime risk entirely. Drain the AUDIT verdict stream to zero before flipping.
5. Validate continuously with negative connectivity tests in CI and a Prometheus alert on hubble_drop_total{reason="POLICY_DENIED"}.

The payoff is not only prevention. Once default-deny is in place, every policy drop becomes a high-signal event: legitimate traffic no longer gets denied, so a drop means either an undeclared dependency or an intrusion. You have converted a flat, silent network into one that tells you when something is wrong.