As parents in tech, we’ve learned that neither children nor applications thrive without clear boundaries. There are no “good” or “bad” kids, just as there are no inherently “good” or “bad” applications, only behaviors shaped by the guardrails around them. In parenting, we establish rules to encourage safe, responsible decisions while still allowing independence. Modern cloud-native platforms require the same approach. Policy as Code provides those guardrails, defining how applications can operate, what resources they can access, and how they remain compliant at scale without slowing down innovation.

Just as parents establish boundaries to help their children make safe and responsible decisions, platform engineering teams rely on policies to guide application behavior. The challenge is that setting rules alone is not enough; you also need visibility into whether they are being followed, where they are failing, and whether they are creating unintended consequences. Without real-time observability, policy enforcement can become a black box, making it difficult to identify misconfigurations, compliance gaps, or obstacles to developer productivity before they impact production.

This is where Policy as Code and Observability come together. Much like a parent needs feedback to understand how rules are working in practice, platform teams need continuous insight into policy outcomes across their clusters. By combining Kyverno, a Kubernetes-native policy engine, with VictoriaMetrics, an open-source monitoring solution, teams can eliminate policy blind spots and gain a unified view of both compliance and system health, enabling them to scale governance without sacrificing visibility or developer velocity.

Why This Matters: Community Impact and Practitioner Value

Historically, cluster security and infrastructure observability have existed as two distinct silos within platform organizations. Security teams write policy rules, while reliability teams build dashboards.

This framework changes that dynamic by treating policy enforcement as a primary telemetry data source. Instead of relying on passive, text-heavy log files to figure out why a deployment failed, this setup directly transforms admission control webhooks into real-time metric streams. It treats policy as data, allowing organizations to observe compliance trends with the exact same tooling used to monitor CPU usage or network latency.

How It Directly Helps Practitioners

• Removes “Invisible Wall” Friction: Developers frequently encounter admission controllers as sudden, unexplained blocks during deployments. By exposing policy metrics on shared Grafana dashboards, developers can instantly see exactly which validation rules are failing across the cluster, transforming security from a roadblock into self-service feedback.
• Simplifies Governance via Native Tooling: Practitioners do not need to learn specialized, complex programming languages to audit cluster states. Because Kyverno relies entirely on standard YAML and CEL (Common Expression Language), and VictoriaMetrics natively supports standard PromQL queries, any practitioner can modify policies or adjust monitoring dashboards without deep proprietary expertise.
• Prevents Operational Overhead and Budget Bloat: High-volume clusters can trigger thousands of admission requests every minute. For practitioners running traditional monitoring systems, storing these high-cardinality metrics can lead to massive RAM usage and steep infrastructure bills. Using a lightweight time-series backend ensures that teams gain comprehensive policy visibility without incurring high data storage costs.

Operational Challenges Solved

1. Eliminating Specialized Language Overhead

Adopting policy-as-code often introduces a steep learning curve due to specialized, non-native languages. This creates operational bottlenecks where only a small number of engineers can write, maintain, or audit compliance rules. Kyverno addresses this by managing policies as standard Kubernetes manifests using declarative YAML and CEL.

2. Managing High-Cardinality Metrics Scaling

Tracking every admission request, mutation, validation failure, and image verification at scale generates a significant volume of time-series data. VictoriaMetrics is a resource-efficient time-series database that can process millions of data points with low memory overhead. Its native UI Cardinality Explorer helps teams identify which policy labels consume the most storage, ensuring the monitoring infrastructure remains performant.

3. Surface Visibility for Enforced Rules

Policy enforcement can sometimes function as an invisible wall for developers. An admission controller rejects workloads, and there is no central dashboard to track why or how often violations occur. This architecture uses a five-policy implementation that spans the resource lifecycle and forwards raw engine telemetry to real-time dashboards.

Architectural Deployment

Phase 1: Deploying the Core Components

First, install the Kyverno Policy Engine to handle admission control, then install VictoriaMetrics as the time-series database backend.

# 1. Install Kyverno Policy Engine
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace



# 2. Install VictoriaMetrics Single-Node Server
helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update
helm install vm vm/victoria-metrics-single 

Phase 2: Configuring Metric Collection

The vmagent gathers telemetry by scraping the dedicated Kyverno metrics service endpoint (port 8000) and forwarding those data points directly to VictoriaMetrics.

helm upgrade --install vmagent victoriametrics/victoria-metrics-agent \
  --set 'remoteWrite[0].url=http://vm-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write' \
  --set 'extraArgs.promscrape.config-global=\n  scrape_interval: 10s\n\nscrape_configs:\n  - job_name: kyverno\n    metrics_path: /metrics\n    static_configs:\n      - targets:\n          - kyverno-svc-metrics.kyverno.svc.cluster.local:8000' \
  --set 'podLabels.app=vmagent' \
  --set 'podLabels.environment=demo' \
  --set 'podLabels.team=observability'

Verify that vmagent is utilizing the correct target configuration and successfully scraping endpoints:

curl http://localhost:8429/targets

Deploying the good and the bad apps

Just as clear expectations help guide behavior, platform policies help ensure applications follow organizational standards from the moment they are deployed. To demonstrate this, we deploy an application that intentionally violates organizational standards by omitting required Kubernetes labels.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bad-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: bad-app
  template:
    metadata:
      labels:
        app: bad-app
        random-id: "12345"
    spec:
      containers:
        - name: nginx
          image: nginx

In contrast, we deploy an application that follows the organization’s labeling and metadata standards.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: good-app
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: good-app
  template:
    metadata:
      labels:
        app: good-app
        environment: demo
        team: platform
    spec:
      containers:
        - name: nginx
          image: nginx:stable

The Five-Policy Lifecycle Framework

To move beyond simple enforcement and make policy behavior observable, we introduce a structured approach that follows the full lifecycle of a workload in the cluster.

1. Validating Policy

We create a validating policy that enforces the presence of required labels.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-observability-labels
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: check-required-labels
      match:
        resources:
          kinds:
            - Pod
      validate:
        message: "Missing observability labels: app, environment, team"
        pattern:
          metadata:
            labels:
              app: "?*"
              environment: "?*"
              team: "?*"

2. Mutating Policy

To reduce developer friction, we introduce a mutation policy that automatically injects missing compliance labels.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-safe-observability-labels
spec:
  rules:
    - name: add-labels-to-pods-only
      match:
        any:
          - resources:
              kinds:
                - Pod
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              demo: "true"
              observability: "enabled"
              environment: "demo"
              team: "observability"

3. Generating Policy

Generating policies extends governance beyond enforcement by automatically creating required resources at namespace initialization.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-configmap
spec:
  rules:
    - name: generate-config
      match:
        any:
          - resources:
              kinds:
                - Namespace
      generate:
        apiVersion: v1
        kind: ConfigMap
        name: demo-config
        namespace: "{{request.object.metadata.name}}"
        data:
          data:
            message: "Hello from Kyverno"

4. Validating Image Policy

Securing the software supply chain requires restricting pod creation to trusted container registries.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  rules:
  - name: validate-image-registry
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Only images from Docker Hub library are allowed"
      pattern:
        spec:
          containers:
          - image: "nginx*"

5. Deleting Policy

Deleting policies are designed to prevent unsafe operations by explicitly restricting high-risk actions, such as deleting critical workloads.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: protect-deployments
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: block-deletion-of-deployments
      match:
        any:
          - resources:
              kinds:
                - Deployment
      validate:
        message: "Deletion of Deployments is not allowed in this demo"
        deny:
          conditions:
            any:
              - key: "{{ request.operation }}"
                operator: Equals
                value: DELETE

Dashboard Visualization and Telemetry Analysis

Once metrics are flowing, enforcement events translate directly into real-time Grafana observability dashboards via standardized PromQL queries routed to VictoriaMetrics.

Key Monitoring Metrics

• Total Webhook Throughput: Tracks the total volume of admission requests. kyverno_admission_requests_total
• Total count of admission requests grouped by whether they were allowed or blocked sum by (request_allowed)(kyverno_admission_requests_total)
• Resource Distribution: Aggregated via sum by (resource_kind), shows which resource kinds are most frequently affected by policy enforcement.

Operational Summary

To summarize: policies are not just enforcement mechanisms. They are signals about how your platform behaves in the real world. And when you observe those signals, you transform them into operational intelligence.