Debugging Kong Gateway configuration

Kubernetes resources can request configuration that Kong Ingress Controller can’t translate into a valid Kong Gateway configuration. While the admission webhook can reject some invalid configurations during creation and the controller can fix some invalid configurations on its own, some configuration issues require you to review and fix them. When such issues arise, Kong Ingress Controller creates Kubernetes Events to help you identify problem resources and understand how to fix them.

To determine if there are any translation failures that you might want to fix, you can monitor the ingress_controller_translation_count Prometheus metric.

Kong Ingress Controller’s Prometheus metrics include ingress_controller_translation_count and ingress_controller_configuration_push_count counters. Issues that require human intervention add success=false tallies to these counters.

Kong Ingress Controller also generates error logs with a could not update kong admin for configuration push failures.

Kong Ingress Controller provides Kubernetes Events to help understand the state of your system. Events occur when an invalid configuration is rejected by Kong Gateway (KongConfigurationApplyFailed) or when an invalid configuration such as an upstream service that doesn’t exist is detected (KongConfigurationTranslationFailed).

For more information, see Events.

If the controller generates configuration that it cannot apply to Kong successfully, reviewing the generated configuration manually and/or applying it in a test environment can help locate potential causes.

Under normal operation, the controller doesn’t store generated configuration; it is only sent to Kong’s Admin API. The --dump-config flag enables a diagnostic mode where the controller also saves generated configuration and Kong Gateway responses to memory. You can retrieve these via the web interface of the diagnostic server at host:10256/debug/config.

To use the diagnostic mode:

1. Set the --dump-config flag (or CONTROLLER_DUMP_CONFIG environment variable) to true. Optionally set the --dump-sensitive-config flag to true to include un-redacted TLS certificate keys and credentials.

   If you’re deploying with the Helm chart, add the following to your values.yaml file under the controller key:

   ingressController:
    env:
      dump_config: "true"
      dump_sensitive_config: "true"
   

   To enable configuration dumping temporarily for an existing deployment, run the following command:

    kubectl set env -n kong deployment/kong-controller \
      CONTROLLER_DUMP_CONFIG="true" \
      CONTROLLER_DUMP_SENSITIVE_CONFIG="true" \
      -c ingress-controller
   

2. (Optional) Make a change to a Kubernetes resource that you know will reproduce the issue. If you are unsure what change caused the issue originally, you can omit this step.

3. Port forward to the diagnostic server:

   kubectl port-forward -n kong deployments/kong-controller 10256:10256
   

4. Retrieve successfully, or unsuccessfully, applied configurations:

   curl -s localhost:10256/debug/config/successful | jq .config > last_good.json
   curl -s localhost:10256/debug/config/failed | jq .config > last_bad.json
   

5. v3.4+ Retrieve the last error response body received from Kong Gateway:

   curl -so raw_error_body.json localhost:10256/debug/config/raw-error
   

Using dumped configuration

Once you have dumped configuration, take one of the following approaches to isolate issues:

• If you know of a specific Kubernetes resource change that reproduces the issue, diffing last_good.json and last_bad.json will show the change the controller is trying to apply unsuccessfully.

  diff -u last_good.json last_bad.json
  

• You can apply dumped configuration via the /config Admin API endpoint (DB-less mode) or using decK (DB-backed mode) to a test instance not managed by the ingress controller. This approach lets you review requests and responses (passing --verbose 2 to decK will show all requests).

• To run a DB-less Kong Gateway instance with Docker for testing purposes, run curl https://get.konghq.com/quickstart | bash -s -- -D.

  Once this image is running, run curl http://localhost:8001/config --json @last_bad.json to try applying the configuration and see any errors.

  You can also analyze the returned error response body from Kong Gateway to understand the issue.