This guide walks you through preparing for a Kong Gateway upgrade, helps you determine which upgrade path to use, and helps you decide on the best strategy for each Kong Gateway deployment mode.
Additionally, this guide lists some fundamental factors that play important roles in the upgrade process, and explains how to back up and recover data.
This guide uses the following terms in the context of Kong Gateway:
Upgrade: The overall process of switching from an older to a newer version of Kong Gateway.
Migration: The migration of your data store data into a new environment.
For example, the process of moving 2.8.x data from an old PostgreSQL instance to a new one for 3.4.x is referred to as database migration.
Note: If you are interested in upgrading between the Kong Gateway Enterprise 2.8.x and 3.4.x long-term
support (LTS) versions, see the LTS upgrade guide.
The actual execution of the upgrade depends on type of deployment you have with Kong Gateway.
In this part of the upgrade journey, you will use the strategy you determined during the preparation phase.
Kong Gateway adheres to a structured approach to versioning its products, which makes a
distinction between major, minor, and patch versions.
The upgrade from 2.x to 3.x is a major upgrade.
The lowest version that Kong Gateway 3.0.x supports migrating from is 2.1.x.
Kong Gateway does not support directly upgrading from 1.x to 3.x.
If you are running 1.x, upgrade to 2.1.0 first at a minimum, then upgrade to 3.0.x from there.
While you can upgrade directly to the latest version, be aware of any breaking changes
between the 2.x and 3.x series (both this version and prior versions) and in the Gateway changelog.
Upgrade paths are subject to a wide spectrum of conditions, and there is not a one-size-fits-all way applicable to all users.
Factors include, but are not limited to:
Deployment modes
Custom plugins
Technical capabilities or limitations of the environment
Hardware capacities
SLA
You should discuss the upgrade process thoroughly and carefully with Kong’s support engineers before you take any action.
We encourage you to stay up-to-date with Kong Gateway releases, as that helps maintain a smooth upgrade path.
The smaller the version gap is, the less complex the upgrade process becomes.
Kong Gateway maintains LTS versions and guarantees upgrades between adjacent LTS versions.
The current LTS in the 2.x series is 2.8, and the current LTS in the 3.x series is 3.4.
If you want to upgrade between the 2.8 and 3.4 LTS versions,
see the LTS Upgrade guide.
Always back up your database or declarative configuration files before an upgrade.
The kong migrations commands used during upgrade and database migration are not reversible.
There are two main types of backup for Kong Gateway entities:
Database backup: PostgreSQL has native exporting and importing tools that are reliable and performant, and that ensure consistency when backing up or restoring data. If you’re running Kong Gateway in Traditional or Hybrid mode, you should always take a database-native backup.
Declarative backup: Kong ships two declarative backup tools: decK and the Kong CLI, which support managing Kong Gateway entities in the declarative format.
For Traditional and Hybrid mode deployments, use these tools to create secondary backups. For DB-less mode deployments, use the Kong CLI and manually manage declarative configuration files.
We highly recommend backing up your data using both methods if possible, as this offers you recovery flexibility.
The database-native tools are robust and can restore data instantly compared to the declarative tools.
In case of data corruption, try to do a database-level restore first.
Otherwise, bootstrap a new database and use declarative tools to restore configuration from backup files.
Review the Backup and restore guide to
prepare backups of your configuration.
If you run into any issues and need to roll back, you can also reference that guide to restore your old data store.
Though you could define your own upgrade procedures, we recommend using one of the strategies in this section.
Any custom upgrade requirements may require a well-tailored upgrade strategy.
For example, if you only want a small group of objects to be directed to the new version, use the
Canary plugin and a load balancer that supports traffic interception.
Whichever upgrade strategy you choose, you should account for management downtime for Kong Gateway, as
Admin API operations and database updates are not allowed during the upgrade process.
Based on your deployment type, we recommend one of the following upgrade strategies.
Carefully read the descriptions for each option to choose the upgrade strategy that works best for your situation.
Here’s a flowchart that breaks down how the decision process works:
flowchart TD
A{Deployment type?} --> B(Traditional mode)
A{Deployment type?} --> C(Hybrid mode)
A{Deployment type?} --> D(DB-less mode)
A{Deployment type?} --> E(Konnect DP)
B ---> F{Enough hardware to
run another cluster?}
C --> G(Upgrade CP first) & H(Upgrade DP second)
D ----> K([Rolling upgrade])
E ----> K
G --> F
F ---Yes--->I([Dual-cluster upgrade])
F ---No--->J([In-place upgrade])
H ---> K
click K "/gateway/upgrade/rolling/"
click I "/gateway/upgrade/dual-cluster/"
click J "/gateway/upgrade/in-place/"
Figure 1: Choose an upgrade strategy based on your deployment type. For Traditional mode, choose a dual-cluster upgrade if you have enough resources, or an in-place upgrade if you don’t have enough resources. For DB-less mode and Konnect DPs, use a rolling upgrade. For Hybrid mode, use one of the Traditional mode strategies for CPs, and the rolling upgrade for DPs.
See the following sections for breakdowns and links to each upgrade strategy guide.
A Traditional mode deployment is when all Kong Gateway components are running in one environment,
and there is no Control Plane/Data Plane separation.
You have two options when upgrading Kong Gateway in Traditional mode:
Dual-cluster upgrade:
A new Kong Gateway cluster of version Y is deployed alongside the current version X, so that two
clusters serve requests concurrently during the upgrade process.
In-place upgrade: An in-place upgrade reuses
the existing database and has to shut down cluster X first, then configure the new cluster Y to point
to the database.
We recommend using a dual-cluster upgrade if you have the resources to run another cluster concurrently.
Use the in-place method only if resources are limited, as it will cause business downtime.
Important: While the blue-green upgrade strategy is an option,
we do not recommend it. Support from Kong for upgrades using this strategy is limited.
It is nearly impossible to fully cover all migration tests, because we have to cover all
combinations, given the number of Kong Gateway versions, upgrade strategies, features adopted, and deployment modes.
If you must use this strategy, only use it to upgrade between patch versions.
In DB-less mode, each independent Kong Gateway node loads a copy of declarative Kong Gateway
configuration data into memory without persistent database storage, so failure of some nodes doesn’t spread to other nodes.
Deployments in this mode should use the rolling upgrade strategy.
You could parse the validity of the declarative YAML contents with version Y, using the deck gateway validate or the kong config parse command.
You must back up your current kong.yaml file before starting the upgrade.
Hybrid mode deployments consist of one or more Control Plane (CP) nodes, and one or more Data Plane (DP) nodes.
CP nodes use a database to store Kong Gateway configuration data, whereas DP nodes don’t, since they get all of the needed information from the CP.
The recommended upgrade process is a combination of different upgrade strategies for each type of node, CP or DP.
The major challenge with a Hybrid mode upgrade is the communication between the CP and DP.
As Hybrid mode requires the minor version of the CP to be no less than that of the DP, you must upgrade CP nodes before DP nodes.
The upgrade must be carried out in two phases:
Upgrade the CP according to the recommendations in the section Traditional mode,
while DP nodes are still serving API requests.
Upgrade DP nodes using the recommendations from the section DB-less mode.
Point the new DP nodes to the new CP to avoid version conflicts.
The role decoupling feature between CP and DP enables DP nodes to serve API requests while upgrading CP.
With this method, there is no business downtime.
Custom plugins (either your own plugins or third-party plugins that are not shipped with Kong Gateway)
need to be installed on both the Control Plane and the Data Planes in Hybrid mode.
Install the plugins on the Control Plane first, and then the Data Planes.
See the following sections for a breakdown of the options for Hybrid mode deployments.
CP nodes must be upgraded before DP nodes. CP nodes serve an admin-only role and require database support.
You can select from the same upgrade strategies nominated for Traditional mode (dual-cluster or in-place),
as described in figure 2 and figure 3 respectively.
flowchart TD
DBA[(Current
database)]
DBB[(New
database)]
CPX(Current Control Plane X)
Admin(No admin
write operations)
CPY(New Control Plane Y)
DPX(fa:fa-layer-group Current Data Plane X nodes)
API(API requests)
DBA -.- CPX -."DP connects to either \nCP X...".- DPX
Admin -.X.- CPX & CPY
DBB --pg_restore--- CPY -."...OR to CP Y".- DPX
API--> DPX
style API stroke:none!important,fill:none!important
style DBA stroke-dasharray:3
style CPX stroke-dasharray:3
style Admin fill:none!important,stroke:none!important,color:#d44324!important
linkStyle 2,3 stroke:#d44324!important,color:#d44324!important
Figure 2: The diagram shows a CP upgrade using the dual-cluster strategy.The new CP Y is deployed alongside the current CP X, while current DP nodes X are still serving API requests.
flowchart
DBA[(Database)]
CPX(Current Control Plane X \n #40;inactive#41;)
Admin(No admin \n write operations)
CPY(New Control Plane Y)
DPX(fa:fa-layer-group Current Data Plane X nodes)
API(API requests)
DBA -..- CPX -."DP connects to either \nCP X...".- DPX
Admin -.X.- CPX & CPY
DBA --"kong migrations up \n kong migrations finish"--- CPY -."...OR to CP Y".- DPX
API--> DPX
style API stroke:none!important,fill:none!important
style CPX stroke-dasharray:3
style Admin fill:none!important,stroke:none!important,color:#d44324!important
linkStyle 2,3 stroke:#d44324!important,color:#d44324!important
Figure 3: The diagram shows a CP upgrade using the in-place strategy, where the current CP X is directly replaced by a new CP Y.The database is reused by the new CP Y, and the current CP X is shut down once all nodes are migrated.
From the two diagrams, you can see that DP nodes X remain connected to the current CP node X, or alternatively switch to the new CP node Y.
Kong Gateway guarantees that new minor versions of CPs are compatible with old minor versions of the DP,
so you can temporarily point DP nodes X to the new CP node Y.
This lets you pause the upgrade process if needed, or conduct it over a longer period of time.
This setup is meant to be temporary, to be used only during the upgrade process.
We do not recommend running a combination of new versions of CP nodes and old versions of DP nodes in a long-term production deployment.
After the CP upgrade, cluster X can be decommissioned. You can delay this task to the very end of the DP upgrade.
Once the CP nodes are upgraded, you can move on to upgrade the DP nodes.
The only supported upgrade strategy for DP upgrades is the rolling upgrade.
The following diagrams, figure 4 and 5, are the counterparts of figure 2 and 3 respectively.
flowchart TD
DBX[(Current \n database)]
DBY[(New \n database)]
CPX(Current Control Plane X)
CPY(New Control Plane Y)
DPX(Current Data Planes X)
DPY(New Data Planes Y)
API(API requests)
LB(Load balancer)
Admin(No admin \n write operations)
Admin2(No admin \n write operations)
subgraph A [ ]
Admin -.X.- CPX
DBX -.- CPX
DBY --- CPY
CPX -."Current DP connects to \neither CP X...".- DPX
Admin2 -.X.- CPY
CPY -."...OR to CP Y".- DPX
DPX -.90%..- LB
CPY --- DPY --10%---- LB
end
subgraph B [ ]
API --> LB & LB & LB
end
linkStyle 0,4 stroke:#d44324!important,color:#d44324!important
linkStyle 8,9 stroke:#b6d7a8!important
style CPX stroke-dasharray:3
style DPX stroke-dasharray:3
style DBX stroke-dasharray:3
style API stroke:none!important,fill:none!important
style A stroke:none!important,display:none!important
style B stroke:none!important,display:none!important
style Admin fill:none!important,stroke:none!important,color:#d44324!important
style Admin2 fill:none!important,stroke:none!important,color:#d44324!important
Figure 4: The diagram shows a DP upgrade using the dual-cluster and rolling strategies.The new CP Y is deployed alongside with the current CP X, while current DP nodes X are still serving API requests.In the image, the background color of the current database and CP X is grey instead of white, signaling that the old CP is already upgraded and might have been decommissioned.
flowchart
DBA[(Database)]
CPX(Current Control Plane X \n #40;inactive#41;)
CPY(New Control Plane Y)
DPX(Current Data Planes X)
DPY(New Data Planes Y)
API(API requests)
LB(Load balancer)
Admin(No admin \n write operations)
Admin2(No admin \n write operations)
subgraph A [ ]
Admin -.X.- CPX
DBA -.X.- CPX
DBA --- CPY
CPX -."Current DP connects to \neither CP X...".- DPX
Admin2 -.X.- CPY
CPY -."OR to CP Y".- DPX -.90%..- LB
CPY --- DPY --10%---- LB
end
subgraph B [ ]
API --> LB & LB & LB
end
linkStyle 0,1,4 stroke:#d44324!important,color:#d44324!important
linkStyle 8,9 stroke:#b6d7a8!important
style CPX stroke-dasharray:3,stroke:#c1c6cdff!important
style DPX stroke-dasharray:3
style A stroke:none!important,color:#fff!important
style B stroke:none!important,color:#fff!important
style Admin fill:none!important,stroke:none!important,color:#d44324!important
style Admin2 fill:none!important,stroke:none!important,color:#d44324!important
Figure 5: The diagram shows a DP upgrade using the in-place and rolling strategies.The diagram shows that the database is reused by the new CP Y, while current DP nodes X are still serving API requests.
There is a special case if you deployed Kong Gateway in Hybrid mode and the version you are using is 3.1.0.0 or 3.1.1.1.
Kong removed the legacy WebSocket protocol between the CP and DP, replaced it with a new WebSocket protocol in 3.1.0.0,
and added back the legacy one in 3.1.1.2.
So, upgrade to 3.1.1.2 first before moving forward to later versions.
Additionally, the new WebSocket protocol has been completely removed since 3.2.0.0.
See the following table for the version breakdown:
Review the breaking changes and changelogs for the release or
releases that you are upgrading to.
Make any preparations or adjustments as directed in the breaking changes.
There are some universal factors that may also influence the upgrade, regardless of your deployment mode.
Selecting a strategy for the target deployment mode doesn’t guarantee that you can start the upgrade immediately.
You must also account for the following factors:
During the upgrade process, no changes can be made to the database.
Until the upgrade is completed:
Carefully review all changelogs starting from your current version X,
all the way up to the target version Y.
Look for any potential conflicts, especially for schema changes and functionality removal.
When configuring the new cluster, update kong.conf directly or via environment variables based on the changelog.
Breaking changes in kong.conf in a minor version upgrade are infrequent, but do happen.
For example, the parameter pg_ssl_version defaults to tlsv1 in 2.8.2.4, but in 3.2.2.1, tlsv1 is not a valid argument anymore.
If you were depending on this setting, you would have to adjust your environment to fit one of the new options.
If you have custom plugins, review the code against changelog and test the custom plugin using the new version Y.
If you have modified any Nginx templates like nginx-kong.conf and nginx-kong-stream.conf, also make those changes to the templates for the new version Y.