Back to top
Less than a minute read

Configuring kubectl for multiple clusters

How to use kubectl with more than one cluster

For some people once you have setup kubectl with your cluster you are done, but what happens when you have multiple clusters to manage? That is our situation here at ngineered and so we thought we should post an article about using kubectl with contexts.

As with most things the answer to "How do you setup kubectl to use multiple clusters?" is: it depends. It primarily depends on whether you set your cluster up to use certificates or not. We are going to assume that you have, because that's best practice.

kubectl typically writes it's config file to the .kube directory in your home directory. When you have set this up for the first cluster the config file will look something like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /some/path/to/ca.pem
    server: https://default-master.domain....
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    user: default-admin
  name: default-system
current-context: default-system
kind: Config
preferences: {}
users:
- name: default-admin
  user:
    client-certificate: /some/path/to/admin.pem
    client-key: /some/path/to/admin-key.pem

So once we have our second cluster setup for a customer, that we have
imaginatively called "customer-cluster", we need to add in a cluster
entry:

- cluster:
    certificate-authority: /some/path/to/customer/ca.pem
    server: https://customer-master.domain...
  name: customer-cluster

This names the cluster and points kubectl to the CA certificate and the API server's URL.

And then a context:

- context:
    cluster: customer-cluster
    user: customer-admin
  name: customer-cluster

The context ties together the cluster and the user kubectl will use in this context to access it.

Finally a user for that cluster:

- name: customer-admin
  user:
    client-certificate: /some/path/to/customer/admin.pem
    client-key: /some/path/to/customer/admin-key.pem

And here we specify the path to the certificate and key for that user.

So the whole file would look like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /some/path/to/ca.pem
    server: https://default-master.domain....
  name: default-cluster
- cluster:
    certificate-authority: /some/path/to/customer/ca.pem
    server: https://customer-master.domain...
  name: customer-cluster
contexts:
- context:
    cluster: default-cluster
    user: default-admin
  name: default-system
- context:
    cluster: customer-cluster
    user: customer-admin
  name: customer-cluster
current-context: default-system
kind: Config
preferences: {}
users:
- name: default-admin
  user:
    client-certificate: /some/path/to/admin.pem
    client-key: /some/path/to/admin-key.pem
- name: customer-admin
  user:
    client-certificate: /some/path/to/customer/admin.pem
    client-key: /some/path/to/customer/admin-key.pem

Now when we use kubectl you can use --context to specify which cluster
you are using. You do not need to use --context when using the default cluster, so for example:

kubectl --context customer-cluster get nodes

would get the nodes on the customer-cluster, using the cutomer-admin user certificate and key to authenticate against the customer's API server.

You can set the context using:

kubectl config use-context customer-cluster

Now this command:

kubectl get nodes

will show the nodes on the customer-cluster rather than the default-system cluster, switch back to default by:

kubectl config use-context default-system

You can display what context you are in using:

kubectl config current-context

We tend to use --context in our kubectl commands rather than changing context around, because it is easy to forget which context you are in if you have multiple clusters and, of course, you don't want to be creating or deleting objects on the wrong cluster.

Contact us

If you've found this article useful perhaps ngineered can help you. Please feel free to contact us using the details below if you want to discuss what services ngineered can provide to your company.

Ian MacDougall

Ian is responsible for Ngineered’s customer builds. An expert in the requirements of the enterprise, he is known for his rock-solid builds, and his sensitivity to legacy systems.

Discussion