Manging Resources in a Kubernetes Cluster¶
Kubernetes resources include compute resources and storage resources.
By default, a pod in Kubernetes will run with no limits on compute resources (CPU, GPU, memory) in the default namespace.
To avoid exhausting all available resources, the use of resources can be controlled either at the container level or the namespave level:
users can specify how much compute resources (CPU and memory) each container needs
administrators may set resource quotas for each namespace.
Resource Quotas *¶
A resource quota is a limit to aggregate resource consumption per namespace. Resource Quotas are normally enabled by default, otherwise they can be enabled by starting the Kubernetes apiserver including ResourceQuota in the flag –enable-admission-plugins=.
The following example shows how to set cpu and memory quotas for pods in a namespace. Create a YAML file quota.yaml with the following content:
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-resources
spec:
hard:
pods: "4"
requests.cpu: "1"
requests.memory: 1Gi
limits.cpu: "2"
limits.memory: 2Gi
Apply these limits to a specific <namespace> with the command:
$ kubectl create -f quota.yaml --namespace=<namespace>
The difference bewteen requests and limits is illustrated in this figure:
A request is the amount of a resource that the system will guarantee for the container, and Kubernetes will use this value to decide on which node to place the pod.
A limit is the maximum amount of resources that Kubernetes will allow the container to use.
When allocating compute resources, each container may specify a request and a limit value for either CPU or memory. The quota can be configured to limit either value.
If the quota has a value specified for requests.cpu or requests.memory, then it requires that every incoming container makes an explicit request for those resources. If the quota has a value specified for limits.cpu or limits.memory, then it requires that every incoming container specifies an explicit limit for those resources
Viewing quotas¶
To view the quotas limits and consumptions issue:
$ kubectl get quota --namespace=<namespace>
NAME AGE
compute-resources 30s
object-counts 32s
which will list the values set for quotas in the object ResourceQuotas for that namespace.
More details on the resource usage and limits are provided by the command:
$ kubectl describe quota compute-resources --namespace=<namespace>
Name: compute-resources
Namespace: <namespace>
Resource Used Hard
-------- ---- ----
limits.cpu 0 2
limits.memory 0 2Gi
pods 0 4
requests.cpu 0 1
requests.memory 0 1Gi
requests.nvidia.com/gpu 0 4
Note that GPU are Extended Resources specified by a fully-qualified resource name.
- ..warning:: if quota is enabled in a namespace for compute resources,
users must specify requests or limits for those values when creating pods; otherwise, the quota system may reject pod creation.
Setting Default Requests and Limits for a Namespace¶
In order to set a default value for requests and limits in a namespace you have to create a LimitRange object.
Here is an example of a configuration file that specifies a default CPU request and a default CPU limit:
apiVersion: v1
kind: LimitRange
metadata:
name: cpu-limit-range
spec:
limits:
- default:
cpu: 1
defaultRequest:
cpu: 0.5
type: Container
Then issue the command:
$ kubectl create -f <yaml file> --namespace=<namespace>
If a container is created in that namespace without specifying values for CPU request and CPU limit, the container is given a default CPU request of 0.5 and a default CPU limit of 1.
Note that for GPU the default must match defaultRequest in LimitRange if both are specified.
What if you specify a container’s limit, but not its request?¶
The container’s CPU request is set to match its CPU limit.
What if you specify a container’s request, but not its limit?¶
The container’s CPU request is set to the value specified in the container’s configuration file.
