The volume storage in NAIS is supported using the storage class rook-ceph. Rook handles the ceph cluster which in turn provides persistent voluments and persistent volume claims. This is most commonly used for elasticsearch installations and other solutions that require persistent volumes for databases. This is persistent storage and available in all clusters. The preferred solution is to use GCP for both applications and persistent volumes, but it is supported on-prem as well.
The installation is done through [navikt/nais-tpa] with helm. This also supports dynamic resizing of the persistent volumes.
On NAIS on-prem we use nais-tpa to install helm charts with the defined persistent volume claims. When the chart is installed rook will initiate the persistent volumes required by the chart. Contact the nais team for questions using slack channel #nais-tpa.
Example yaml for elasticsearch install:
name: examplerelease:chart: nais/opendistro-elasticsearch:0.3configuration:odfe:image:repository: amazon/opendistro-for-elasticsearchtag: 1.0.0pullPolicy: IfNotPresentstateful:enabled: trueclass: "rook-ceph-block"serviceType: ClusterIPenv:TEAM: "team"cluster.name: "example"security:enabled: truemaster:replicas: 3resources:limits:cpu: 2memory: 2048Mirequests:cpu: 0.2memory: 1152Mistateful:size: 4Gienv:ES_JAVA_OPTS: "-Xms1024m -Xmx1024m"ingest:replicas: 2resources:limits:cpu: 2memory: 2048Mirequests:cpu: 0.2memory: 1152Mienv:ES_JAVA_OPTS: "-Xms1024m -Xmx1024m"data:replicas: 3antiAffinity: "soft"resources:limits:cpu: 2memory: 4096Mirequests:cpu: 0.2memory: 2176Mistateful:size: 24Gienv:ES_JAVA_OPTS: "-Xms2048m -Xmx2048m"kibana:replicas: 1resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100menv:ES_JAVA_OPTS: "-Xmx1024m"ingress:domain: nais.example.com
If you are installing elasticsearch and the two client pods does not start as expected, log in to the master-1 pod using kubectl exec and run
Resizing volumes requires changing the persistent volume claim in kubernetes and then changing the helm chart definitions. Contact the nais team if you need the change the storage parameters.
General ceph metrics are available from several dashboards in grafana:
Feel free to help us out by adding examples!