Warning
on-premises disk with rook/ceph has been discontinued and is not available for general use
On-premises disk¶
The volume storage in NAIS is supported using the storage class rook-ceph. Rook handles the ceph cluster which in turn provides persistent voluments and persistent volume claims. This is most commonly used for elasticsearch installations and other solutions that require persistent volumes for databases. This is persistent storage and available in all clusters. The preferred solution is to use GCP for both applications and persistent volumes, but it is supported on-prem as well.
The installation is done through [navikt/nais-tpa] with helm. This also supports dynamic resizing of the persistent volumes.
Info
This feature is only available in on-premises clusters.
How to¶
Installation of helm chart¶
On NAIS on-prem we use nais-tpa to install helm charts with the defined persistent volume claims. When the chart is installed rook will initiate the persistent volumes required by the chart. Contact the nais team for questions using slack channel #nais-tpa.
Example yaml for elasticsearch install:
example-opendistro-elasticsearch.yaml
name: example
release:
chart: nais/opendistro-elasticsearch:0.3
configuration:
odfe:
image:
repository: amazon/opendistro-for-elasticsearch
tag: 1.0.0
pullPolicy: IfNotPresent
stateful:
enabled: true
class: "rook-ceph-block"
serviceType: ClusterIP
env:
TEAM: "team"
cluster.name: "example"
security:
enabled: true
master:
replicas: 3
resources:
limits:
cpu: 2
memory: 2048Mi
requests:
cpu: 0.2
memory: 1152Mi
stateful:
size: 4Gi
env:
ES_JAVA_OPTS: "-Xms1024m -Xmx1024m"
ingest:
replicas: 2
resources:
limits:
cpu: 2
memory: 2048Mi
requests:
cpu: 0.2
memory: 1152Mi
env:
ES_JAVA_OPTS: "-Xms1024m -Xmx1024m"
data:
replicas: 3
antiAffinity: "soft"
resources:
limits:
cpu: 2
memory: 4096Mi
requests:
cpu: 0.2
memory: 2176Mi
stateful:
size: 24Gi
env:
ES_JAVA_OPTS: "-Xms2048m -Xmx2048m"
kibana:
replicas: 1
resources:
limits:
cpu: 1000m
memory: 2048Mi
requests:
cpu: 100m
env:
ES_JAVA_OPTS: "-Xmx1024m"
ingress:
domain: nais.example.com
If you are installing elasticsearch and the two client pods does not start as expected, log in to the master-1 pod using kubectl exec and run /sgadmin.sh
Resizing volumes requires changing the persistent volume claim in kubernetes and then changing the helm chart definitions. Contact the nais team if you need the change the storage parameters.
Metrics¶
General ceph metrics are available from several dashboards in grafana:
Open Distro Elasticsearch metric referencel
Code examples¶
Feel free to help us out by adding examples!