> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-parallel-read-in-order-multi-part.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy ClickHouse with FIPS Compliance (Government)

This tutorial covers deploying ClickHouse Private with FIPS 140-3 compliance on AWS. It builds on the standard AWS deployment, with stricter node architecture and AMI requirements, then adds the FIPS-specific Kubernetes configuration.

<Note>
  **Deploying on GKE?**

  The FIPS-specific Kubernetes steps below, including certificate generation, certificate secrets, and the `openSSL` Helm flags, are cloud-agnostic. For GCP, provision x86\_64 infrastructure with [Deploy on GCP GKE](/cloud/clickhouse-private/tutorials/deploy-gcp), then apply those steps here, substituting the GCS storage settings and Workload Identity from the GCP guide.
</Note>

***

## Prerequisites

Before starting this tutorial, you must have completed the following from [tutorials/deploy-aws.md](/cloud/clickhouse-private/tutorials/deploy-aws):

* **Step 1:** IAM role for ECR access
* **Step 2:** Container images copied to your ECR using the FIPS version tags below
* **Step 3:** VPC created
* **Step 4:** EKS cluster created
* **Step 5:** FIPS node groups created using the x86\_64 requirements below instead of the standard ARM64 recommendations
* **Step 6:** S3 bucket and IAM roles created

### FIPS node group requirements

Use x86\_64 nodes for ClickHouse Server, ClickHouse Keeper, and the ClickHouse Operator. The cryptographic provider validation for the current Server and Keeper FIPS images does not include Linux on ARM64. A FIPS-enabled ARM node image does not extend that validation to the cryptographic module inside the ClickHouse containers.

<Note>
  **Keeper topology requirements also apply to FIPS deployments**

  For production HA, run the three Keeper nodes in three distinct availability zones. If any one zone is lost, the remaining two replicas still form a quorum. See [Keeper quorum and failure domains](/cloud/clickhouse-private/reference/infrastructure-requirements#keeper-quorum-and-failure-domains).
</Note>

AWS supports two x86\_64 node image paths for this deployment:

* Use `BOTTLEROCKET_x86_64_FIPS`, which is available directly as an EKS managed node group AMI type.
* Build a custom EKS-optimized Amazon Linux 2023 x86\_64 AMI with [FIPS mode enabled](https://docs.aws.amazon.com/linux/al2023/ug/fips-mode.html), then select it through an EC2 launch template. AWS publishes `AL2023_x86_64_STANDARD`, but does not expose a separate `AL2023_x86_64_FIPS` managed node group AMI type. See the [EKS AMI build scripts](https://docs.aws.amazon.com/eks/latest/userguide/eks-ami-build-scripts.html).

The Bottlerocket option avoids maintaining a custom AMI. Use the AL2023 option when your organization standardizes on AL2023 and already has a custom AMI build and patching process.

| Node group | Architecture | FIPS node AMI options | Instance guidance | Kubernetes label |
| - | - | - | - | - |
| Server | x86\_64 | `BOTTLEROCKET_x86_64_FIPS` or custom FIPS-enabled AL2023 x86\_64 | x86 "d" series with local NVMe and a 1:4 vCPU-to-memory ratio; [`m5d.16xlarge`](https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html) is a compatible GovCloud example | `clickhouseGroup: server` |
| Keeper | x86\_64 | `BOTTLEROCKET_x86_64_FIPS` or custom FIPS-enabled AL2023 x86\_64 | `m8i.2xlarge` when available; `m7i.2xlarge`, `m6i.2xlarge`, or `m5.2xlarge` are compatible alternatives | `clickhouseGroup: keeper` |
| Operator | x86\_64 | `BOTTLEROCKET_x86_64_FIPS` or custom FIPS-enabled AL2023 x86\_64 | Minimum xlarge, using an x86-compatible instance type | No dedicated label required |

Configure instance-store NVMe using the matching AL2023 or Bottlerocket tab in [Step 5 of the standard AWS deployment](/cloud/clickhouse-private/tutorials/deploy-aws#server-node-group). In the Helm command below, set `SSD_CACHE_IS_ON_EMPTY_DIR=true` for Bottlerocket, which binds instance-store NVMe to `/var/lib/kubelet`. Set it to `false` for AL2023, which mounts instance-store NVMe at `/nvme/disk`.

### FIPS Version Tags

Use the following FIPS-specific image tags when copying artifacts:

| Artifact | Tag |
| - | - |
| clickhouse-server | `26.4.1.2596-fips` |
| clickhouse-keeper | `26.2.1.258-fips` |
| clickhouse-operator | `main-1.22166.1-fips` |
| helm/clickhouse-operator-helm | `1.22166.1` |
| helm/onprem-clickhouse-cluster | `1.11.6` |

### S3 Bucket Naming Restriction

**The S3 bucket name cannot contain any periods (`.`)** when using FIPS. The S3 FIPS endpoints require virtual-host-style addressing, which does not work when the bucket name contains periods.

***

## Step 1: Create FIPS Certificates

FIPS compliance requires TLS certificates generated using FIPS-approved algorithms in a FIPS-enabled environment. You need three sets of certificates:

| Certificate | Private Key | Public Key | SAN Required |
| - | - | - | - |
| Server | `server.key` | `server.crt` | `*.c-${CLUSTER_NAME}-server-headless.ns-${CLUSTER_NAME}.svc.cluster.local`, `c-${CLUSTER_NAME}-server-any.ns-${CLUSTER_NAME}.svc.cluster.local` |
| Client | `client.key` | `client.crt` | (none) |
| Keeper | `keeper.key` | `keeper.crt` | `*.c-${CLUSTER_NAME}-keeper-headless.ns-${CLUSTER_NAME}.svc.cluster.local` |

You also need a CA certificate (`ca.crt` and `ca.key`), either self-signed or from your organization's certificate authority.

**Requirements:**

* Minimum RSA 3072-bit key size
* SHA-256 or higher signature algorithm
* Generated in a FIPS-enabled environment
* A unique set of certificates (excluding CA) must be created for each ClickHouse cluster

For the full certificate generation procedure using a Red Hat UBI8 container with FIPS crypto policy, see [how-to/configure-fips-certificates.md](/cloud/clickhouse-private/how-to/configure-fips-certificates).

***

## Step 2: Install Kubernetes Prerequisites

### Install VolumeSnapshot CRDs

```bash theme={null}
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
```

### Install StorageClass (Optional)

Only if you do not have a custom or existing StorageClass:

```bash theme={null}
ECR_HOST=0000000000.dkr-ecr-fips.us-west-2.on.aws
CHART_VERSION=1.11.6

helm install clickhouse-prerequisites \
    oci://$ECR_HOST/helm/onprem-clickhouse-cluster \
    --version=$CHART_VERSION \
    -n default \
    --set-json="storageClass.create=true" \
    --set-json="createCluster=false" \
    --set-json="serviceAccount.create=false"
```

***

## Step 3: Install Operator

Note the FIPS ECR endpoint format: `0000000000.dkr-ecr-fips.us-west-2.on.aws` (not the standard `.amazonaws.com` endpoint).

```bash theme={null}
# update ECR_HOST as needed
ECR_HOST=0000000000.dkr-ecr-fips.us-west-2.on.aws

# should use the version of the operator's helm chart, not of the operator itself (eg 1.22166.1, not main-1.22166.1)
OPERATOR_VERSION=1.22166.1

# image tag: the FIPS operator build (note the main- prefix and the -fips suffix).
# Always set this explicitly: the FIPS variant is selected by the tag, so an
# unset tag will not give you a FIPS build.
OPERATOR_IMAGE_TAG=main-1.22166.1-fips

# set AZs as determined by VPC subnets
AZ_LIST='["us-west-2a","us-west-2b","us-west-2c"]'

helm install clickhouse-operator \
   oci://$ECR_HOST/helm/clickhouse-operator-helm \
   --version=$OPERATOR_VERSION \
   --create-namespace \
   -n clickhouse-operator-system \
   --set-json="image.repository=\"$ECR_HOST/clickhouse-operator\"" \
   --set-json="image.tag=\"$OPERATOR_IMAGE_TAG\"" \
   --set-json='cilium.enabled=false' \
   --set-json='idleScalerEnabled=false' \
   --set-json='webhooks.enabled=false' \
   --set-json='operator.debug=true' \
   --set-json='operator.metricsScraper.enabled=false' \
   --set-json="operator.availabilityZones=$AZ_LIST" \
   --set-json='operator.featureFlags.backupOnASeparatePod=true' \
   --set-json='operator.featureFlags.serverCMEKEnabled=true'
```

***

## Step 4: Create Certificate Secrets

Secrets must be installed in the same Kubernetes namespace where the ClickHouse cluster will run. If your cluster name is `default-xx-01`, the namespace will be `ns-default-xx-01`.

Ensure the namespace exists, then create the secrets. **Do not change the secret names or key names.**

```bash theme={null}
CERT_DIR=.
CLUSTER_NAME=default-xx-01

# server
kubectl create secret generic -n ns-${CLUSTER_NAME} ${CLUSTER_NAME}-server-cert-secret \
  --from-file=ca.crt="$CERT_DIR/ca/ca.crt" \
  --from-file=server.crt="$CERT_DIR/server/server.crt" \
  --from-file=server.key="$CERT_DIR/server/server.key" \
  --from-file=client.crt="$CERT_DIR/client/client.crt" \
  --from-file=client.key="$CERT_DIR/client/client.key"

# keeper
kubectl create secret generic -n ns-${CLUSTER_NAME} ${CLUSTER_NAME}-keeper-cert-secret \
  --from-file=ca.crt="$CERT_DIR/ca/ca.crt" \
  --from-file=keeper.crt="$CERT_DIR/keeper/keeper.crt" \
  --from-file=keeper.key="$CERT_DIR/keeper/keeper.key"
```

***

## Step 5: Deploy ClickHouse Cluster

Use the correct FIPS version of the endpoints for ECR and S3. FIPS endpoints can be found at [https://aws.amazon.com/compliance/fips/](https://aws.amazon.com/compliance/fips/).

<Info>
  **Guaranteed QoS (recommended)**

  ClickHouse workloads should run with matching `requests` and `limits` for
  both CPU and memory. The single `SERVER_CPU`/`SERVER_MEMORY`/`KEEPER_CPU`/
  `KEEPER_MEMORY` values below are applied to **both** `resources.requests`
  and `resources.limits` in the helm invocation that follows, which places
  the pods in the [Guaranteed](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/)
  QoS class. If you need to run with a different QoS class, review
  [Pod QoS: Guaranteed (recommended)](/cloud/clickhouse-private/reference/infrastructure-requirements#pod-qos-guaranteed-recommended)
  first for the trade-offs.
</Info>

<Note>
  **FIPS deployments set the image tags explicitly**

  The standard tutorials leave `server.image.tag` / `keeper.image.tag` unset and take
  the versions pinned in the cluster Helm chart. FIPS is the exception: the FIPS
  variant is selected by the image tag rather than by a separate chart, so the `-fips`
  tags below must be set explicitly. Keep them aligned with the versions in the
  [Component Versions reference](/cloud/clickhouse-private/reference/versions).
</Note>

### Create the Admin Password Secret

The `default` user's password is read from a Kubernetes Secret in the cluster's namespace. Create it alongside the certificate secrets from [Step 4](#step-4-create-certificate-secrets), before installing the chart, because the operator reads it as soon as it reconciles the cluster.

```bash theme={null}
CLUSTER_NAME=default-xx-01
PASSWORD_SECRET_NAME=clickhouse-admin-password

kubectl create secret generic $PASSWORD_SECRET_NAME -n ns-$CLUSTER_NAME --from-literal=password='p@$$w0rd'
```

Use the password itself, not a hash. Single-quote it so the shell does not expand characters such as `$` in the value.

<Note>
  The chart only needs the Secret's name, so anything that produces a Secret with a `password` key in that namespace works, including External Secrets Operator, a Vault injector, or a sealed secret. Changing the value later is applied automatically; see [Reset passwords](/cloud/clickhouse-private/how-to/reset-passwords).
</Note>

### Deploy the Cluster

```bash theme={null}
# update values below as needed,
# the ecr host should be the one in your account
# the s3 bucket should be the one created earlier
# the password secret is the one created above
CLUSTER_NAME=default-xx-01
PASSWORD_SECRET_NAME=clickhouse-admin-password
ECR_HOST=0000000000.dkr-ecr-fips.us-west-2.on.aws
SA_IAM_ROLE_ARN=arn:aws:iam::0000000000:role/CH-S3-${CLUSTER_NAME}-uw2-00-Role

S3_BUCKET_REGION=us-gov-west-1
S3_BUCKET_URL=https://my-clickhouse-data.us-gov-west-1.amazonaws.com
S3_BUCKET_NAME=my-clickhouse-data

# s3 key prefix can use any UUID value, but must be unique for all clusters storing data in the bucket specified above
S3_KEY_PREFIX=ch-s3-$(uuidgen | tr '[:upper:]' '[:lower:]')

# these must match the x86_64 clickhouseGroup labels added to the node groups
SERVER_NG_GROUP=server
KEEPER_NG_GROUP=keeper

# these values should change depending on selected instance sizes
# be sure to take daemon set requirements into account when setting CPU and MEMORY values
SERVER_CPU=64
SERVER_MEMORY=256Gi
# Keeper: high-load profile; for moderate load 2-4 CPU / 8Gi also works (see the infrastructure reference)
KEEPER_CPU=4
KEEPER_MEMORY=16Gi

# Update the `verificationMode` field to control how TLS certificates are verified.
# Defaults to `none`. See Context.h in the ClickHouse source code for more details.
SERVER_VERIFICATION_MODE=none
KEEPER_VERIFICATION_MODE=none

# Currently, usage of `strict` is not recommended and you might encounter issues with the Operator.
# If you require mTLS cert auth, use `relaxed` (Teleport integration requires it)
# SERVER_VERIFICATION_MODE=relaxed
# KEEPER_VERIFICATION_MODE=relaxed

CHART_VERSION=1.11.6

# As a general rule, set cache size to 70-80% of the allocatable instance-store SSD capacity,
# accounting for DaemonSets and cloud-provider reserved disk space.
# The example below sizes the cache for the four 600 GB NVMe devices on m5d.16xlarge.
CACHE_DISK_SIZE_GI=1700Gi

# Select according to the Server node AMI:
# - Bottlerocket with instance-store NVMe bound to /var/lib/kubelet: true
# - AL2023 with instance-store NVMe mounted at /nvme/disk: false
SSD_CACHE_IS_ON_EMPTY_DIR=true

helm install $CLUSTER_NAME \
    oci://$ECR_HOST/helm/onprem-clickhouse-cluster \
    --version=$CHART_VERSION \
    -n ns-$CLUSTER_NAME \
    --create-namespace \
    --set-json="account.passwordSecretRef.name=\"$PASSWORD_SECRET_NAME\"" \
    --set-json="server.image.repository=\"$ECR_HOST/clickhouse-server\"" \
    --set-json="server.storage.s3.endpoint=\"$S3_BUCKET_URL\"" \
    --set-json="server.storage.s3.region=\"$S3_BUCKET_REGION\"" \
    --set-json="server.storage.s3.bucketName=\"$S3_BUCKET_NAME\"" \
    --set-json="server.storage.s3.keyPrefix=\"$S3_KEY_PREFIX\"" \
    --set-json='server.arm64=false' \
    --set-json="server.image.tag=\"26.4.1.2596-fips\"" \
    --set-json="server.podPolicy.nodeSelector.clickhouseGroup=\"$SERVER_NG_GROUP\"" \
    --set-json="server.openSSL.enabled=true" \
    --set-json="server.openSSL.required=true" \
    --set-json="server.openSSL.verificationMode=$SERVER_VERIFICATION_MODE" \
    --set-json="keeper.image.repository=\"$ECR_HOST/clickhouse-keeper\"" \
    --set-json='keeper.arm64=false' \
    --set-json="keeper.image.tag=\"26.2.1.258-fips\"" \
    --set-json="keeper.podPolicy.nodeSelector.clickhouseGroup=\"$KEEPER_NG_GROUP\"" \
    --set-json="keeper.openSSL.enabled=true" \
    --set-json="keeper.openSSL.required=true" \
    --set-json="keeper.openSSL.verificationMode=$KEEPER_VERIFICATION_MODE" \
    --set-json="serviceAccount.annotations={\"eks.amazonaws.com/role-arn\":\"$SA_IAM_ROLE_ARN\"}" \
    --set-json='server.tolerations=[{"effect":"NoSchedule","key":"clickhouse.com/do-not-schedule","operator":"Exists"}]' \
    --set-json='keeper.tolerations=[{"effect":"NoSchedule","key":"clickhouse.com/do-not-schedule","operator":"Exists"}]' \
    --set-json="server.ssdCacheConfiguration.cacheDiskSize=\"$CACHE_DISK_SIZE_GI\"" \
    --set-json="server.ssdCacheConfiguration.isOnEmptyDir=$SSD_CACHE_IS_ON_EMPTY_DIR" \
    --set-json="server.podPolicy.resources.limits.cpu=\"$SERVER_CPU\"" \
    --set-json="server.podPolicy.resources.limits.memory=\"$SERVER_MEMORY\"" \
    --set-json="server.podPolicy.resources.requests.cpu=\"$SERVER_CPU\"" \
    --set-json="server.podPolicy.resources.requests.memory=\"$SERVER_MEMORY\"" \
    --set-json="keeper.podPolicy.resources.limits.cpu=\"$KEEPER_CPU\"" \
    --set-json="keeper.podPolicy.resources.limits.memory=\"$KEEPER_MEMORY\"" \
    --set-json="keeper.podPolicy.resources.requests.cpu=\"$KEEPER_CPU\"" \
    --set-json="keeper.podPolicy.resources.requests.memory=\"$KEEPER_MEMORY\""
```

**Key differences from the standard AWS deployment:**

| Setting | Standard | FIPS |
| - | - | - |
| Node architecture | ARM64 recommended on AWS | x86\_64 required |
| Node AMI | Standard EKS-optimized AMI | `BOTTLEROCKET_x86_64_FIPS` or a custom FIPS-enabled x86\_64 AMI |
| ECR endpoint | `0000000000.dkr.ecr.us-west-2.amazonaws.com` | `0000000000.dkr-ecr-fips.us-west-2.on.aws` |
| Server image tag | `26.4.1.2596` | `26.4.1.2596-fips` |
| Keeper image tag | `26.2.1.258` | `26.2.1.258-fips` |
| Operator image tag | `main-1.22166.1` | `main-1.22166.1-fips` |
| S3 endpoint | (default) | Explicit FIPS endpoint URL |
| `server.openSSL.enabled` | not set | `true` |
| `server.openSSL.required` | not set | `true` |
| `keeper.openSSL.enabled` | not set | `true` |
| `keeper.openSSL.required` | not set | `true` |
| Architecture flags | `server.arm64=true`, `keeper.arm64=true` | `server.arm64=false`, `keeper.arm64=false` |
| Cache configuration | `bytesPerGiRAM` | `cacheDiskSize` (explicit size); `isOnEmptyDir=true` for Bottlerocket or `false` for AL2023 |

***

## Step 6: Verify Installation

### Port-forward on TLS Port

With FIPS mode, ClickHouse uses TLS on port **9440** instead of the standard port 9000.

```bash theme={null}
kubectl port-forward svc/c-default-xx-01-server-any 9440:9440 -n ns-default-xx-01
```

### Create Client Configuration

Create a `client-config.xml` file to tell the ClickHouse client where to find the CA certificate. Update the path to your `ca.crt` file:

```xml theme={null}
<config>
    <openSSL>
        <client>
            <caConfig>/path/to/certs/ca.crt</caConfig>
        </client>
    </openSSL>
</config>
```

### Connect and Run a Query

```bash theme={null}
clickhouse client --host localhost --port 9440 --user default --ask-password --secure --config=client-config.xml
```

Run a test query:

```sql theme={null}
clickhouse-cloud :) select 1;
```

Expected output:

```
SELECT 1
Query id: 825591bf-a8e5-4995-ac9f-afb864854ba2
   ┌─1─┐
1. │ 1  │
   └───┘
1 row in set. Elapsed: 0.001 sec.
```

**Important:** Port 9000 (plaintext native) and port 8123 (plaintext HTTP) will refuse connections when `openSSL.required=true`. Use port 9440 (native TLS) or port 8443 (HTTPS) instead.

***

## Next Steps

* **Teleport integration** (optional): For secure database access via Teleport Enterprise, see [Integrate Teleport with ClickHouse Private](/cloud/clickhouse-private/how-to/integrate-teleport). This requires `SERVER_VERIFICATION_MODE=relaxed` and `KEEPER_VERIFICATION_MODE=relaxed`.