Using S3 Storage with ClickHouse

This guide walks through configuring CodeXray's ClickHouse to use S3-compatible object storage for telemetry data. Works with AWS S3, MinIO, Ceph, GCS (via S3 API), and any other S3-compatible backend.

Overview

By default, ClickHouse stores everything on local disk. Moving to S3 lets you:

Reduce storage costs — hot data stays on local SSD; cold data moves to cheap object storage
Scale storage independently from compute — no more resizing PVCs when telemetry volume grows
Retain longer — hold months of hot-queryable data without hitting local disk limits

CodeXray supports two S3 modes, both native ClickHouse features:

Mode	Description	Best for
Tiered	Recent data on local SSD, older data auto-moved to S3	Best query performance on recent data; cost-effective long-term retention
S3-only	All data on S3, local disk used only for caching	Minimal local storage; maximum cost optimisation

Prerequisites

CodeXray deployed — either via Docker Compose or via the Kubernetes manifest
An S3-compatible bucket — AWS S3, MinIO, Ceph, etc.
S3 credentials — access key + secret key, or IAM role / IRSA / workload identity if you prefer

One bucket per ClickHouse deployment

Use a dedicated bucket for ClickHouse data. Do not share it with other applications. ClickHouse manages its own file lifecycle, and any external bucket-level lifecycle policy (e.g., S3 Lifecycle rules that delete objects after N days) can corrupt your telemetry data.

Step 1 — Create the S3 Bucket

AWS CLI example:

aws s3 mb s3://my-codexray-clickhouse --region us-east-1

For MinIO, Ceph, or other compatible storage, create the bucket using your provider's UI or client.

Step 2 — Write the ClickHouse Storage Configuration

ClickHouse reads XML configuration files from /etc/clickhouse-server/config.d/. Create a new file — storage.xml — with the S3 disk definitions and storage policies.

Tiered mode (recommended)

Recent data stays on the default local disk. Once free space drops below the move_factor threshold, the oldest data auto-moves to S3.

<clickhouse>
  <storage_configuration>
    <disks>
      <!-- The S3 object-storage disk -->
      <s3_disk>
        <type>s3</type>
        <endpoint>https://s3.us-east-1.amazonaws.com/my-codexray-clickhouse/</endpoint>
        <region>us-east-1</region>
        <access_key_id from_env="AWS_ACCESS_KEY_ID"/>
        <secret_access_key from_env="AWS_SECRET_ACCESS_KEY"/>
        <metadata_path>/var/lib/clickhouse/disks/s3_disk/</metadata_path>
        <!-- Safety: never use zero-copy replication; each replica writes its own S3 objects -->
        <send_metadata>true</send_metadata>
      </s3_disk>

      <!-- Local cache layer in front of S3 -->
      <s3_cache>
        <type>cache</type>
        <disk>s3_disk</disk>
        <path>/var/lib/clickhouse/disks/s3_cache/</path>
        <max_size>10Gi</max_size>
        <cache_on_write_operations>1</cache_on_write_operations>
      </s3_cache>
    </disks>

    <policies>
      <s3_tiered>
        <volumes>
          <hot>
            <disk>default</disk>
          </hot>
          <cold>
            <disk>s3_cache</disk>
          </cold>
        </volumes>
        <!-- Move oldest data to S3 when free space drops below 10% of the hot volume -->
        <move_factor>0.1</move_factor>
      </s3_tiered>
    </policies>
  </storage_configuration>

  <!-- Make s3_tiered the default policy for new tables -->
  <merge_tree>
    <storage_policy>s3_tiered</storage_policy>
  </merge_tree>
</clickhouse>

S3-only mode

All data goes straight to S3. Local disk is used only for the read/write cache. Use this when you want the absolute minimum local footprint.

<clickhouse>
  <storage_configuration>
    <disks>
      <s3_disk>
        <type>s3</type>
        <endpoint>https://s3.us-east-1.amazonaws.com/my-codexray-clickhouse/</endpoint>
        <region>us-east-1</region>
        <access_key_id from_env="AWS_ACCESS_KEY_ID"/>
        <secret_access_key from_env="AWS_SECRET_ACCESS_KEY"/>
        <metadata_path>/var/lib/clickhouse/disks/s3_disk/</metadata_path>
      </s3_disk>

      <s3_cache>
        <type>cache</type>
        <disk>s3_disk</disk>
        <path>/var/lib/clickhouse/disks/s3_cache/</path>
        <!-- Larger cache recommended for s3-only mode -->
        <max_size>20Gi</max_size>
        <cache_on_write_operations>1</cache_on_write_operations>
      </s3_cache>
    </disks>

    <policies>
      <s3_only>
        <volumes>
          <main>
            <disk>s3_cache</disk>
          </main>
        </volumes>
      </s3_only>
    </policies>
  </storage_configuration>

  <merge_tree>
    <storage_policy>s3_only</storage_policy>
  </merge_tree>
</clickhouse>

Using IAM role / IRSA / workload identity

If your environment provides credentials automatically (AWS IRSA, EC2 instance profile, GCP workload identity), replace the <access_key_id> and <secret_access_key> lines with:

<use_environment_credentials>true</use_environment_credentials>

No secret is required on the Kubernetes side — ClickHouse picks up credentials from the pod's environment or instance metadata automatically.

Step 3a — Apply on Docker Compose

Drop the config file into the ClickHouse container

Save the XML from Step 2 as storage.xml next to your docker-compose.yaml. Then add a volume mount and credentials to the ClickHouse service:

services:
  codexray-clickhouse:
    image: clickhouse/clickhouse-server:24.3
    environment:
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
    volumes:
      - clickhouse_data:/var/lib/clickhouse
      - ./storage.xml:/etc/clickhouse-server/config.d/storage.xml:ro

Set the credentials in the environment (or in a .env file Docker Compose reads automatically):

export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"

Then recreate the ClickHouse container:

docker compose up -d codexray-clickhouse

Step 3b — Apply on Kubernetes

1. Create the credentials Secret

kubectl create secret generic clickhouse-s3-creds \
  --from-literal=AWS_ACCESS_KEY_ID=your-access-key \
  --from-literal=AWS_SECRET_ACCESS_KEY=your-secret-key \
  -n codexray

If you're using IRSA or workload identity, skip this step and omit the envFrom block below — ClickHouse will resolve credentials from instance metadata.

2. Create the storage ConfigMap

Save your storage.xml from Step 2, then:

kubectl create configmap clickhouse-storage-config \
  --from-file=storage.xml \
  -n codexray

3. Patch the ClickHouse Deployment

Edit the existing ClickHouse Deployment to mount the ConfigMap and load the credentials:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: clickhouse
  namespace: codexray
spec:
  template:
    spec:
      containers:
        - name: clickhouse
          image: clickhouse/clickhouse-server:24.3
          envFrom:
            - secretRef:
                name: clickhouse-s3-creds
          volumeMounts:
            - name: clickhouse-pv
              mountPath: /var/lib/clickhouse
              subPath: data
            - name: clickhouse-pv
              mountPath: /var/log/clickhouse-server
              subPath: logs
            - name: storage-config
              mountPath: /etc/clickhouse-server/config.d/storage.xml
              subPath: storage.xml
      volumes:
        - name: clickhouse-pv
          persistentVolumeClaim:
            claimName: clickhouse-pv
        - name: storage-config
          configMap:
            name: clickhouse-storage-config

Apply it:

kubectl apply -f deploy/k8s/codexray.yaml

The ClickHouse pod will restart with the new storage configuration.

Step 4 — Verify

Connect to the running ClickHouse instance:

# Docker Compose
docker compose exec codexray-clickhouse clickhouse-client

# Kubernetes
kubectl -n codexray exec -it deploy/clickhouse -- clickhouse-client

Verify the S3 disks are configured

SELECT name, path, type, free_space, total_space FROM system.disks;

You should see ObjectStorage type disks for s3_disk and s3_cache alongside the built-in default disk.

Verify the storage policy

SELECT * FROM system.storage_policies;

You should see either s3_tiered or s3_only depending on the mode you chose.

Check data distribution

SELECT disk_name, sum(bytes_on_disk) AS bytes
FROM system.parts
WHERE active = 1
GROUP BY disk_name
ORDER BY bytes DESC;

In tiered mode, you'll see data on both default (hot) and s3_cache (cold). In S3-only mode, all data lives on s3_cache.

Watch the migration happen (tiered mode)

Give ClickHouse time to generate telemetry — then run the distribution query above periodically. As local free space drops below your move_factor threshold (10% by default), you'll see bytes migrating from default to s3_cache.

How It Works

The cache layer

Every S3 read and write goes through a local cache (s3_cache disk). Frequently accessed files are served from cache without a round-trip to S3; the cache_on_write_operations flag ensures newly written data is also pre-populated, so recent telemetry is immediately available from cache.

Cache size tuning:

Tiered mode — 10 GB is a reasonable starting point (recent data lives on default anyway)
S3-only mode — bump to 20+ GB since every read touches this cache

Move factor (tiered mode only)

<move_factor>0.1</move_factor> means: "when less than 10% of free space remains on the hot volume, start moving the oldest data to cold (S3)." Lower values keep more data hot; higher values keep more data cold. The default of 0.1 is conservative and works well for most workloads.

Data isolation (multi-replica deployments)

If you scale ClickHouse beyond a single replica, each replica must write to its own S3 path prefix. ClickHouse has a "zero-copy replication" feature that lets replicas share S3 objects, but it's been disabled by default since version 22.8 due to known data-corruption issues — don't re-enable it. Instead, use the default per-replica path behaviour, which is safe.

Using MinIO or Other S3-Compatible Storage

Any S3-compatible endpoint works. Swap the <endpoint> URL and point your credentials at the right provider:

<s3_disk>
  <type>s3</type>
  <endpoint>https://minio.example.com/codexray-clickhouse/</endpoint>
  <region>us-east-1</region>
  <access_key_id from_env="AWS_ACCESS_KEY_ID"/>
  <secret_access_key from_env="AWS_SECRET_ACCESS_KEY"/>
  <metadata_path>/var/lib/clickhouse/disks/s3_disk/</metadata_path>
</s3_disk>

The <region> field is required by the S3 protocol but isn't meaningful for many self-hosted backends — set it to us-east-1 as a safe default.

Troubleshooting

ClickHouse pod keeps restarting after config change

Most often a syntax error in storage.xml. Check logs:

kubectl -n codexray logs deploy/clickhouse --tail=100
# or
docker compose logs codexray-clickhouse --tail=100

Look for <Error> lines near startup — ClickHouse prints the exact XML validation error.

"ObjectStorage disks" aren't showing up

The config file didn't load. Confirm the mount path is correct (/etc/clickhouse-server/config.d/storage.xml) and that the file is actually present inside the container:

kubectl -n codexray exec deploy/clickhouse -- cat /etc/clickhouse-server/config.d/storage.xml

Existing tables still write to local disk

Setting <storage_policy> in merge_tree applies to newly created tables. Existing tables keep their original policy. To migrate existing CodeXray tables:

ALTER TABLE your_database.your_table MODIFY SETTING storage_policy = 's3_tiered';

Ask the CodeXray team before doing this in production — the exact table names depend on your deployment version.

S3 credentials errors

NoSuchBucket — the <endpoint> URL has a typo or the bucket doesn't exist
SignatureDoesNotMatch — wrong secret key, or the <region> doesn't match the bucket's region
AccessDenied — the IAM user / role doesn't have s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket on the bucket

Next Steps

Architecture — where ClickHouse sits in the CodeXray platform
Requirements — sizing guidance for different tiers
Need help designing for very large ingest volumes or regulated retention? Email hello@codexray.io.

Overview​

Prerequisites​

Step 1 — Create the S3 Bucket​

Step 2 — Write the ClickHouse Storage Configuration​

Tiered mode (recommended)​

S3-only mode​

Step 3a — Apply on Docker Compose​

Drop the config file into the ClickHouse container​

Step 3b — Apply on Kubernetes​

1. Create the credentials Secret​

2. Create the storage ConfigMap​

3. Patch the ClickHouse Deployment​

Step 4 — Verify​

Verify the S3 disks are configured​

Verify the storage policy​

Check data distribution​

Watch the migration happen (tiered mode)​

How It Works​

The cache layer​

Move factor (tiered mode only)​

Data isolation (multi-replica deployments)​

Using MinIO or Other S3-Compatible Storage​

Troubleshooting​

ClickHouse pod keeps restarting after config change​

"ObjectStorage disks" aren't showing up​

Existing tables still write to local disk​

S3 credentials errors​

Next Steps​