clickhouse-copier

The description of the utility and its parameters, as well as examples of the config files that you need to create for the copier are in the official doc ClickHouse copier utility

The steps to run a task:

  1. Create a config file for clickhouse-copier (zookeeper.xml)

    ZooKeeper config format

  2. Create a config file for the task (task1.xml)

    Copy task configuration

  3. Create the task in ZooKeeper and start an instance of clickhouse-copier

    clickhouse-copier --daemon --base-dir=/opt/clickhouse-copier --config=/opt/clickhouse-copier/zookeeper.xml --task-path=/clickhouse/copier/task1 --task-file=/opt/clickhouse-copier/task1.xml

    If the node in ZooKeeper already exists and you want to change it, you need to add the task-upload-force parameter:

    clickhouse-copier --daemon --base-dir=/opt/clickhouse-copier --config=/opt/clickhouse-copier/zookeeper.xml --task-path=/clickhouse/copier/task1 --task-file=/opt/clickhouse-copier/task1.xml --task-upload-force=1

    If you want to run another instance of clickhouse-copier for the same task, you need to copy the config file (zookeeper.xml) to another server, and run this command:

    clickhouse-copier --daemon --base-dir=/opt/clickhouse-copier --config=/opt/clickhouse-copier/zookeeper.xml --task-path=/clickhouse/copier/task1

The number of simultaneously running instances is controlled be the max_workers parameter in your task configuration file. If you run more workers superfluous workers will sleep and log messages like this:

<Debug> ClusterCopier: Too many workers (1, maximum 1). Postpone processing

See also

clickhouse-copier 20.3 and earlier

clickhouse-copier 20.4 - 21.6

Kubernetes job for clickhouse-copier

Last modified 2022.11.09: Update _index.md (02938ab)