How to move older clickhouse partitions to S3 disk [closed] - amazon-s3

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Improve this question
I'm currently starting to work with clickhouse for our in-house analytics system, but it looks like there are no automated ways to configure policies for data retention. The only thing I saw was the ALTER ... MOVE PARTITION (https://clickhouse.tech/docs/en/sql-reference/statements/alter/partition/#alter_move-partition), but it looks like the process has to be manual / implemented in our application layer.
My objective is to move data older than 3 months directly to an S3 cluster for archival and price reasons, while still being able to query it.
Is there any native way to do so directly in clickhouse with storage policies?
Thanks in advance.

Answer 1

This answer was based out of #Denny Crane's comment: https://altinity.com/blog/clickhouse-and-s3-compatible-object-storage, where I did put comments where there were not enough explanations, and keeping it in the event that the link dies.
Add your S3 disk to a new configuration file (Let's say /etc/clickhouse-server/config.d/storage.xml:
<yandex>
<storage_configuration>
<disks>
<!-- This tag is the name of your S3-emulated disk, used for the rest of this tutorial -->
<your_s3>
<type>s3</type>
<!-- Set this to the endpoint of your S3-compatible provider -->
<endpoint>https://nyc3.digitaloceanspaces.com</endpoint>
<!-- Set this to your access key ID provided by your provider -->
<access_key_id>*****</access_key_id>
<!-- Set this to your access key Secret provided by your provider -->
<secret_access_key>*****</secret_access_key>
</your_s3>
</disks>
<!-- Don't leave this file yet! We still have things to do there -->
...
</storage_configuration>
</yandex>
Add a storage policy for your data storage:
<!-- Put this after the three dots in the snippet above -->
<policies>
<shared>
<volumes>
<default>
<!-- Default is the disk that is present in the default question -->
<disk>default</disk>
</default>
<your_s3>
<disk>your_s3</disk>
</your_s3>
</volumes>
</shared>
</policies>
Once that is done, you can create your tables with the following insert statement:
CREATE TABLE visits (...)
ENGINE = MergeTree
TTL toStartOfYear(time) + interval 3 year to volume 'your_s3'
SETTINGS storage_policy = 'shared';
Where shared is your policy name, and your_s3 is the name of your disk in that policy.