Kafka s3 connector to minio in parquet format - amazon-s3

Question

I use docker compose to start 3 services in their respect containers:
zookeeper, kafka broker, and minio-connector
The three services can be started and connected successfully when I use the following configurations in minio-connector to consume from kafka and dump record in JSON format to minio:
start up command:
root#e1d1294c6fe6:/opt/bitnami/kafka/bin# ./connect-standalone.sh /plugins/connector.properties /plugins/s3-sink.properties
connector.properties file:
bootstrap.servers=kafka:9092
plugin.path=/plugins
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect.offsets
s3-sink.properties file:
name=s3-sink
connector.class=io.confluent.connect.s3.S3SinkConnector
tasks.max=1
topics=202208.minio.connector.test
s3.region=us-east-1
s3.bucket.name=minioUsr
s3.part.size=5242880
flush.size=1
store.url=https://minio.kube.url
storage.class=io.confluent.connect.s3.storage.S3Storage
format.class=io.confluent.connect.s3.format.json.JsonFormat
schema.compatibility=NONE
schema.generator.class=io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
Now I'd like the connector to consume records and dump to minio in parquet format.
kafka and zookeeper services remain the same. I modified the connector.properties and s3-sink.properties but the connector cannot start.
new connector.properties file:
bootstrap.servers=kafka:9092
plugin.path=/plugins
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect.offsets
key.converter=io.confluent.connect.avro.AvroConverter
value.converter=io.confluent.connect.avro.AvroConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=true
key.converter.schema.registry.url=https://registry...:1443
value.converter.schema.registry.url=https://registry...:1443
new s3-sink.properties:
name=s3-sink
connector.class=io.confluent.connect.s3.S3SinkConnector
tasks.max=1
topics=202208.minio.connector.test
s3.region=us-east-1
s3.bucket.name=minioUsr
s3.part.size=5242880
flush.size=1
store.url=https://minio.kube.url
storage.class=io.confluent.connect.s3.storage.S3Storage
schema.generator.class=io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
schema.compatibility=NONE
format.class=io.confluent.connect.s3.format.parquet.ParquetFormat
enhanced.avro.schema.support=true
My questions are:
With the above configuration, the connector fails to start with an exception
[2022-08-29 15:21:10,610] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectStandalone:130)
org.apache.kafka.common.config.ConfigException: Invalid value io.confluent.connect.avro.AvroConverter for configuration value.converter: Class
io.confluent.connect.avro.AvroConverter could not be found.
at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:728)
at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:474)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:467)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:129)
at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:385)
at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.<init>(StandaloneConfig.java:42)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:81)
I installed the connector by downloading the confluentinc-kafka-connect-s3-10.1.0 zip file and manually unzip and copy all jars to /plugins/lib. I found the following jars related to parquet:
root#e1d1294c6fe6:/opt/bitnami/kafka/bin# ls -1 /plugins/lib/*parquet*
/plugins/lib/parquet-avro-1.11.1.jar
/plugins/lib/parquet-column-1.11.1.jar
/plugins/lib/parquet-common-1.11.1.jar
/plugins/lib/parquet-encoding-1.11.1.jar
/plugins/lib/parquet-format-structures-1.11.1.jar
/plugins/lib/parquet-hadoop-1.11.1.jar
What is missing in installation?
Any further change is required in the configuration?

Answer 1

I have to manually install avro converter using confluent-hub
download confluent-hub client:
https://docs.confluent.io/5.5.1/connect/managing/confluent-hub/client.html
using confluent-hub to install avro converter, following answers in:
Kafka Connect Confluent S3 Sink Connector: Class io.confluent.connect.avro.AvroConverter could not be found