Flink 1.9 SQL Client throws ClassNotFoundException: org.apache.kafka.clients.consumer.ConsumerRecord - sql

Question

Trying to use Flink 1.9 SQL-Client with Kafka without success.
After figuring out about the required jar files and copying them into the lib directory, I am getting the following run-time exception when doing SELECT * FROM table-name:
Flink SQL> select * from default_catalog.default_database.member_customer_newsletters ;
[ERROR] Could not execute SQL statement. Reason:
java.lang.ClassNotFoundException: org.apache.kafka.clients.consumer.ConsumerRecord
Doing jar -tf of the jar-file, I can see the class ConsumerRecord being there:
jar -tf ./lib/flink-sql-connector-kafka-0.11_2.12-1.9.0.jar|grep 'ConsumerRecord'
org/apache/flink/kafka011/shaded/org/apache/kafka/clients/consumer/ConsumerRecord.class
So, I am not sure why it is trhowing a ClassNotFoundException as the class is already in the jar-file?
I just need to add the run-time is looking for "org.apache.kafka.clients.consumer.ConsumerRecord" but because this jar-file is shaded the full qualified name for the class is "org.apache.flink.kafka011.shaded.org.apache.kafka.clients.consumer.ConsumerRecord.class"
But, this should be true with any other class in this shaded jar as well!

Answer 1

There are two different classes:
org.apache.kafka.clients.consumer.ConsumerRecord
org.apache.flink.kafka011.shaded.org.apache.kafka.clients.consumer.ConsumerRecord
In flink-sql-connector-kafka-0.11_2.12-1.9.0.jar, you found the class
org.apache.flink.kafka011.shaded.org.apache.kafka.clients.consumer.ConsumerRecord
while Flink is complaining about:
org.apache.kafka.clients.consumer.ConsumerRecord
The first is a class used internally by Flink, after a kind of copy-paste from Kafka.
The second one is a class in kafka-clients-0.11.0.2.jar.
So Flink is right to complain about a missing library.