Summary
The main question is how does one secure the setting of of ACLs on Kafka. ACLs can be used to restrict who can consume / produce to topics, but how is the setting of ACLs restricted? E.g. some user on another network machine using kafka-acls.sh
Details
I am quite new to kafka and I've just setup up my first kafka 1.0.0 cluster and I am using the Kafka admin CLI(kafka-acls.sh) to grant acls for principals.
Here is problem I found: I can use this kafka-acls.sh on any other machine to manipulate my kafka cluster, without any permission required?! Is this an existing security issue?
My requirement is, as an admin, for my kafka topics I would grant read permission to the consumers. But if the consumer owners can use the kafka-acls.sh, they could add that permission by themselves.
I've tried these:
kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --cluster --operation Create --deny-principal User:*
kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --cluster --operation Alter--deny-principal User:*
Current ACLs for resource Cluster:kafka-cluster:
User:* has Deny permission for operations: Create from hosts: *
User:* has Deny permission for operations: Alter from hosts: *
I was hoping this can stop anyone to change ACLs on any topics; but I still can grant permission to any principal. I am expecting some settings in kafka properties file that can do the work. My properties settings regarding ACL are:
# Switch to enable topic deletion or not, default value is false
#delete.topic.enable=true
###To enable ZooKeeper authentication on brokers
zookeeper.set.acl=true
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:Admin
Any Ideas to restrict the admin CLI, guys?
Any advice would be appreciated.
ACLs are stored in Zookeeper so you need to run Zookeeper in secure mode with authenticated access (requires Apache Kafka 0.9 or higher).
https://cwiki.apache.org/confluence/display/KAFKA/KIP-38%3A+ZooKeeper+Authentication
This will allow you to restrict the ACL admin tools from working anywhere and by anyone since they include a zookeeper client which will have to be configured with valid admin credentials in order to connect and change Kafka ACLs stored in Zookeeper.
There is an example secure Kafka broker and Zookeeper setup explained in this blog post
https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/
In more recent versions of Apache Kafka there is also an API called AdminClient which allows apps to be written without direct Zookeeper dependency or connections. In 1.0.0 the AdminClient includes methods to create ACLs.
https://kafka.apache.org/10/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html
However at the moment the CLI commands in /bin have not been rewritten to use this new API which is why they still connect directly to Zookeeper.
Related
I am trying to access AWS elasticache(REDIS). I followed this instruction:
https://redsmin.uservoice.com/knowledgebase/articles/734646-amazon-elasticache-and-redsmin
Redis is connected now but when I click on configuration. I got this error:
"Redsmin can't load the configuration. Check with your provider that you have access to the configuration command."
edit 1:
config Redis command is sadly not available on AWS Elasticache, see their documentation:
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/RestrictedCommands.html
To deliver a managed service experience, ElastiCache restricts access to certain cache engine-specific commands that require advanced privileges. For cache clusters running Redis, the following commands are unavailable:
[...]
config
That's why Redsmin configuration module (it's the only module impacted) cannot display current your Redis AWS Elasticache configuration.
After enabling authentication, able to login to SQLLine with passing the superuser i.e. ignite credentials. After logging I can see schemas like IGNITE, PUBLIC etc. Now, how I connect to any of the schema existing in the Ignite Cluster? Can I grant particular user to access the particular schema and does ignite super user has full access to any schemas?
Ignite does not have ACLs right now. Successful authentication means you are allowed to see and change everything.
GridGain, which is built on top of Ignite, has some advanced security, though.
I use typical stack YARN/Ranger with atomic policies for accessing YARN queues. Having Hadoop user access, how to get list of queues that user has access to? I can see how it's usually done from admin side, but what about user? I went through yarn APIs, but found nothing. Ranger - user usually doesn't have enough permissions to get more details about itself. Is the only way to do it is to bruteforce all queues in cluster until u find accessible one?
Unfortunately, the user queue policy is not visible through the REST APIs for Fair Scheduler. You can double-check by running:
curl RM-ADDR:PORT/ws/v1/cluster/scheduler
but looking at ResourceManager REST API’s:Cluster Scheduler API I think you're out of luck.
If you use Ambari or Cloudera Manager, those might have APIs that will allow you to download the Fair Scheduler's XML file.
I'm setting up a new RabbitMQ service in iAPC (Swisscom app cloud) and I need to control the user access of the different producer/consumer application.
My access control requirement:
Application A can only write to queue X.
Application B can only read from queue X.
RabbitMQ provides usually user management functionalities. However, the whole user management in the admin section, RabbitMQ management GUI, is not available.
What solution does exist in iAPC to manage read/write permissions for different applications which have an app binding?
Is it even possible to setup different users?
I believe there is no way to add additional users in these managed RabbitMQ service deployments provided by Swisscom. This is quite similar across all of the available shared services (e.g. ElasticSearch or MariaDB) which come with a preset of defined users. I assume that this is true because those are actually shared services (as opposed to dedicated ones), where there may be authentication / security concerns if you are allowed to administer existing users.
For anyone who is interested the way to access your RabbitMQ CloudFoundry service admin interface via the provided environment parameters to see what is possible:
bind your RabbitMQ service to a running app instance (e.g. MY-APP)
look at the environment of that app with cf env MY-APP
tunnel the RabbitMQ management port to your localhost:
cf ssh -N -T -L 15000:rabbitmq.service.consul:15672 MY-APP
open a webbrowser and look at http://localhost:15000
Use the Username and Password you found in step (2) under rabbitmqent > credentials > management to log in
I'm currently using connect-redis in my Sails.js project to leverage a locally-installed redis instance. In the future, I'd like to use a common redis instance for multiple server instances (behind a load balancer), so I've been looking at AWS Elasticache. I'm having trouble with the configuration, though.
sails-project\config\session.js:
adapter: 'connect-redis',
host: 'primary-endpoint.xxxxxx.ng.0001.apse1.cache.amazonaws.com',
port: 6379,
ttl: <redis session TTL in seconds>,
db: 0,
pass: <redis auth password>,
prefix: 'sess:',
What should the TTL value be? Should the pass attribute point to IAM somehow?
I tried creating a user in IAM with AmazonElastiCacheFullAccess permissions and putting its access key ID in the pass attribute, but I got this error in my server console (testing on my Windows box):
C:\repos\sails-project\node_modules\connect-redis\lib\connect-redis.js:83
throw err;
^
AbortError: Redis connection lost and command aborted. It might have been processed.
at RedisClient.flush_and_error (C:\repos\sails-project\node_modules\redis\index.js:362:23)
...
Any ideas on what to change?
I'm going to assume your "windows box" is outside of AWS.
For Elasticache you can't access it from outside AWS. See the Security Section here : https://aws.amazon.com/elasticache/faqs/#Can_I_access_Amazon_ElastiCache_from_outside_AWS
The most common use case is to have EC2 instances within a VPC access and consume the Elasticache service. Along with this the Elasticache Redis service doesn't employ authentication and only allows lock down via security groups.
If you need something that differentiates from this configuration then you should look at putting Redis on EC2 so that you have full control.