I am new to cloudera. I am using cloudera manager console 5.4.
Under dynamic resource pool, I find two tabs status and configuration.
In status, I see various queues under resource pool usage. However I can't edit info over there.
And in configuration, I find no queues configured, only default queue.
Can you please tell me whether queues are configured or not?
In Cloudera Manager, go to the "Web UI" tab and click on "Resource Manager Web UI ...". Then on the RM web UI, click on the left side bar link for "Scheduler".
If your queues are created, you should see them there.
Related
"Dynamic resource pool configuration" of yarn queue is present in Cloudera Enterprise.
Does "Dynamic resource pool configuration" option is also present in HDP 3.1.0? Is there any way to enable it.
Thanks in advance.
This looks like it's just marketting and "Dynamic Resource Pools" are actually just yarn que's by a different name. (And they're is tooling in Ambari to manage them.)
I use typical stack YARN/Ranger with atomic policies for accessing YARN queues. Having Hadoop user access, how to get list of queues that user has access to? I can see how it's usually done from admin side, but what about user? I went through yarn APIs, but found nothing. Ranger - user usually doesn't have enough permissions to get more details about itself. Is the only way to do it is to bruteforce all queues in cluster until u find accessible one?
Unfortunately, the user queue policy is not visible through the REST APIs for Fair Scheduler. You can double-check by running:
curl RM-ADDR:PORT/ws/v1/cluster/scheduler
but looking at ResourceManager REST API’s:Cluster Scheduler API I think you're out of luck.
If you use Ambari or Cloudera Manager, those might have APIs that will allow you to download the Fair Scheduler's XML file.
I'm setting up a new RabbitMQ service in iAPC (Swisscom app cloud) and I need to control the user access of the different producer/consumer application.
My access control requirement:
Application A can only write to queue X.
Application B can only read from queue X.
RabbitMQ provides usually user management functionalities. However, the whole user management in the admin section, RabbitMQ management GUI, is not available.
What solution does exist in iAPC to manage read/write permissions for different applications which have an app binding?
Is it even possible to setup different users?
I believe there is no way to add additional users in these managed RabbitMQ service deployments provided by Swisscom. This is quite similar across all of the available shared services (e.g. ElasticSearch or MariaDB) which come with a preset of defined users. I assume that this is true because those are actually shared services (as opposed to dedicated ones), where there may be authentication / security concerns if you are allowed to administer existing users.
For anyone who is interested the way to access your RabbitMQ CloudFoundry service admin interface via the provided environment parameters to see what is possible:
bind your RabbitMQ service to a running app instance (e.g. MY-APP)
look at the environment of that app with cf env MY-APP
tunnel the RabbitMQ management port to your localhost:
cf ssh -N -T -L 15000:rabbitmq.service.consul:15672 MY-APP
open a webbrowser and look at http://localhost:15000
Use the Username and Password you found in step (2) under rabbitmqent > credentials > management to log in
I ask early about notification on client here.
Now I am interesting in notification for server side. In particular I am interesting in fact that notification inform all servers.
My problem is cluster of servers. I have some database elements cashed on all servers. If some user on any server update database element cash need to be refresh. Notification could do the job.
Or is there another way to deal with cluster of servers ?
Marko
There is no complete tutorial on this topic, I'm afraid.
Scout however does have the functionality you are looking for in the form of the IClusterSynchronizationService.
You can use it to register listeners and send messages between Eclipse Scout servers.
For it to work, you'll need a message passing system (message queue) such as ApacheMQ or RabbitMQ. You simply have to install the necessary connector from the Eclipse Marketplace and register them for integration in your application. A detailed explanation is in this tutorial. (You need to add the new connector as dependencies to your product files, register the cluster synchronization service, and configure it with properties for host, port, ...).
The "BahBah" Demo chat application on GitHub has an implementation of these listeners and how to register them.
The (inofficial) fork of the BahBah Chat demo has some of these changes already built in.
how to use "activemq-admin" to view the list of queues; number of messages in the queue;
I read through the tutorial : http://activemq.apache.org/activemq-command-line-tools-reference.html
didn't find a working solution...
and my web console on the slave machine does not work... the web console seems always go with the master machine (in the master/slave structure)
I just want to test that if I send messages into queues on master, slave could update.
so I am trying to use activemq-admin.
The way it works is that Slave is waiting to get a lock to the DB (Kahadb by default) , you will not be able to check the slave , bring down the master and now the slave will become the master broker and you should be able to see all the queues and messages dropped in them (assuming you are using persistence)
You can use JMX, web console or programatically, as you can find here. The easiest solution, I think, is to use web console like here.
I can't understand why isn't accessible the web console. Check for ActiveMQ config xml.
Also you can connect via JMX like:
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
Go here for more information.