I have couple questions:
in local docker runner for web console, I do not see the monitoring, is that only specific to online tool:
For online console, I downloaded the java code and tries to run the LoadCaches, but got a lot errors:
[09:13:51,089][SEVERE][mgmt-#47%ImportedCluster1%][GridTaskWorker] Failed to obtain remote job result policy for result from ComputeTask.result(..) method (will fail the whole task): GridJobResultImpl [job=C2 [c=LoadCacheJobV2 [keepBinary=false]], sib=GridJobSiblingImpl [sesId=943eb92f561-a8fe7c19-f1d7-4fae-80ad-6a22fe23707a, jobId=a43eb92f561-a8fe7c19-f1d7-4fae-80ad-6a22fe23707a, nodeId=00d08e30-944b-4ec5-a6d6-a429e142e5f4, isJobDone=false], jobCtx=GridJobContextImpl [jobId=a43eb92f561-a8fe7c19-f1d7-4fae-80ad-6a22fe23707a, timeoutObj=null, attrs={}], node=TcpDiscoveryNode [id=00d08e30-944b-4ec5-a6d6-a429e142e5f4, addrs=[0:0:0:0:0:0:0:1%lo0, 10.1.2.120, 127.0.0.1], sockAddrs=[/10.1.2.120:47500, /0:0:0:0:0:0:0:1%lo0:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1537373630293, loc=false, ver=2.5.2#20180622-sha1:cca47098, isClient=false], ex=class o.a.i.IgniteException: Failed to load cache: DepartmentsCache, hasRes=true, isCancelled=false, isOccupied=true]
class org.apache.ignite.IgniteException: Remote job threw user exception (override or implement ComputeTask.result(..) method if you would like to have automatic failover for this exception): Failed to load cache: DepartmentsCache
at org.apache.ignite.compute.ComputeTaskAdapter.result(ComputeTaskAdapter.java:102)
at org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1047)
at org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1040)
at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6742)
at org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1040)
at org.apache.ignite.internal.processors.tas
Is this because my local ignite is not in the same version as the web agent? Online portal gives web agent at 2.4.9, but my local ignite is at version 2.6, if that is the issue, where is apache ignite v2.4.9?
Thanks
Monitoring comes with GridGain Web Console. It is deployed on console.gridgain.com website. It uses GridGain v8.4.9, so you should use it with Ignite 2.4.
If you use an open source Apache Ignite Web Console, then enterprise features are not available there. But you can deploy it in your environment and use it with any version.
Related
I'm facing error while starting kafka server,
have setup the SSL and it's working fine for kafka 3 brokers. And zookeeper is also setup with SSL
Now tried to setup the SCRAM with SASL_SSL for kafka broker from server property file.
It's not working I have created a user with following command
kafka-configs.sh --zookeeper localhost:2182 --zk-tls-config-file zookeeper-client.properties --entity-type users --entity-name broker-admin --alter --add-config 'SCRAM-SHA-512=[password=DEM123]'
and I can see user is created.
but while trying to run the command to run kafka broker
kafka-server-start.sh -daemon server-0.properties
It is having some error while I have checked server.log file
[2021-10-05 16:21:38,369] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/users/broker-admin
Can anyone support me?
let me share my zookeeper.proerpties file
dataDir=/var/www/kafka/data/zookeeper
clientPort=2181
secureClientPort=2182
authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.trustStore.location=/var/www/kafka/ssl/kafka.zookeeper.truststore.jks
ssl.trustStore.password=zookeepbook
ssl.keyStore.location=/var/www/kafka/ssl/kafka.zookeeper.keystore.jks
ssl.keyStore.password=zookeepbook
ssl.clientAuth=need
maxClientCnxns=0
admin.enableServer=true
admin.serverPort=9090
server.1=localhost:2888:3888
server.properties file content :
broker.id=0
listeners=SASL_SSL://localhost:9092
advertised.listeners=SASL_SSL://localhost:9092
zookeeper.connect=localhost:2182
log.dirs=/var/www/kafka/data/broker-0
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.client.enable=true
zookeeper.ssl.protocol=TLSv1.2
zookeeper.ssl.truststore.location=/var/www/kafka/ssl/kafka.broker-0.truststore.jks
zookeeper.ssl.truststore.password=zookeepbookbrk0
zookeeper.ssl.keystore.location=/var/www/kafka/ssl/kafka.broker-0.keystore.jks
zookeeper.ssl.keystore.password=zookeepbookbrk0
zookeeper.set.acl=true
ssl.truststore.location=/var/www/kafka/ssl/kafka.broker-0.truststore.jks
ssl.truststore.password=zookeepbookbrk0
ssl.keystore.location=/var/www/kafka/ssl/kafka.broker-0.keystore.jks
ssl.keystore.password=zookeepbookbrk0
ssl.key.password=zookeepbookbrk0
security.inter.broker.protocol=SASL_SSL
ssl.client.auth=none
ssl.protocol=TLSv1.2
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username='broker-admin' password=DEM123;
super.users=User:broker-admin
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
Can you try to set 'skipACL=yes' to your zookeeper.properties ?
If you authenticated with Zookeeper by using SSL client certs when you created 'broker-admin' user, I think it because access from other than the place where you executed the command is denied.
I want to connect aws-Kafka with s3 using confluence connector on my ec2 server. I try to configure everything like in tutorials. When I run connect-standalone or connect-distributed, at first everything goes well, I don't get any errors in the logs but after information about connection starting, my connector died instantly without any information. Has anybody got same problem?
config/connect-standalone.properties
bootstrap.servers=msk-connection-string
plugin.path=/home/ubuntu/connectors/confluentinc-kafka-connect-s3
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
offset.storage.file.filename=/tmp/connect.offsets
connector.properties
connector.class=io.confluent.connect.s3.S3SinkConnector
format.class=io.confluent.connect.s3.format.bytearray.ByteArrayFormat
flush.size=1
topics=SomeTopic
s3.bucket.name=bucket-name-here
s3.region=us-west-2
s3.part.size=5242880
aws.access.key.id=****
aws.secret.access.key=****
behavior.on.null.values=ignore
storage.class=io.confluent.connect.s3.storage.S3Storage
topics.dir=../topics
store.url=http://bucket-name.s3-website-Region.amazonaws.com
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
logs:
[2021-08-20 06:32:35,954] INFO Kafka version: 2.7.0 (org.apache.kafka.common.utils.AppInfoParser:119)
[2021-08-20 06:32:35,954] INFO Kafka commitId: 448719dc99a19793 (org.apache.kafka.common.utils.AppInfoParser:120)
[2021-08-20 06:32:35,954] INFO Kafka startTimeMs: 1629441155953 (org.apache.kafka.common.utils.AppInfoParser:121)
Killed
Please help!
MSK requires TLS connection
When adding few lines with ssl configuration to config/connect-standalone.properties
producer.security.protocol=SSL
consumer.security.protocol=SSL
security.protocol=SSL
ssl.protocol=TLS
ssl.truststore.location=/your/path/to/truststore/kafka.client.truststore.jks
It starts working properly!
We use ActiveMQ 5.15.6, and I need your guidance to extract the ActiveMQ statistics via command line. At the moment we use the web console to get the ActiveMQ statistics which can be accessed via:
http://<IPAddress>:8161/admin/queues.jsp
And when I run ./activemq bstat it gives the below output
$./activemq bstat
INFO: Loading '/etc/default/activemq'
INFO: Using java '/bin/java'
Java Runtime: Oracle Corporation 1.8.0_252 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
Heap sizes: current=62976k free=62319k max=932352k
JVM args: -Xms64M -Xmx1G -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/apps/activemq/current/conf/login.config -Dactivemq.classpath=/apps/activemq/current/conf:/apps/activemq/apache-activemq-5.15.6//../lib/: -Dactivemq.home=/apps/activemq/current -Dactivemq.base=/apps/activemq/current -Dactivemq.conf=/apps/activemq/current/conf -Dactivemq.data=/apps/activemq/current/data
Extensions classpath:
[/apps/activemq/current/lib,/apps/activemq/current/lib/camel,/apps/activemq/current/lib/optional,/apps/activemq/current/lib/web,/apps/activemq/current/lib/extra]
ACTIVEMQ_HOME: /apps/activemq/current
ACTIVEMQ_BASE: /apps/activemq/current
ACTIVEMQ_CONF: /apps/activemq/current/conf
ACTIVEMQ_DATA: /apps/activemq/current/data
Connecting to JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
INFO: Broker not available at: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
Can you please advise what command or script do I need to run to get the stats via command line?
The output is telling you what is wrong namely that the command line client cannot connect to the JMX port where the broker should be exposing its JMX mbeans which the 'bstat' command uses to collect broker metrics. You either need to enable JMX on the broker or configure the bstat command to point to where you've configure the JMX port to be:
activemq bstat –jmxurl service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
To understand the broker JMX configuration please read the docs which are located here.
I am trying to get the node and cluster details in the Apache Ignite WebConsole. Below are the steps i have performed:
1. Download the Apache Ignite WebConsole.
2. My applications is running the ignite node as a cache layer(Ignite node started OK (id=ac87a66c,)
3. Ignite is running on Ignite discovery url 192.168.0.102:47500
4. I ran the bat file: web-console-agent.bat. But it is not able to connect to the agent and hence the web console:
[2020-05-26T18:05:33,245][INFO ][main][AgentLauncher] Starting Apache GridGain Web Console Agent...
[2020-05-26T18:05:33,415][INFO ][main][AgentLauncher]
[2020-05-26T18:05:33,416][INFO ][main][AgentLauncher] Web Console Agent configuration :
[2020-05-26T18:05:33,535][INFO ][main][AgentLauncher] User's security tokens : ********************************af05
[2020-05-26T18:05:33,539][INFO ][main][AgentLauncher] URI to Ignite node REST server : http://localhost:8080
[2020-05-26T18:05:33,540][INFO ][main][AgentLauncher] URI to GridGain Web Console : https://console.gridgain.com
[2020-05-26T18:05:33,548][INFO ][main][AgentLauncher] Path to properties file : default.properties
[2020-05-26T18:05:33,548][INFO ][main][AgentLauncher] Path to JDBC drivers folder : C:\pluralsight\gridgain-web-console-agent-2020.03.01\jdbc-drivers
[2020-05-26T18:05:33,557][INFO ][main][AgentLauncher] Demo mode : enabled
[2020-05-26T18:05:33,560][INFO ][main][AgentLauncher]
[2020-05-26T18:05:33,621][INFO ][main][WebSocketRouter] Starting Web Console Agent...
[2020-05-26T18:05:33,635][INFO ][Connect thread][WebSocketRouter] Connecting to server: wss://console.gridgain.com
[2020-05-26T18:05:35,996][INFO ][http-client-16][WebSocketRouter] Successfully completes handshake with server
[2020-05-26T18:05:40,035][WARN ][pool-2-thread-1][ClusterHandler] Failed to connect to cluster.
[2020-05-26T18:05:40,036][WARN ][pool-2-thread-1][ClusterHandler] Check that '--node-uri' configured correctly.
[2020-05-26T18:05:40,039][WARN ][pool-2-thread-1][ClusterHandler] Ensure that cluster nodes have [ignite-rest-http] module in classpath (was copied from libs/optional to libs folder).
[2020-05-26T18:05:40,045][INFO ][pool-2-thread-1][ClustersWatcher] Failed to establish connection to node
Please let me know where i am missing steps
I have a topology that runs well on a Local cluster.
But when I try to run it on a production cluster the following things happens:
The nimbus is up
The storm UI is up
The two workers I use are up
Zookeper is up
I run storm with
storm jar myjar.jar MyClass
Nimbus submits the topology
The topologies and the workers appears in the storm UI
BUT:
The topology does not start despite the fact that its status is ACTIVE
The log file of the topology does not appear in the workers.
I have the following log in the worker on the supervisor.log:
2016-04-15 13:18:19.831 o.a.s.d.supervisor [WARN] There was a connection problem with nimbus. #error {
:cause jobs-rec-storm-nimbus
:via
[{:type java.lang.RuntimeException
:message org.apache.storm.thrift.transport.TTransportException: java.net.UnknownHostException: jobs-rec-storm-nimbus
:at [org.apache.storm.security.auth.TBackoffConnect retryNext TBackoffConnect.java 64]}
{:type org.apache.storm.thrift.transport.TTransportException
:message java.net.UnknownHostException: jobs-rec-storm-nimbus
:at [org.apache.storm.thrift.transport.TSocket open TSocket.java 226]}
{:type java.net.UnknownHostException
:message jobs-rec-storm-nimbus
:at [java.net.AbstractPlainSocketImpl connect AbstractPlainSocketImpl.java 184]}]
:trace
[[java.net.AbstractPlainSocketImpl connect AbstractPlainSocketImpl.java 184]
[java.net.SocksSocketImpl connect SocksSocketImpl.java 392]
[java.net.Socket connect Socket.java 589]
[org.apache.storm.thrift.transport.TSocket open TSocket.java 221]
[org.apache.storm.thrift.transport.TFramedTransport open TFramedTransport.java 81]
[org.apache.storm.security.auth.SimpleTransportPlugin connect SimpleTransportPlugin.java 103]
[org.apache.storm.security.auth.TBackoffConnect doConnectWithRetry TBackoffConnect.java 53]
[org.apache.storm.security.auth.ThriftClient reconnect ThriftClient.java 99]
[org.apache.storm.security.auth.ThriftClient <init> ThriftClient.java 69]
[org.apache.storm.utils.NimbusClient <init> NimbusClient.java 106]
[org.apache.storm.utils.NimbusClient getConfiguredClientAs NimbusClient.java 78]
[org.apache.storm.utils.NimbusClient getConfiguredClient NimbusClient.java 41]
[org.apache.storm.blobstore.NimbusBlobStore prepare NimbusBlobStore.java 268]
[org.apache.storm.utils.Utils getClientBlobStoreForSupervisor Utils.java 462]
[org.apache.storm.daemon.supervisor$fn__9590 invoke supervisor.clj 942]
[clojure.lang.MultiFn invoke MultiFn.java 243]
[org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9351$fn__9369 invoke supervisor.clj 582]
[org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9351 invoke supervisor.clj 581]
[org.apache.storm.event$event_manager$fn__8903 invoke event.clj 40]
[clojure.lang.AFn run AFn.java 22]
[java.lang.Thread run Thread.java 745]]}
2016-04-15 13:18:19.831 o.a.s.d.supervisor [INFO] Finished downloading code for storm id jobs-KafkaMigration-topology-3-1460740616
2016-04-15 13:18:19.850 o.a.s.d.supervisor [INFO] Missing topology storm code, so can't launch worker with assignment ...(some more numbers)
So I asume that I have a connection problem with nimbus, but the properties file in the worker is:
storm.zookeeper.servers:
- "192.168.22.209"
- "192.168.22.216"
- "192.168.22.217"
storm.local.dir: "/app/home/storm"
storm.zookeeper.root: "/storm-prod"
#
nimbus.seeds: ["192.168.120.96"]
And if I make a ping to the nimbus ip from the workers, it returns OK
Where is the error, How can I fix it?
Thanks!
Whats appears to happen in this context is that Storm supervisor resolves nimbus from whatever is configured in storm.yaml seeds/host the first time and from then on uses nimbus host name to download the topology artifacts.
If that is correct, DNS is mandatory for a cluster setup. This is far from ideal, specially when using containers in an orchestrated environment like kubernetes.
Current workaround i'm using is adding
storm.local.hostname: "<local.ip.value>"
to the storm.yaml
Thanks to #bastien who provided the tip on storm user mailing list
I ran into the similar issue. Turns out my firewall rules were blocking the supervisor ports. Make sure the supervisor and nimbus are able to talk to each other.
I found that I need to have the hostnames of the boxes match what I was calling them in the /etc/hosts file
in host file i had
xxx.xxx.xxx.xxx nimbus
but the host name on the box was different and it was pulling the hostname from the os
changing the host name on the os of the nimbus server resolved my issue.