ERR Slot xxx is already busy (Redis::CommandError) - redis

I want setup redis cluster with 6 nodes (node1, node2, node3, node4, node5, node6), which has 3 masters and 3 slaves. Each node has this configuration file
redis.conf
port 6379
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 10000
appendonly yes
I get error when create the cluster. Create command:
redis-trib.rb create --replicas 1 node1:6379 node2:6379 node3:6379 node4:6379 node5:6379 node6:6379
Error:
>>> Creating cluster
Connecting to node node1:6379: OK
Connecting to node node2:6379: OK
Connecting to node node3:6379: OK
Connecting to node node4:6379: OK
Connecting to node node5:6379: OK
Connecting to node node6:6379: OK
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
node6:6379
node5:6379
node4:6379
Adding replica node3:6379 to node6:6379
Adding replica node2:6379 to node5:6379
Adding replica node1:6379 to node4:6379
S: 1f13819038ba983bb8355f54cb8cec19d2b29e01 node1:6379
replicates 534745088c8b403b81d7e48a22d2e317fb420a38
S: 711461862393664b46d73db6561631f40de29561 node2:6379
replicates f503fe6fd52c73e446267795111ae6ea95495829
S: 204fa4e23b08e2c6ad80b0aca271fc380bc6885d node3:6379
replicates fe6a8e88afdb2796c09fcc873b37ba90c2ba6d79
M: 534745088c8b403b81d7e48a22d2e317fb420a38 node4:6379
slots:10923-16383 (5461 slots) master
M: f503fe6fd52c73e446267795111ae6ea95495829 node5:6379
slots:5461-10922 (5462 slots) master
M: fe6a8e88afdb2796c09fcc873b37ba90c2ba6d79 node6:6379
slots:0-5460,6918 (5462 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
/var/lib/gems/1.8/gems/redis-3.2.2/lib/redis/client.rb:114:in `call': ERR Slot 16011 is already busy (Redis::CommandError)
from /var/lib/gems/1.8/gems/redis-3.2.2/lib/redis.rb:2646:in `method_missing'
from /var/lib/gems/1.8/gems/redis-3.2.2/lib/redis.rb:57:in `synchronize'
from /usr/lib/ruby/1.8/monitor.rb:242:in `mon_synchronize'
from /var/lib/gems/1.8/gems/redis-3.2.2/lib/redis.rb:57:in `synchronize'
from /var/lib/gems/1.8/gems/redis-3.2.2/lib/redis.rb:2645:in `method_missing'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:205:in `flush_node_config'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:667:in `flush_nodes_config'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:666:in `each'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:666:in `flush_nodes_config'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:1007:in `create_cluster_cmd'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:1388:in `send'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:1388
I also did these, but also got same error message
use ip address instead of hostname
remove nodes.conf in each nodes

How said #thepirat000 (in all nodes did FLUSHALL and then CLUSTER RESET SOFT) i also changed hostname to ip address

Related

How to connect to a remote Redis cluster within another network?

Redis cluster is created with internal ip addresses.
M: 24ff344338e4abb4f7f2e888ee9d57843fc46e62 10.0.9.19:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 9aa005c2d914db12a394af0cb8a0d8e218730099 10.0.9.15:6379
slots: (0 slots) slave
replicates f67bbf56d98c2bff9eba343356e2b52bd5e59b12
S: aedd33304e59cbe7091fb36befdb230f3956f03e 10.0.9.16:6379
slots: (0 slots) slave
replicates 9cd64e70f9fd7fffb44b79186e09e3872ea3ebb4
M: f67bbf56d98c2bff9eba343356e2b52bd5e59b12 10.0.9.20:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 2af640e7072b255786b47337e7ca171e0506f5f9 10.0.9.14:6379
slots: (0 slots) slave
replicates 24ff344338e4abb4f7f2e888ee9d57843fc46e62
M: 9cd64e70f9fd7fffb44b79186e09e3872ea3ebb4 10.0.9.18:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
There is only one public ip for external connection, 54.174.xxx.xxx.
After connecting to the cluster, there is problem for redirecting.
52.71.xxx.xxx:6379> lrange mylist 0 -1
-> Redirected to slot [5282] located at 10.0.9.18:6379
Could not connect to Redis at 10.0.9.18:6379: Connection timed out
Could not connect to Redis at 10.0.9.18:6379: Connection timed out
(254.54s)
not connected>
It look like redis is asking the client to connect to another node for the data. Is it possible to let the current connected node to fetch data on behave? or Any another solution?
As least, I know Cassandra will fetch data from another node to return instead of redirecting.

Datanode error : NameSystem.getDatanode

Help please Folks
I am trying to set up my Hadoop multinode env (1 master, 1 secondary and 3 slaves - hadoop 2.7.1/Ubuntu 14 on AWS) and i am getting "NameSystem.getDatanode" ERROR message. I browsed and read and tried but reach my limits. Could you point me at least in some direction
Logs (extract) from master - xxx-141/142/143 are the ip of the slaves
'''''''''''''''''''''''''''''''''''
Line 134: 2016-01-23 17:36:19,432 ERROR org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.getDatanode: Data node DatanodeRegistration(XXX.XX.XX.143:50010, datanodeUuid=6826238d-9213-4b19-a6eb-13115e3bea8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-57295bbd-e78e-4265-99f7-fdacccbcb33a;nsid=1674724909;c=0) is attempting to report storage ID 6826238d-9213-4b19-a6eb-13115e3bea8d. Node 172.31.22.141:50010 is expected to serve this storage.
Line 135: 2016-01-23 17:36:19,457 ERROR org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.getDatanode: Data node DatanodeRegistration(XXX.XX.XX.142:50010, datanodeUuid=6826238d-9213-4b19-a6eb-13115e3bea8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-57295bbd-e78e-4265-99f7-fdacccbcb33a;nsid=1674724909;c=0) is attempting to report storage ID 6826238d-9213-4b19-a6eb-13115e3bea8d. Node 172.31.22.141:50010 is expected to serve this storage.
Line 159: 2016-01-23 17:36:20,988 ERROR org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.getDatanode: Data node DatanodeRegistration(XXX.XX.XX.141:50010, datanodeUuid=6826238d-9213-4b19-a6eb-13115e3bea8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-57295bbd-e78e-4265-99f7-fdacccbcb33a;nsid=1674724909;c=0) is attempting to report storage ID 6826238d-9213-4b19-a6eb-13115e3bea8d. Node XXX.XX.XX.143:50010 is expected to serve this storage.
Extract From SLAVE2 SERVER logs
'''''''''''''''''''''''''''''''''''''
2016-01-23 17:36:14,812 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
`2016-01-23 17:36:18,607 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Unsuccessfully sent block report 0x3c90bbfe60c, containing 1 storage report(s), of which we sent 0. The reports had 1 total blocks and used 0 RPC(s). This took 4 msec to generate and 144 msecs for RPC and NN processing. Got back no commands.
2016-01-23 17:36:18,608 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1050309752-MAST.XX.XX.169-1453113991010 (Datanode Uuid 6826238d-9213-4b19-a6eb-13115e3bea8d) service to master/MAST.XX.XX.169:9000 is shutting down
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.UnregisteredNodeException): Data node DatanodeRegistration(1XX.XX.XX.142:50010, datanodeUuid=6826238d-9213-4b19-a6eb-13115e3bea8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-57295bbd-e78e-4265-99f7-fdacccbcb33a;nsid=1674724909;c=0) is attempting to report storage ID 6826238d-9213-4b19-a6eb-13115e3bea8d. Node 1XX.XX.XX.141:50010 is expected to serve this storage.
at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanode(DatanodeManager.java:495)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1791)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1315)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:163)
at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28543)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1407)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy13.blockReport(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:199)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:463)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:688)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:823)
at java.lang.Thread.run(Thread.java:745)
2016-01-23 17:36:18,610 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-1050309752-MAST.XX.XX.169-1453113991010 (Datanode Uuid 6826238d-9213-4b19-a6eb-13115e3bea8d) service to master/MAST.XX.XX.169:9000
2016-01-23 17:36:18,611 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-1050309752-MAST.XX.XX.169-1453113991010 (Datanode Uuid 6826238d-9213-4b19-a6eb-13115e3bea8d)
2016-01-23 17:36:18,611 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Removing block pool BP-1050309752-MAST.XX.XX.169-1453113991010
2016-01-23 17:36:20,611 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2016-01-23 17:36:20,613 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2016-01-23 17:36:20,614 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at ip-SLAV-XX-XX-142/1XX.XX.XX.142
************************************************************/`
it looks like you have three slaves
172.31.22.141:50010
172.31.22.142:50010
172.31.22.143:50010
and you created two of them from a clone of the first slave, after the slave was already included in the cluster.
The two clones now already have a copy of the DFS and use the same storage ID as the first slave. Only one slave with the same ID is expected by the name server. It is trying to tell you this by logging:
[...] is attempting to report storage ID [...].
Node [...]:50010 is expected to serve this storage.
You can try removing the dfs directory on two of the slaves, then restarting them.
i.e. stop the slaves, do a rm -rf on the dfs directory like:
rm -rf /tmp/hadoop-hadoop/dfs/
You can then restart and check that all slaves are connecting and test file replication, e.g. by setting the replication level to 4 for some files like:
hdfs dfs -setrep -w 4 -R /user/somedir
The -w option causes the command to wait until replication has succeeded.

Spring XD distributed mode Redis Sentinel

I am trying to run spring xd in distruibuted mode on 2 Ubuntu VMs. My goal is to deploy a module on one of the VMs running spring xd and make it visible to the container on the other VM(hostname: container1). On the main VM(hostname: xd-admin) I am running redis-sentinel with this configuration in servers.yml
spring:
redis:
port: 6379
host: 127.0.0.1
sentinel:
master: 127.0.0.1:26379
nodes: 127.0.0.1:26379,127.0.0.1:26380,127.0.0.1:26381
In container1 I have the following in servers.yml
spring:
redis:
port: 6379
host: 127.0.0.1
sentinel:
master: xd-admin:26379
nodes: 127.0.0.1:26379,127.0.0.1:26380,127.0.0.1:26381
zk:
namespace: xd
client:
connect: xd-admin:2181
sessionTimeout: 60000
connectionTimeout: 30000
initialRetryWait: 1000
retryMaxAttempts: 3
When I run xd-container on the xd-admin host I get
Caused by: redis.clients.jedis.exceptions.JedisException: Can connect to sentinel, but 127.0.0.1:26379 seems to be not monitored...
at redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:150) ~[jedis-2.6.2.jar:na]
at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:69) ~[jedis-2.6.2.jar:na]
at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:47) ~[jedis-2.6.2.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:215) ~[spring-data-redis-1.5.0.RELEASE.jar:1.5.0.RELEASE]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:202) ~[spring-data-redis-1.5.0.RELEASE.jar:1.5.0.RELEASE]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:195) ~[spring-data-redis-1.5.0.RELEASE.jar:1.5.0.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1633) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1570) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE]
... 30 common frames omitted
When I run xd-container on the container1 host I get
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: All sentinels down, cannot determine where is 192.168.33.10:26379 master is running...
at redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:153)
at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:69)
at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:47)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:215)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:202)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:195)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1633)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1570)
... 31 more
I have zookeeper and rabbit-mq running on xd-admin. I have redis running on container1. I know xd-admin is accessible from container1 because I have apache2 installed on xd-admin and I receive a response when I run curl xd-admin from container1. How do I configure redis and/or my servers.yml properly for my containers to communicate?
In my servers.yml files I commented redis.sentinel and its children and the exception went away.

Redis3 cluster infinite waiting for the cluster to join

I have 2 servers and 3 instances of redis3 in each of them. I have a cluster-nodes directory, where I have all the data of each instance. Here it is.
cluster-nodes/
|-- 7777
| |-- db01
| | -- nodes-7777.conf
| -- redis.conf
|-- 7778
| |-- db02
| | -- nodes-7778.conf
| -- redis.conf
-- 7779
|-- db03
| -- nodes-7779.conf
-- redis.conf
Here is my config file redis.conf under the 7777 directory
pidfile /var/run/redis/redis-7777.pid
port 7777
dir /opt/redis/cluster-nodes/7777/db01/
cluster-enabled yes
cluster-config-file nodes-7777.conf
cluster-node-timeout 15000
When I try to start redis I get
./redis-trib.rb create --replicas 1 127.0.0.1:7777 127.0.0.1:7778 127.0.0.1:7779 192.168.56.41:7777 192.168.56.41:7778 192.168.56.41:7779
>>> Creating cluster
Connecting to node 127.0.0.1:7777: OK
Connecting to node 127.0.0.1:7778: OK
Connecting to node 127.0.0.1:7779: OK
Connecting to node 192.168.56.41:7777: OK
Connecting to node 192.168.56.41:7778: OK
Connecting to node 192.168.56.41:7779: OK
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7777
192.168.56.41:7777
127.0.0.1:7778
Adding replica 192.168.56.41:7778 to 127.0.0.1:7777
Adding replica 127.0.0.1:7779 to 192.168.56.41:7777
Adding replica 192.168.56.41:7779 to 127.0.0.1:7778
M: 209d68fae9c64855d34972f660232eb96370a669 127.0.0.1:7777
slots:0-5460 (5461 slots) master
M: 62e2b167a287b94b5154f7b9b0f226345baa81b7 127.0.0.1:7778
slots:10923-16383 (5461 slots) master
S: 36ed59deceb01788db76abc0c2f22925a27295fc 127.0.0.1:7779
replicates 2760b5fcc99c6563a7cf8deea159efb012309238
M: 2760b5fcc99c6563a7cf8deea159efb012309238 192.168.56.41:7777
slots:5461-10922 (5462 slots) master
S: 16bf95ba9cb743c2a3caecaab5c2fd5121d80557 192.168.56.41:7778
replicates 209d68fae9c64855d34972f660232eb96370a669
S: 30e7a5b4a94b5ff3a09f4809d6fd62edb2279b0e 192.168.56.41:7779
replicates 62e2b167a287b94b5154f7b9b0f226345baa81b7
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join.....................................................................................................................................................................................................................................................................^C./redis-trib.rb:534:in `sleep': Interrupt
from ./redis-trib.rb:534:in `wait_cluster_join'
from ./redis-trib.rb:1007:in `create_cluster_cmd'
from ./redis-trib.rb:1373:in `<main>'
Here is the output from cluster nodes on the first server
62e2b167a287b94b5154f7b9b0f226345baa81b7 127.0.0.1:7778 master - 0 1435144555558 2 connected 10923-16383
36ed59deceb01788db76abc0c2f22925a27295fc 127.0.0.1:7779 master - 0 1435144554554 3 connected
209d68fae9c64855d34972f660232eb96370a669 127.0.0.1:7777 myself,master - 0 0 1 connected 0-5460
And this is from the second
16bf95ba9cb743c2a3caecaab5c2fd5121d80557 127.0.0.1:7778 master - 0 1435144648065 5 connected
30e7a5b4a94b5ff3a09f4809d6fd62edb2279b0e 127.0.0.1:7779 master - 0 1435144647057 6 connected
2760b5fcc99c6563a7cf8deea159efb012309238 127.0.0.1:7777 myself,master - 0 0 4 connected 5461-10922
It seems that all of them are started as masters? Is there something wrong in my configs?
Thank you.
p.s. when I try the same configs and start all instances in one server, everything works fine.
The problem in my case was that I was starting the service with localhost address,
./redis-trib.rb create --replicas 1 127.0.0.1:7777 127.0.0.1:7778 127.0.0.1:7779 192.168.56.41:7777 192.168.56.41:7778 192.168.56.41:7779
In order to fix that 127.0.0.1 should be substituted with ip address of the local node, i.e.
./redis-trib.rb create --replicas 1 192.168.56.40:7777 192.168.56.40:7778 192.168.56.40:7779 192.168.56.41:7777 192.168.56.41:7778 192.168.56.41:7779
Please check the 17777 17778...port,cluster need those port to communication。

Cassandra Cluster with Multiple Datacenter fail Authenticaion only for a specific user

I've created a cassandra cluster as follow:
Datacenter: MG
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 192.168.0.120 128.45 KB 256 13.3% e1c9e29f-b6f4-4e9f-89f2-bd19153e3253 RACK01
UN 192.168.0.121 115.01 KB 256 12.6% a45f35b7-dbcc-4b09-a35f-5836cabfdedb RACK01
Datacenter: SP
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 192.168.0.101 143.25 KB 256 13.1% 3b3bccf9-63bf-4a33-8efb-412efec35f3d RACK01
UN 192.168.0.100 126.63 KB 256 12.4% 1123cc2f-4ae3-4045-bfe5-1395c36692de RACK01
UN 192.168.0.103 151.64 KB 256 11.2% a9baf020-a1af-4b08-825c-b0e49e938802 RACK02
UN 192.168.0.102 150.65 KB 256 12.3% ce96514f-6f23-4c02-b246-86c0be717ca5 RACK02
Datacenter: RJ
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 192.168.0.111 155.03 KB 256 12.9% 2157445b-d455-4e19-8394-0a9f67397f2e RACK01
UN 192.168.0.110 131.14 KB 256 12.3% 9aaa5de2-37fc-4810-9563-4d711a9457e6 RACK01
For every node I've enabled PasswordAuthenticator and created a "username" with password "password" as superuser. Also I've configured the rpc_address to the node ip address.
When typing ./cqlsh -u username -p password ip_addr on any node from "SP" datacenter I can get connected. However if I try connect to any other node from "MG" or "RJ" datacenter the response is a AuthenticationException: Cannot Achieve consistency level LOCAL_ONE.
Two things are bugging me:
When I try to connect using username "cassandra" and password "cassandra" I can connect on any node of any datacenter
I doubled check and every keyspace on every node has a consistency level of ONE
Does someone has any clue on what can be wrong?
Thanks
I've just performed some research and found out that in despite of creating a cluster, the keyspace for "system_auth" is configured by default using SimpleStrategy; however such a strategy shall not be used when using multiple datacenters.
So I changed the system_auth keyspace by using ALTER KEYSPACE command and setting a replication of type 'class':'NetworkTopologyStrategy', 'MG':1, 'SP':2, 'RJ':1
Now I am able to connect on every node since all username/password data was replicted for all datacenters.
I'm still learning some tricks about cassandra, so excuse the lack of more deep information.
Anyway hope this can help.