I have 6 nodes in our data center of which 3 node also act as seed nodes.I was planning to remove 3 nodes from rack2 and also reduce seed node count to two. With "nodetool decommission" we can remove nodes but is there any extra steps involve in removing seed nodes.
UN 10.10.1.56 339.96 MiB 1 ? rack1
UN 10.10.1.46 334.72 MiB 1 ? rack1
UN 10.10.2.76 307.72 MiB 1 ? rack2
UN 10.10.2.66 296.15 MiB 1 ? rack2
UN 10.10.2.86 316.89 MiB 1 ? rack2
UN 10.10.1.36 375.69 MiB 1 ? rack1
You need to update seed list on nodes before decomissioning seed node - change list, optionally do the rolling restart one node at time, or in the new versions use nodetool reloadseeds
Related
I have a Centos7 VM with ZFS on linux installed.
The VM has a disk /dev/sdb, that I've added to a pool named 'backup', and in this pool created a dataset.
Now, I wanted to increase the size of the disk in VMware, and then expand the size of the pool, but I'm not getting this to work.
I've tried 'zpool online -e backup sdb', but nothing changes.
I've tried running 'partprobe /dev/sdb' before and after the live above, but nothing changes.
I've tried rebooting + the above, nothing changes.
I've tried "parted /dev/sdb",resizing the partition (it suggests the actual new size of the volume), and then all of the above. But nothing changes
I've tried 'zpool export backup' + 'zpool import backup' in various combinations with all of the above. No luck
And also: 'lsblk' and 'df -h' reports the old/wrong size of /dev/sdb, even if parted seems to understand that it has been increased.
PS: autoexpand=on
What to do?
I faced a similar issue today and had to try a lot before finding the solution.
When I tried the known solutions (using zpool) of setting autoexpand as on and also restarting the partprobe, system would not auto expand (even after a restart).
Finally, I could solve it using parted instead of getting into zpool at all.
We need to be careful here since wrong partition selections can cause data loss.
What worked for me in your situation
Step 1: Find which pool you are trying to expand. In my case, it is 5 as seen below (unallocated space is after this pool). Use parted -l
parted -l
Output
Model: VMware, VMware Virtual S (scsi)
Disk /dev/sda: 69.8GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 2097kB 1049kB bios_grub
2 2097kB 540MB 538MB fat32 EFI System Partition boot, esp
3 540MB 2009MB 1469MB swap
4 2009MB 3592MB 1583MB zfs
5 3592MB 32.2GB 28.6GB zfs
Step 2: Instructing explictly to expany pool number 5 to 100% available. Note that '5' is not static. You need to use the pool id you wish to expand. Double-check this. Use parted /dev/XXX resizepart YY 100%
parted /dev/sda resizepart 5 100%
After this, I was able to use the entire space in VM.
For reference:
LSBSK Before
sda 8:0 0 65G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 513M 0 part /boot/grub
│ /boot/efi
├─sda3 8:3 0 1.4G 0 part
│ └─cryptoswap 253:1 0 1.4G 0 crypt [SWAP]
├─sda4 8:4 0 1.5G 0 part
└─sda5 8:5 0 29.5G 0 part
LSBSK After
sda 8:0 0 65G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 513M 0 part /boot/grub
│ /boot/efi
├─sda3 8:3 0 1.4G 0 part
│ └─cryptoswap 253:1 0 1.4G 0 crypt [SWAP]
├─sda4 8:4 0 1.5G 0 part
└─sda5 8:5 0 61.7G 0 part
We have 3 nodes in our existing Datastax cluster. Sometime the ring status showing wrongly, after dse service restart on wrong nodes the issue got solved.
Correct :
nodetool status
Datacenter: SearchGraph
=======================
UN 10.10.1.56 1015.41 MiB 1 ? 936a1ac0-6d5e-4a94-8953-d5b5a2016b92 rack1
UN 10.10.1.46 961.43 MiB 1 ? 3f41dc2a-2672-47a1-90b5-a7c2bf17fb50 rack1
UN 10.10.1.36 1013.72 MiB 1 ? 0822145f-4225-4ad3-b2be-c995cc230830 rack1
Wrong :
Datacenter: DC1
===============
?N 10.10.1.46 ? 1 ? null r1
?N 10.10.1.36 ? 1 ? null r1
Datacenter: SearchGraph
=======================
UN 10.10.1.56 1005.33 MiB 1 ? null rack1
Configuration Details is same on all nodes :
cat /etc/dse/cassandra/cassandra.yaml | grep endpoint_snitch:
endpoint_snitch: GossipingPropertyFileSnitch
cat /etc/dse/cassandra/cassandra-rackdc.properties |grep -E 'dc=|rack='
dc=SearchGraph
rack=rack1
Issue got Resolved after restarting DSE nodes with saved_caches and hints folder content removed. I had tried to change the DC name in cassandra-rackdc.properties and revert it back earlier, may be issue caused by these.Thanks
To create and start a cluster in Redis, I use create-cluster.sh file inside
/redis-3.04/utils/create-cluster
With the use of this I can create as many nodes I want by changing the:
Settings
PORT=30000
TIMEOUT=2000
NODES=10
REPLICAS=1.
I wonder if I can create for example 10 nodes (5 masters with 5 slaves) in the beginning but start only 4 masters and 4 slaves (meet and join).
Thanks in advance.
Yes. You can add more nodes if load increase on your existing cluster .
Basic Steps are :
Start new redis instances - Let's say you want to add 2 more master and there slaves (Total 4 redis instances)
Then using redis-trib utility do following :
redis-trib.rb add-node <new master node:port> <any existing master>
e.g. ./redis-trib.rb add-node 192.168.1.16:7000 192.168.1.15:7000
After this new node will be assigned an id . Note that id and run following command to add slave to node that we added in prev step
/redis-trib.rb add-node --slave --master-id <master-node-id> <new-node> <master-node>
./redis-trib.rb add-node --slave --master-id 6f9db976c3792e06f9cd252aec7cf262037bea4a 192.168.1.17:7000 192.168.1.16:7000
where 6f9db976c3792e06f9cd252aec7cf262037bea4a is id of 192.168.1.16:7000.
Using similar steps you can add 1 more master-slave pair .
Since these node do not contains any slots to serve, you have move some of the slots from existing masters to new masters .( Re-Sharding)
To that you can run following command/Resharding steps :
6.1 ./redis-trib.rb reshard <any-master-ip>:<master-port>
6.2 It will ask : How many slots do you want to move (from 1 to 16384)? => Enter number of slots you want to move
6.3 Then it will ask : What is the receiving node ID?
6.4 Enter node id to which slots need to be moved . (new masters)
6.5 It will prompt :
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1: (enter source node id or all)
6.6 Then it will prompt info saying Moving slot n to node node-id like
Moving slot 10960 from 37d10f18f349a6a5682c791bff90e0188ae35e49
Moving slot 10961 from 37d10f18f349a6a5682c791bff90e0188ae35e49
Moving slot 10962 from 37d10f18f349a6a5682c791bff90e0188ae35e49
6.7 It will ask : Do you want to proceed with the proposed reshard plan (yes/no)? Type Yes and enter and you are done .
Note : If data is large it might take some time to reshard.
Few Commands :
To know all nodes in cluster and cluster nodes with node ids:
redis-cli -h node-ip -p node-port cluster nodes
e.g. redis-cli -h 127.0.0.1 -p 7000 cluster nodes
To know all slots in cluster :
redis-cli -h 127.0.0.1 -p 7000 cluster slots
Ref : https://redis.io/commands/cluster-nodes
Hope this will help .
because something unexpected accidents redis memory has increased up a lot , and we delete the unused keys a lot , but the memory does not release, is there a way to manually release it, except restart of redis.
redis info
# Memory
used_memory:14166381000
used_memory_human:13.19G
used_memory_rss:41278218240
used_memory_peak:50044293760
used_memory_peak_human:46.61G
used_memory_lua:31744
mem_fragmentation_ratio:2.91
mem_allocator:jemalloc-3.2.0
nmon t
│ Top Processes Procs=288 mode=3 (1=Basic, 3=Perf 4=Size 5=I/O)──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│
│ PID %CPU Size Res Res Res Res Shared Faults Command │
│ Used KB Set Text Data Lib KB Min Maj │
│ 1732 15.9 58932952 40309580 756 58921404 0 944 9 0 redis-server
free -m
total used free shared buffers cached
Mem: 64378 61914 2464 0 11 22
-/+ buffers/cache: 61879 2498
Swap: 32255 16710 15545
According to antirez https://groups.google.com/forum/#!topic/redis-db/ibhYDLT_n68 (1st answer) Redis would always use used_memory_peak as a reference to how much memory it might need (and how much memory it wants allocated).
If you really need to free that memory with zero downtime — you can setup a slave, make it read-write CONFIG SET slave-read-only no reroute your application traffic to the slave and restart the master. Or just continue using the slave (by making it master SLAVEOF NO ONE)
I am using redis for caching but recently I ran into a problem with the amount of memory used - had to restart my server since all ram had been consumed.
It's not the biggest machine but how should I configure redis to avoid the same problem again?
free -m
total used free shared buffers cached
Mem: 240 222 17 0 6 38
-/+ buffers/cache: 177 62
Swap: 255 46 209
I have changed the following settings:
timeout 60
databases 1
save 300 1
save 60 100
maxmemory 104857600
top
top - 14:15:28 up 1:19, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 49 total, 1 running, 48 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 245956k total, 228420k used, 17536k free, 6916k buffers
Swap: 262136k total, 47628k used, 214508k free, 39540k cached
you can use the "maxmemory" directive in the config file: when this amount of memory is exceeded then Redis will expire earlier keys having already an expire set (the keys that would expire sooner are the first that will be removed).
Unlike memcached, redis is supposed to be a databse; so it won't automatically remove old values to make room for new ones.
You have to explicitly set the expire time for each key/value, and even then you could overflow if you create key/values faster than that.
Use Redis virtual memory in Redis 2.0:
http://antirez.com/post/redis-virtual-memory-story.html