Redis wrong "used_memory_human" on Sentinel Slave - redis

A slave node reports:
redis-cli -p 7001 info memory | grep used_memory
used_memory:10741368904
used_memory_human:10.00G
With no keys:
redis-cli -p 7001 info Keyspace
# Keyspace
It's master reports the correct size:
# Memory
used_memory:4963584
used_memory_human:4.73M
Persistence files have all the same size on every server:
-rw-r--r-- 1 root root 178 Feb 28 17:43 /var/lib/redis/dump_7001.rdb
Actions taken: on the Slave the .rdb file was deleted and redis restarted. Master-Slave synced ok but still reporting the difference.
Failovered but didn't solve it.
jira#fr4redistaskmp03:~$ redis-cli -h redistaskmp01 -p 7001 info memory | grep used_memory_human
used_memory_human:4.73M
jira#fr4redistaskmp03:~$ redis-cli -h redistaskmp02 -p 7001 info memory | grep used_memory_human
used_memory_human:4.77M
jira#fr4redistaskmp03:~$ redis-cli -h redistaskmp03 -p 7001 info memory | grep used_memory_human
used_memory_human:10.00G
jira#fr4redistaskmp03:~$ redis-cli -h redistaskmp03 -p 7001 info replication
# Replication
role:slave
master_host:172.25.8.17
master_port:7001
Any idea?
The complete output:
The bad one:
redistaskmp03:~$ redis-cli -p 7001 info memory
# Memory
used_memory:10741368904
used_memory_human:10.00G
used_memory_rss:6864896
used_memory_rss_human:6.55M
used_memory_peak:10741430888
used_memory_peak_human:10.00G
used_memory_peak_perc:100.00%
used_memory_overhead:10741311316
used_memory_startup:3658312
used_memory_dataset:57588
used_memory_dataset_perc:0.00%
allocator_allocated:10741442744
allocator_active:10741768192
allocator_resident:10751008768
total_system_memory:135206285312
total_system_memory_human:125.92G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:10737418240
maxmemory_human:10.00G
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:325448
allocator_rss_ratio:1.00
allocator_rss_bytes:9240576
rss_overhead_ratio:0.00
rss_overhead_bytes:-10744143872
mem_fragmentation_ratio:0.00
mem_fragmentation_bytes:-10734483056
mem_not_counted_for_evict:0
mem_replication_backlog:10737418240
mem_clients_slaves:0
mem_clients_normal:234764
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
The good one:
# Memory
used_memory:4963584
used_memory_human:4.73M
used_memory_rss:4591616
used_memory_rss_human:4.38M
used_memory_peak:5168376
used_memory_peak_human:4.93M
used_memory_peak_perc:96.04%
used_memory_overhead:4908920
used_memory_startup:3658352
used_memory_dataset:54664
used_memory_dataset_perc:4.19%
allocator_allocated:4993456
allocator_active:5312512
allocator_resident:10412032
total_system_memory:135206285312
total_system_memory_human:125.92G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:10737418240
maxmemory_human:10.00G
maxmemory_policy:noeviction
allocator_frag_ratio:1.06
allocator_frag_bytes:319056
allocator_rss_ratio:1.96
allocator_rss_bytes:5099520
rss_overhead_ratio:0.44
rss_overhead_bytes:-5820416
mem_fragmentation_ratio:0.94
mem_fragmentation_bytes:-309056
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:0
mem_clients_normal:201992
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
redis_version:5.0.4
Maybe a bug?

Solved.
The problem was "repl-backlog-size 10gb" on .conf file.

Related

Redis server on diffrent node using docker on ubuntu 20

i want to setup redis cluster on 3 diffrent nodes using docker..can any one suggest the commands or some documents for setup?
i am finding the documents for single node but i want it for multiple nodes
any help would be appriciated
Thank you!
so i followed these steps to setup redis cluster on 3 diffrent servers.
add entries in nano /etc/hosts file on all the nodes 3 nodes
ip1 hostname1
ip2 hostname2
ip3 hostname3
then save and exit
on node1
docker run --name redis-node-1 --hostname hostname1 --net test -p 6379:6379 -p 16379:16379 -d redis redis-server --cluster-enabled yes --cluster-config-file nodes.conf --cluster-node-timeout 5000
on node2
docker run --name redis-node-2 --hostname hostname2 --net test -p 6379:6379 -p 16379:16379 -d redis redis-server --cluster-enabled yes --cluster-config-file nodes.conf --cluster-node-timeout 5000
on node3
docker run --name redis-node-3 --hostname hostname3 --net test -p 6379:6379 -p 16379:16379 -d redis redis-server --cluster-enabled yes --cluster-config-file nodes.conf --cluster-node-timeout 5000
now once all 3 containers are up run the below command on any node.i am running it on 1st node.
docker exec -it redis-node-1 redis-cli --cluster create ip1:6379 ip2:6379 ip3:6379 --cluster-replicas 0
after the setup is complete verify it with below command
docker exec -it redis-node-1 redis-cli cluster info
This worked for me ..Thank you!

mariadb galera node stuck at WSREP state transfer ongoing

I restarted mariadb galera cluster node, and now its taking ages to start, however I noticed
Status: "WSREP state transfer ongoing, current seqno: 9331 waited 510.000000 secs" and also noticed Memory: 71.3M memory usage is going up everytime I recheck mariadb service status
mariadb.service - MariaDB 10.1.47 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: activating (start) since Fri 2020-11-27 21:26:43 GMT; 8min ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 19409 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, statu
Process: 19407 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Process: 19497 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /u
Process: 19495 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status
Process: 19494 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=
Main PID: 19701 (mysqld)
Status: "WSREP state transfer ongoing, current seqno: 9331 waited 510.000000 secs"
Tasks: 14 (limit: 4573)
Memory: 71.3M
CPU: 49.181s
CGroup: /system.slice/mariadb.service
├─19701 /usr/sbin/mysqld --wsrep_start_position=5b96f94b-2dcb-11eb-8e8b-eff8238871c4:9331
├─19764 sh -c wsrep_sst_rsync --role 'joiner' --address '192.168.5.165' --datadir '/var/lib/mysql/'
├─19765 /bin/bash -ue /usr//bin/wsrep_sst_rsync --role joiner --address 192.168.5.165 --datadir /var
├─19824 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf
├─19865 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf
├─19884 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf
└─23405 sleep 1
what should I do, what is the best possbile way to avoid this in the future ?
First of all support for MariaDB 10.1 has ended but you could increase the systemd timeout.

Timeout waiting for privilege escalation prompt

Context
I'm asking this question because those two questions/answerse (one and two) have little context, and I'd liket o expand on it.
I'm trying to provision CentOS7 production servers with ansible client 2.8 . My environment consist one master node (NIS Server) and one compute node (NIS Client). Before provisioning live servers, I setted up a virtual lab (vlab) from VirtualBoxes, which mimics production environment. I copied my public keys to both production and vlab environment.
Problem
My playbooks work with vlab. Unfortunately, the same commands/playbooks are failing against production nodes to which I have ssh password less access and sudo rights. The only feedback that I get is this error "Timeout (7s) waiting for privilege escalation prompt: \u001b[?1h\u001b=\r\r"
Question
Does anyone know what causes this behaviour and how to fix it? I tried SeLinux permissions on home directory, I ended up disabling it (sudo setenforce 0). I'm arriving to a conclusion that there has to be something external that stops ansible but?
Debugging, logging
Ansible Config
# config file for ansible -- https://raw.githubusercontent.com/ansible/ansible/devel/examples/ansible.cfg
[defaults]
timeout = 5
inventory = ./config/hosts
remote_user = lukas
SSHD Config
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTHPRIV
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials no
UsePAM yes
X11Forwarding yes
UseDNS no
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem sftp /usr/libexec/openssh/sftp-server
Ansible log
# checking kernel version
$ uname -a
Linux compute01 3.10.0-514.26.2.el7.x86_64
# pinging compute node
$ ansible -m ping compute01
123.123.123.123 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
# installing vim
$ ansible compute01 -m yum -a 'name=vim state=installed' -b -K -u lukas
ansible 2.8.4
config file = /home/lukas/Coding/projects/nebula-provision/ansible/producion/ansible.cfg
configured module search path = [u'/home/lukas/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.12 (default, Nov 12 2018, 14:36:49) [GCC 5.4.0 20160609]
Using /home/lukas/Coding/projects/nebula-provision/ansible/producion/ansible.cfg as config file
BECOME password:
setting up inventory plugins
host_list declined parsing /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts as it did not pass it's verify_file() method
script declined parsing /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts as it did not pass it's verify_file() method
auto declined parsing /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts as it did not pass it's verify_file() method
Not replacing invalid character(s) "set([u'-'])" in group name (kubernetes-master)
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
....
Parsed /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts inventory source with ini plugin
Loading callback plugin minimal of type stdout, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/minimal.pyc
META: ran handlers
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861 `" && echo ansible-tmp-1566570853.88-235437120093861="` echo /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861 `" ) && sleep 0'"'"''
<123.123.123.123> (0, 'ansible-tmp-1566570853.88-235437120093861=/home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861\n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12697\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<123.123.123.123> Attempting python interpreter discovery
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<123.123.123.123> (0, 'PLATFORM\nLinux\nFOUND\n/usr/bin/python\n/usr/bin/python2.7\n/usr/bin/python\nENDFOUND\n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12697\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<123.123.123.123> (0, '{"osrelease_content": "NAME=\\"CentOS Linux\\"\\nVERSION=\\"7 (Core)\\"\\nID=\\"centos\\"\\nID_LIKE=\\"rhel fedora\\"\\nVERSION_ID=\\"7\\"\\nPRETTY_NAME=\\"CentOS Linux 7 (Core)\\"\\nANSI_COLOR=\\"0;31\\"\\nCPE_NAME=\\"cpe:/o:centos:centos:7\\"\\nHOME_URL=\\"https://www.centos.org/\\"\\nBUG_REPORT_URL=\\"https://bugs.centos.org/\\"\\n\\nCENTOS_MANTISBT_PROJECT=\\"CentOS-7\\"\\nCENTOS_MANTISBT_PROJECT_VERSION=\\"7\\"\\nREDHAT_SUPPORT_PRODUCT=\\"centos\\"\\nREDHAT_SUPPORT_PRODUCT_VERSION=\\"7\\"\\n\\n", "platform_dist_result": ["centos", "7.3.1611", "Core"]}\n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12697\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/system/setup.py
<123.123.123.123> PUT /home/lukas/.ansible/tmp/ansible-local-12685VGhDEA/tmpwDBaIn TO /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py
<123.123.123.123> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 '[123.123.123.123]'
<123.123.123.123> (0, 'sftp> put /home/lukas/.ansible/tmp/ansible-local-12685VGhDEA/tmpwDBaIn /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py\n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12697\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename#openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs#openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs#openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink#openssh.com" revision 1\r\ndebug2: Server supports extension "fsync#openssh.com" revision 1\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/lukas size 0\r\ndebug3: Looking up /home/lukas/.ansible/tmp/ansible-local-12685VGhDEA/tmpwDBaIn\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:8 O:131072 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:9 O:163840 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:10 O:196608 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:11 O:229376 S:23124\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 32768 bytes at 98304\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 8 32768 bytes at 131072\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 9 32768 bytes at 163840\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 10 32768 bytes at 196608\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 11 23124 bytes at 229376\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'chmod u+x /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/ /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py && sleep 0'"'"''
<123.123.123.123> (0, '', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12697\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 -tt 123.123.123.123 '/bin/sh -c '"'"'sudo -H -S -p "[sudo via ansible, key=kuswqyltevcovqytnefnxinbrwvcydkq] password:" -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-kuswqyltevcovqytnefnxinbrwvcydkq ; /usr/bin/python /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
123.123.123.123 | FAILED! => {
"msg": "Timeout (7s) waiting for privilege escalation prompt: \u001b[?1h\u001b=\r\r"
sshd log
Aug 26 13:36:19 123.123.123.123 sudo: pam_unix(sudo:auth): conversation failed
Aug 26 13:36:19 123.123.123.123 sudo: pam_unix(sudo:auth): auth could not identify password for [lukas]
I believe the problem you are having is that you haven't set the permission escalation user password. In Ansible, when we need root permissions, we set the become variable. The method for becoming root may vary, so you can set the way using the ansible_become_method.
Regarding your problem, I think you need to set up the ansible_become_user and ansible_become_password before running your playbook. You can do it in your inventory, or wherever it feels right for your case.
Here is the link to the variable list you can configure to modify how Ansible will connect to the hosts.
You can get more information about privilege escalation on the "Understanding Privilege Escalation" of Ansible Docs.
I hope it helps.
Turned out that the problem relates to PAM auth module. In short, I added auth sufficient pam_permit.so line to /etc/pam.d/sudo file. In long this is how I arrived to the solution.
Be aware that there can be a million of reasons for PAM to fail, thus this solution might not work for you.
$ cat /etc/pam.d/sudo
#%PAM-1.0
# Fixing "auth could not identify password for [username]" ssh problem.
auth sufficient pam_permit.so
# Original config below
auth include system-auth
account include system-auth
password include system-auth
session optional pam_keyinit.so revoke
session required pam_limits.so
session include system-auth
Just now I had the same issue and my command was
./ansible-playbook playbook.yml -i hosts -b --become-user root --extra-vars "ansible_become_pass= myPass" --become-method su
after removing the space in
"ansible_become_pass=[space]myPass" --> "ansible_become_pass=myPass"
I tried executing again and got
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:[random string].
Are you sure you want to continue connecting (yes/no/[fingerprint])?
And I was like wt fun?
and then I saw that when executing this playbook I already was a root user
so I've changed to a different user and tried executing this command again and thank God it worked.

Redis hash structure occupies more memory in cluster mode

Hash:
./redis-cli -c -p 7000 hlen 0
(integer) 7746812
./redis-cli -c -p 7000 hlen 1
(integer) 7746812
./redis-cli -c -p 7000 hlen 2
(integer) 7746812
./redis-cli -c -p 7000 hlen 3
(integer) 7746812
./redis-cli -c -p 7000 hlen 4
(integer) 7746812
./redis-cli -c -p 7000 hlen 5
(integer) 0
Memory for each hash:
./redis-cli -c -p 7000 keys '*'
1) "3"
./redis-cli -c -p 7000 memory usage 3
(integer) 415715543
./redis-cli -c -p 7001 keys '*'
1) "2"
2) "1"
memory usage for each keys:
./redis-cli -c -p 7001 memory usage 1
(integer) 415715543
./redis-cli -c -p 7001 memory usage 2
(integer) 415715543
./redis-cli -c -p 7002 memory usage 0
(integer) 415715543
./redis-cli -c -p 7002 memory usage 4
(integer) 415715543
Memory usage cluster level:
./redis-cli -c -p 7001 info memory
# Memory
used_memory:1004513344
used_memory_human:**957.98M**
used_memory_rss:1030799360
used_memory_rss_human:983.05M
used_memory_peak:1004615496
used_memory_peak_human:958.08M
used_memory_peak_perc:99.99%
used_memory_overhead:2568042
used_memory_startup:1449576
used_memory_dataset:1001945302
used_memory_dataset_perc:99.89%
allocator_allocated:1004619400
allocator_active:1004859392
allocator_resident:1022844928
total_system_memory:75798228992
total_system_memory_human:70.59G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:239992
allocator_rss_ratio:1.02
allocator_rss_bytes:17985536
rss_overhead_ratio:1.01
rss_overhead_bytes:7954432
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:26347944
mem_not_counted_for_evict:3162
mem_replication_backlog:1048576
mem_clients_slaves:16922
mem_clients_normal:49694
mem_aof_buffer:3162
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
Same for node 7002
And 480MB for node 7000 which has only one hash.
Question:
Each hash takes 415MB
But why memory used is 480MB for one hash and 958MB for 2 hashses.
I printed the list of keys also in the same cluster.
Calculations are not tallying properly.
What am I missing here? Kindly advice.
It is not because of this also. I did memory purge. After that also, memory remains the same.
Redis has internal structure which occupies memory apart from names and values. It is called "Memory Overhead" in redis.
This is the reason for memory change for the hash and the cluster.
We can utilize ziplist to make hashes memory efficient.

redis-cli is not working

I've used redis-cli before (not sure if on this box), but now it's giving me this:
nak#none:~$ redis-cli
usage: redis-cli [-h host] [-p port] [-r repeat_times] [-n db_num] cmd arg1 arg2 arg3 ... argN
usage: echo "argN" | redis-cli [-h host] [-p port] [-r repeat_times] [-n db_num] cmd arg1 arg2 ... arg(N-1)
If a pipe from standard input is detected this data is used as last argument.
example: cat /etc/passwd | redis-cli set my_passwd
example: redis-cli get my_passwd
example: redis-cli -r 100 lpush mylist x
So, I netcat to the redis server and am able to communicate:
nak#none:~$ ncat 127.0.0.1 6379
info
$336
redis_version:1.2.0
arch_bits:32
multiplexing_api:epoll
uptime_in_seconds:332
uptime_in_days:0
connected_clients:1
etc...
I've tried specifying IP and port: redis-cli -h 127.0.0.1 -p 6379 (without success)
Not a big deal, but weird.
Any ideas as to why this is happening? Thanks stackoverflow!
According to your "info" you are using redis 1.2, which is a really old version. That version may not have supported interactive cli. I cannot confirm since that version is not even in the repository.
the version 1.2 seems to be to old:
Visit http://www.redis.io/download and if you are using ubuntu you can follow this guide http://wiki.ubuntuusers.de/Redis or https://askubuntu.com/questions/68576/how-does-one-upgrade-redis-2-2-to-2-4