Redis keys disappear at random times - redis

Question

I have an issue with Redis which appears only in production and I am not able to reproduce it locally.
I have 11 servers that send data to Redis and each one increments members of an hash map (each server has its own hash map).
At random times the hash maps disappear and I see all the counts starting from 0 again.
Note that:
keys are not expired: neither expiration nor ttl are set on any key;
keys are not evicted: maxmemory is not set and maxmemory-policy is no-eviction anyway;
Redis never has memory problems because it's on a server with 15GB of free RAM and it never crashes anyway;
INFO reports 13 connected clients which makes sense: 11 servers + 1 monitoring application that I have locally + the connection used to get the output of the INFO command.
I don't know where to look anymore.
Here is the output of the INFO command:
# Server
redis_version:3.2.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:5a93b34a97c2cde8
redis_mode:standalone
os:Linux 4.9.0-6-amd64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:6.3.0
process_id:1394
run_id:ff6063b446dab8248fe9db118d2993a9de4252c8
tcp_port:6379
uptime_in_seconds:186923
uptime_in_days:2
hz:10
lru_clock:2982223
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
# Clients
connected_clients:13
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
# Memory
used_memory:1067336
used_memory_human:1.02M
used_memory_rss:3784704
used_memory_rss_human:3.61M
used_memory_peak:1471928
used_memory_peak_human:1.40M
total_system_memory:27401003008
total_system_memory_human:25.52G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:3.55
mem_allocator:jemalloc-3.6.0
# Persistence
loading:0
rdb_changes_since_last_save:13854793
rdb_bgsave_in_progress:0
rdb_last_save_time:1529530575
rdb_last_bgsave_status:err
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
# Stats
total_connections_received:185
total_commands_processed:19637940
instantaneous_ops_per_sec:121
total_net_input_bytes:752885632
total_net_output_bytes:1197081334
instantaneous_input_kbps:4.61
instantaneous_output_kbps:9.27
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:1333722
keyspace_misses:120814
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:185
migrate_cached_sockets:0
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:262.40
used_cpu_user:207.39
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=14,expires=0,avg_ttl=0

Answer 1

Very likely Redis server is restarting and you are losing data because redis isn't able to save to disk.
Rdb last save status is error, so data was never created. Also, aof is disabled. If redis restarts, it will start with all data wiped out.
Check your logs - very likely redis doesn't have permissions to write to disk. Also I'm sure you will see entries that suggest redis is restarting.