My web app runs out of connection slots to the database when enough requests occur. This is despite setting it to run what seems to be a conservative size of connection pool, and I have limited the number of processes, threads. Am I correct that the connection pool is shared across threads, but not processes? And what is a good strategy for choosing a good combination of connection pool size, number of processes and threads, whilst avoiding running out of DB connections?
Error I'm seeing:
OperationalError: (OperationalError) FATAL: remaining connection
slots are reserved for non-replication superuser connections
/etc/postgresql/9.1/main/postgresql.conf:
max_connections = 100
app.ini:
sqlalchemy.pool_size = 1
sqlalchemy.max_overflow = 5
Apache config:
WSGIDaemonProcess test1 processes=5 threads=10 maximum-requests=10000
WSGIProcessGroup test1
Looking at the processes:
$ ps aux |grep postgres |wc
This can increase to 102 under a reasonable load and stay there, despite many connections being idle, and errors come in.
Related
We saw "Connection refused to ip:263*" to redis instances from the application logs. To solve we changed the port number from 26** to 6379 and it worked fine.
Upon analysis we found one one of the slave redis servers have the port number 26380 opened using
netstat -tupln
command. but the other server is not. Upon reading found that 26380, 26379, 26381 are ports used by sentinel. We suspect this 2**** ports should be opened on all servers and sue to some reasons it is not.
Please tell us how to check the logs in sentinel
checking if sentinel is configured.
checking if it is running.
checking what could have caused this to stop suddenly.
redis logs for port
EDIT
this is what I can see from the sentinel logs
2907:signal-handler (1653294181) Received SIGTERM scheduling shutdown...
2907:X 23 May 16:23:01.105 # User requested shutdown...
2907:X 23 May 16:23:01.105 * Removing the pid file.
2907:X 23 May 16:23:01.106 # Sentinel is now ready to exit, bye bye...
433:X 23 May 16:25:08.364 # Creating Server TCP listening socket ipaddress:26379: bind: Cannot assign requested address
anotheripaddress
I am occasionally getting SSH failures in my Ansible 2.6.19 playbook during operations that that use file or copy with large with_items. Several items will succeed then at some point I will get
Shared connection to xxx.xyz.com closed
sudo: PAM account management error: Authentication service cannot retrieve authentication info
Then 2 seconds later there is a SUCCESS message for each of the rest of the files. This suggests to me that something must have happened on the server to cause the issue and then it resolved itself.
I have pipelining = True in my ansible.cfg.
How do I make Ansible playbook try again on SSH errors like this so the playbook doesn't fail?
EDIT: To address the comment, I am investigating the source but since I don't have control of it I need a backup. The retry/until is at the task level, however, there are too many tasks to put it on each one. I really need something at a playbook level. e.g. in ansible.cfg
One option at configuration level is use retry files. This will allow you to rerun the playbooks with the --limit #path/to/retry-file option.
Excerpt from ansible.cfg:
retry_files_enabled = True
retry_files_save_path = ~/.ansible-retry
This will cause a <playbook>.retry file to be created (in ~/.ansible-retry/ directory) when a playbook failure occurs. Though it doesn't make Ansible automatically retry, the playbook can be rerun with --limit option to cover the hosts on which failure occurred. This can be combined with error handling (as #Zeitounator commented).
The other option is to use the wait_for_connection module.
- name: wait for connection to host for 2 mins
wait_for_connection:
timeout: 120
I have an application using Redis as a cache.
I have sporadic episodes where Redis suddenly becomes unresponsive. Even a simple curl test fails:
~ curl localhost:6379
curl: (7) Failed to connect to localhost port 6379: Connection refused
Redis console logs show no errors,
I am using this metrics exporter: https://github.com/oliver006/redis_exporter
During these episodes, redis_up is 0 and redis_exporter_last_scrape_error is 1 with no error detail.
FYI, I have 46 million short keys in Redis with empty string values.
How do I go about diagnosing and hopefully resolving this?
I've got redis installed on my VM, and I haven't used it in a while. (Last I was using it, it did work, and now it doesn't.. nothing's changed in that time (about a month)). Needless to say I'm deeply confused but I'll post as much info as I can.
$ redis-server
Server starts, but throws a warning about overcommit memory being set to 0. I'm on a VM, so I can't change this setting from 0 to 1 if I wanted, which I wouldn't want to anyway for my purposes. I've written a custom redis.config file though, which I want it to use (and which I was using in the past), so starting it with the default config file doesn't do me much good. Let's try this again.
$ redis-server redis.config
$
Nothing. Silence. No error message, just didn't start.
$ nohup redis-server redis.config > nohup.out&
I get a process ID, but then $ ps and I see the the process is listed as stop and shortly disappears. Again, no errors, and no output in nohup.out nor in the log file for redis. Below is the redis.config I'm using (without the comments to keep it short)
daemonize yes
pidfile [my-user-account-path]/redis/redis.pid
port 0
bind 127.0.0.1
unixsocket [my-user-account-path]/tmp/redis.sock
unixsocketperm 770
timeout 10
tcp-keepalive 60
loglevel warning
logfile [my-user-account-path]/redis/logs/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error no
rdbcompression no
rdbchecksum no
dbfilename dump.rdb
dir [my-user-account-path]/redis/db
slave-serve-stale-data yes
slave-priority 100
appendonly no
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
# ADVANCED CONFIG is set to all default settings#
I'm sure it's probably something stupid, probably even a permissions thing somewhere (I've tried executing this as root, fyi), to no avail. Anyone ever experience something similar with Redis?
i have been experiencing redis crashes as well. just an fyi - the guy responsible for much of redis' development, Salvatore Sanfilippo, aka antirez, keeps an interesting blog that has some insight on redis crashes:
http://antirez.com/news/43
I want to timeout and kill idle redis clients. Is there a setting I can set to do this? I seem to remember setting a configuration somewhere but I can't seem to find it again.
I want this to be done automatically, rather than manually calling the client kill command.
Have a look into the Redis configuration file (the one you use to launch Redis).
# Close the connection after a client is idle for N seconds (0 to disable)
timeout 0
Just check the parameter is not commented out, and change the timeout parameter to put a non zero value in seconds. The instance should be restarted to take this parameter in account.
To change this parameter on a running Redis instance, you can use a client command:
> src/redis-cli config set timeout 10
OK
> src/redis-cli config get timeout
1) "timeout"
2) "10"