Redis db permission issue while running GitLab - redis

I am trying to setup GitLab, Redis and PostgreSQL containers in Kubernetes using Gluster for persistence. GlusterFS nodes are setup on machines (CentOS) external to Kubernetes cluster (running on RancherOS host). Issue is that when GitLab tries starting up, the login page doesn't load. It's a fresh setup and not something that stopped working now.
root#gitlab-2797053212-ph4j8:/var/log/gitlab/gitlab# tail -50 sidekiq.log
...
...
2017-09-07T11:53:03.098Z 547 TID-1fdjck ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-07T11:53:03.098Z 547 TID-1fdjck ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:73:in `run'
2017-09-07T11:53:03.098Z 547 TID-1fdjck ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/util.rb:17:in `watchdog'
2017-09-07T11:53:03.098Z 547 TID-1fdjck ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/util.rb:26:in `block in safe_thread'
2017-09-07T11:53:03.099Z 547 TID-1fdf1k ERROR: Error fetching job: ERR Error running script (call to f_7b91ed9f4cba40689cea7172d1fd3e08b2efd8c9): #user_script:7: #user_script: 7: -MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:121:in `call'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/redis-3.3.3/lib/redis.rb:2399:in `block in _eval'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
...
So i checked for Redis container logs.
[root#node-a ~]# docker logs -f 67d44f585705
...
...
[1] 07 Sep 14:43:48.140 # Background saving error
[1] 07 Sep 14:43:54.048 * 1 changes in 900 seconds. Saving...
[1] 07 Sep 14:43:54.048 * Background saving started by pid 2437
[2437] 07 Sep 14:43:54.053 # Failed opening .rdb for saving: Permission denied
...
Checked online for this issue and then noticed the following permissions and owner details inside of Redis pod:
[root#node-a ~]# docker exec -it 67d44f585705 bash
groups: cannot find name for group ID 2000
root#redis-2138096053-0mlx4:/# ls -ld /var/lib/redis/
drwxr-sr-x 12 1000 1000 8192 Sep 7 11:51 /var/lib/redis/
root#redis-2138096053-0mlx4:/#
root#redis-2138096053-0mlx4:/# ls -l /var/lib/redis/
total 22
drwxr-sr-x 2 1000 1000 6 Sep 6 10:37 backups
drwxr-sr-x 2 1000 1000 6 Sep 6 10:37 builds
drwxr-sr-x 2 redis redis 6 Sep 6 10:14 data
-rw-r--r-- 1 redis redis 13050 Sep 7 11:51 dump.rdb
-rwxr-xr-x 1 redis redis 21 Sep 5 11:00 index.html
drwxrws--- 2 1000 1000 6 Sep 6 10:37 repositories
drwxr-sr-x 5 1000 1000 55 Sep 6 10:37 shared
drwxr-sr-x 2 root root 8192 Sep 6 10:37 ssh
drwxr-sr-x 3 redis redis 70 Sep 7 10:20 tmp
drwx--S--- 2 1000 1000 6 Sep 6 10:37 uploads
root#redis-2138096053-0mlx4:/#
root#redis-2138096053-0mlx4:/# grep 1000 /etc/passwd
root#redis-2138096053-0mlx4:/#
Ran following and all looked fine.
root#redis-2138096053-0mlx4:/# chown redis:redis -R /var/lib/redis/
However, when i deleted and ran the GitLab deployment YAML again, the permissions inside the Redis container again got skewed up. I am not sure whether Gluster is messing up with the Redis file/folders permissions. Can't think of any other reason right now.
One thing i would like to highlight is that all the three containers are using the same PVC
- name: gluster-vol1
persistentVolumeClaim:
claimName: gluster-dyn-pvc
Above is common for all three. What differs is shown below:
a) postgresql-deployment.yaml
volumeMounts:
- name: gluster-vol1
mountPath: /var/lib/postgresql
b) redisio-deployment.yaml
volumeMounts:
- name: gluster-vol1
mountPath: /var/lib/redis
c) gitlab-deployment.yaml
volumeMounts:
- name: gluster-vol1
mountPath: /home/git/data
Any suggestion?

With the following steps, i was able to resolve the 'Permission denied' issue for Redis:
Created separate volumes for PostegreSQL, Redis and GitLab in GlusterFS.
Created separate Storage Class for all three.
Created PersistentVolumeClaim (PVC) for them and mapped /var/lib/postgresql, /var/lib/redis and /home/git/data to their respective PVCs.
Earlier, all three paths mentioned above were pointing to the same volume in GlusterFS. Somehow, it seems, they were causing issue for Redis.

Related

Trying to setup ssh keys and encountering some issues in giving proper permissions

After generating ssh keys, I have tried to login but, I am getting a message Permission denied (publickey). This seems to be an error in access permissions.
After generating ssh keys for logging into cluster, when i give the command below from the terminal
$> ls -l ~/.ssh/id_*
I should get in return:
-rw------- 1 git git 751 Mar 1 20:16 /home/username/.ssh/id_rsa
-rw-r--r-- 1 git git 603 Mar 1 20:16 /home/username/.ssh/id_rsa.pub
Instead I am getting:
-rw-r--rw- 1 ubuntu ubuntu 3381 févr. 15 18:35 /home/ubuntu.ssh/id_rsa
-rw-r--rw- 1 ubuntu ubuntu 737 févr. 15 18:35 /home/ubuntu/.ssh/id_rsa.pub
Hence login fails with message:
Permission denied (publickey).

Mount host directory to docker/podman container with correct permissions

Using:
podman version 4.2.0
AlmaLinux 8.7
I've created an image based on redhat/ubi8 with the following Dockerfile:
FROM docker.io/redhat/ubi8
RUN dnf install -y gcc-c++ cmake python39 openssh git
RUN useradd -ms /bin/bash foobar -g users
USER foobar
WORKDIR /home/foobar/
RUN mkdir -p .ssh
$ docker build -t mount_test_image .
I run the image from a directory that contains a directory ssh, and I want to mount that directory to /home/foobar/.ssh with ownership of foobar.users
$ ls -l
-rw-r--r--. 1 host_user users 269 Dec 7 09:10 Dockerfile
drwxrwxr-x. 2 host_user users 18 Dec 2 10:41 ssh
docker run -it -d --rm --mount type=bind,src=ssh,target=/home/foobar/.ssh --name=mount_test mount_test_image
However when I enter the container via
docker exec -it mount_test '/bin/sh'
The home directory looks like this:
drwx------. 1 foobar users 18 Dec 7 17:10 .
drwxr-xr-x. 1 root root 21 Dec 7 17:10 ..
-rw-r--r--. 1 foobar users 18 Jun 20 11:31 .bash_logout
-rw-r--r--. 1 foobar users 141 Jun 20 11:31 .bash_profile
-rw-r--r--. 1 foobar users 376 Jun 20 11:31 .bashrc
drwxrwxr-x. 2 root root 18 Dec 2 18:41 .ssh
I obviously get a "permission denied" when trying to access that directory.
sh-4.4$ ls /home/foobar/.ssh
ls: cannot open directory '/home/foobar/.ssh': Permission denied
I tried changing the ownership of the directory on the host to match the uid of the container user, but then it just looks like this:
drwxrwxr-x. 2 nobody root 18 Dec 2 18:41 .ssh
My host user uid:gid is 501:100 and the container user is 1000:100. Right now I'm just trying to generate an ssh key to upload to bitbucket, but this seems like a simple feature a container should be have. All the tutorials and examples just stop after the --mount command instruction and say "there ya go!". What good is the mount point if you can't read/write it?
EDIT:
I tried on Archlinux using docker instead of podman and it works like one would expect with both -v and --mount. The owner of the mounted directory in the container matches the uid and gid of the host. Is this then a bug in podman or is it just done differently?
You are using a non-root user (foobar) in a rootless container. You must use --userns=keep-id for the container user to see the mounted volumes.
https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md#using-volumes

Systemd service using screen: 'No screen session found'

I'm trying to setup a minecraft server for my kid and screen is giving me fits. I've been following this tutorial and I've already consulted this AskUbuntu answer. I'm running on Ubuntu 16.04.5 x64. I have created the following Service file and linked it to /etc/systemd/system/minecraft.service.
[Unit]
Description=Minecraft Server
Documentation=
Wants=network.target
After=network.target
[Service]
User=minecraft
Group=minecraft
Nice=5
EnvironmentFile=-/var/minecraft/unit.conf
KillMode=none
SuccessExitStatus=0 1
ProtectHome=true
ProtectSystem=full
PrivateDevices=true
NoNewPrivileges=true
PrivateTmp=true
InaccessibleDirectories=/root /sys /srv -/opt /media -/lost+found
ReadWriteDirectories=/var/minecraft/server /usr/bin/screen
WorkingDirectory=/var/minecraft/server
#ExecStartPre= /usr/bin/screen -dmS Minecraft
ExecStart= /usr/bin/screen -dm -S Minecraft '/usr/bin/java -Xmx1500M -Xms1024M -jar /var/minecraft/server/craftbukkit-1.13.jar'
ExecStop=/usr/bin/screen -S minecraft -p 0 -X stuff "stop^M"
[Install]
WantedBy=multi-user.target
When I monitor the service startup, I get the following in the journalctl logs:
Aug 05 14:55:41 spigot-1 systemd[1]: Started Minecraft Server.
Aug 05 14:55:41 spigot-1 screen[9869]: No screen session found.
Per the SO post I referenced, I already changed the permissions in /var/run/screen. This is the current setup:
root#spigot-1:/var/minecraft/server# ls -alR /var/run/screen/
/var/run/screen/:
total 0
drwxrwxrwx 4 root root 80 Aug 5 14:12 .
drwxr-xr-x 23 root root 860 Aug 5 14:11 ..
drwx------ 2 minecraft minecraft 40 Aug 5 14:55 S-minecraft
drwx------ 2 root root 40 Aug 5 14:50 S-root
/var/run/screen/S-minecraft:
total 0
drwx------ 2 minecraft minecraft 40 Aug 5 14:55 .
drwxrwxrwx 4 root root 80 Aug 5 14:12 ..
/var/run/screen/S-root:
total 0
drwx------ 2 root root 40 Aug 5 14:50 .
drwxrwxrwx 4 root root 80 Aug 5 14:12 ..
** Debug Notes **
You'll see the commented ExecStartPre command in there... all that happens if I try to start the screen session with that command is I get a second 'No screen session found' error.
I can launch the server from the command line using 'screen -dmS Minecraft [..]' and reattach. I can do this both as root and as the 'minecraft' user.
Any help would be greatly appreciated
I don't have Ubuntu to hand, but I tried something similar on Fedora 24 and had a similar error. To investigate I added an strace to the screen command and removed PrivateTmp so that I could get the trace output.
ExecStart=/bin/strace -o /tmp/s -f screen -dm -S me bash -c 'sleep 999'
In the output I found that at the end, screen was trying to open a pseudo-tty and failed to do so:
open("/dev/ptmx", O_RDWR) = -1 EACCES (Permission denied)
...
write(1, "No more PTYs.\r\nSorry, could not "..., 52) = 52
I added some ls -l /dev commands to the Unit to see what the effect of the PrivateDevices=true option had been. There was indeed a /dev/ptmx file, but unlike my real /dev it was a symbolic link to /dev/pts/ptmx, instead of just being the special character device. For some reason on my system I have:
$ ls -l /dev/ptmx /dev/pts/ptmx
crw-rw-rw- 1 root tty 5, 2 Aug 6 14:29 /dev/ptmx
c--------- 1 root root 5, 2 Jun 27 08:29 /dev/pts/ptmx
(This is probably a bug depending on the version of systemd and udev.)
So when /dev/ptmx is replaced in the private namespace by a symbolic link to /dev/pts/ptmx, you can no longer open a new pty. A simple fix for testing was sudo chmod a+rw /dev/pts/ptmx which indeed made the Unit work and start the screen process.
This may not be the case for you, but you could undertake a similar debug method to try to find the problem.

impala-shell just hangs when invoked in cli

root#master:/usr/lib# impala-shell
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
Starting Impala Shell without Kerberos authentication
Then it hangs here. Who know how to diagnose this problem?
Impala related service have been started.
root#master:/usr/lib# ps -ef | grep impala
root 7551 1 0 05:46 ? 00:00:03 /usr/lib/impala/sbin/impalad
impala 11328 1 0 06:44 ? 00:00:00 /usr/lib/impala/sbin/statestored -log_dir=/var/log/impala -state_store_port=24000
impala 11395 1 0 06:45 ? 00:00:03 /usr/lib/impala/sbin/catalogd -log_dir=/var/log/impala
root 11703 1 0 06:55 ? 00:00:00 grep --color=auto impala
/tmp/impalad.ERROR:
Log file created at: 2016/06/14 07:09:31
Running on machine: master
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0614 07:09:31.558585 11834 logging.cc:120] stderr will be logged to this file.
E0614 07:09:32.755619 11918 thrift-server.cc:176] ThriftServer 'backend' (on port: 22000) exited due to TException: Could not bind: Transport endpoint is not connected
E0614 07:09:32.755651 11834 thrift-server.cc:165] ThriftServer 'backend' (on port: 22000) did not start correctly
E0614 07:09:32.755666 11834 impalad-main.cc:78] ThriftServer 'backend' (on port: 22000) did not start correctly

Production log not working

In the past I thought I was just crazy. I may be, but my production log is not at all responding to some requests. I am POSTing images to my rails app from a mobile client, then GETting a refreshed view in a web browser. The changed record is plainly visible. None of the above is showing up in my production log, but similar requests were logged an hour ago. I haven't changed any config files. I haven't restarted my server. Any suggestions as to why this is happening?
Check what value is set to
config.log_level = :warn
in your
/rails_app/config/enviroments/production.rb
FYI
The available log levels are: :debug,
:info, :warn, :error, :fatal.
So if you set the level to fatal only serious errors will be logged. If you set it to debug almost everything will be logged. By default in production it is set to warn. This is with good reason. As you don't want huge log files piling up on your production server for no reason.
In my case first i added config.log_level = :debug to production.rb file then i had to create the production.log file (in the log folder), then giving appropriate permissions with chmod command.
In my case I had the following gem rails_12factor in my Gemfile for production.
So I had to
Remove rails_12factor from my Gemfile
gem uninstall rails_12factor
bundle install
And on the next push to production the logs worked for me. FYI: I am using rails 5.2.1, but the rails web app started in rails 3.
Low memory is the most likely cause that I have been able to come up with. I will post here if I can prove it.
I solved the Problem by putting
Rails.logger.instance_variable_get(:#logger).instance_variable_get(:#log_dest).sync = true if Rails.logger
into my config/initializers/...
I was having the same problem. Tried everything out there on this website.
Finally i checked the value of Rails.logger in console. It gave following output.
#<RailsStdoutLogging::StdoutLogger:0x00007fe3b5bc3540
#default_formatter=#<Logger::Formatter:0x00007fe3b5bc3658 #datetime_format=nil>,
#formatter=#<ActiveSupport::Logger::SimpleFormatter:0x00007fe3b5bc34c8 #datetime_format=nil>,
#level=0,
#logdev=
#<Logger::LogDevice:0x00007fe3b5bc3608
#dev=#<IO:<STDOUT>>,
#filename=nil,
#mon_count=0,
#mon_mutex=#<Thread::Mutex:0x00007fe3b5bc35b8>,
#mon_mutex_owner_object_id=70307991657220,
#mon_owner=nil,
#shift_age=nil,
#shift_period_suffix=nil,
#shift_size=nil>,
#progname=nil>
Then I crosschecked with some other rails project and that gave the below output
#<ActiveSupport::Logger:0x00005566367b77c0 #level=0,
#progname=nil,
#default_formatter=#<Logger::Formatter:0x00005566367b7950 #datetime_format=nil>, #formatter=#<Logger::Formatter:0x00005566367b7770 #datetime_format=nil>,
#logdev=#<Logger::LogDevice:0x00005566367b7900 #shift_period_suffix=nil, #shift_size=nil,
#shift_age=nil, #filename=nil,
#dev=#<File:/home/rajanverma/workspace/aarvy/log/production.log>,
#mon_mutex=#<Thread::Mutex:0x00005566367b7888>, #mon_mutex_owner_object_id=46948744543360,
#mon_owner=nil, #mon_count=0>>
You can clearly see that both projects were using different loggers to show output. I don't want to output log on STDOUT so I changed my logger to ActiveSupport by creating an initializer file.
config/initializer/logger.rb
Rails.logger = ActiveSupport::Logger.new('log/production.log')
Now it started logging on production.log.
EDIT: I tried doing the same in production.rb, but don't know why it always got overwritten. So i decided to put in initializer. I don't know weather it has any performance issues. Also I don't know how it was changing automatically to #<RailsStdoutLogging::StdoutLogger:0x00007fe3b5bc3540 as writing on production.log should be a default behaviour. Please let me know if you know better reason.
I had the same problem related to production log.
To trace back issue with web server:
tail -f /var/log/apache2/error.log
Message from application: Permission denied - /var/www/APPLICATION-NAME/log/invitation.log (Errno::EACCES)
cd /var/www/APPLICATION-NAME/log
using the comman ll you will get log permissions
total 28
drwxr-sr-x 2 www-data www-data 4096 Sep 18 03:55 ./
drwxrwsr-x 14 www-data www-data 4096 Sep 24 23:52 ../
-rw-r--r-- 1 root www-data 71 Sep 18 03:55 agent_invitation.log
-rwxrwxrwx 1 root www-data 71 Sep 18 03:55 email.log*
-rw-r--r-- 1 root www-data 71 Sep 18 03:55 invitation.log
-rw-r--r-- 1 root www-data 71 Sep 18 03:55 messages.log
-rw-r--r-- 1 root www-data 51 Sep 13 01:32 production.log
Solution:
Have to change owner from root to other user:
chown www-data:www-data ./ -R
Now use ll to see the permissions
total 28
drwxr-sr-x 2 www-data www-data 4096 Sep 18 03:55 ./
drwxrwsr-x 14 www-data www-data 4096 Sep 24 23:52 ../
-rw-r--r-- 1 www-data www-data 71 Sep 18 03:55 agent_invitation.log
-rwxrwxrwx 1 www-data www-data 71 Sep 18 03:55 email.log*
-rw-r--r-- 1 www-data www-data 71 Sep 18 03:55 invitation.log
-rw-r--r-- 1 www-data www-data 71 Sep 18 03:55 messages.log
-rw-r--r-- 1 www-data www-data 51 Sep 13 01:32 production.log
Restart you web server:
sudo service apache2 restart
now check production logs.