Doing my gitlab backup the backuped files have:
no timestamp
should be like this: The filename will be [TIMESTAMP]_gitlab_backup.tar
here the files::
root#gitlab:~# ll /mnt/backup-git/ -h
total 1.9G
-rw------- 1 git git 57M Nov 29 15:57 1480431448_gitlab_backup.tar
-rw------- 1 git git 57M Nov 29 15:57 1480431473_gitlab_backup.tar
-rw------- 1 git git 452M Nov 30 02:00 1480467623_gitlab_backup.tar
Here my configuration values for the backup::
$ grep -i backup /etc/gitlab/gitlab.rb | grep -v '^#'
gitlab_rails['backup_path'] = "/mnt/backup-git/"
gitlab_rails['backup_keep_time'] = 604800
To create them, following the documentation here, (omnibus installation):
root#gitlab:~# crontab -l | grep -v '^#'
0 2 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1
The files clearly have a timestamp already:
1480431448_gitlab_backup.tar
The bold is the unix time for the backup
Related
Using:
podman version 4.2.0
AlmaLinux 8.7
I've created an image based on redhat/ubi8 with the following Dockerfile:
FROM docker.io/redhat/ubi8
RUN dnf install -y gcc-c++ cmake python39 openssh git
RUN useradd -ms /bin/bash foobar -g users
USER foobar
WORKDIR /home/foobar/
RUN mkdir -p .ssh
$ docker build -t mount_test_image .
I run the image from a directory that contains a directory ssh, and I want to mount that directory to /home/foobar/.ssh with ownership of foobar.users
$ ls -l
-rw-r--r--. 1 host_user users 269 Dec 7 09:10 Dockerfile
drwxrwxr-x. 2 host_user users 18 Dec 2 10:41 ssh
docker run -it -d --rm --mount type=bind,src=ssh,target=/home/foobar/.ssh --name=mount_test mount_test_image
However when I enter the container via
docker exec -it mount_test '/bin/sh'
The home directory looks like this:
drwx------. 1 foobar users 18 Dec 7 17:10 .
drwxr-xr-x. 1 root root 21 Dec 7 17:10 ..
-rw-r--r--. 1 foobar users 18 Jun 20 11:31 .bash_logout
-rw-r--r--. 1 foobar users 141 Jun 20 11:31 .bash_profile
-rw-r--r--. 1 foobar users 376 Jun 20 11:31 .bashrc
drwxrwxr-x. 2 root root 18 Dec 2 18:41 .ssh
I obviously get a "permission denied" when trying to access that directory.
sh-4.4$ ls /home/foobar/.ssh
ls: cannot open directory '/home/foobar/.ssh': Permission denied
I tried changing the ownership of the directory on the host to match the uid of the container user, but then it just looks like this:
drwxrwxr-x. 2 nobody root 18 Dec 2 18:41 .ssh
My host user uid:gid is 501:100 and the container user is 1000:100. Right now I'm just trying to generate an ssh key to upload to bitbucket, but this seems like a simple feature a container should be have. All the tutorials and examples just stop after the --mount command instruction and say "there ya go!". What good is the mount point if you can't read/write it?
EDIT:
I tried on Archlinux using docker instead of podman and it works like one would expect with both -v and --mount. The owner of the mounted directory in the container matches the uid and gid of the host. Is this then a bug in podman or is it just done differently?
You are using a non-root user (foobar) in a rootless container. You must use --userns=keep-id for the container user to see the mounted volumes.
https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md#using-volumes
I cannot find a way to use the running SSH server on GH Actions.
When I try to connect to 127.0.0.1 via ssh, there is a server, and responds, but
somehow ignores the configuration files in .ssh (or whatever the case may be).
Here is what script I used (the general setup does not seem to influence the results):
ssh-keygen -t ed25519 -f ~/.ssh/whatever -N ''
cat > ~/.ssh/config <<EOF
Host host.example
User $USER
HostName 127.0.0.1
IdentityFile ~/.ssh/whatever
EOF
echo -n 'from="127.0.0.1" ' | cat - ~/.ssh/whatever.pub > ~/.ssh/authorized_keys
ssh -o 'StrictHostKeyChecking no' host.example id
I am not satisfied with the results, since I cannot reproduce the log locally
(every machine I have behaves normally, i.e. allows to execute the command).
Generating public/private ed25519 key pair.
Created directory '/home/runner/.ssh'.
Your identification has been saved in /home/runner/.ssh/whatever.
Your public key has been saved in /home/runner/.ssh/whatever.pub.
The key fingerprint is:
SHA256:2ZCprVg5rZXp0IguQlCanUVTlCX7IFt2TPTnimdk0gM runner#fv-az60
The key's randomart image is:
+--[ED25519 256]--+
| ..+o+++ |
| = o ..= + |
|+ o . = E . . |
|. * # O o |
| . o B S * . |
|. . o B = o |
|. . o o o + |
| . . o |
| |
+----[SHA256]-----+
Warning: Permanently added '127.0.0.1' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
runner#127.0.0.1: Permission denied (publickey,password).
##[error]Process completed with exit code 255.
This is a permissions issue. By default, the permissions on the home folder in the container are too broad for the ssh daemon to accept (world/others read/write), so the server-side rejects your connection. Removing world/others read/write permission on your home directory fixes ths issue.
To fix, add the following to your script, just before the ssh call. This command removes the others read/write permission on the home directory:
chmod og-rw ~
Evidence:
name: ssh-example
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Run a multi-line script
run: |
ssh-keygen -t ed25519 -f ~/.ssh/whatever -N ''
cat > ~/.ssh/config <<EOF
Host host.example
User $USER
HostName 127.0.0.1
IdentityFile ~/.ssh/whatever
EOF
echo -n 'from="127.0.0.1" ' | cat - ~/.ssh/whatever.pub > ~/.ssh/authorized_keys
echo "Before fixing permissions on authorized_keys, notice home directory is world read/write"
ls -la ~/.ssh
ssh -o 'StrictHostKeyChecking no' host.example id || echo "ssh failed as expected... trying to fix permissions"
chmod og-rw ~
echo "After fixing permissions on home folder ~ ..."
ls -la ~/.ssh
ssh -o 'StrictHostKeyChecking no' host.example id
Output from the Github Action:
Generating public/private ed25519 key pair.
Created directory '/home/runner/.ssh'.
Your identification has been saved in /home/runner/.ssh/whatever.
Your public key has been saved in /home/runner/.ssh/whatever.pub.
The key fingerprint is:
SHA256:vKl342+LK4YP7Kj00Eqm1Jnst/7ED3Pzu/6TPOiHoUc runner#fv-az76
The key's randomart image is:
+--[ED25519 256]--+
| |
| |
| |
| . |
| S |
| o.o.. o E |
| .==. o*ooo = . |
|.=.+ +ooO.==.* |
|. oo=o==.=B#Boo |
+----[SHA256]-----+
Before fixing permissions on authorized_keys, notice home directory is world read/write
total 24
drwx------ 2 runner docker 4096 Feb 23 21:58 .
drwxrwxrwx 8 runner docker 4096 Feb 23 21:58 ..
-rw-r--r-- 1 runner docker 113 Feb 23 21:58 authorized_keys
-rw-r--r-- 1 runner docker 89 Feb 23 21:58 config
-rw------- 1 runner docker 411 Feb 23 21:58 whatever
-rw-r--r-- 1 runner docker 96 Feb 23 21:58 whatever.pub
Warning: Permanently added '127.0.0.1' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
runner#127.0.0.1: Permission denied (publickey,password).
ssh failed as expected... trying to fix permissions
After fixing permissions on home folder ~ ...
total 28
drwx------ 2 runner docker 4096 Feb 23 21:58 .
drwx--x--x 8 runner docker 4096 Feb 23 21:58 ..
-rw-r--r-- 1 runner docker 113 Feb 23 21:58 authorized_keys
-rw-r--r-- 1 runner docker 89 Feb 23 21:58 config
-rw-r--r-- 1 runner docker 222 Feb 23 21:58 known_hosts
-rw------- 1 runner docker 411 Feb 23 21:58 whatever
-rw-r--r-- 1 runner docker 96 Feb 23 21:58 whatever.pub
uid=1001(runner) gid=115(docker) groups=115(docker)
Permission Denied can be caused by multiple reasons.
this is code from github repo
if (options.control_master == SSHCTL_MASTER_ASK ||
options.control_master == SSHCTL_MASTER_AUTO_ASK) {
if (!ask_permission("Allow shared connection to %s? ", host)) {
debug2("%s: session refused by user", __func__);
reply_error(reply, MUX_S_PERMISSION_DENIED, rid,
"Permission denied");
this is caused by refused connection
probable causes.
sshd daemon i.e. ssh server is not running
user has no permission to ssh.
only root has permission to ssh.
check
# systemctl status sshd.service |grep Active
also check
# cat /etc/ssh/sshd_config
I don't think .ssh permissions are issue.
as they are created by user.
user permission mask is mostly 755. which is why not having permission for same user
to its own .ssh directory is highly unlikely.
Do let me know if problem persists.
I am making a deployment script using GitLab's CD. I've got a script:
- ssh USER#HOST "cd domains/$DOMAIN/ && mkdir build-$CI_JOB_ID"
- rsync -ar --port=22 * USER#HOST :domains/$DOMAIN/build-$CI_JOB_ID
- ssh USER#HOST "cd domains/$DOMAIN/ && ln -sfv build-$CI_JOB_ID/public public_html && ls -la"
- ssh USER#HOST "cd domains/$DOMAIN/ && ls | grep '^build\-.*$' | grep -Ev '^build-$CI_JOB_ID$' | xargs rm -rf"
everything works fine but not ln command. It works only 50% of the time. Here are logs from ` jobs runnning one after another.
Job 1 with $CI_JOB_ID = 76337215 worked properly. Link is correct.
$ ssh USER#HOST "cd domains/$DOMAIN/ && ln -sfv build-$CI_JOB_ID/public public_html && ls -la"
public_html -> build-76337215/public
total 20
drwx--x--x 5 USER 1000 7 Jun 20 22:15 .
drwx--x--x 23 USER 1000 23 Jun 19 16:34 ..
-rw-r--r-- 1 USER 1000 39 Jun 17 22:12 .htaccess
drwxr-xr-x 12 USER 1000 20 Jun 20 22:07 build-76335972
drwxr-xr-x 12 USER 1000 20 Jun 20 22:14 build-76337215
drwxr-xr-x 2 USER 1000 4 Jun 20 11:48 logs
lrwxr-xr-x 1 USER 1000 21 Jun 20 22:15 public_html -> build-76337215/public
Job 2 with $CI_JOB_ID = 76339729 did not work. Link is still to old 76337215 from Job 1.
$ ssh USER#HOST "cd domains/$DOMAIN/ && ln -sfv build-$CI_JOB_ID/public public_html && ls -la"
public_html/public -> build-76339729/public
total 20
drwx--x--x 5 USER 1000 7 Jun 20 22:28 .
drwx--x--x 23 USER 1000 23 Jun 19 16:34 ..
-rw-r--r-- 1 USER 1000 39 Jun 17 22:12 .htaccess
drwxr-xr-x 12 USER 1000 20 Jun 20 22:14 build-76337215
drwxr-xr-x 12 USER 1000 20 Jun 20 22:28 build-76339729
drwxr-xr-x 2 USER 1000 4 Jun 20 11:48 logs
lrwxr-xr-x 1 USER 1000 21 Jun 20 22:15 public_html -> build-76337215/public
What I am doing wrong? Why it is not working 100% of the time?
The problem is that if public_html already exists and is a directory (or a symlink to a directory), then your ln command creates a new link in that directory, rather than replacing public_html.
Use the -T option to avoid this:
ln -sfTv build-$CI_JOB_ID/public public_html
alternately, you can use the -n option to not dereference a link
ln -sfnv build-$CI_JOB_ID/public public_html
this will replace public_html if it is a symlink and create the symlink in the subdirectory if it is a real directory (-T would give an error in the latter case).
Backup files I get with gitlab-rake are tar files how can I get tar.gz ?
Here the files::
root#gitlab:~# ll /mnt/backup-git/ -h
total 1.9G
-rw------- 1 git git 57M Nov 29 15:57 1480431448_gitlab_backup.tar
-rw------- 1 git git 57M Nov 29 15:57 1480431473_gitlab_backup.tar
-rw------- 1 git git 452M Nov 30 02:00 1480467623_gitlab_backup.tar
Here my configuration values for the backup::
$ grep -i backup /etc/gitlab/gitlab.rb | grep -v '^#'
gitlab_rails['backup_path'] = "/mnt/backup-git/"
gitlab_rails['backup_keep_time'] = 604800
To create them, following the documentation here, (omnibus installation):
root#gitlab:~# crontab -l | grep -v '^#'
0 2 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1
It doesn't really make sense to compress the gitlab backup tar files. The gitlab backup tar files are the final tarball made during the backup process and the contents are all files compressed during the backup process. You can read more here
Is there some equivalent in Mercurial to NIX soft- or hard- links to directories or files.
Basically that a file (or directory) is linked to a file "somewhere else" and follows the version of that location (Unlike a regular branch I think, where one would have to merge)
Mercurial versions soft links that are internal to the repository just great. It'll detect them, record them, and create them for you. Is there a specific use case you're looking for? The closest thing to an link that reaches outside the repository is a subrepo, which is a pointer to a specific version of another repo.
Symlinks work
(df)Ry4ans-MacBook-Air:~ ry4an$ hg init olav
(df)Ry4ans-MacBook-Air:~ ry4an$ cd olav/
(df)Ry4ans-MacBook-Air:olav ry4an$ echo this > target
(df)Ry4ans-MacBook-Air:olav ry4an$ ln -s target link
(df)Ry4ans-MacBook-Air:olav ry4an$ ls -l
total 16
lrwxr-xr-x 1 ry4an staff 6B Feb 16 19:25 link# -> target
-rw-r--r-- 1 ry4an staff 5B Feb 16 19:25 target
(df)Ry4ans-MacBook-Air:olav ry4an$ hg commit -A -m "link and its target"
adding link
adding target
(df)Ry4ans-MacBook-Air:olav ry4an$ hg log -p
changeset: 0:42a41a431661
tag: tip
user: Ry4an Brase <ry4an-hg#ry4an.org>
date: Sat Feb 16 19:26:17 2013 -0500
summary: link and its target
diff -r 000000000000 -r 42a41a431661 link
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/link Sat Feb 16 19:26:17 2013 -0500
## -0,0 +1,1 ##
+target
\ No newline at end of file
diff -r 000000000000 -r 42a41a431661 target
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/target Sat Feb 16 19:26:17 2013 -0500
## -0,0 +1,1 ##
+this
(df)Ry4ans-MacBook-Air:olav ry4an$ hg update null
0 files updated, 0 files merged, 2 files removed, 0 files unresolved
(df)Ry4ans-MacBook-Air:olav ry4an$ ls -l
(df)Ry4ans-MacBook-Air:olav ry4an$ hg update tip
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
(df)Ry4ans-MacBook-Air:olav ry4an$ ls -l
total 16
lrwxr-xr-x 1 ry4an staff 6B Feb 16 19:26 link# -> target
-rw-r--r-- 1 ry4an staff 5B Feb 16 19:26 target
But hardlinks don't
$hg commit -Am "hardlinks target"
adding link
adding target
$hg log -p
changeset: 0:ec9407634133
tag: tip
user: Chris Wesseling <chris.wesseling#cwi.nl>
date: Wed Mar 13 23:14:44 2013 +0100
summary: hardlinks target
diff -r 000000000000 -r ec9407634133 link
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/link Wed Mar 13 23:14:44 2013 +0100
## -0,0 +1,1 ##
+foo
diff -r 000000000000 -r ec9407634133 target
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/target Wed Mar 13 23:14:44 2013 +0100
## -0,0 +1,1 ##
+foo
$ls -lin
total 8
276702 -rw-r--r-- 2 1204653 5900 4 13 mrt 23:14 link
276702 -rw-r--r-- 2 1204653 5900 4 13 mrt 23:14 target
$hg update null
0 files updated, 0 files merged, 2 files removed, 0 files unresolved
$hg update tip
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ls -lin
total 8
276719 -rw-r--r-- 1 1204653 5900 4 13 mrt 23:15 link
276721 -rw-r--r-- 1 1204653 5900 4 13 mrt 23:15 target
Path auditing on *nix
On *nix systems, hg Mercurial audits symbolic links ("symlinks") for referred path security.
For example, absolute and empty paths are considered unsafe and will therefore not be added to the repository.
Mercurial developers have not documented this feature. However, the source code contains a comment with a somewhat vague explanation:
class pathauditor(object):
'''ensure that a filesystem path contains no banned components.
the following properties of a path are checked:
- ends with a directory separator
- under top-level .hg
- starts at the root of a windows drive
- contains ".."
- traverses a symlink (e.g. a/symlink_here/b)
- inside a nested repository (a callback can be used to approve
some nested repositories, e.g., subrepositories)
'''
On Windows, symbolic links are not supported for various reasons, see:
https://www.mercurial-scm.org/bts/issue1825
https://www.mercurial-scm.org/bts/issue2579