I am making a deployment script using GitLab's CD. I've got a script:
- ssh USER#HOST "cd domains/$DOMAIN/ && mkdir build-$CI_JOB_ID"
- rsync -ar --port=22 * USER#HOST :domains/$DOMAIN/build-$CI_JOB_ID
- ssh USER#HOST "cd domains/$DOMAIN/ && ln -sfv build-$CI_JOB_ID/public public_html && ls -la"
- ssh USER#HOST "cd domains/$DOMAIN/ && ls | grep '^build\-.*$' | grep -Ev '^build-$CI_JOB_ID$' | xargs rm -rf"
everything works fine but not ln command. It works only 50% of the time. Here are logs from ` jobs runnning one after another.
Job 1 with $CI_JOB_ID = 76337215 worked properly. Link is correct.
$ ssh USER#HOST "cd domains/$DOMAIN/ && ln -sfv build-$CI_JOB_ID/public public_html && ls -la"
public_html -> build-76337215/public
total 20
drwx--x--x 5 USER 1000 7 Jun 20 22:15 .
drwx--x--x 23 USER 1000 23 Jun 19 16:34 ..
-rw-r--r-- 1 USER 1000 39 Jun 17 22:12 .htaccess
drwxr-xr-x 12 USER 1000 20 Jun 20 22:07 build-76335972
drwxr-xr-x 12 USER 1000 20 Jun 20 22:14 build-76337215
drwxr-xr-x 2 USER 1000 4 Jun 20 11:48 logs
lrwxr-xr-x 1 USER 1000 21 Jun 20 22:15 public_html -> build-76337215/public
Job 2 with $CI_JOB_ID = 76339729 did not work. Link is still to old 76337215 from Job 1.
$ ssh USER#HOST "cd domains/$DOMAIN/ && ln -sfv build-$CI_JOB_ID/public public_html && ls -la"
public_html/public -> build-76339729/public
total 20
drwx--x--x 5 USER 1000 7 Jun 20 22:28 .
drwx--x--x 23 USER 1000 23 Jun 19 16:34 ..
-rw-r--r-- 1 USER 1000 39 Jun 17 22:12 .htaccess
drwxr-xr-x 12 USER 1000 20 Jun 20 22:14 build-76337215
drwxr-xr-x 12 USER 1000 20 Jun 20 22:28 build-76339729
drwxr-xr-x 2 USER 1000 4 Jun 20 11:48 logs
lrwxr-xr-x 1 USER 1000 21 Jun 20 22:15 public_html -> build-76337215/public
What I am doing wrong? Why it is not working 100% of the time?
The problem is that if public_html already exists and is a directory (or a symlink to a directory), then your ln command creates a new link in that directory, rather than replacing public_html.
Use the -T option to avoid this:
ln -sfTv build-$CI_JOB_ID/public public_html
alternately, you can use the -n option to not dereference a link
ln -sfnv build-$CI_JOB_ID/public public_html
this will replace public_html if it is a symlink and create the symlink in the subdirectory if it is a real directory (-T would give an error in the latter case).
Related
I'm setting up a new website on a new Ubuntu droplet at Digital Ocean. I set this up previously on a different droplet and rsynced the website contents to it. I followed directions I found on the web, but didn't keep detailed-enough notes. I'm just trying to replicate this on the new Droplet. But I've messed up somehow. My knowledge of permissions, etc. is rudimentary :-(
My server has: /var/www/html
This is what I did:
• sudo usermod -a -G www-data [myusername]
• sudo chown -R www-data:www-data /var/www
• sudo chmod -R g+rw /var/www
• sudo chmod -R g+rws /var/www
I then created “/var/www/howardmann.us/public_html” for my website.
(I didn't put the public_html directory in var/www/html. I didn't on my other Droplet. Perhaps I should.)
sudo ls -la /var/www
total 16
drwxrwsr-x 4 www-data www-data 4096 Feb 25 15:43 .
drwxr-xr-x 14 root root 4096 Feb 24 14:27 ..
drwxr-sr-x 3 root www-data 4096 Feb 25 15:43 howardmann.us
drwxrwsr-x 2 www-data www-data 4096 Feb 25 15:42 html
sudo ls -la /var/www/howardmann.us/public_html
total 8
drwxr-sr-x 2 root www-data 4096 Feb 25 15:43 .
drwxr-sr-x 3 root www-data 4096 Feb 25 15:43 ..
Now, excerpts of rsync script (successful to my other Droplet) result from my laptop now:
1.
howardm$ ./deploy.sh
sending incremental file list
rsync: failed to set times on "/var/www/howardmann.us/public_html/.": Operation not permitted (1)
rsync: recv_generator: mkdir "/var/www/howardmann.us/public_html/css" failed: Permission denied (13)
* Skipping any contents from this failed directory *
3.
rsync: mkstemp "/var/www/howardmann.us/public_html/.DS_Store.8zaaQg" failed: Permission denied (13)
Question: How do I fix ownership and permissions on the server directories for 1) proper permissions for a website and 2) permit a rsync deploy?
For comparison, here is the status of my current working web site on the other Droplet:
howardm#howardmann:~$ ls -la /var/www/
total 20
drwxrwxr-x 5 howardm www-data 4096 Mar 5 2016 .
drwxr-xr-x 13 root root 4096 Mar 28 2015 ..
drwxrwxr-x 4 howardm www-data 4096 Jun 8 2016 howardmann.us
drwxrwxr-x 2 howardm www-data 4096 Mar 29 2015 html
Thanks!
Howard
Well, I fixed this after reading some more about permissions for a website.
My website folder is titled howardmann.us, which contains a public_html folder with the website contents.
So: /var/www/howardmann.us/public_html. (/var/www also contains a html directory)
I achieved the relevant ownership/permissions with:
sudo chown -R howardm:www-data /var/www/
I'm the owner, and www-data the group.
sudo usermod -a -G www-data [myusersername]
I'm a member of the www-data group
sudo chmod -R 2775 /var/www/
This yields the proper permissions, I believe for a web site.
I made sure the same owner:group applied to all these directories:
ls -l /var/www
total 8
drwxr-sr-x 3 howardm www-data 4096 Feb 26 19:23 howardmann.us
drwxrwsr-x 2 howardm www-data 4096 Feb 25 15:42 html
ls -l /var/www/howardmann.us
total 4
drwxr-xr-x 11 howardm www-data 4096 Feb 5 2017 public_html
Now rsync works without errors.
If anyone believes these ownership/permissions for a website are (substantially) improper, please comment.
Howard
here is my system information:
$ apachectl -v
Server version: Apache/2.4.18 (Ubuntu)
Server built: 2016-07-14T12:32:26
$ cat /etc/*release* | grep -i dist
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
$ uname -r
4.4.0-57-generic
i wish to install mod_maxminddb. i installed geoipupdate as a prerequisites and configured it to include not only geolite database, but either commercial one (using this)
$ ls -l /etc/GeoIP.conf
-rw-r--r-- 1 root root 818 Dec 24 18:29 /etc/GeoIP.conf
$ ls -l /usr/share/GeoIP/
total 0
$ geoipupdate
$ ls -l /usr/share/GeoIP/
total 187444
-rw-r--r-- 1 root root 112192399 Dec 24 18:46 GeoIP2-City.mmdb
-rw-r--r-- 1 root root 3012279 Dec 24 18:46 GeoIP2-Country.mmdb
-rw-r--r-- 1 root root 47721533 Dec 24 18:46 GeoIPCity.dat
-rw-r--r-- 1 root root 1699494 Dec 24 18:45 GeoIP.dat
-rw-r--r-- 1 root root 4189407 Dec 24 18:45 GeoIPISP.dat
-rw-r--r-- 1 root root 4299547 Dec 24 18:45 GeoLiteASNum.dat
-rw-r--r-- 1 root root 17760694 Dec 24 18:45 GeoLiteCity.dat
-rw-r--r-- 1 root root 1054583 Dec 24 18:45 GeoLiteCountry.dat
then complied and installed the mod_maxminddb, and when apache is configured to use the commercial databases, the following error is thrown
$ apachectl -M
AH00526: Syntax error on line 12 of /etc/apache2/mods-enabled/maxminddb.conf:
MaxMindDBFile: Failed to open /usr/share/GeoIP/GeoIPCity.dat: The MaxMind DB file contains invalid metadata
but that is not the case when apache is configured to use the geolite databases.
any ideas?
You didn't include your Apache config, but you appear to be trying to use mod_maxminddb with GeoIPCity.dat (GeoIP Legacy). It only works with GeoIP2. Adjust line 12 of maxminddb.conf to use GeoIP2-City.mmdb instead.
Doing my gitlab backup the backuped files have:
no timestamp
should be like this: The filename will be [TIMESTAMP]_gitlab_backup.tar
here the files::
root#gitlab:~# ll /mnt/backup-git/ -h
total 1.9G
-rw------- 1 git git 57M Nov 29 15:57 1480431448_gitlab_backup.tar
-rw------- 1 git git 57M Nov 29 15:57 1480431473_gitlab_backup.tar
-rw------- 1 git git 452M Nov 30 02:00 1480467623_gitlab_backup.tar
Here my configuration values for the backup::
$ grep -i backup /etc/gitlab/gitlab.rb | grep -v '^#'
gitlab_rails['backup_path'] = "/mnt/backup-git/"
gitlab_rails['backup_keep_time'] = 604800
To create them, following the documentation here, (omnibus installation):
root#gitlab:~# crontab -l | grep -v '^#'
0 2 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1
The files clearly have a timestamp already:
1480431448_gitlab_backup.tar
The bold is the unix time for the backup
I'm trying to get user directories to work with Lighttpd on Arch Linux. But after creating the public_html directory, placing an index.html file in it, setting permissions, configuring Lighttpd to use the user directory module, and restarting lighttpd, it still gives 404 for one specific user (yet another one works).
Here are my configuration files:
$ cat /etc/lighttpd/lighttpd.conf
# This is a minimal example config
# See /usr/share/doc/lighttpd
# and http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs:ConfigurationOptions
server.port = 80
server.username = "http"
server.groupname = "http"
server.document-root = "/srv/http"
server.errorlog = "/var/log/lighttpd/error.log"
dir-listing.activate = "enable"
index-file.names = ( "index.html" )
mimetype.assign = (
".html" => "text/html",
".txt" => "text/plain",
".css" => "text/css",
".js" => "application/x-javascript",
".jpg" => "image/jpeg",
".jpeg" => "image/jpeg",
".gif" => "image/gif",
".png" => "image/png",
"" => "application/octet-stream"
)
include "conf.d/userdir.conf"
include "conf.d/cgi.conf"
$ cat /etc/lighttpd/conf.d/userdir.conf
server.modules += ( "mod_userdir" )
userdir.path = "public_html"
This is what things look like for user aardbei:
$ cat /home/aardbei/public_html/index.html
doot doot
$ ls -ld /home/aardbei/public_html
drwxrwxrwx 2 aardbei aardbei 4096 Mar 27 13:10 /home/aardbei/public_html
$ ls -ld /home/aardbei/public_html/index.html
-rwxrwxrwx 1 aardbei aardbei 37 Mar 27 13:11 /home/aardbei/public_html/index.html
But even after restarting the server with sudo systemctl restart lighttpd I still get 404 at URI /~aardbei/index.html and not what I should get: "doot doot"
However, this is what things look like for the user madeline:
$ cat /home/madeline/public_html/index.html
blah blah blah
$ ls -ld /home/madeline/public_html/
drwxrwxrwx 19 madeline madeline 4096 Mar 27 13:33 /home/madeline/public_html/
$ ls -ld /home/madeline/public_html/index.html
-rw-r--r-- 1 madeline madeline 15 Mar 27 13:33 /home/madeline/public_html/index.html
So the important parts are the same. And yet going to URI /~madeline/index.html does what it should do: it shows "blah blah blah"
Nothing looks relevant here, but here are groups for the two users:
$ groups madeline
wheel video audio wireshark madeline
$ groups aardbei
wheel aardbei
What is going on? Why doesn't the user directory for the user aardbei work in Lighttpd?
Following the instructions on the Arch Linux wiki for Apache worked: https://wiki.archlinux.org/index.php/Apache_HTTP_Server#User_directories
$ chmod o+x /home/aardbei
$ chmod o+x /home/aardbei/public_html
$ chmod -R o+r /home/aardbei/public_html
I'm still not sure I understand the permissions at play here, but it solves my problem.
The reason that the accepted answer works is due to the user that the web server is running as requiring access to the user's home directory in order to access their public_html.
Granting o+x allows other users to access a file and/or sub-directory within but not allow them to list the contents of a directory. Basically they can pass through it so long as they know what they're looking for, public_html, but they cannot get a list of the contents otherwise.
Example
Here's my home directory:
$ ls -dl /home/sam
drwx-----x. 3 sam sam 4096 Nov 3 11:08 /home/sam
$ ls -dl /home/sam/public_html
drwxr-xr-x. 2 sam users 4096 Nov 3 11:09 /home/sam/public_html
Now as the user of the web server, lighttpd, cannot list the contents of my home directory:
$ sudo -u lighttpd ls /home/sam
ls: cannot open directory /home/sam: Permission denied
But can see a specific directory if they happen to know its name:
$ ls -dl /home/sam/public_html
drwxr-xr-x. 2 sam users 4096 Nov 3 11:09 /home/sam/public_html
Here's another directory that the web server can see too:
$ sudo -u lighttpd ls -ld /home/sam/someotherdir
drwx------. 2 sam users 4096 Nov 3 11:22 /home/sam/someotherdir
And files within the public_html are visible as well:
$ ls -dl /home/sam/public_html/index.html
-rw-r--r--. 1 sam users 3 Nov 3 11:09 /home/sam/public_html/index.html
Normal permissions apply here, so if you don't want the web server to see something, make it readonly to your user & groups but not everyone else (others).
I'm using Docker to run an Apache instance. My docker file goes something like this:
FROM ubuntu
MAINTAINER your.face#gmail.com
RUN cat /etc/passwd
RUN cat /etc/group
RUN apt-get update && apt-get install -yq apache2 php5 libapache2-mod-php5 php5-mysql
RUN apt-get install -yq openssh-server
RUN mkdir /var/run/sshd
ENV APACHE_RUN_USER www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR /var/log/apache2
EXPOSE 80
ADD config/apache2/000-default.conf /etc/apache2/sites-available/000-default.conf
ADD config/php5/php.ini /etc/php5/apache2/php.ini
ADD config/start.sh /tmp/start.sh
ADD src /var/www
RUN chown -R root:www-data /var/www
RUN chmod u+rwx,g+rx,o+rx /var/www
RUN find /var/www -type d -exec chmod u+rwx,g+rx,o+rx {} +
RUN find /var/www -type f -exec chmod u+rw,g+rw,o+r {} +
#essentially: CMD ["/usr/sbin/apache2ctl", "-D", "FOREGROUND"]
CMD ["/tmp/start.sh"]
However, when I build the container and run it, I only ever get 403 errors.
Notice that I've specified that Apache should run as www-data in www-data group, and that /var/www has been recursively chownd to belong to root:www-data.
Also, all directories are searchable and readable, and all files are readable and writeable by the www-data group (well, according to ls -la and namei -m they are anyways).
How do I fix these permissions issues? I cant figure it out.
Actual error from the Apache error.log:
[Fri May 23 18:33:27.663087 2014] [core:error] [pid 14] (13)Permission denied: [client 11.11.11.11:61689] AH00035: access to /index.php denied (filesystem path '/var/www/index.php') because search permissions are missing on a component of the path
EDIT:
output of ls -laR /var/www at the end of the Dockerfile:
Step 21 : RUN ls -laR /var/www
---> Running in 74fd3609dfc8
/var/www:
total 1036
drwxr-xr-x 67 root www-data 4096 May 23 18:38 .
drwxr-xr-x 26 root root 4096 May 23 18:38 ..
-rw-rw-r-- 1 root www-data 28 May 23 12:22 .gitignore
-rw-rw-r-- 1 root www-data 501 May 23 12:22 .htaccess
-rw-rw-r-- 1 root www-data 7566 May 23 12:22 index.php
Output of namei -m /var/www/index.php at the end of the Dockerfile:
Step 22 : RUN namei -m /var/www/index.php
---> Running in 1203f0353090
f: /var/www/index.php
drwxr-xr-x /
drwxr-xr-x var
drwxr-xr-x www
-rw-rw-r-- index.php
EDIT2
After trying a whole bunch of things, including chmod -R 777 just to see if I could get anything to work, I tried putting the source files added from the Dockerfile into /var/www/html, the default location for Apache files to be served.
I matched the default file permissions exactly (I think), and it still isn't working. The default index.html that comes with Apache loads just fine, but the added src folder still have a 403 access denied error.
I changed the Dockerfile to ADD src /var/www/html/src and the permissions were set using:
RUN find /var/www/html -type d -exec chmod u+rwx,g+rx,o+rx {} +
RUN find /var/www/html -type f -exec chmod u+rw,g+r,o+r {} +
No luck. Below is some of the output of ls -laR on /var/www. Notice that the permissions for the html folder and index.html that come with an apache2 install match those of the added src folder:
Step 19 : RUN ls -laR /var/www/
---> Running in 0520950d0426
/var/www/:
total 12
drwxr-xr-x 6 root root 4096 May 23 19:23 .
drwxr-xr-x 24 root root 4096 May 23 19:23 ..
drwxr-xr-x 5 root root 4096 May 23 19:23 html
/var/www/html:
total 24
drwxr-xr-x 5 root root 4096 May 23 19:23 .
drwxr-xr-x 6 root root 4096 May 23 19:23 ..
-rw-r--r-- 1 root root 11510 May 23 18:28 index.html
drwxr-xr-x 47 root root 4096 May 23 19:23 src
/var/www/html/src:
total 1032
drwxr-xr-x 47 root root 4096 May 23 19:23 .
drwxr-xr-x 5 root root 4096 May 23 19:23 ..
-rw-r--r-- 1 root root 28 May 23 12:22 .gitignore
-rw-r--r-- 1 root root 501 May 23 12:22 .htaccess
-rw-r--r-- 1 root root 7566 May 23 12:22 index.php
Perhaps chmod doesn't work quite the way I thought it does??
EDIT3
A final bit of information. The Docker container is being built by buildbot, which I've been assuming runs as root. I haven't been able to reproduce this scenario without using buildbot to do the building.
Building everything via sudo docker build -t apache . type commands on my laptop works fine, but the problems arise when buildbot does it. No idea why :^/
I just ran into this after posting a similar question at Running app inside Docker as non-root user.
My guess is you can't chmod/ chown files that were added via the ADD command. – thom_nic Jun 19 at 14:14
Actually you can. You just need to issue a a RUN command after the ADD for the file location that will be INSIDE your container. For example
ADD extras/dockerstart.sh /usr/local/servicemix/bin/
RUN chmod 755 /usr/local/bin/dockerstart.sh
Hope that helps. It worked for me.
I encountered a similar issue; however my container was using VOLUME to map directories across the container.
Changing the permissions on the directory that maps to /var/www/html itself remedied the 403 Forbidden errors.
docker-host$ ls -ld /var/www/html
drwxr--r-- 53 me staff 1802 Mar 8 22:33 .
docker-host$ chmod a+x /var/www/html
docker-host$ ls -ld /var/www/html
drwxr-xr-x 53 me staff 1802 Mar 8 22:33 .
Note that chmod must be applied on the Docker host, not within the container. Executing it within the container effects no change to the directory.
docker-container$ chmod a+x /var/www/html
docker-container$ ls -ld /var/www/html
drwxr--r-- 53 me staff 1802 Mar 8 22:33 .