Where to put robots.txt on my web server? - robot

I have 2 domains that points to my webserver, see below:
domain1.com --> /var/www/html/live
domain2.com --> /var/www/html/dev
My robots.txt file
User-agent: *
Disallow: /dev/
Now I want domain2.com not to be indexed by search engines, where should I put my robots.txt?

Related

Apache virtualhost with a domain

I have an apache server with a bought domain.
I want to know if it is possible to redirect some web pages... For example
I have a NextCloud Server that I want to access by www.example.com/nextcloud
And a plex server I want to access by www.example.com/plex
PD: I don't have the possibilities of subdomains like www.plex.example.com because I didn't hire it when I bought the domain
Is this possible? How do I need to configure apache virtualhost? Thanks!
You mention that you want to access by www.example.com/nextcloud and by www.example.com/plex. I will therefore take for granted that you do not want the site address to change in your client's browser. So no redirection here. Redirection would change the address bar value.
Then the option you want is a reverse proxy. It will "hide" the fact that the client is being served pages by another site or application.
Assumptions:
You have system 1 with an Apache server that responds to http://www.example.com
You have system 2 with an application that responds to http://www.domain1.com/nextcloud.
You have system 3 with an a plex application that reponds to http://www.domain2.com/plex
Therefore on system 1, in the configuration file for your Apache (most probably httpd.conf), you will:
load the proxy modules
add these lines in your <VirtualHost>:
<VirtualHost *:80>
ServerName www.example.com
ServerAlias example.com
[... SOME OTHER CONFIG ...]
ProxyPass "/nextcloud" "http://www.domain1.com/nextcloud"
ProxyPassReverse "/nextcloud" "http://www.domain1.com/nextcloud"
ProxyPass "/plex "http://www.domain2.com/plex"
ProxyPassReverse "/plex" "http://www.domain2.com/plex"
[... SOME OTHER CONFIG ...]
</VirtualHost>
Now domain1.com and domain2.com can be IP addresses, but using dns is so much better for flexibility. Adjust this sample as required.
Complete mod_proxy documentation: https://httpd.apache.org/docs/2.4/mod/mod_proxy.html

How to make a Permanent redirect to an internal ip address with apache in Ubuntu?

I have the following problem: I want to redirect any request received on the external ip lets say 192.168.x.x, to an internal ip of the same machine, lets say 172.0.0.1. The internal ip has been associated to a name editing the hosts file as follows:
$/etc/hosts
...
...
172.0.0.1 www.example.com
I want to have www.example.com in the HTTP_HOST field of the request header unregarding to which external ip has been requested (In order to have django sites working properly without adding the current ip every time to the site list). Consequently I tried the following:
$/etc/apache2/sites-enabled/000-default.conf
<VirtualHost *:443>
ServerName entrypoint.com
Redirect 301 / https://www.example.com:443
</VirtualHost>
<VirtualHost 172.0.0.1:443>
ServerName www.example.com
...
...
</VirtualHost>
When I try to connect from a browser the redirect seems to look for a web domain instead of the local ip. How can I fix this?

How to protect my server from false proxy requests

I have a website, call it
http://www.example.com
Users create a site for themselves and they each get a sub-domain automatically, so something like
http://user1.example.com
http://user2.example.com
In addition, some users can use their own domain name, by creating a A-Record in their domain name DNS setup to point to my server's IP address.
http://www.myownname.com > 10.10.10.10 (my site's server IP address)
In my Apache VirtualHost, it's set up as follows
<VirtualHost 10.10.10.10:80>
ServerName www.example.com
ServerAlias *
UseCanonicalName Off
DirectoryIndex index.php index.html
DocumentRoot /home/public/
<Directory /home/public/>
Require all granted
Allowoverride ALL
</Directory>
</VirtualHost>
Line "ServerAlias *" is there so that I can catch any domain name pointing to my IP address and then handle it via my site.
Everything is working well up to this point.
Now, recently, I've checked my Apache access log and I see strange access such as follows
10.10.10.10 - - [25/Jul/2014:12:48:04 -0700] "GET http://www.some-random-site.com/ HTTP/1.0" 200 1456
It looks like someone is trying to use my server as a proxy to access other sites. I've read quite a few pages in order to block this proxy requests (e.g. https://wiki.apache.org/httpd/ProxyAbuse)
For example, if I use "default virtual host" to block everything except what's specified in my virtual host, all the custom domain names pointing to my server's IP address doesn't work anymore. But it does block the proxy requests.
How can I block proxy requests, while still being able to allow custom domain names via A-Record (or CNAME) pointing to my IP address?

Domain and subdomain wildcards on Apache as dynamic subfolders

I'm creating website builder like weebly and wix format. I would like to dynamic subfolders. For example:
domain.com /public_html/sites/domain.com/
sub1.root.com /public_html/sites/sub1/
test.com /public_html/sites/test.com/
sub2.root.com /public_html/sites/sub2/
I can do only static with following and forced me to add one by one whole websites.
<VirtualHost *:80>
DocumentRoot "/public_html/sites/domain.com"
ServerName domain.com
ServerAlias *.domain.com
</VirtualHost>
How can i do dynamic both domains that client owner and subdomains that i'll give user.
For dynamic domain name configuration based on the names of folders you create in
/public_html/sites/, you can use the ServerAlias and VirtualDocumentRoot directives.
Add something like this in your apache virtualhost configuration file:
<VirtualHost *:80>
...
ServerAlias *
VirtualDocumentRoot /public_html/sites/%0
...
</VirtualHost>
To understand why %0 is used, consider the domain m.rate.movies.net split into parts
Index: %1 %2 %3 %4
Domain: m rate movies net
Negative Index: %-4 %-3 %-2 %-1
The one I chose to use, %0 represents the entire domain name - m.rate.movies.net. This allows you to support domain names that can have varying number of parts separated by dots. This makes sub-directories created inside /public_html/sites/ be the document root for any domain name pointed to your server's IP.
/public_html/sites/domain.com --> http://domain.com/
/public_html/sites/blog.writers.org --> http://blog.writers.org/
/public_html/sites/m.rate.movies.net --> http://m.rate.movies.net/
If you used something like VirtualDocumentRoot /public_html/sites/%1
You will have to create the sub-directories this way
/public_html/sites/domain --> http://domain.com/
/public_html/sites/blog --> http://blog.writers.org/
/public_html/sites/m --> http://m.rate.movies.net/
You can also use a combination: VirtualDocumentRoot /public_html/sites/%-2\.%-1
/public_html/sites/domain.com --> http://domain.com/
/public_html/sites/writers.org --> http://blog.writers.org/
/public_html/sites/movies.net --> http://m.rate.movies.net/
Yet another way: VirtualDocumentRoot /public_html/sites/%-1/%-2/
/public_html/sites/com/domain --> http://domain.com/
/public_html/sites/org/writers --> http://blog.writers.org/
/public_html/sites/net/movies --> http://m.rate.movies.net/
With these setup, you won't have to edit the virtual host configuration every time you add a new domain. You won't be able to use separate log files for each domain, but for that, you can use a custom log format that includes the host name.

Apache named-based VirtualHost configuration for multiple sites in one host

My problem is simple and i think the solution also.
I search the forums about 2-3 hours and didnt my answer... -_-' .
I got "space" in a host provider. So they gave me a "www" directory.
I have multiple folders in this directory so it's like:
www--
|
/Folder1
/Folder2
/Folder3 etc.etc.....
Each folder represents a website.
So i want to redirect each website to each folder e.g. :
www.example1.com -> Folder1
www.example2.com -> Folder2
www.example3.com -> Folder3 etc....
-BUT- without showing the subfolder of each ...
So this is NOT a solution: www.example1.com/Folder1
This IS a solution: www.example1.com .
How can i modify my .htaccess file in root WWW ?
Thanks in advance !
This has to be done through your http server configuration (like apache).
VirtualHosts is how you split multiple domains across one computer, and their respective .htaccess configuration will give you protection for your site (username/password).
If you just want to split the computer into multiple hosts, you don't need to use .htaccess.
See the following links
Here are some VirtualHost Configuration Examples
Apache HTTP Server Tutorial
Server Config Files
Excerpt from the "example's" link on "Name-Based Virtual Hosts":
<VirtualHost 172.20.30.50>
DocumentRoot /www/example1
ServerName www.example.com
# Other directives here ...
</VirtualHost>
<VirtualHost 172.20.30.50>
DocumentRoot /www/example2
ServerName www.example.org
# Other directives here ...
</VirtualHost>
The configuration file is usually located in /etc/apache2/apache2.conf which then reads from /etc/apache2/sites-enabled/*.conf which are essentially symbolic links to /etc/apache2/sites-available/*.conf - those files are used with the above code to accomplish the results you describe.
You may also be interested looking into nginx, an alternative to apache2.
.htaccess is not what you're looking for. Instead, read up on Name-based Virtual Host Support. You will need to add some entries to your httpd.conf file to point each domain to the correct folder in the www directory. It should be pretty straight forward.
http://httpd.apache.org/docs/2.2/vhosts/name-based.html