How to use dejavu for elasticsearch on local running it from opensource.appbase.io - apache

I wanted to use the wonderful dejavu software to audit a local elasticsearch instance.
Wondefull idea, but currently it looks that it's not easy as it should be. But I found a way to do it, and that's what I want to share with Community.
Basically you can run dejavu from the opensource.appbase.io but first you need to set SSL to access your local ES. Here is how I did it in less than 1 hour (saving you other 4 hours of googling/reasearch/try&fail).
This is little bit like issue https://github.com/appbaseio/dejavu/issues/106 but running it from the dejavu's website instead of local extension.
Just in case, I've tested this on Chrome and Firefox, both succesfully. I've done it all in Windows, but I suppose Linux should work because nothing described here is Windows-specific.
Bad news: the google extension fails (no idea because the error reporting does not exist)
More: I don't want to deal with Docker, that is the other "easy" way that is provided.
So this is what I did:
1.- here is the link to the main web site: https://opensource.appbase.io/dejavu/
2.- click to run it live. Here is the link: https://opensource.appbase.io/dejavu/live?default=true
It will start inmediately, but won't be able to get any data. Don't worry.
3.- point it to local, in my case was http://localhost:9200 plus my ES index document. Failed, but don't worry.
If you look at the console and network tools of your browser, dejavu looks to be trying SSL to connect with your ES. Here is where setting SSL on your ES (or in front of it) became required.
There are several ways to do this. In my case (because it was not for production) I've used Apache httpd, specifically the already packaged (portable) version of xampp, so it's just about minutes to have it downloaded and running. I donwloaded from here: https://portableapps.com/apps/development/xampp
4.- after installed Apache, open a new tab in your browser and try the SSL like https://localhost and it will give you a correct warning about the certificate. Just trust on it to check SSL is up and running. Don't worry about the warning.
5.- to set the SSL to redirect ES requests correctly, here you have these simple settings to add in apache config file, then restart apache. I took the idea from here: Elasticsearch with apache2 ssl proxy
<Proxy http://127.0.0.1:9200>
ProxySet connectiontimeout=5 timeout=90
</Proxy>
<LocationMatch "^(/_aliases|.*/_search|.*/_mapping|/_nodes|/_settings)$">
ProxyPassMatch http://127.0.0.1:9200
ProxyPassReverse http://127.0.0.1:9200
</LocationMatch>
6.- now it is required to set the cross-origin authorization at your ES. Add these lines into your elasticsearch.yml file and restart ES. The instructions comes from dejavu's github site: https://github.com/appbaseio/dejavu
http.cors.allow-origin: "https://opensource.appbase.io"
http.cors.enabled: true
http.cors.allow-headers : X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization,Access-Control-Allow-Origin
http.cors.allow-credentials: true
7.- test how is it going: get your index mapping with something like this: https://localhost/your_es_doc_index/_mapping
You will get the JSON maping of your index. Everything is working (Apache and ES)
8.- now, in your dejavu, point it to https://localhost:443 (yes, the port must be set because if not it will assume to do https://localhost:9200, believe it or not). Don't forget to set your ES index document in the "header" box ni dejavu. Now click in "Connect".
After couple of seconds, you will have the list of your types ready and accesible in dejavu

Related

Apache2 will only serve small files

I just know this is one of those questions where the solution should have been obvious and I'm going to feel like an idiot but I'm stuck.
I've set up a basic apache2 web sever under openSUSE Leap 15.1 on my LAN with a single virtual host (for the moment).
If I create a simple html file of 255 bytes or less a browser on another workstation picks it up without problem. If the file gets to anything larger than 255 bytes apache doesn't serve it. The GET request shows up on the server but nothing shows in Firefox, Konqueror at least gives me a 1b Object not found error.
I should say that running a browser on the server itself shows all these files perfectly well should I use 127.0.0.1, localhost or the server name in the URL.
Is this an Apache2 directive I've missed or something else entirely?
After setting the LogLevel to trace1 and then hunting around on the web I came across what appears to be the solution. Set the Apache directive ...
EnableSendfile Off
Apparently Apache is compiled with this set to On and the kernel sendfile doesn't do the job. Not a detailed explanation I know but I haven't followed this all the way through - I just needed to get Apache working!

How do you remove SSL encryption after letsencrypt set up forced SSL?

I chose to force https on my server when setting up let's encrypt, not realizing what a NIGHTMARE it would cause for my Node.js WebSocket server. Nothing that I've found online works to get a secure WebSocket connection up and running. (Probably because I'm using Apache for everything aside from WebSockets)
So I've decided to just disable SSL site-wide. I don't have any real use for it as my server is strictly for development anyway.
I've tried the method here But Apache reports that the site is already disabled. So I cleared the cache on my browser and even tried to visit the site incognito to no avail.
I also looked at the instructions here but NONE of my site-available contain the lines that are mentioned.
Then I saw this post but my apache2.conf file did not contain the lines he mentioned either.
I'm puling my hair out.
If anyone else has this issue, here's what I did to solve it.
I saw a configuration file named <sitenam>-le-ssl.conf in /etc/apache2/sites-enabled So I tried a2dissite <sitename>-le-ssl.conf
But Apache said "the file doesn't exist"
So I copied that file to my home directory with cp <sitename>-le-ssl.conf ~(AND YOU SHOULD TOO! just in case...)
Then I deleted that file with sudo rm <sitename>-le-ssl.conf
Then, restart apache with sudo service apache2 restart
Finally, clear the cache on your browser and visit your site. Everything should be back to normal now.
No idea why letsencrypt set up my server in such an odd way. It didn't follow ANY of the standards that others said it should.
Hope this helps someone in the future.

Forwarded Tomcat through Apache uses wrong Context path

Okay let me explain my problem really fast. I have a JEE Programm running on my tomcat server. The server has some user defined in the tomcat-users.xml When i test my programm on my local machine everything works fine.
However if i deploy the .war on my server and i want to access a Rest Endpoint i get a 401 unauthorized error. If i remove the users security check i can work fine with the program. So the URLs and server setup is correct.
I think that the problem is somehow related to the forwarding of tomcat through my apache.
So lets assume i have an apache running on http://myIp.de
then i forwarded tomcat with following apache config:
ProxyRequests off
ProxyPass /tomcat http://localhost:8181/ nocanon
ProxyPassReverse /tomcat http://localhost:8181/
so now i can reach tomcat through: http://myIp.de/tomcat
also i can "speak" to my app via: tomcat/myApp
But somehow the Authentizication now fails. And i think the problem is
somehow related to wrong context path. Because tomcat/manager
also fails to login.
Make your life easier by deploying your app under /tomcat on tomcat too. This way there's no path-translation required. Keep in mind that you'll get all the session cookies tied to a specific path and this path is not necessarily translated once forwarded to the client.
Also, sooner or later you might need
ProxyPreserveHost On
(look it up) or utilize mod_jk to preserve this header (and more information) automatically.
Edit: Following your comment, Basic Auth headers seem not to be forwarded to tomcat as well. I haven't attempted this myself, but all the places that I've looked up seem to imply that there'd be some duplication (e.g. second credentials file for Apache) - that doesn't look good. In this case I'd suggest to try out mod_jk rather than mod_proxy. You'll use the JkMount directive, rather than ProxyPass and need a workers.properties, but mod_jk is a lot better in keeping the full context of the request when forwarding to tomcat. I've had good experience with it so far and have only heard little complaints about it - largely in situations that were pretty huge and complex/complicated anyway. At least you should try if it solves your problems.

What's a simple way to serve http://localhost as https://someOtherDomain?

I run a local development web server for testing out code changes.
Often I have to test my local changes with remote services that can only connect securely to another domain.
e.g. https://external1.com will only talk to https://someOtherDomain.com, but I've got to test integration of my new code changes with https://external1.com
While I've got a setup configured that works, it seems complex, and took a bit to get setup right. It seems to me that many developers would want to do this same thing, so my question is this:
Is there an easy way to proxy my local webserver as https://someOtherDomain.com ?
EDIT: So maybe this should be asked this way - Does a command line or GUI tool exist that you can pass a local port and a domain name, and it serves your local port securely over https://someOtherDomain.com - no config or SSL cert creation required? Of course it'd be nice if the SSL cert could be replaced through configuration if need be, but by default, it'd work automatically, by using a precanned SSL cert. And even though I'm using Apache, I'm looking for a solution that actually doesnt use Apache - it uses something else. Why? Because I want this solution to work well with any other webserver that's being used by people on our team - as we all run different stacks, and I'd like to be able to let any of us securely serve our sites without having to configure each webserver individually.
Here's my current setup for taking my local webserver and serving it up at https://www.someOtherDomain.com
To test this locally, I've been:
editing my hosts file, and adding an entry to make www.someOtherDomain.com point to my local machine, which of course is running my dev server. This makes it so my local site is now available at http://www.someOtherDomain.com
127.0.0.1 www.someOtherDomain.com
Running Apache with a SSL Cert setup and mod_proxy to redirect all https requests to my local http server, thus making my site available at https://www.someOtherDomain.com. Here's my Apache config for this:
ServerName www.someOtherDomain.com
<Location /balancer-manager>
SetHandler balancer-manager
</Location>
ProxyPass /balancer-manager !
ProxyPass / balancer://mycluster/
<Proxy balancer://mycluster>
BalancerMember http://localhost route=1
</Proxy>
ProxyPass / balancer://mycluster
ProxyPassReverse / balancer://mycluster
SSLHonorCipherOrder On
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite RC4-SHA:HIGH:!ADH
# Rewrite all http requests to https
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
I run this on a mac, but am interested in solutions for linux as well. I've seen various Man in the Middle proxy's that sound like they'd work with some configuration... but I'm looking for something really simple to install and run - not just for me, but something I can tell team members about too, as we may all have to do this a lot in the future.
IMPORTANT NOTE: My local webserver isn't running on Port 80, though I've put it this way in the above example, to keep it simple. I understand port 80 on a mac is a bit special, but am very happy with solutions that work fine on all ports but port 80.
I think mitmproxy can do this for you, as least on linux and os-x. I haven't tried it myself but this question seems to show how it is done. It's still not a trivial program though.
There are however other approaches, which I have used:
The first one is the most pretty simple one, create a DNS entry for develop.mydomain.com which points to 127.0.0.1 and a single certificate for a (sub)domain where you control the DNS. Spread that certificate to all your developers. They'll still need to setup SSL themself but they don't need to generate certificates anymore. It has the added benefit that everybody is developing against https://develop.mydomain.com which allows them to share the configuration.
For bonus points, create a DNS entry for *.develop.mydomain.com and a wildcard certificate and your developers can have different sites (e.g. https://project1.develop.mydomain.com and https://project2.develop.mydomain.com) on there local machine. (Contrary to what the internet sometimes tells you, name based virtual hosting works fine with SSL as long as the certificate is valid for all the named hosts). Since the domain is the same for everybody you can consider getting a valid wildcard certificate to get rid of the warnings.
Unlike the solutions below this works even outside of the office network, which may be relevant when people are working from home or at the customer.
Building on this you could also create DNS entries for the internal IP's of the developer machines (if those are fixed). This does add some work, but it means the ongoing work of a developer can be reached by others in the local network, which can be very convenient for demo's, testing on mobile devices, etc.
An other option is to configure a single machine to proxy for all your developers. Create a DNS entry pointing to the internal IP of this box on something like *.develop.mydomain.com, a matching wildcard certificate and configure that box once with the correct certificate. Now you can create a virtual host for each proxied server, and again, all sites will be reachable throughout the local network, but it does require the developer to have fixed ip addresses (or hostnames added to DNS through DHCP).
Combined with the ability of apache to include all files in a directory in it's configuration makes it trivial to create a script which adds a new site based on a template. All it has to do is write a new file based on the requested subdomain plus the destination and reload the apache config. This means something like a simple PHP script can do what you want the application to do.
If you want to expose your web on your local machine to the internet try Runscope Passageway, it's easy to setup and "just works" (from experience).
Another alternative is ngrok which I also used, but it didn't always work for me.

Neo4j not functional through apache proxy

I am able to run neo4j fine through port 7474 on my server including cypher queries. Though when I access neo4j through the apache proxy it will load just fine but any requests done through cypher will only return an "Unknown error". I have other proxies such as rstudio running just fine.
I have tried the default values on the neo4j website for proxy configuration with no success. I am at a loss for what to try. Please let me know for more information needed, or how I can get additional information on the cypher error.
I tried the sample Query:
CREATE (n {name:"World"}) RETURN "hello", n.name
And this returns "Unknown error" when done through the proxy, but when done through port 7474 it works fine
This is a Linux Ubuntu LTR 12.04.4 machine.
Neo4j 2.1.1
Apache 2.2.22
Sorry if this is vague but I have not found any help for this issue nor do I know what additional information would be relevant.
Thank you.
Update:
It now works with the case provided by Stefan (Thank you!). But I am unsure how to change it from being on the root of my domain to "/database/" in your example you can change it to "/neo4j" How would I change the other parts of this config file for this to function?
As it looks now (non functional with change of proxy from "/"):
ProxyPass /database/ http://localhost:7474/
ProxyPassReverse /database/ http://localhost:7474/
RedirectMatch permanent ^/database /database/
<Location /db/manage>
AddOutputFilterByType SUBSTITUTE application/json
Substitute "s|http://localhost:7474|http://localhost:8080|n"
</Location>
I tried to change the substitute rule from "localhost:8080" to "localhost:8080/database" and to "/database" to no avail.
In closing what worked is to make it a subdomain and still have it on the root. Not sure why this has to be the case, but it is functional. Thank you again Stefan!
Some time ago I've setup a example config for using mod_proxy and mod_substitute, see https://github.com/sarmbruster/vagrant_neo4j_modproxy. See esp the Apache config file.
Be aware that mod_substitute will not work with huge responses > 1M.