I am currently investigating the ELK (Elasticsearch, Logstash, Kibana) stack for centralized log file analysis.
The plan is to store logs of multiple applications in the same Elasticsearch cluster using logstash and day-based indexes.
All documents contain a field called application, e.g."application": "superapp".
Now we are looking for a way to implement access control like this:
A) Superuser: is able to see log entries of all applications.
B) Developer: can only see log entries of the applications he is allowed to. For example the dev team for application "superapp" should only be able to see the entries for this application.
To wrap it up: we need access control based on the value in the field application.
While reading the documentation for Elastisearch and Shield I could not find an obvious way to do it.
Any ideas how we could realize this in a way that would also work with Kibana 3 and 4?
My first idea was to use aliases which are being automatically assigned to documents using index templates. I am wondering if this is the right direction.
I asked this question here on the elasticsearch Google Group and got this reply:
"You can separate out the different types of logs into their own indices which would make things much easier, you could also setup an alias with a filter and then provide access to that alias to certain users.
Currently KB isn't multi-tenanted but it is a feature that is going to be added, you'd have to setup multiple instances with each going to their own alias."
To sum it up: multi-tenancy needs to addressed at the frontend (Kibana) and the backend (Elasticsearch).
Frontend: Use Proxies for Kibana
https://github.com/salyh/elastic-defender
https://github.com/fangli/kibana-authentication-proxy
Backend: Several approaches using filtered alias and alias templates
Limiting Indexes and Operations
Faking Index per User with Aliases -
http://engineering.aweber.com/using-elasticsearchs-aliases/
http://opennomad.com/content/controlling-access-elasticsearch-filtered-aliases-nginx-and-tokens
Related
I dont have much experience with elk stack I basically only know the basics.
Something i.e. filebeat gets data and sends it to logstash
Logstash processes it and sends it Elastic search
Kibana uses elastic search to visualise data
(I hope that thats correct)
I need to create an elk system where data from three different projects is passed, stored and visualised.
Project no1. Uses MongoDB and I need to get all the information from 1 table into kibana
Project no2. Also uses MongoDB and I need to get all the information from 1 table into kibana
Project no3. Uses mysql and I need to get a few tables from that database into kibana
All three of these projects are on the same server
The thing is for Projects 1 and 2 I need the data flow to be constant (i.e. if a user registers I can see that in kabana)
But for Project no3. I only need the data when I need to generate a report (this project functions as a BI of sorts)
So my question is how does one go about creating an elk architecture that gets the inputs from these 3 sources and is able to combine into one elk project.
My best guess is :
Project No1 -> filebeat -> logstash
Project No2 -> filebeat -> logstash
Project No3 -> logstash
(logstash here being a single instance that then feeds into elastic)
Would this be a realistic approach?
I also stumbled upon redis, and from the looks of it it looks like it can combine all the data sources into one and then feed the output to logstash.
What would be the better approach?
Finally, I mentioned filebeat, but from what I understand it basically reads the data from a log file. Would that mean that I would have to re-write all my database entries into a log file in order to feed them into logstash or can logstash tap into the DB without an intermediary.
I tried looking for all of this online, but for some reason the internet is a bit scarce on ELK stack beginner questions.
Thanks
filebeat is used for shipping logs to logstash, you can't use it for reading items from DB. But you can read from DB using logstash's input plugins.
From what you're describing you'll need a logstash instance with 3 pipelines (one per project)
For project 3 you can use Logstash JDBC input plugin to connect to your mysql DB and read new/updated lines based on some "last_updated" column.
JDBC input plugin has a cron confguration value, that allows you to set it up to run periodically and read updated lines with an SQL query that you define in configuration.
For projects 1-2 you can also use the JDBC input plugin with mongoDB.
There is also an Open Source implementation for a mongoDB input plugin on git. You can check this post for how to use it here.
(see the full list of input plugins here)
If that works for you and you manage to set it up, then the rest will be about the same for all three configurations.
i.e. using filter plugins to modify data, and Elasticsearch output plugin to push data to an elastic index.
I would like to have two separate Parse servers (configured with a different app ID) connect to the same mongodb, so they can see the same set of users, so that I can create 2 different apps that share the same userbase.
Is this something Parse would support? Are there any expected conflicts or config caveats? I was unable to find info about this on Parse's github..
thanks
There's nothing to do, besides setting the database URL option to the same value on both servers, and that your database is accessible from both servers.
I'm not sure why you would need two different applicationId's as you want the same data and likely, logic running on both apps.
No, Parse Server does not support sharing classes between applications.
What you could do is have one of the instances or maybe a third one handle authentication and store your user information. I am pretty sure this would mean you will have to manually set user info on your requests and objects to save on the other two instances.
Another option is for each of the instances have an afterSave hook on the user class that saves and updates the info at the other instance. This seems easier to do and maintain.
I would choose the second option.
My company decided to migrate to the Oracle Fusion Middleware, and we are starting to use the Oracle Webcenter components, especially the WCC (v11.1).
We can access our documents via 2 different entry points :
http://server:port_1/wcc
http://server:port_2/cs (and this URL is also accessed via RIDC)
The GUI are very different from one app to the other, but the main functionalities seem to be the same on both of them : I can browse, view, download stored documents, I have access to the metadata, I can do an advanced search or upload a file...
Why does Oracle provide 2 webapps to manage the content ? Why are these URLs on 2 different Weblogic servers (or domains) ? Should we use one address rather than the other ?
Thank you.
They are simply two different GUIs. Some functionality is available in one that may not be available in the other.
Use whatever interface best fits your needs (including what user's may prefer).
/cs/ is for the native (original) UI.
/wcc/ is for the new(ish) ADF WebUI. Also see this blog post.
I'm trying to run a number of applications with known failure rates through Sonar, with hopes of deciding which metrics are most valuable in determining whether a particular application will fail. Ultimately I'll be making some sort of algorithm that will look at the outputs of whatever metrics I'm using and generate a score from 1 - 100. I've got about 21 applications put through Sonar, and the results have been stored in a MySQL database. I originally planned to use PowerPivot to find relationships in the data, but it seems like the formatting of the tables doesn't lend itself well to that. Other questions on stackoverflow have told me that Sonar's tables are unformatted, and I should instead use the Web Service API to get the information. I'm unfamiliar with API and was unsuccessful in trying to do what I wanted by looking at Sonar's documentation for API.
From an answer to another question:
http://nemo.sonarsource.org/api/timemachine?resource=org.apache.cxf:cxf&format=csv&metrics=ncloc,violations_density,comment_lines_density,public_documented_api_density,duplicated_lines_density,blocker_violations,critical_violations,major_violations,minor_violations
This looks very similar to what I'd like to have, except I'm only looking at each application once (I'm analyzing a sample of all the live applications on a grid), which means Timemachine isn't really what I'm looking for. Would it be possible to generate a similar table, except instead of the stats for a particular application per date, it showed the statistics for an application and all of its classes, etc?
If you're not familiar with the WS API, you can also create your own Sonar plugin to achieve whatever you want: it is written in Java and it will execute on every analysis you run. This way, in the code ot this custom plugin, you can do whatever you want: flush the metrics you need in an output file, push them into a third party system, ... etc.
Just take a look on how to write a plugin (most probably you will create a Decorator). You have concrete examples also to get started faster.
So here's the situation. I want to add 'old' news from our previous website in to an asset publisher portlet on our new Liferay 6.1 site. The problem is that I want them to show up as if I had added them in the past.
So, I figure, how hard can it be to modify the createDate? I've since been able to directly access the MySQL database and perform updates on the article object's createDate field. However, it doesn't seem to propagate to my Liferay deployment, regardless of clearing caches, reindexing search indices, and restarting Liferay. The web content still maintains it's 'original' createDate even though the database shows it as the value I have changed it to.
Here's the query I used:
mysql> UPDATE JournalArticle SET createDate='2012-03-08 15:17:12' WHERE ArticleID = 16332;
I have since learned that it is a no-no to directly manipulate the database, as the dynamics of database/Liferay isn't as straight forward as Liferay performing lookups. So it looks like I might need to use the Liferay API, namely, setCreateDate as seen here.
But I have absolutely no idea where and how to leverage the API. Do I need to create a dummy portlet with the sole purpose of using this API call? Or can I create a .java file somewhere on the server running my Liferay deployment and run it to leverage this method?
I only have like 15 articles I need to do this to. I can find them by referencing the ArticleID and GroupID.
Any help would be greatly appreciated. I've grepped the Liferay deployment and found setCreateDate being used heavily within .java files inside the knowledge-base-portlet, but I can't tell how else to directly use them without creating a portlet.
On the other hand, if anybody knows how to get my database to propagate it's changes to the Liferay deployment, even though I know it's a dirty hack, that would probably be the easiest.
Thanks; I really appreciate it.
Using of Liferay API is of course the clear and better way, but for only 15 articles I would try to change it directly through the database.
I checked the database and it seems that Liferay stores the data in these tables: JOURNALARTICLE and ASSETENTRY.
Try to change the created date in both these tables.
Then reload the cache: Control Panel -> Server Administration --> Clear Database Cache.
You can write hook for application startup event. This way whenever liferay is first started it will change the create date as you desire. Later if you want to remove the hook it can be done easily. See here on how to create a hook and deploy it.
http://www.liferay.com/community/wiki/-/wiki/Main/Portal+Hook+Plugins
Also, changing in database itself is not at all recommended even for 1 value/article. Always use Liferay provided service api to modify.