We are using Grafana for data visualization and would like to have some overview about what our users are doing. There is a statistics page at /admin/stats, accessible for administrators that has some numbers about active sessions, logins, dashboards, etc. Is it possible to easily access this data and visualize the numbers in a dashboard?
If you happen to be using Prometheus, Grafana now exposes metrics for Prometheus on /metrics endpoint - included in here are the statistics you're interested in. See https://grafana.com/docs/grafana/latest/features/datasources/prometheus/#getting-grafana-metrics-into-prometheus
I'm not familiar with other monitoring systems, sorry.
Hope this helps!
Related
I am building a service to handle a large number of devices, for a large number of users.
We have a complex schema of access roles assigned to each entity. Some data entries can be written to by certain users, while some users can only read from some entities (but can write to others).
This is a cloud service: there are more devices, and users than can be handled by a single server machine (we are using non relational Cloud databases for this).
I was wondering if there was an established cloud-scale user/role management backend system which I could integrate to enforce the access rules, instead of writing my own. This tech should preferably be cloud agnostic, so I would prefer not to use a SAAS solution, but deploy my own.
I am looking for a system which can scale to millions of users, and billions of data entities
I think authentication is not going to be a big issue, there are very robust cloud based solutions available for storing identities and authenticating millions of users. Authorization will be trickier, and will depend a lot on how granular you want it to be. You could look at Apigee for example as a very scalable proxy that might help you implement this. So getting to the point where you have a token that you can verify the users identity with and that might contain some scopes is not going to be hard imo. If that is enough for you then I would just look at Auth.0, Okta and the native IDM solution of whatever cloud platform you are using (Cognito, Cloud Identity etc.).
I think you will find that more features come with a very hefty pricetag. So Auth.0 is far superior compared to Cognito, but Cognito still has enough features for basic use cases and will end up costing a fraction of Auth.0 in large deployments. So everything comes with pros and cons. If you have very complex requirements such as a bunch of big legacy repositories that you need to integrate then products like Auth.0 rapidly start looking more attractive.
Personally I would look at Auth.0, Cognito and Apigee and my decision would depend massively on parameters that you haven't mentioned in your question. Obviously these are all SaaS solutions, which I think you should definitely be using anyways. I would not host this myself unless I had no other choice, and going that route will radically limit your choices and probably increase expenses. All the cool stuff is happening in the cloud.
We have our applications (Map Reduce Jobs, microservices) completely running out of AWS.
We intend to use a single service for viewing (for debug purposes), monitoring and alarms (notifications based on a threshold) on logs.
Is there any specific benefits of using external service providers like sumo logic over the one provided by AWS itself (cloudwatch in this case)
In full disclosure, I'm an engineer at Sumo Logic. But here is an analysis done by one of my colleagues a few months ago as to why you would want to use Sumo Logic specifically over AWS Cloudwatch itself:
You can’t easily search across multiple Cloudwatch Log Groups in Cloudwatch. In Sumo, you can define metadata to easily query across log groups, and even log sources outside of AWS within the same query.
Cloudwatch does not include any pre-built apps (out of the box queries, dashboards, and alerts). Sumo Logic provides a wide variety of apps and integrations, all included with your subscription: https://www.sumologic.com/applications/
With Cloudwatch, you pay for dashboards, you pay to query your data, and you pay to use its query API. These are all included in your Sumo Logic subscription (depending on the subscription level you choose).
Sumo provides 30 days of log retention out of the box. Data retention is another a la carte cost when using CloudWatch. Sumo Logic also provides you with the ability to forward your logs off to S3 for long-term storage outside of our platform.
Cloudwatch does not include advanced analytics operators. Sumo Logic includes operators like Outlier, Log Reduce, and Log Compare, which are all part of the Sumo Logic platform.
Regarding search time, Sumo Logic vs AWS CloudWatch Insights (AWS CloudWatch log search): Here is a quote from a customer with 100 AWS accounts worth of CloudTrail logs: "We can search all 100 of our accounts in Sumo in the same amount of time it takes us to search 1 account with AWS's CloudWatch.”
Sumo Logic provides Threat Intelligence as part of your subscription as well, to be able to check all of your logs against Crowdstrike’s threat database of known IoC’s.
Sumo training and certification is included with your subscription.
On a personal note, I can also say that Sumo Logic's post-sales support is top-notch, as we put a huge emphasis on customer success.
Please keep in mind that this analysis by my colleague is a few months old, so some of these items may have been addressed by AWS since then.
I study Vue and Vuex. In the official documentation there is a simple example of using a Vuex with saving data to localStorage.
To better understand the material I studied, I decided to consolidate the knowledge into practice and write a mini application - a clone of trello (SPA).
Namely:
Create three routes:
General dashboard (/dashboard) where are boards
Board (/board) where one or several columns are located, each column has a button for
creating a task in it
Task (/:task-id) that are in columns, tasks can be moved between columns
Sidebar in which all notice with the board are displayed (CRUD by tasks and columns, changes in the status of a task, and so on.)
Sockets so that other users can see the
changes on the board in real time.
Questions!
What data should I store exclusively in the storage Vuex? Excluding authorization. It is obvious.
For what data in this application can localStorage be useful?
What should I use so that data is not discarded when I refresh the page or navigate? I can use localStorage, but hypothetically there can be a lot of data. The fourth question follows from this.
Is a better solution to use persistent remote storage on server / cloud? If so, could you give me information on how to do this? And in this case, interaction with the database is of interest, at what point is it better to save data in the database?
I'm interested in how to properly build such an application, as in a real commercial application.
I use and learn the stack MEVN
1- you can store any type of data in your store, 2 - I don't thing is useful. Because if users remove browser cache all them data will be forget. So you need configure an database for this. 3 - You need a Database and some Backend to provide your data. 4 - It's depends. if you need only for developement, you can install any things in your machine. If you need some thing more robust, could you take a cloud server. But for configure the server you need a little bit system administrator skills.
I have a project in BigQuery where many people update/add Views.
Other access Views/Tables from 3rd party softwares like Tableau.
I have no control for example if the Analysit who wrote the query in Tableau used the Partition of the table or not.
Is it possible somehow to ask BigQuery to send email for each query that passes threshold? For example 20GB. Then I can check this specific query and user to see if it's OK or not (I'm not forcing partition as it's not always what we need)
I know that it's possible to use the Stackdriver Logging export to download logs into BigQuery tables / storage but I don't see anything there that can tell me if query passed this specific criteria.
There are different solutions available but the best is using Cloud Pub/Sub topics and piece of Cloud Function:
Enable programmatic notifications to receive Cloud Pub/Sub messages with the current status of your budget
Programmatic Budgets Notification Examples
and how do you decide on what monitoring method to use (LDAP monitoring, SNMP, JMX or the logs). Is it worth it to implement all of them? How different are they?
I pulled the metrics through all of these methods and they seem to be very similar to me. I'm new to working with directory servers so I'm interested in what others would think.
Also, after you pull the metrics, how do you make use of them (view them with jconsole for example?)
With OpenDJ, all metrics are equally available through LDAP and JMX. SNMP only has a subset of the metrics, i.e. the ones that are defined in the standard directory MIB.
When monitoring any server, I don't think a single capture of metrics is useful. What you want to compute average and running average over sample periods, define thresholds for some of the metrics and raise alert when those thresholds are met.