Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
So this might be a bit of a strange one, but I'm trying to find a tool that would help me visualize real time data in a form of a table rather than a graph/chart. There are a lot of tools out there like Grafana, Kibana, Tableau that kind of fit a similar purpose, but for a very different application and they're primarily made for aggregated data visualization.
I am essentially looking to build something like what a departure board is at an airport. You got flight flight AAA that landed 20 minutes ago, XXX departing in 50 minutes, once flight AAA is clear it disappears from the departure board etc. Only I want to have that real-time, as the input will be driven by actions users are performing on the shop floor on their RF guns.
I'd be connecting to a HANA database for this. I know it's definitely possible to build it using HTML5, Ajax and Websocket but before I get on the journey of building it myself I want to see if there's anything out there that somebody else has already done better.
Surely there's something there already - especially in the manufacturing/warehousing space where having real-time information on big screens is of big benefit?
Thanks,
Linas M.
Based on your description I think you might be looking for a dashboard solution.
Dashboards are used in many scenarios, especially where an overview of the past/current/expected state of a process is required.
This can be aggregated data (e.g. how long a queue is, how many tellers are occupied/available, what the throughput of your process is, etc.) or individual data (e.g. which cashier desk is open, which team player is online, etc.).
The real-time part of your question really boils down to what you define to be real-time.
Commonly, it’s something like “information delivery that is quick enough to be able to make a difference”.
So, if, for example, I have a dashboard that tells me that I will likely be short of, say, service staff tomorrow evening (based on my reservations) then to make a difference I need to know this as soon as possible (so I can call more staff for tomorrows shift). It won’t matter much if the data takes 5 or 10 minutes from the system entry to the dashboard, but when I only learn about it tomorrow afternoon, that’s too late.
Thus, if you’re after a dashboard solution, then there are in fact many tools available and you already mentioned some of them.
Others would be e.g. SAP tools like Business Objects Platform or SAP Cloud Analytics. To turn those into “real-time” all you need to do is to define how soon the data needs to be present in the dashboard and set the auto-refresh period accordingly.
Related
The question is about using a chat-bot framework in a research study, where one would like to measure the improvement of a rule-based decision process over time.
For example, we would like to understand how to improve the process of medical condition identification (and treatment) using the minimal set of guided questions and patient interaction.
Medical condition can be formulated into a work-flow rules by doctors; possible technical approach for such study would be developing an app or web site that can be accessed by patients, where they can ask free text questions that a predefined rule-based chat-bot will address. During the study there will be a doctor monitoring the collected data and improving the rules and the possible responses (and also provide new responses when the workflow has reached a dead-end), we do plan to collect the conversations and apply machine learning to generate improved work-flow tree (and questions) over time, however the plan is to do any data analysis and processing offline, there is no intention of building a full product.
This is a low budget academy study, and the PHD student has good development skills and data science knowledge (python) and will be accompanied by a fellow student that will work on the engineering side. One of the conversational-AI options recommended for data scientists was RASA.
I invested the last few days reading and playing with several chat-bots solutions: RASA, Botpress, also looked at Dialogflow and read tons of comparison material which makes it more challenging.
From the sources on the internet it seems that RASA might be a better fit for data science projects, however it would be great to get a sense of the real learning curve and how fast one can expect to have a working bot, and the especially one that has to continuously update the rules.
Few things to clarify, We do have data to generate the questions and in touch with doctors to improve the quality, it seems that we need a way to introduce participants with multiple choices and provide answers (not just free text), being in the research side there is also no need to align with any specific big provider (i.e. Google, Amazon or Microsoft) unless it has a benefit, the important consideration are time, money and felxability, we would like to have a working approach in few weeks (and continuously improve it) the whole experiment will run for no more than 3-4 months. We do need to be able to extract all the data. We are not sure about which channel is best for such study WhatsApp? Website? Other? and what are the involved complexities?
Any thoughts about the challenges and considerations about dealing with chat-bots would be valuable.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I have multiple databases that sometimes interact with each other but are mostly independent. I now need build a new application that allows users to search though the data of the rest of the application (sort of searching through the history of the other applications).
So I'm going to need a dozen or so stored procedures/views that will access data from various databases.
Should I have each stored procedure/view on the database that is being queried? Or do I have a brand new database for this part of the application that gathers data from all other databases in views/SPs and just query that?
I think it should be the first option, but then where do I put the Login table that tracks user logins into this new report application? It doesn't belong in any other database. (each database has it's own login table, its just the way it was setup).
What you are asking here fits into the wide umbrella of business intelligence.
The problem you are going to hit quickly...reporting queries tend to be low number of queries and relatively resource intense (from a hardware point of view). If you will, low volume high intensity.
The databases you are hitting are most likely high transaction databases. IE they are dealing with a large number of smaller queries, either as a large number of single (or multiple) inserts or quick selects. If you will, high volume low intensity queries.
Of course, these two models conflict heavily when trying to optimize them. Running a reporting query that joins multiple tables and runs for several minutes will often lock tables or consume resources that prevent (or severely inhibit) the database from performing its day to day job. If the system is configured for high number of small transactions, then your reporting query simply isn't going to get the resources it requires and the time lines on reporting results will be horribly long.
The answer here is the centralized data warehouse that collects the data from several sources and brings it together so it can be reported on. It's usually 3 components, a centralized data model, an etl platform to load that data model from the several data sources, and a reporting platform that interacts with this data. There are several third party potentials (listed in comments) that somewhat mimic the functionality of all three, or you can create these separately.
There are a few scenarios (usually due to an abundance of resources or a lack of traffic) where reporting direct from the production data of multiple data sources works, but those scenarios are pretty far and few between (usually never in an actual production environment).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have a program with a database with information about people that contains million records.
One of the tasks were to filter the results by birth date, then group
them by city and finally compare the population of each city with the
given numbers.
I started to write everything in SQL query, but then I started to wonder, that it may make server too busy and maybe it's better to do some calculations with the application itself.
I would like to know if there are any rules/recommendations
when to use server to make calculations ?
when to use tools like LINQ in the application ?
For such requirements, there's no fixed rule or strategy, it is driven by application / business requirements, couple of suggestions that may help:
Normally Sql Query does a good job in churning lots of data to deliver a smaller result set post filtering / Grouping / Sorting. However it needs
correct table design, indexing to optimize. As the data size increase Sql may under perform
Transferring data over the network, from hosted database to application is what kills the performance, since network can be big bottleneck, especially if the data is beyond certain size
In memory processing using Linq2Objects can be very fast for repetitive calls, which needs to apply filters, sort data and do some more processing
If the UI is a rich client, then you can afford to bring lots of data in the memory and keep working on it using Linq, it can be part of in memory data structures, if the UI is Web then you need to Cache the data
For having the same operations as sql, for in memory data, for multiple types, you need custom code, which preferably use Expression trees along with linq, else a simple linq would do for a known fixed type
I have a similar design in one of my web application, normally it is a combination, which works best in the most of the practical scenarios
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
I would just like to ask or get a few expert advices/directions on building a database and architecture for a telecommunication app.
Basiclly there are 3 parts:
Switch (low level, calls, signaling)
Backend (CRM)
Mobile
Mostly focused now on the backend part so would like to know if anyone had some experience and would be able to point in some directions for research and so on. Nothing special, things like SQL or NoSQL and just some good points where I could learn and research more! Thx
I have worked for both Intergraph and Ericsson on their telecommunications apps and can suggest a few places for you to start your research.
Domain
The telecom domain is very large, so I would suggest you determine what your audience wants you to focus on first. Are you interested in facilities management, which is where poles, lines, and equipment are located? Are you interested in logical network management, where you are worried less about the physical cables themselves and worried more about the logical circuits that "ride" these physical media? Narrowing your domain will help you tremendously.
There are quite a few resources online, but I would suggest a reference, like this one: Fundamentals of Telecom Book
Do your best to research and know your domain.
Data Modeling
For telecom data modeling, check out ESRI at www.esri.com. They specifically publish a telecom data model white paper that you might want to check out (you'll have to register to see it) at: http://downloads2.esri.com/support/TechArticles/Telecommunications_Data_Model.pdf
Also, check out the offerings from IBM at: [https://www-01.ibm.com/marketing/iwm/iwm/web/signup.do?source=sw-infomgt&S_PKG=500019725&S_CMP=is_bro11&S_TACT=109HF36W][3]
Finally, check out the products from Intergraph, Smallworld, and 3-GIS at:
http://www.intergraph.com/communications/
http://www.gedigitalenergy.com/geospatial/catalog/smallworld_network.htm
http://www.3-gis.com/
Graphs
My final bit of advice to you is to know your graph theory. Modeling telecom networks requires advanced knowledge of node-edge, edge-edge, and directed graphs. You need this knowledge to model pole-line graphs, underground duct networks, cable sheath-to-sheath connections, fiber and copper stand-to-strand and pair-to-pair connections, and the relationships between various parts of the network.
Beyond the relationships that you model with graphs, you will employ this knowledge to trace networks quickly.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Suppose I have an online store application that contains millions of items that are maintained by the application. The application is so famous that millions of items get sold for each hour. I store all this information in a database, say Oracle DB.
Now if I want to show the top 5 items sold in the last 1 Hour then I can write a query something like :
Get the list of products that were sold in last 1 Hour.
Find the count of each product from above result and order by that count value, then display the top 5 records.
This seems to be a working query, but the problem is, for each 1 Hour if I am having millions of items sold then running this query against the table that contains all the transactional information will definitely hit performance issues. How can we fix such issues? Are there any other way of implementing it.
As a note, Amazon at its peak on Cyber Monday is selling a bit over a million items per hour. You must have access to an incredible data store.
Partitioning is definitely one solution, but it can be a little complicated. When you say "the last hour" that can go over a partitioning boundary. Not a big deal, but it would mean accessing multiple partitions for each query.
Even one million items and hour is just a few hundred items per second. This might give you enough leeway to add a trigger (or probably logic to an existing trigger) that would maintain a summary table of what you are looking for.
I offer this as food-for-thought.
I doubt that you are actually querying the real operational system. My guess is that any environment that is handling even a dozen sales per second is not going to have such queries running on the operational system. The architecture is more likely a feed into a decision support system. And, that gives you the leeway to implement an additional summary table as data goes into the system. This is not question of creating triggers on a load. It is, instead, a question of loading detailed data into one table and summary information into another table, based on how the information is being passed from the original operation system to the decision support system.
I think you should try the partitioning.
E.g. you can split the data for each month/week/whatever into different partitions using maybe range partitioning and then for the last hour it is quite easy to run the query only for a specific, last partition. See partitioning-wise joins to learn more about it.
Of course, you'll need to perform some specific implementation steps, but every war can require some sacrifice...