Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
I need to generate reports based on a dataset pulled from a third party API, but I can't store the data on disk. (If not storing the data were not a requirement, I would be storing the data in a relational database, and writing a query to join several tables to generate the export as a CSV, for instance.) I've been reading about Redis and I wanted to know if that is a potential solution here as a temporary datastore? Or would I have a hard time putting the tables in the dataset together? If not Redis, what is the recommended way to cache data for reporting purposes in an Azure environment?
I'm filling in a lot of the gaps with assumptions, but to answer your question, yes.
Azure Redis Cache could be used to run your reports "in memory" generally speaking.
For the solution, "it depends" on:
the type of data
how you ingest the data
the type of reports you are trying to run
You have a platform that can run reports with Azure Redis Cache, but you still need to model the data properly to build your reports. Redis is not a relational database. Without more details, you should start here: https://redis.io/topics/data-types-intro
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I'm looking for the best sql client to use with AWS Athena (with JDBC driver).
One of my needs is to share the queries between users, and I would like to know how can I do it in the best way.
The options I see are:
Using the web Athena query editor, and save the queries in 'Saved Queries' tab. Generally I prefer to work with a DB software, and not this web tool.
Using a simple DB client tool (I use DBeaver right now), and share the queries using git. (I find it uncomfortable, because the user will need to choose queries based only on their name, and it might get tricky when there are a lot of queries)
Is there a better way to do it? for example an SQL client that imports the Athena saved queries
Also- I would be interested to find a way and generalize my SQL queries - for example, to edit values in them without changing the query itself. I guess it can be done using some scripting mechanism over the SQL or the DB itself. Any idea how to do this with the client too?...
Thanks!
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
I am looking for scalable alternative to traditional DBMS like PostgreSQL or MySQL.
In traditional databases I don't have the following features:
Auto sharding to ensure linear scalability.
Replication with automatic failover and recovery to ensure high availability.
No single point of failure.
MongoDB looks like good candidate if I can sacrifice transactions.
Also I've looked at several newSQL databases. NewSQL seems suitable for my purposes: VoltDB, TiDB, cockroachDB. But I'm worried about whethever they are production-ready.
May be there are extensions allowing to run postgreSQL or MySQL in clustered mode out of box.
You should check out Vitess. It's used at YouTube and by a few other companies.
PS: I work on that project.
TiDB
Compatibility with MySQL
It supports the MySQL Protocol so that you can transfer your MySQL scripts running on TiDB without change.
Use cases
It was used by many big name company such as Mobike, uber，pinterest etc. In Mobike, the big data team uses TiDB as a slave for synchronizing data with online DB. After that, OLTP query, consisting of analysis and gathering request, was executed in such circumstance. Last but not the least, the cloud computing platform belongs to Tencent, the technology giant, recommend customers use HTAP based on TiDB for OLTP and OLAP.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I'm trying to "open" (?) a 20gb database with a .sql extension and can't find any documentation for beginners that doesn't already assume database access. I think as a first step I need some database management system. It don't need to have any development type capabilities - just the ability to compile (right word?) SQL.
Any suggestions?
Thanks,
Erin
The best RDBMS system to use as a beginner is probably going to be either MySQL or SQLite.
MySQL is an excellent database system and is capable of holding an extremely large amount of data. The data however is not stored in a movable file as you have described here.
SQLite is also an excellent database system and is capable of creating database files that you can move from one machine to another quite easily. The downside is that for a data set that large, you will likely have performance issues.
Based on the size of the file you mentioned, it sounds like what you have is a file with a whole bunch of SQL statements in it. Without seeing the contents of the file, it is extremely difficult to say which RDBMS it came from, but at the very least you should install MySQL and learn how to use it.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
I need to store logs in a distributed file system.
Let's say that I have many types of logs. Each log type is recorded in file. But this file can be huge, so it must be distributed across many nodes (with replication for data durability).
These files must support append/get operations.
Is there a distributed system that achieves my needs?
Thanks!
I would recommend Flume, a log pulling infrastructure from the folks at Cloudera:
http://github.com/cloudera/flume
You can also try out Scribe from Facebook:
http://github.com/facebook/scribe
Combine a NAS with a no-sql database like MongoDB and you'll have distributed, large, and fault tolerant.
Of course, without more specific details like how much data, structure of the logs (or lack thereof), etc, it's really hard to recommend a real product.
For example, if by "huge" you really mean 2TB or less, and the data is highly structured, then a regular SQL server in a 2 machine clustered environment for fail over will do just fine.
However, if by "huge" you mean exabyte level or more and/or unstructured data then several large (and very expensive) NAS devices are needed. On which you run a set of no-sql databases that are clustered for fail/over and/or multi-master relationships...
You can use Logstash to collect the logs and centralize them with an Elasticsearch cluster. The local logs could be rolling log files, so that they remain small.
Further you can use Graylog2 to analyze and view your logs.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I'm helping a friend learn SQL, and need more databases to help him get more practice.
We have of course AdventureWorks, Northwind, and Pubs.
Does anybody know of any other SQL databases samples that might be available to download?
I know some sites have some databases where you can practice queries on the site, but I was looking for something he can run locally.
Try downloading and importing some of the free data sets the Census bureau, US Geological Survey, National Oceanic and Atmospheric Administration, or other large data gathering agencies provide for free on the web.
Why not work with a billion record table of all the elevation points in the US? Not only do you get the practice your DB skills, you can get real world types of indexing and performance issues.
Here's one called the Chinook Database. It has scripts for SQLServer, Oracle, MySQL, SQLServer Compact.
http://chinookdatabase.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=21111
As I find more, I'll post here.
I think it is much better Idea to create a database from scratch ,Taking sample project say a simple inventory management system and then building tables finally creating then in db ,It would be much practical way to learn then to directly play with a ready made database.