What is the CTRL + S in SQL. I am currently using BigQuery and having a hard time understanding the tutorial provided in my online course.
I followed the steps but I am still confused. I wanted to save the current dataset before I proceed for another data editing and cleaning.
Related
I need to copy a BigQuery dataset from one project to another. However, when I follow the documentation
https://cloud.google.com/bigquery/docs/copying-datasets#required_permissions
I am only able to transfer the tables and not the procedures that are also stored in the same dataset. Is there a way to accomplish that?
In order to further contribute to the community I am posting #MikhalBerlyant`s answer as Community Wiki.
Currently there is an open Feature request for copying procedures between datasets. Although, there is no ETA for evaluation and implementation, you can follow any updates by following this thread, link.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I am building a shiny dashboard for the Company I work for that displays key metrics for the Company. The dashboard currently loads quite fast, but as time goes by the dashboard is getting slower, since the tables in the database that I am querying (I am using RMySQL for this) are getting larger (it's collecting more data over time).
I have done my best to make the SQL queries run fast (making use of index columns, repartitioning etc.), but that only takes it so far. After querying the data, the dashboard then needs to process that data, which is also slowing down over time due to the size of the data increasing.
Are there any options out there to work around this? An engineer I work with suggested using Ajax functions in R, that would allow the dashboard to load only a subset of data at first, (ie. last 2 months) enough to populate the dashboard's charts, and in the background would continue to load the rest of the data (on a separate process? I'm not sure). However, I don't know how to get started here. (what libraries to use, how to call the query and run the subsequent data processing code in an ajax function, etc.)
Any thoughts on this would be highly appreciated.
(Side Note: My queries would take long enough that it would cause the shiny dashboard to 'Disconnect from Server' when I tried to launch it from an AWS EC2 instance. I got around this by calling the queries within 'reactivePoll' function which I am sure is probably not the best way to do this.)
Edit 1: The dashboard also grabs some data from an API endpoint as well, and I'm not sure how this could impact the use of any ajax-like operations or functions to improve my dashboard.
One option would be to use the fst package and store the sql data in fst file format. The front-end application would then proceed to extract the data from the fst file as oppose to hitting the sql server.
I had the same problem a while ago actually. Depending on the complexity of the dashboard the problem of the dashboard being slow can be overcome by having all the data aggregations computed overnight and stored in a temporary table on which Shiny picks from.
Instead of getting shiny to do the calculations each time, you manually create all the possible aggregations using R or Python then upload the results to the temporary database each day/hour using crontab.
This way, the R Shiny dashboard becomes just a tool to visualize data has been pre-computed.
Again, this depends on the complexity of your dashboard and whether or not you have a temporary database / table onto which you can store the aggregations.
Broadly speaking, can someone tell me if I'm headed in the right direction?
I now know how to write SQL Queries pretty well.
I would like to start aggregating multiple queries onto one "form"/template (not sure if that's the correct terminology).
I have access to lots of clean data in the form of Excel Files.
I should be able to load the excel files into Visual Studio and then write reports that refer to those excel files as databases, am I right?
I haven't been able to find a great SSDT tutorial yet, but I'll keep looking. Thanks for any feedback.
First off, I apologize that I'm writing a bit of a novel here. My understanding of your question is that you're looking for architectural guidance on the best way to go, and that's not a quick answer.
If, however, I've misinterpreted your intent and you are actually just looking for how to code up an excel file as a database, there is already a lot of articles online that you can google.
Regarding your architectural question...it is really going to come down to choosing the best trade-offs for what you're building. I will give you some pointers that I have learned and hopefully it is helpful to you and others in the community.
I would be very hard pressed to advise that you use an excel file as a database.
While it might seem like the most straight forward solution, the trade-offs here are very costly in debugging file locking issues and dealing with excel specific errors, it becomes a death by a thousand cuts. It is certainly possible, but this is a trap that I personally fell into early in my career.
Here's is a link to some descriptions of the problems that you'd have with an excel file database and here is a 2nd link.
To paraphrase your question, it sounds like you're developing a personal ETL application for improving your productivity and your company's metrics. Spreadsheets come into your e-mail inbox and transformed versions of the spreadsheets go out of your e-mail outbox. You are wanting to look at the departments' data from a historical and comparative perspective. I have done this many times in the past as well and it is a very reasonable goal.
The best way that I have found to do this is to use a SQL Server database. You can start this out in phases of minimal viable product to do this in small easy chunks.
Phase I:
Download and install SQL Server 2016 Express free. Make sure
to install localdb when you install SQL Server 2016. See the localdb
instructions for more information.
Create the localdb instance on the command line.
Connect to the new localdb instance in SQL Server
Create a new Database that you'll use for importing the data. Give it a name like "ReportData"
Import the excel files received from the variety of businesses into the new database. This stackoverflow answer gives a great description of how to do it. Here is an alternate example.
If you get any error messages about drivers you may need to download the correct drivers.
Develop your SQL queries that you want to use. For simplicity, I'm just showing a basic select statement here, but you can build some sophisticated SQL queries for aggregating the data in this step.
Export the data from the excel file into a CSV file or an excel file. You do this by right clicking in the "Results" area and selecting "Save Results As..."
Manually copy and paste the resulting values into the excel templates that you would like.
Note step 9 will be automated soon, but it is better for now to understand your domain objects and be thinking about the business logic that you're building in a quick iterative manner.
Phase II:
Create a new Console application in Visual Studio that will transform the data from the database into an Excel file output. The most powerful way to do this is to use EPPlus. Here is a detailed explanation on how to do this.
Note, when you run the source code from the detailed explanation link, you need to change the output path first, for example to c:\temp. Note also that there are plenty of other Excel spreadsheet helper packages out there, so feel free to look around at other packages as well. EPPlus is simply one that I have been successful with in my projects. This Console application will be querying your SQL Server database using the queries that you built in step 7 above.
Phase III:
In time, you many find that co-workers and managers within your company want to start accessing your data directly through a web page...
At a high level, the steps you would take are:
Backup the database and restore it onto a server.
Implement a simple MVC application
Perhaps even build web pages to allow users to import excel so that they don't need to e-mail them to you any longer.
An additional note, there are Enterprise level ETL and reporting tools out there as well, such as SSIS/SSRS, etc that you could look into if you're looking for a more sophisticated tool set, but I didn't get that impression with your question.
I hope that this answer helps and isn't too long winded. Please let me know if any of the steps are unclear, I know it's a lot of information in one post.
I'm making my project for a-level computing and I reached a problem that im not sure how to solve. Im making something similar to a student information management system and I want a way to store a small list of students maybe 5 to 10 preferably in my program and make reference to them from all forms and so that if something is changed on one of the students maybe a piece of information, that these information is carried to the next form. My teacher has very little knowledge on programming and so im kind of stuck, i have no previous experience with databases or sql however if someone is willing to break it down ill be very grateful and ive got a good understanding of arrays. My deadline is the 10th of may so asap please, thanks
- kyle
If you prefer simplicity I would stick to a CSV file or XML. I found a website which an end-to-end tutorial on how to create the XML and add items (students) to it using VB.net as according to your tag. You need only after to read them from the file, but might just as well add new:
http://vb.net-informations.com/xml/how-to-xml-in-vb.net.htm
I would stick with XML (or CSV as preffered) as it is a text file basically so you can see and make changes directly if needed.
I'm looking someone who can help me with this issue, that seems very easy to achieve but there is no much information around the internet.
I'm working with MS Access and Visual Basic, and I would like to create a simple bar graph, giving params that I've already gathered from many querys.
In this case: A chart about Litres per Week. I have a query to gather the amount of litres of 4 different weeks. I also provide the information for the X Axis (Last week, Last two weeks, and so on.. ). It would look like this:
So I'm wondering if this can be done, as well a pie charts and line charts.
I'm aware that this can be easilly done with Excel, but I don't want open another program to read information.
Thank you very much.
How to make Charts in MS Access would be my first key word search :)
Then I will check on MSDN for proper tools/properties availability/updates on software and so on.
Here is a simple basic tutorial to follow
This is the entire google search that would provide you anything you want to check on the topic.
Please note the version of MS Access you are using against the tutorials described. In certain versions there are certain changes have been done to the software itself. You are much lucky if it's not 2007...
EDIT Based on the comments with OP:
Since OP has exhaustive/heavy queries, it's recommended to create a temp table based on the final query.
Then temp table can act as a view and become the data source for the Chart
dynamic binding of chart from database in VS2010 in C#
Bind query as a datasource to MS Access chart. There is a method called databindtable which is a chart property. You may use it:
e.g. chart1.DataBindTable(yoursource,columname)
This article provies a good sample code.