I would like to make an app in Xcode where users get scored based on football results. For this, I need to update a table of football scores every weekend. What would be the best way to go about this? I assume it would be through SQL queries, but what is the industry standard for grabbing up to date data?
The solution is going to be dependent upon your DBMS but a couple ideas to get you started.
Utilizing SQL server you can created a job and schedule it run on a server a specific schedule, you can write TSQL code to look at CSV files and update your database tables
You can write an integration package with SSIS and schedule that package to run weekly
You can write a solution in Python or another scripting language and schedule that to run on your server with windows task manager
Related
I need to collect data from a SQL Server table, format it, and then put it into a different table.
I have access to SQL Server but cannot setup triggers or scheduled jobs.
I can create tables, stored procedures, views and functions.
What can I setup that will automatically collect the data and insert it into a SQL Server table for me?
I would probably create a stored procedure to do this task.
In the stored procedure you can create a CTE or use temp tables (depending on the task) and do all the data manipulation you require and once done, you can use the SELECT INTO statement to move all the data from the temp table into the SQL Server table you need.
https://www.w3schools.com/sql/sql_select_into.asp
You can then schedule this stored procedure to run at a time desired by you
A database is just a storage container. It doesn't "do" things automatically all by itself. Even if you did have the access to create triggers, something would have to happen to the table to cause the trigger to fire, typically a CRUD operation on the parent table. And something external needs to happen to initiate that CRUD operation.
When you start talking about automating a process, you're talking about the function of a scheduler program. SQL Server has one built in, the SQL Agent, and depending on your needs you may find that it's appropriate to enlist help from whoever in your organization does have access to it. I've worked in a couple of organizations, though, that only used the SQL Agent to schedule maintenance jobs, while data manipulation jobs were scheduled through an outside resource. The most common one I've run across is Control-M, but there are other players in that market. I even ran across one homemade scheduler protocol that was just built in C#.NET that worked great.
Based on the limitations you lay out in your question, and the comments you've made in response to others, it sounds to me like you need to do socialize your challenge within your organization to find out what their routine mechanism is for setting up data transfers. It's unlikely that this is the first time it's come up, unless the company was founded in the last week or two. It will probably require that you set up your code, probably a stored procedure or maybe an SSIS package, and then work with someone else, perhaps a DBA or a Site Operations team or some such, to get that process automated to fire when you need it to, whether through an Agent job or maybe a file listener.
Well you have two major options, SP and SSIS.
Both of them can be scheduled to run at a given time with a simple Job from the SQL Server Agent. Keep in mind that if you are doing this on a separate server you might need to add the source server as a Linked Server so you can access it from the script.
I've done this approach in the past and it has worked great. Note, for security reasons, I am not able to access the remote server's task scheduler, so I go through the SQL Server Agent:
Run a SQL Server Agent on a schedule of your choice
Use the SQL Server Agent to call an SSIS Package
The SSIS Package then calls an executable which can pull the data you want from your original table, evaluate it, and then insert a formatted version of it, one record at a time. Alternatively, you can simply create a C# script within the SSIS package via a Script Task.
I hope this helps. Please let me know if you need more details.
I am trying to find the easiest and simplest and quickest way to upload a sheet from Excel to a table in SQL Server 2012 automatically every morning as a job from a location on my folder to the table.
SSIS is the ETL tool you could use, but if it’s a very simple job you can just write a BCP command.
https://learn.microsoft.com/en-us/sql/tools/bcp-utility?view=sql-server-2017
The way the schedule it is to add the task to the agent job on the server. A few things to bear in mind with ETL:
Will your file be named the same each day?
Do you need to retain archived versions of the file?
How do you do error handling if it’s absent or malformed?
Does the DDL need to change periodically to accomodate new date ranges (I.e. new day/month year)
Will this pattern be reused in the future?
Do you need to test logically (duplicates/logical fallacies/referential integrity etc)?
Under whose account will the job run (hint, don’t use your own - get a service account)?
The more complex the answers are to these types of questions the more likely you’ll need a real ETL tool like SSIS
This is a general question and probably there are some solutions already. Most of the things I have found are related to database development, deployment, etc..
I am looking for a process that runs daily and performs some checks against some tables of a database. The data loaded in these tables is loaded by a lot of users, but the idea is that defining some rules, the process will detect "wrong" values loaded by the user.
I know this is a very open question, but do you know if this possible with some tools: Jenkins, DBGhost, etc...?
Thank you,
Kat
You have many options. Here's one train of thought.
Create a table called data_audit with fields like so:
audit_datetime
table
field
wrong_value
rule_violated
issue_description
Create stored procedures/functions that can detect wrong values and store the data into this audit table.
Depending on your database, you can run the stored procedure upon schedule. For example, if you have SQL Server, you can run the job using SQL Agent. Once the job is finished, you can run another job that finds count(*) from audit table for today's date. If count was higher than zero, use Database Mail feature to email relevant people to take action.
If you have a database like MySQL or PostgreSQL, write a short program in a language of your choice (PHP/Python/.NET/whatever) to execute the stored procedure, then do count(*) and then email if count was higher than zero. You can run this program using either cron on Linux or Linux-like systems or Task Scheduler in Windows.
You could use tools like Jenkins to schedule such activity. Task Scheduler/cron are built into your operating system and are easy to use. Additional installation like Jenkins is not necessary. If you already have Jenkins installed, you can certainly piggy-back on it.
I am importing data from a third party database. Following various actions carried out on the data by my software application, I then need to automatically produce update and insert scripts to amend / append to the original 3rd party database. There will be in the region of 20 scripts to amend 20 tables.
The live 3rd party database is held remotely of my server which holds a backup of the data only. The data to be amended will be static on the live database during the exercise I am performing.
Does anyone know of an application that will assist in preparing these scripts?
Thanks
Microsoft SQL Server Integration Services was designed for ETL jobs and can help with complicated workflows (lots of dependancies and ordering etc.);
https://en.wikipedia.org/wiki/SQL_Server_Integration_Services
However I personally prefer to simply write SQL scripts and schedule them using SQL Server Agent. If you do wish to use a graphical tool though - SSIS is probably your best bet.
If you want to get clever and dynamically generate these (if the procedure for the 20 tables is standard) you can use BIML to ninja up one package which will work for all tables: https://www.mssqltips.com/sqlservertip/3094/introduction-to-business-intelligence-markup-language-biml-for-ssis/
I have an SQL Azure Database instance which provide data to a windows 8 app. The data in my database should be updated periodically (weekly). Is the any way to make it? I'm thinking of write a app which will run weekly and update the database. But still don't know how to make it run on Window Azure? Please help!
Thank you,
There are a number of ways to achieve this, does the data however need to come from a different source or can it be calculated?
Either way, seeing as you're already knee deep in SQL Azure I would suggest putting your logic into a worker role that can be scheduled to run your updates once a week. This would give you a great opportunity to do calculations and/or fetch data externally.
Azure gives you the flexibility to scale this worker role into numerous instances as well depending on the work load.
Here is a nice intro tutorial on creating a worker role on Azure: link
Write the application and set it to run through your cron job manager on a weekly time schedule. The cron job manager should be provided by your host.