i have to insert and update some values which is daily coming from
excel file but as everyday excel file format is different
so tell me other possible ways to automate insert update ?
Are the excel files really in different formats or does Excel just think they are different? If the columns are still in the same ordinal positions but they are being interpreted as having different data types, then yes, you can provide hints to the driver to overcome.
Otherwise, you could use C#/vb.net and query the worksheet, dump that into an dataset, write that to a variable and then shred that object but it's ugly. In fact, dealing with Excel in a programmatic fashion is always ugly and best avoided.
If your file is a different format each day then you are out of luck. That is a problem, and there is really no easy or efficient way to parse and insert/update based on that. Whatever the source of the data is, you need to ensure that it becomes consistent.
If it is a handful of formats that you can test and handle accordingly then you could always have some data flow logic inside the SSIS package, but if this isn't predetermined then you would have no way of handling these cases.
YOu deal with this by returning the file to the provider and requiring them to provide in the same fashion every day. Then your SSIS pacakge should reject the file if it is not in the correct format. While you aer at it you will have far fewer problems if they send .txt or .csv file. Excel support is exceedingly poor.
Related
I am tasked to create a template that will be Filled up by Business Users with Employee Information, then our program will load this into the Database using External Tables.
However, our Business Users constantly change the template by adding, removing or reordering fields.
I am convinced to use XLSX instead of CSV so that I can lock the Column Headers so they cannot remove, add and reorder the columns.
However, When i query the External Table, it shows Non-ASCII Characters when reading XLSX because its in Binary.
How can i do either of the following?
Effectively Read Excel Files from External Tables
Lock the Headers of CSV Files?
What you have here is a political problem, but you are looking for a technical fix. Not a good fit.
The problem comes in two halves:
Somebody decided it was a good idea to collect user input in a spreadsheet, which it is generally not.
Users are fiddling with the input format, which they should not.
Fixes are:
Strictly enforce the data structure. Reject any CSV which doesn't natch and make the users edit them. They will quickly tire of tweaking the spreadsheets when they realise they're just creating more work for themselves. But they will also get resentful, so consider ...
Building a data input screen. It's pretty simple to knock up a spreadsheet-like grid UI. You don't need anything complicated in Java: Oracle's Apex is intended for exactly this sort of thing. Find out more.
However, if you are stuck with Excel as a UI I suggest you have a look at Anton Scheffer's excellent PLSQL as_read_xlsx package on the Amis site. Check it out. You'll probably need to replace your external table with a view over a table (perhaps pipelined) function.
I suppose this is somewhat of an extension of the question asked here.
However, I believe the linked OP and my own reason for reading a file with SQL Developer are different. I am learning SQL and databases and am attempting to create a model database (as in, I won't be editing the data after insertion, just set up search queries and what not). I want it to be large (over 100,000 entries), so I've created a C++ program that wrote randomly generate entries for the database on a .txt file (one entry per line in the .txt file) instead of hard coding the insertion of each entry. Now what I want to do is read the .txt file in SQL Developer and inserts it into a table.
My problem lies in the fact that I am not able to create directories. I am using a university oracle connection and I do not have the privileges to actually make a directory so that I can use UTL_FILE on my .txt file as was answered in the linked question. Assuming there is no way for me to gain this permission, is there an alternate way I can accomplish reading a .txt file for data for my table? Is there a better way to go about creating "dummy data" for my database?
What I ended up doing to insert my mock data was change the way the .txt file was formatted. Instead of having my C++ code write the data one entry per row, I actually made the code so that it wrote SQL code to the .txt file as I think #toddlermenot was suggesting, more or less. After I had the C++ code write as many inserts-with-mock-entries as I needed to the text file, I just copy/pasted it to SQL developer and achieved the desired results.
My problem is a classic case of making the process more complicated than it needed to be.
Also, even though I did not use the method,#Multisync provided an interesting way to go about achieving my goal. I had no idea SQL had the tools for me to generate mock data. Thanks for introducing me to that.
Ok, so the background to the story. I am largely self taught the bits of SQL i do know, and it tends to be just enough to make things work that need to work - albeit with a fair bit of research for the most basic jobs!
I am using a piece of software which grabs a string of data, and then passes it straight to an SQL stored procedure to move the data around, perform a few tasks on the string to make it the format i need it to be, and then grabs lumps of this data and places it in various SQL tables as outlined by the SP. I get maybe half a million lines of data each day, and this process works perfectly well and quickly. However, should data be lost, or not manage to make it through to the SQL database correctly, i do still have a log of the 500,000 lines of raw data in CSV file format.
I cant seem to find a way to simply bulk import this data into the various tables in the various formats it needs to be in. Assuming there is no way to re-pass this data through the 3rd party software (i have tried and failed), what is the best (read easiest for a relative lamen) way to send each line of this CSV file through my existing stored procedure, which can then process and import the data as normal? i have looked at the bcp utility, but that didnt seem to be viable (or i am not well enough informed to make it do what i need). I have also done the usual trawling of the web (and these forums) to see if anything jumped out at me as the obvious way forward, but come up a bit dry.
Apologies if i am asking something a bit 101, but i would certainly be grateful if anyone could help me out with this - if i missed out any salient bits of information, let me know! :)
Cheers.
The SQL Server Import/Export Wizard is a point and click solution that can be used to import CSV files into SQL Server.
The wizard builds an SSIS package behind the scenes, which can be saved and scheduled to run as needed. The wizard doesn't give you much in the way of data transformation, but the data could be loaded into a staging table and then processed by your existing stored procedure.
Why excel? Well excel is what is used to import the player salaries.
Now I need the spreadsheet to do the following.
Create teams within salary cap.
Include/exclude specific player function
Build multiple lineups from a selected list of players within the cap
Can I do all of this with excel? or do I need to know excel vba as well?
Also which parts of excel or if necessary excel vba must I need to know to code such a thing? Also if someone could give me a short summary of the steps needed to hypothetically make such a thing it would be great. Thanks.
I'm posting this as a reply because it is too long for a comment window.
Just because Excel has a grid doesn't mean that it is fit for data storage and data handling, on the contrary.
What you typically want to do is create a transparent structure that guarantees the integrity of your data and that allows dynamic portability when needed one day.
Excel is meant to be a spreadsheet, people forget this all the time or they just avoid the topic: although Excel has a grid, doesn't mean that it is a good fit for reliable data storage. It is not even the least complex way of storing data, depending on the amount of VBA that you need to manage all these data and the gates that you unnecessarily open towards potential bugs.
This is why an RDBMS is what will fit your needs, in this case Access would be a good option as it preserves your data integrity if you get the table structure right and it executes a lot of tasks for you that you should otherwise need to program yourself to protect the integrity of your data.
Although you can perform SQL on spreadsheets too, note that Excel does NOT cover related tables (what you typically seem to need for building your teams and salary limits), so what many Excel programmers will typically do is to create their own code to make this cross-table data storage thing work.
Don't do this if the alternative is available and much more reliable and future-proof.
At first sight, it seems that you won't even need any VBA; I'm not sure of all the things you want to do, but my first impression is that you can manage everything with SQL syntax and stored queries in MS Access. You can import Excel sheets into Access if you get their format right so that should not be a problem.
Once your data is stored there, rest assured that you have made your life a lot easier.
I have a few challenges I need help on. I need to pull data in to my SQL database from arbitrary sources.
The details are: I know the exact structure of my database and the structure will not change. When I do take in new data, it will occur only one time, at the time I set up an instance of my database. I will make many instances of my database and each time it will have to pull data from a different source, and those sources will be structured in different ways.
The data will most likely contain thousands of rows of records. The data source will most likely be held in Excel, Access, more rare Word and even rarer, it'll be held in a SQL database.
I can assume that most of the core data will be the same, just put in different locations. They will follow a general grouping despite how there held.
Essentially, I'm transferring data from legacy systems to a SQL system and this must be done for many groups and they need their own private instance of the database.
Any thoughts on how I would do this? How hard would it be to write a program that would do most of this for me?
This is definitely a real-world question. Is it possible to write a program that will do most of this? Not most of this, I think, but perhaps some of it.
For each table in your target system, create a view that displays the source data you expect to be able to insert. Choose column names that make it easy to tell what has to be done; most likely you'll choose column names that match the target columns in your INSERT statement. Save your INSERT statements as stored procedures.
Now, when you are given a new source of data in a new format, you will still have to recreate your views, but once the views are displaying the right data under your chosen column names, you can run your stored procedures without change.
I have a similar type of project where data is being retrieved from Access, .ini file, file modification dates, and MySql. I scrape this data every morning and basically append to a set SqlServer schema.
I created a DataTable and as I iterate a set of directories, insert the data into each new row. Once I have the DataTable complete, I perform a bulkcopy to append to the database.
I hope that helps you out a bit. I know my project doesn't cover all the aspects of your question; but also don't have a DBA to provide views, stored procedures, etc. Nor do I have the additional time to devote to such things. Not the most favorable of conditions, but that's the way it is.
HTH...
The best way of solving this problem is with and ETL (Extract-Transform-Load) solution. A good choice is SSIS which is through Microsoft's BI suite.
This is the building blocks for consciousness or the base......
1 A data base that organizes thousands of files similar to dna,
2 user interface
3 parts are hidden, preventing a system breach/crash