I suppose this is somewhat of an extension of the question asked here.
However, I believe the linked OP and my own reason for reading a file with SQL Developer are different. I am learning SQL and databases and am attempting to create a model database (as in, I won't be editing the data after insertion, just set up search queries and what not). I want it to be large (over 100,000 entries), so I've created a C++ program that wrote randomly generate entries for the database on a .txt file (one entry per line in the .txt file) instead of hard coding the insertion of each entry. Now what I want to do is read the .txt file in SQL Developer and inserts it into a table.
My problem lies in the fact that I am not able to create directories. I am using a university oracle connection and I do not have the privileges to actually make a directory so that I can use UTL_FILE on my .txt file as was answered in the linked question. Assuming there is no way for me to gain this permission, is there an alternate way I can accomplish reading a .txt file for data for my table? Is there a better way to go about creating "dummy data" for my database?
What I ended up doing to insert my mock data was change the way the .txt file was formatted. Instead of having my C++ code write the data one entry per row, I actually made the code so that it wrote SQL code to the .txt file as I think #toddlermenot was suggesting, more or less. After I had the C++ code write as many inserts-with-mock-entries as I needed to the text file, I just copy/pasted it to SQL developer and achieved the desired results.
My problem is a classic case of making the process more complicated than it needed to be.
Also, even though I did not use the method,#Multisync provided an interesting way to go about achieving my goal. I had no idea SQL had the tools for me to generate mock data. Thanks for introducing me to that.
Related
I am tasked to create a template that will be Filled up by Business Users with Employee Information, then our program will load this into the Database using External Tables.
However, our Business Users constantly change the template by adding, removing or reordering fields.
I am convinced to use XLSX instead of CSV so that I can lock the Column Headers so they cannot remove, add and reorder the columns.
However, When i query the External Table, it shows Non-ASCII Characters when reading XLSX because its in Binary.
How can i do either of the following?
Effectively Read Excel Files from External Tables
Lock the Headers of CSV Files?
What you have here is a political problem, but you are looking for a technical fix. Not a good fit.
The problem comes in two halves:
Somebody decided it was a good idea to collect user input in a spreadsheet, which it is generally not.
Users are fiddling with the input format, which they should not.
Fixes are:
Strictly enforce the data structure. Reject any CSV which doesn't natch and make the users edit them. They will quickly tire of tweaking the spreadsheets when they realise they're just creating more work for themselves. But they will also get resentful, so consider ...
Building a data input screen. It's pretty simple to knock up a spreadsheet-like grid UI. You don't need anything complicated in Java: Oracle's Apex is intended for exactly this sort of thing. Find out more.
However, if you are stuck with Excel as a UI I suggest you have a look at Anton Scheffer's excellent PLSQL as_read_xlsx package on the Amis site. Check it out. You'll probably need to replace your external table with a view over a table (perhaps pipelined) function.
Ok, so the background to the story. I am largely self taught the bits of SQL i do know, and it tends to be just enough to make things work that need to work - albeit with a fair bit of research for the most basic jobs!
I am using a piece of software which grabs a string of data, and then passes it straight to an SQL stored procedure to move the data around, perform a few tasks on the string to make it the format i need it to be, and then grabs lumps of this data and places it in various SQL tables as outlined by the SP. I get maybe half a million lines of data each day, and this process works perfectly well and quickly. However, should data be lost, or not manage to make it through to the SQL database correctly, i do still have a log of the 500,000 lines of raw data in CSV file format.
I cant seem to find a way to simply bulk import this data into the various tables in the various formats it needs to be in. Assuming there is no way to re-pass this data through the 3rd party software (i have tried and failed), what is the best (read easiest for a relative lamen) way to send each line of this CSV file through my existing stored procedure, which can then process and import the data as normal? i have looked at the bcp utility, but that didnt seem to be viable (or i am not well enough informed to make it do what i need). I have also done the usual trawling of the web (and these forums) to see if anything jumped out at me as the obvious way forward, but come up a bit dry.
Apologies if i am asking something a bit 101, but i would certainly be grateful if anyone could help me out with this - if i missed out any salient bits of information, let me know! :)
Cheers.
The SQL Server Import/Export Wizard is a point and click solution that can be used to import CSV files into SQL Server.
The wizard builds an SSIS package behind the scenes, which can be saved and scheduled to run as needed. The wizard doesn't give you much in the way of data transformation, but the data could be loaded into a staging table and then processed by your existing stored procedure.
i have to insert and update some values which is daily coming from
excel file but as everyday excel file format is different
so tell me other possible ways to automate insert update ?
Are the excel files really in different formats or does Excel just think they are different? If the columns are still in the same ordinal positions but they are being interpreted as having different data types, then yes, you can provide hints to the driver to overcome.
Otherwise, you could use C#/vb.net and query the worksheet, dump that into an dataset, write that to a variable and then shred that object but it's ugly. In fact, dealing with Excel in a programmatic fashion is always ugly and best avoided.
If your file is a different format each day then you are out of luck. That is a problem, and there is really no easy or efficient way to parse and insert/update based on that. Whatever the source of the data is, you need to ensure that it becomes consistent.
If it is a handful of formats that you can test and handle accordingly then you could always have some data flow logic inside the SSIS package, but if this isn't predetermined then you would have no way of handling these cases.
YOu deal with this by returning the file to the provider and requiring them to provide in the same fashion every day. Then your SSIS pacakge should reject the file if it is not in the correct format. While you aer at it you will have far fewer problems if they send .txt or .csv file. Excel support is exceedingly poor.
This question is in continuation to my previous question related to File I/O.
I am using RFile to open a file and read/write data to it. Now, my requirement is such that I would have to modify certain fields within the file. I separate each field within a record with a colon and each record with a newline. Sample is below:
abc#def.com:Albert:1:2
def#ghi.com:Alice:3:1
Suppose I want to replace the '3' in the second record by '2'. I am finding it difficult to overwrite specific field in the file using RFile because RFile does not provide its users with such facility.
Due to this, to modify a record I have to delete the contents of the file and serialize ( that is loop through in memory representation of records and write to the file ). Doing this everytime there is a change in a record's value is quite expensive as there are hundreds of records and the change could be quite frequent.
I searched around for alternatives and found CPermanentFileStore. But I feel the API is hard to use as I am not able to find any source on the Internet that demonstrates its use.
Is there a way around this. Please help.
Depending on which version(s) of Symbian OS you are targetting, you could store the information in a relational database. Since v9.4, Symbian OS includes an SQL implementation (based on the open source SQLite engine).
Using normal files for this type of records takes a lot of effort no matter the operating system. To be able to do this efficiently you need to reserve space in the file for expansion of each record - otherwise you need to rewrite the entire file if a record value changes from say 9 to 10. Also storing a lookup table in the file will make it possible to jump directly to a record using RFile::Seek.
The CPermamanentFileStore simplifies the actual reading and writing of the file but basically does what you have to do yourself otherwise. A database may be a better choice in this instance. If you don't want to use a database I think using stores would be be a better solution.
My company has a legacy micro-simulation program that simulates a population and changes to that population over a period of years.
For each year, the program produces a binary file with a record for each individual that holds their characteristics (e.g., age, maritial status, income ... about 20 fields).
We currently have several utility programs that read these files and produce summary reports. Problem is that each time somebody wants a new report, a new utility program has to be written.
Changing the program so that the records are stored in a database instead of binary files is out of the question (I have asked ... several times). I have written a few programs that import the binary files into a database and then run queries on the tables I have created. The problem here is that it always takes longer to import the data and run the query than it does to run a utility program written in c++ that just read the records one by one and accumulate the desired data. Often the binary files contain over 30 million records and the import step alone takes forever.
So here is my question. Is there anything out there that would allow me to specify the structure of my binary file and then run SQL queries on the file? I think you can use ODBC to run queries on plain text files, but I've never seen anything like that for binary files.
If there isn't anything available, what are the steps I would need to take to build something that could run a query directly on my file? I understand this would probably be way beyond my ability, but it can't hurt to know where I would need to start.
OpenAccess is a toolkit that you can use to build ODBC or JDBC drivers for arbitrary systems. Disclaimer: I've not used it, and another division of my company sells it.
It's possible using SSIS: Loading Binary Files into SQL Server Using SSIS
This amight also be of interest: Reading and Writing Files in SQL Server using T-SQL
I do not have much experience with LINQ, but couldn't you use InteropServices to parse the binary files into C# objects and then query stuff out with LINQ's SQL?