I have a text file of tab separated data. Millions of rows so I can't open it in Excel.
Is there a way to use some SQL client to load the file and run queries on it without uploading the file to a database? I.e. can my local machine function as my database?
I know I could solve this with command line scripting but I'm trying to find a solution that I can then share with the accounting department and they're scared of the command line.
You can create an ODBC datasource based on the Microsoft Text Driver. That driver should come pre-installed on any Windows PC. This uses the Jet Engine which is still somewhere in the Windows OS. Then you can use any Query tool (even MS Query) to analyse your data.
Related
I am trying to use Excel pull data from a large Oracle Data Warehouse via an ODBC connection. I have a query that works using the editor in Access. I've tried using Power Query and Microsoft SQL to use this query to get this data into Excel and I get errors.
Therefore:
Does SQL executed from Excel need to be in a different syntax? Shouldn't it still be Oracle?
How can I use this pre-written query to ping the data warehouse and get what I need?
Here is the SQL that I have so far. I had to change some table names...sorry if that makes it weird.
The SQL you posted uses Access-specific functions. That is NOT a valid SQL query if run directly against Oracle. If you have a bunch of linked tables in Access, that would allow this.
The ODBC datasource connection in Excel works differently. ODBC executes the query directly at the datasource. It does some validation first and supports a limited subset of SQL language supported at the destination. What's included in the subset is determined in part by the driver selected for the connection.
So what you want to do is use a tool that lets you build the query directly in an Oracle environment, like Quest Toad or Oracle SQL Developer. Once you have the query working there, it should be easier to port it to Excel.
One thing I like to do is put my query into a view on the database. Then I can just select everything from the view when creating the Excel connection.
It's also worth pointing you to the My Data Sources folder. When you first setup an ODBC connection in Excel, the connection is saved by default in Windows in a folder called "My Data Sources" located just under your user profile folder. For example: C:\Users\UserName\My Data Sources\Data Source Name.odc.
You can open these *.odc files in any text editor, and you should be able to manually edit the SQL here. Especially look for the <odc:CommandText> element. In this way you can build a simple query up front, and then improve on the SQL command in your favorite environment and easily move the updated SQL to the existing ODBC connection.
Disclaimer: I am somewhat of a n00b when it comes to database programming, so bear with me.
I've been attempting to batch process a rather large amount (~20 gb) of data all contained in .MDF SQL database files. The files contain meteorological data obtained through weather balloons, with each table consisting of ~1 second observations of winds, pressure, height, temps, etc, and are created with our radiosonde tracking software on an unnetworked Windows machine. It is possible (and quite easy) to load the files using the associated software and export the tables as an ASCII text file...however, this process involves manually loading each one. As I'm performing a study that requires as many soundings as possible (we have over 2000), doing this process over and over for several years of twice-daily observations is extremely time-prohibitive.
I've been taking the files off of the computer and putting them on my laptop running Linux Mint, and consider myself to be fluent with Perl...I do most of my data analysis with Perl scripts. That said, I've had the darndest time trying to get into the database files!
I've tried to connect to one of the files using the DBI package using variants on
$dbh = DBI->connect("DBI:ODBC:$filename") or die "blahblahblah";
I have unixODBC installed and configured, have downloaded "libmyodbc.so" and "libodbcmyS.so", and keep getting the error
DBI connect('','',...) failed: [unixODBC][Driver Manager]Data source name not found, and no default driver specified (SQL-IM002) at dumpsql.pl line 6.
I've tried remedying this a number of ways over the past couple days, and I won't post them here for the sake of brevity. My odbcinst.ini file is as follows:
[MySQL]
Description = ODBC for MySQL
Driver = /usr/lib/x86_64-linux-gnu/odbc/libmyodbc.so
Setup = /usr/sib/x86_64-linux-gnu/odbc/libodbcmyS.so
FileUsage = 1
I'm seriously confused. I THINK I'm doing everything that various online tutorials are suggesting, but everyone else is connecting to servers and these files are all local and in the same directory! Could anyone attempt to point me in the right direction? All I want is to calculate meteorological values using vertical sounding data! Am I missing something totally obvious?
Any help would be greatly appreciated!
It seems the original database server was a Microsoft SQL Server (MDF files). I am afraid these files alone are useless on a Linux machine. You need a Microsoft SQL Server on a Windows machine to get access to the contained data.
You described that you are able to attach a MDF file on a SQL server manually and then you can export the needed data as text files. Try to automate that. I'm not a MS SQL Server expert but it should be possible.
E.g. here is a tutorial to attach and detach a MDF file via T-SQL. So my approach would be to write a script which iterates over the 2000 MDF files and attach each to the SQL server. Then execute a query to export your data and then detach the MDF.
I have a query in MS Access database. I want to run this query in SQLite database. I wondering whether there is a easy way to convert the code or just change step by step?
You have several options available for viewing the contents of a sqlite database and running queries against it. When building sqlite from source you can choose to build the shell utility.
If you want something more turnkey you can install a free tool such as Sqlite Manager for Firefox or Sqlite Browser.
In both apps you can copy-and-paste your SQL that you want to execute against a SQLITE database file. Good luck!
I've been searching without luck for a MacOS iODBC driver that can read saved .SQL files exported in Microsoft SQL Server format. Does one exist?
We've got a large pile of research data stored in one app that can export as Excel spreadsheets or SQL files (eg, a text file full of SQL CREATE TABLE and INSERT statements). We need to import this data into another app (Stata 9) that runs under MacOS and can import Excel files, its own format, or from an ODBC source. So, I need an ODBC driver that can read plain SQL files as its source. We don't need a driver that actually talks to an MSSQL database, because there is no actual database here; just a plain .SQL file with MSSQL-style commands in it.
Unfortunately, the default MacOS install seems to come with no ODBC drivers whatsoever, not even one for reading flat files or SQLite databases.
The current workflow for moving this data — exporting it from DatStat as an Excel spreadsheet, opening that spreadsheet and fixing it by hand to conform to Stata's need, then saving and reimporting into Stata — is ridiculously labor-intensive and also loses a lot of important metadata like variable descriptions and annotations.
I think that best thing to do here is load the data from DatStat to a database and then load it back into Stata. First, export your data from DatStat to a .sql file. I'm not familiar with DatStat, but if you can do this in bulk or via the command line it would be best. You can access your OS's terminal in Stata by using the -shell- command. After you have a .sql file, say foo.sql, you can use the following Stata code to send it to a database and then import into Stata.
odbc sqlfile("foo.sql"), dsn("DataSourceName")
odbc load, exec("SELECT * FROM CustomerTable") dsn("DataSourceName")
You could even issue a final command to cleanup the tables in the database if you don't think you'll use this database again and you don't want it taking up space. Use something like:
odbc exec("DROP TABLE CustomerTable")
Yes, this will probably be slow if your dataset is large, but it could be nice once your data is in the database because you can query parts of it at a time instead of importing the whole thing.
Lastly, you mentioned that no ODBC driver for Mac exists for MS SQL Server. If that is the case, you may want to install one of the open-source database systems like MySQL or PostgreSQL. I'm not a Mac user but drivers for these must exist for mac.
Good luck!
Similar to this question only the other way of flow.
Insert Picture into SQL Server 2005 Image Field using only SQL
I need to be able to save a image field out into the file system and be able to name the file with only using SQL. I don't want to use TEXTCOPY either because I need to use the connection to the database that is running the query itself.
I'd like this to work in SQL 2005 to support older database clients but if it's available in only 2008 that's fine.
Any ideas?
With SQL 2k8 there is the new FILESTREAM type that covers such cases. Filestreams can be opened via the Win32 file access handle like any other file, but hey are integrated into the database from transaction and backup/restore point of view.
I had a similar issue in SQL 2k5 and my solution was to use a CLR stored procedure with EXTERNAL_ACCESS that was writing into the file system using C# file operations.
I don't believe there is a way to save out using only sql, but there is a command-line utility, bcp.