Firebird External Tables - sql

Question

I am trying to find a way to quickly load a lot of data into database and one suggested to use Firebird External Tables, I would like know more about this method, I've tried searching online but I'm not getting the useful information about this, I want to know how do they really work? Do the tables have to be exactly the same? and what if you are loading data from more than one database?

Answer 1

Use external tables like this:
CREATE TABLE ext1 EXTERNAL 'c:\myfile.txt'
(
field1 char(20),
field2 smallint
);
To do quick import into regular table, do something like this:
INSERT INTO realtable1 (field1, field2)
SELECT field1, field2 FROM ext1;
Remember to disable triggers and indexes (if possible) before loading, and reactivate them after.
This information is from Firebird FAQ: http://www.firebirdfaq.org/faq209/
Here's more information about using external tables, including information about file format: http://www.delphiman.de/Bin/UsingExternalFilesAsTables.pdf

Answer 2

Using an external file as a table is a great way to get lots of data into Firebird quickly. However, the sample, which is from the Firebird FAQs, seems to me to be either unnecessarily complex or incorrect, because of use of smallint in table definition. As the FB 2.5 documentation points out, "for most purposes, only columns of CHAR types would be useful."
The external file must be a text file of fixed-length records (so a .csv file won't work). The external table def should then use CHAR fields with sizes that match the lengths of the fields in each record.
Any variation in the length of the records in the text file will lead to misery (from bitter experience). I suppose the example possibly might work if all of the smallints were the same number of digits but more generically, things will go more smoothly if other formats (date, numeric) are simply expressed as CHAR in the text file by padding with spaces.
For example, if the raw data looked like this:
Canada 37855702
Central African Republic 4829764
Chad 16425859
Chile 19116209
China 1404676330
Then the text file should look like this:
Canada 37855702
Central African Republic 4829764
Chad 16425859
Chile 19116209
China 1404676330
Countries are right-padded to twenty-five characters and the (big) integers are left-padded to 10 characters, so the records are 35 characters, plus one for a line feed (*nix) or two for Window's CRLF. (Note that things get more complicated if the file uses Unicode encoding.)
The table def would look like this:
CREATE TABLE ext_test EXTERNAL '/home/dave/fbtest.txt'
(
COUNTRY CHAR(25),
POPULATION CHAR(10),
LF CHAR(1)
);
Make sure that the file resides on the same file system as the FB server process, that the server process has rights to the file (maybe through a FB group) and that the ExternalFileAccess parameter in firebird.conf is set appropriately - see the 2.5 documentation for details.
There are some limited things you can do with an external table, but it's most useful as a temporary transfer table, as a source for the ultimate FB table. INSERT each row from the external table into the ultimate target, casting the CHAR fields to the appropriate data types. For data of any real volume, the process runs much faster than, say, some Python code to read and feed each line individually.
If you are using an older version of FB, don't forget to DROP the external table when you're done with it to free up file locks, as outlined in the FAQs. Newer versions do this automatically. There's lots more on external tables in the 2.5 documentation at the above link.
PS - I have emailed the above to the Firebird documentation team.