Import (non-CSV) text data to PostgreSQL, which is separated via spaces and one capital letter - sql

Question

This is the first time I am working with SQL. I am using PostgreSQL on Windows 7 64bit.
I have the following (large) .txt file of tweets built like this:
T 2009-06-07 02:07:41
U http://twitter.com/cyberplumber
W SPC Severe Thunderstorm Watch 339: WW 339 SEVERE TSTM KS NE 070200Z - 070800Z URGENT - IMMEDIATE BROADCAST REQUE.. http://tinyurl.com/5th9sw
As you see, all three "columns" are separated in the following fashion: T \t (same goes for U and W) instead of the traditional comma (,).
I would like to import the whole file into a SQL table with columns named date, user and text_msg.
I am guessing I will probably have to parse it in some way. Any ideas how to get the data into a table in the simplest and most efficient manner? Please also consider that the .txt files in question are rather huge (>4GB) and thus there is no easy way for me to edit them manually.

Answer 1

Quick&dirty hack:
DROP SCHEMA tmp CASCADE;
CREATE SCHEMA tmp ;
SET search_path=tmp;
CREATE TABLE lutser
( id SERIAL NOT NULL PRIMARY KEY
, ztxt text
);
CREATE TABLE tweetdeck
( id SERIAL NOT NULL PRIMARY KEY
, stamp timestamp NOT NULL
, zurl text
, ztxt text
);
COPY lutser(ztxt)
FROM '/tmp/tweet.dat'
;
INSERT INTO tweetdeck (stamp, zurl, ztxt)
SELECT regexp_replace( t.ztxt, E'^[A-Z][ \t]*', '')::timestamp
, regexp_replace( u.ztxt, E'^[A-Z][ \t]*', '')
, regexp_replace( w.ztxt, E'^[A-Z][ \t]*', '')
FROM lutser t
JOIN lutser u ON u.id = t.id+1
JOIN lutser w ON w.id = t.id+2
WHERE t.id %3 = 1
AND LEFT(t.ztxt,1) = 'T' -- Should be redundant, Won't harm
AND LEFT(u.ztxt,1) = 'U'
AND LEFT(w.ztxt,1) = 'W'
;
SELECT * FROM lutser;
SELECT * FROM tweetdeck;
Results:
COPY 9
INSERT 0 3
id | ztxt
----+--------------------------------------------------------------------------------------------------------------------------------------------------
1 | T 2009-06-07 02:07:31
2 | U http://twitter.com/cyberplumber
3 | W SPC Severe Thunderstorm Watch 339: WW 339 SEVERE TSTM KS NE 070200Z - 070800Z URGENT - IMMEDIATE BROADCAST REQUE.. http://tinyurl.com/5th9sw
4 | T 2009-06-07 02:07:41
5 | U http://twitter.com/cyberplumber
6 | W SPC Severe Thunderstorm Watch 339: WW 339 SEVERE TSTM KS NE 070200Z - 070800Z URGENT - IMMEDIATE BROADCAST REQUE.. http://tinyurl.com/5th9sw
7 | T 2009-06-07 02:07:51
8 | U http://twitter.com/cyberplumber
9 | W SPC Severe Thunderstorm Watch 339: WW 339 SEVERE TSTM KS NE 070200Z - 070800Z URGENT - IMMEDIATE BROADCAST REQUE.. http://tinyurl.com/5th9sw
(9 rows)
id | stamp | zurl | ztxt
----+---------------------+---------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------
1 | 2009-06-07 02:07:31 | http://twitter.com/cyberplumber | SPC Severe Thunderstorm Watch 339: WW 339 SEVERE TSTM KS NE 070200Z - 070800Z URGENT - IMMEDIATE BROADCAST REQUE.. http://tinyurl.com/5th9sw
2 | 2009-06-07 02:07:41 | http://twitter.com/cyberplumber | SPC Severe Thunderstorm Watch 339: WW 339 SEVERE TSTM KS NE 070200Z - 070800Z URGENT - IMMEDIATE BROADCAST REQUE.. http://tinyurl.com/5th9sw
3 | 2009-06-07 02:07:51 | http://twitter.com/cyberplumber | SPC Severe Thunderstorm Watch 339: WW 339 SEVERE TSTM KS NE 070200Z - 070800Z URGENT - IMMEDIATE BROADCAST REQUE.. http://tinyurl.com/5th9sw
(3 rows)

Answer 2

Try doing the following:
Firstly, create an appropriate table in SQL like so:
CREATE TABLE tweet(
ts timestamp, -- if inserting the values as timestamps gives errors, change to 'TEXT'
url TEXT, -- There smarted UDTs for URL available too
message TEXT
);
Then go on and try to run a standard COPY statement, something like the following:
COPY tweet
FROM E'c:\\\\my dir\\\filename' -- path of the file using the magic E for escaped text with double backslash between directory names fro Windows
FORMAT text; -- The default delimiter for format text is a tab
Finally, pray that you'll have enough memory and log spaces for > 4GB of files. For more info regarding the COPY command, see http://www.postgresql.org/docs/9.2/static/sql-copy.html