I have multiple JSON files, they all have same format but the values are different based on each transaction. I want to migrate this data to a postgresql table. What is the best way to proceed with this?
Right now, I am using the following query:
CREATE TABLE TEST (MULTIPROCESS VARCHAR(20), HTTP_REFERER VARCHAR(50));
INSERT INTO TEST SELECT MULTIPROCESS, HTTP_REFERER FROM json_populate_record(NULL::test, '{"multiprocess": true,"http_referer": "http://localhost:9000/"}');
But, once the number of files become large, it becomes very difficult to use this technique. Is there any other way to do this effectively?
You could use a LATERAL JOIN to do insert more than one row at a time:
WITH
json AS(
VALUES('{"multiprocess": true,"http_referer":"http://localhost:9000"}')
,('{"multiprocess": false,"http_referer": "http://localhost:9001/"}')
,('{"multiprocess": true,"http_referer": "http://localhost:9002/"}')
) INSERT INTO test
SELECT multiprocess, http_referer
FROM json, LATERAL json_populate_record(NULL::test, json.column1::json);
Or you could insert into a staging table first and then populate your other table.
Related
I have a table that I'm trying to append unique values to. Every month I get list of user logins to import into this table. I would like to keep all the original values and just append the new and unique values onto the existing table. Both the table and the flatfile have a single column, with unique values, built like this:
_____
login
abcde001
abcde002
...
_____
I'm bulk ingesting the flat file into a temp table, with this:
IF OBJECT_ID('tempdb..#FLAT_FILE_TBL') IS NOT NULL
DROP TABLE #FLAT_FILE_TBL
CREATE TABLE #FLAT_FILE_TBL
(
ntlogin2 nvarchar(15)
)
BULK INSERT #FLAT_FILE_TBL
FROM 'C:\ImportFiles\logins_Dec2021.csv'
WITH (FIELDTERMINATOR = ' ');
Is there a join that would give me the table with existing values + new unique values appended? I'd rather not hard code a loop to evaluate it line by line.
Something like (pseudocode):
append unique {login} from temp_tbl into original_tbl
Hopefully it's an easy answer for someone out there.
Thanks!
Poster on Reddit r/sql provided this answer, which I'm pursuing:
Merge statement?
It looks like using a merge statement will do exactly what I want. Thanks for those who already posted replies.
You can check if a record exists using 'EXISTS' clause and insert if it doesn't exist in the target table. You can also use MERGE statement to achieve the same. Depending on what you want to do to the existing records in the target table, you can modify the Merge statement. Here since you only want to insert new records, you need to specify only what you want to do when a new record comes in. Here is an example
MERGE original_tbl T
USING temp_tbl S
ON T.login = S.login
WHEN NOT MATCHED THEN
INSERT (login)
VALUES(S.login)
Another solution would be to left join the target table to the temp table and insert only when the record doesn't exist.
INSERT INTO original_tbl(login)
SELECT S.Login
FROM temp_tbl S
LEFT JOIN original_tbl T
ON S.Login = T.Login
WHERE T.Login IS NULL
Using the Big Query Streaming API its possible to partition tables with a template suffix:
<targeted_table_name> + <templateSuffix>
eg. targettable_suffix
How can this be done from the web ui in an insert statement?
For example:
insert into `project123.dataset123.targettable_suffix`
(`id`, `value`) values ('123', 'abc')
(Where a table exists called targettable but the suffix table has not been created.)
The INSERT statement requires that the target table has been created in advance. Use a CREATE TABLE statement first to create it, then use INSERT. Note that the BigQuery team strongly recommends using partitioned tables instead of multiple tables that share a prefix, however, and if you use a partitioned table, you only need to create it once.
How would I insert into multiple tables with one sql script in db2
For example, insert a row into T1 DOCK_DOOR and then insert into T2 DOCK_DOOR_LANE multiple times based on the dock_door_sysid from the first table.
My first approach was the following. I was attempting to use a with with three inserts. on the other hand, doing to inserts on the second table is not and option if this can be automated with one insert.
thanks for any feedback
sql example
WITH ins AS (
INSERT INTO DBF1.DOCK_DOOR (DOCK_DOOR_SYSID,DOOR_NUMBER,DOOR_NAME,DOCK_SYSID,DOOR_SEQ,ENCRYPTION_CODE,RFID_ENBLD_FLAG,LANES_COUNT,CMNT_TEXT,CREATE_TS,CREATE_USERID,UPDATE_TS,UPDATE_USERID,VER_NUMBER,ACTIVE_FLAG,STATUS_SYSID,DOOR_TYPE_SYSID)
VALUES (nextval for DBF1.DOCK_DOOR_SEQ,'026','DOOR025',61,25,NULL,'N','2',NULL,current timestamp,'SQL_INSERT',current timestamp,'SQL_INSERT',0,NULL,1723,1142)
RETURNING door_number,dock_door_sysid),
ins2 AS (
INSERT INTO SIT.DOCK_DOOR_lane (DOCK_DOOR_LANE_SYSID,DOOR_LANE_ID,DOCK_DOOR_SYSID,LANE_ID,CREATE_TS,CREATE_USERID,UPDATE_TS,UPDATE_USERID,VER_NUMBER)
VALUES (nextval for DBF1.DOCK_DOOR_LANE_seq,door_number||''||'A',dock_door_sysid,'A',current timestamp},'SQL_INSERT',current timestamp,'SQL_INSERT',0)
SELECT door_number,dock_door_sysid FROM DBF1.DOCK_DOOR
RETURNING door_number,dock_door_sysid)
INSERT INTO DBF1.DOCK_DOOR_lane (DOCK_DOOR_LANE_SYSID,DOOR_LANE_ID,DOCK_DOOR_SYSID,LANE_ID,CREATE_TS,CREATE_USERID,UPDATE_TS,UPDATE_USERID,VER_NUMBER)
VALUES (nextval for DBF1.DOCK_DOOR_LANE_seq,door_number||''||'B',dock_door_sysid,'B',current timestamp},'SQL_INSERT',current timestamp,'SQL_INSERT',0)
SELECT door_number,dock_door_sysid FROM DBF1.DOCK_DOOR;
Table 1 = dock_door
Table 2 = Dock_door_lane
You could do it with a trigger on the dock_door table.
However, if you're on a recent, version on IBM i. You might be able to make use of data change table reference
Your statement would look something like this
insert into dock_door_lane
select <....>
from final table (insert into dock_door <...>)
I'm not sure it will work, as this article indicates that at least at a couple of years ago DB2 for i didn't support the secondary insert required.
This old SO question also seems to confirm that at least at v7.1, the double insert isn't supported.
If I get a chance, I'll run a test on a 7.2 system Monday.
I have a local db that I'm trying to insert multiple rows of data, but I do not want duplicates. I do not have a second db that I'm trying to insert from. I have an sql file. The structure is this for the db I'm inserting into:
(db)artists
(table)names-> ID | ArtistName | ArtistURL | Modified
I am trying to do this insertion:
INSERT names (ArtistName, Modified)
VALUES (name1, date),
(name2, date2),
...
(name40, date40)
The question is, how can I insert data and avoid duplication by checking a specific column to this list of data that I want inserted using SQL?
Duplicate what? Duplicate name? Duplicate row? I'll assume no dup ArtistName.
Have UNIQUE(ArtistName) (or PRIMARY KEY) on the table.
Use INSERT IGNORE instead of IGNORE.
(No LEFT JOIN, etc)
I ended up following the advice of #Hart CO a little bit by inserting all my values into a completely new table. Then I used this SQL statement:
SELECT ArtistName
FROM testing_table
WHERE !EXISTS
(SELECT ArtistName FROM names WHERE
testing_table.ArtistName = testing_table.ArtistName)
This gave me all my artist names that were in my data and not in the name table.
I then exported to an sql file and adjusted the INSERT a little bit to insert into the names table with the corresponding data.
INSERT IGNORE INTO `names` (ArtistName) VALUES
*all my values from the exported data*
Where (ArtistName) could have any of the data returned. For example,
(ArtistName, ArtistUrl, Modified). As long as the values returned from the export has 3 values.
This is probably not the most efficient, but it worked for what I was trying to do.
Ok, I am learning SQL and just installed SQL Server. I've read about outer joins and inner joins but am not sure either is what I want. Basically, I want to reconstruct a text file that has been "chopped" up into 5 smaller text files. The columns are the same across all 5 text files, e.g. name, age, telephone #, etc. The only difference is that they have different numbers of rows of data.
What I'd like to do is "append" the data from each file into one "mega-file". Should I create a table containing all of the data, or just create a view? Then, how do I implement this...do I use union? Any guidance would be appreciated, thanks.
Beyond your immediate goal of merging the five files it sounds like you want the data contained in your text files to be generally available for more flexible analysis.
An example of why you might require this is if you need to merge other data with the data in your text files. (If this is not the case then Oded is right on the money, and you should simply use logparser or Visual Log Parser.)
Since your text files all contain the same columns you can insert them into one table*.
Issue a CREATE statement defining your table
Insert data into your newly created table**
Create an index on field(s) which might often be used in query predicates
Write a query or create a view to provide the data you need
*Once you have your data in a table you can think about creating views on the table, but to start you might just run some ad hoc queries.
**Note that it is possible to accomplish Step 2 in other ways. Alternatively you can programmatically construct and issue your INSERT statements.
Examples of each of the above steps are included below, and a tested example can be found at: http://sqlfiddle.com/#!6/432f7/1
-- 1.
CREATE TABLE mytable
(
id int identity primary key,
person_name varchar(200),
age integer,
tel_num varchar(20)
);
-- 2. or look into BULK INSERT option https://stackoverflow.com/q/11016223/42346
INSERT INTO mytable
(person_name, age, tel_num)
VALUES
('Jane Doe', 31, '888-888-8888'),
('John Smith', 24, '888-555-1234');
-- 3.
CREATE UNIQUE INDEX mytable_age_idx ON mytable (age);
-- 4.
SELECT id, person_name, age, tel_num
FROM mytable
WHERE age < 30;
You need to look into using UNION.
SELECT *
FROM TABLE1
UNION
SELECT *
FROM TABLE2
And I would just create a View -- no need to have a stored table especially if the data ever changes.