I have a python list containing last_name, first_name, title
I would like to insert this list into sql db, but the data should be inserted in 3 columns and many rows...
I tried following:
data = updated_values[7:-1] #list has around 200 names
query = "INSERT INTO names(last_name, first_name, titles) VALUES(%s,%s,%s)"
cursor.executemany(query,data)
but it throws error...
can anyone help me in this?
Thank you guys for all your help, I was able to find the solution and here it is:
it = iter(updated_values[7:-1])
updated_values_1 = list(zip(it, it, it))
query = "INSERT INTO names(last_name, first_name, titles) VALUES(%s,%s,%s)"
cursor.executemany(query,updated_values_1)
new_data = []
first_names = data[::3]
last_names = data[1::3]
titles = data[2::3]
x = 0
for first_name in first_names:
new_data.append( (first_names, last_names[x], titles[x] ))
x += 1
query = "INSERT INTO names(last_name, first_name, titles) VALUES(%s,%s,%s)"
cursor.executemany(query,new_data)
Basically separate your data then loop through one of the arrays and build a new arrray. This is untested so I'm sure you'll have to modify something.
Related
My current SQL:
SELECT B.MESSAGENO, B.LINENO, B.LINEDATA
FROM BILL.MESSAGE AS B, BILL.ACTIVITYAS A
WHERE A.MSGNO = D.MESSAGENO AND A.FUPTEAM = 'DBWB'
AND A.ACTIVITY = 'STOPPAY' AND A.STATUS = 'WAIT'
AND A.COMPANY = D.COMPANY
MESSAGENO LINENO LINEDATA
1234567 1 CHEQUE NO : 9999999 RUN NO : 55555
1234567 2 DATE ISSUED: 12/25/2020 AMOUNT : 710.51
1234567 3 PAYEE : LASTNAME, FIRSTNAME
1234567 4 ACCOUNT NO : 12345-67890
1234567 5 USER : USERNAME
there are 550 sets of 5 lines per MESSAGENO
What I am trying to figure out is how I can get something like where LINENO = 1, concatenate LINEDATA so I just get 9999999 as checkno, where LINENO = 2, concatenate LINEDATA so I get 710.51 as amount, where LINENO = 3, concatenate LINEDATA so I get LASTNAME, FIRSTNAME as payee, where LINENO = 4, concatenate LINEDATA so I get LASTNAME, FIRSTNAME as payee, and lastly, the same thing for USERNAME.
I just cannot seems to conceptualize this. Every time I try, my brain starts turning into macaroni. Any help is appreciated.
UPDATED ANSWER, extracts all fields from stored strings:
https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=e22d26866053ea6088aa78dc23c4d809
Check this fiddle.
It uses a SUBSTRING_INDEX in the internal query to split the fields at the : or by a combination of : and " ". I used the two spaces because I wasn't sure what your actual whitespace was, and when I copied the data out of your post it was just spaces.
Then MAX is used in the outer query to get everything on one line, grouping by the messageNo. Since some lines have two pieces of data to extract, a second string parser was added. Here is the code from the fiddle, for reference. In this case, the table test was created from the output of OP's initial query, since I didn't have enough info to build both tables and completely recreate his output.
SELECT MESSAGENO,
MAX(if(LINENO = 1, extractFirst, null)) as checkNo,
MAX(if(LINENO = 1, extractLast, null)) as runNo,
MAX(if(LINENO = 2, extractFirst, null)) as issued,
MAX(if(LINENO = 2, extractLast, null)) as amount,
MAX(if(LINENO = 3, extractFirst, null)) as payee,
MAX(if(LINENO = 4, extractFirst, null)) as accountNo,
MAX(if(LINENO = 5, extractFirst, null)) as username
FROM (
SELECT MESSAGENO, LINENO,
trim(substring_index(substring_index(LINEDATA, ": ", -2), " ", 1)) as extractFirst,
trim(substring_index(LINEDATA, ":", -1)) as extractLast
FROM test
) z
GROUP BY MESSAGENO
Again, you will be much better off to alter your tables so that you can use simpler queries, as shown in the last fiddle.
===============================================
ORIGINAL ANSWER (demo of string parsing, suggestion for data model change)
You can achieve this with some string parsing BUT ABSOLUTELY DO NOT unless you have no choice. The reason you are having trouble is because your data shouldn't be stored this way.
I've included a fiddle incorporating this case statement and substring_index to extract the data. I have assumed mySQL 8 because you didn't specify SQL version.
https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=068a49a2c819c08018691e54bcdf91e5
case when LINENO = 1 then trim(substring_index(substring_index(LINEDATA, "RUN NO", 1), ":", -1))
else trim(substring_index(LINEDATA, ":", -1)) end
as LDATA
See this fiddle for the full statement. I have just inserted the data from your join into a test table, instead of trying to recreate all your tables, since I don't have access to all the data I would need for that. In future, set up a fiddle like this one with some representative data and the SQL version, and it will be easier for people to help you.
=========================================
I think this is a better layout for you, with all data stored as the proper type and a field defined for each one and the extra text stripped out:
https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=11d52189b005740cdc53175466374635
I'm reading and executing sql queries from file and I need to inspect the result sets to count all the null values across all columns. Because the SQL is read from file, I don't know the column names and thus can't call the columns by name when trying to find the null values.
I think using CTE is the best way to do this, but how can I call the columns when I don't know what the column names are?
WITH query_results AS
(
<sql_read_from_file_here>
)
select count_if(<column_name> is not null) FROM query_results
If you are using Python to read the file of SQL statements, you can do something like this which uses pglast to parse the SQL query to get the columns for you:
import pglast
sql_read_from_file_here = "SELECT 1 foo, 1 bar"
ast = pglast.parse_sql(sql_read_from_file_here)
cols = ast[0]['RawStmt']['stmt']['SelectStmt']['targetList']
sum_stmt = "sum(iff({col} is null,1,0))"
sums = [sum_sql.format(col = col['ResTarget']['name']) for col in cols]
print(f"select {' + '.join(sums)} total_null_count from query_results")
# outputs: select sum(iff(foo is null,1,0)) + sum(iff(bar is null,1,0)) total_null_count from query_results
I am trying to download tweets from a list of three different accounts and then store all the informations in a SQL3 database.
I have tried with the code below, but it seems to run forever. Am I missing something? Is this because I used .executemany() instead of .execute()?
step=0
a_list=["A","B","C"]
for s in a_list:
cursor = tweepy.Cursor(api1.user_timeline, id = s, tweet_mode='extended').items(3189)
for tweet in cursor:
tw_text.append(tweet.full_text)
created_at.append(tweet.created_at)
rtws.append(tweet.retweet_count)
favs.append(tweet.favorite_count)
for h in tweet.entities['hashtags']:
hashlist.append(h['text'])
for u in tweet.entities['urls']:
linklist.append(u['expanded_url'])
try:
medialist.append(media['media_url'] for media in tweet.entities['media'])
except:
pass
step+=1
print('step {} completed'.format(step))
#preparing all the data for .executemany()
g = [(s,tw,crea,rt,fv,ha,li,me) for s in ['GameOfThrones'] for tw in tw_text for crea in created_at for rt in rtws for fv in favs for ha in hashlist for li in linklist for me in medialist]
cur.executemany("INSERT INTO series_data VALUES (?,?,?,?,?,?,?,?)", (g))
con.commit()
print('db updated')
I expect the program to write table in SQL3 but I never receive the message 'db updated' (i.e. the very last print() line)
cur.executemany() takes a list of tuples. Each tuple will have as many elements as number of columns you want to insert value for.
For example, if you have a table with following structure
create table tbl_test(firstname varchar(20), lastname varchar(20));
and you want to insert 3 records in it using executemany(), your object and the call should be like following
list = [('Hans', 'Muster'), ('John', 'Doe'), ('Jane', 'Doe')]
cur.executemany('insert into tbl_test values(?, ?)', list)
Let's say I have a Persons table with two columns:
ID (Uniqueidentifier)
Name (NChar)
I need to get all my persons first:
Dim data = (From p In Persons Select p).ToList
Now that I got all persons in the data variable, is it possible to query this result using a string query? Like...
Dim filtered = (From p In data Select p).Where("Name Like '%John%').ToList
?
I need to build the query on the fly.
Thanks
var filtered = data.Where(a =>
SqlMethods.Like(a.name.ToString(), "%" +
Request.QueryString["search"] + "%"));
i use querystring as an example for dynamic value, and use it in sql like method,
which is similiar to sql like, in your case "request.querystring["search"] value is john"
Try this:
from p in data
where SqlMethods.Like(p.Name, "%"+parameter+"%")
select p;
parameter in your example should be John.
Ive tried many queries to find... just one word and I can´t even make that.
Its a DB2 database Im using com.ibm.db2.jcc.DB2Driver
This brings me info:
select *
from JL_ENR
where id_ws = '002'
and dc_dy_bsn = '2014-08-25'
and ai_trn = 2331
the JL_TPE column is the CLOB column where I want to find two strings in that search result ( and dc_dy_bsn = '2014-08-25'
and ai_trn = 2331 ).
So first I tried with one:
select
dbms_lob.substr(clob_column,dbms_lob_instr(JL_TPE,'CEMENTO'),1)
from
JL_ENR
where
dbms_lob.instr(JL_TPE,'CEMENTO')>0;
didnt work
SELECT * FROM JL_ENR WHERE dbms_lob.instr(JL_TPE,'CEMENTO')>0
and ai_trn = 2331
and dc_dy_bsn = '2014-08-25'
didnt work
Select *
From JL_ENR
Where NOT
DBMS_LOB.INSTR(JL_TPE, 'CEMENTO', 1, 1) = 0;
didn´t work
Could someone explain me how to find two strings please?
Or a tutorial link where it is explained how to make it work...
Thanks.
Can you provide some sample data and the version you are using? Your example should work (tested on v10.5.0.1):
db2 "create table test ( x int, y clob(1M) )"
db2 "insert into test (x,y) values (1,cast('The string to find is CEMENTO, how do we do that?')"
db2 "insert into test (x,y) values (2,cast('The string to find is CEMENT, how do we do that?' as clob))"
db2 "select x, DBMS_LOB.INSTR(y, 'CEMENTO', 1) from test where DBMS_LOB.INSTR(y, 'CEMENTO', 1) > 0"
X 2
----------- -----------
1 23
1 record(s) selected.
I had to search for a specific value in the where clause. I used TEXTBLOB LIKE '%Search value%' and it worked! This was for db2 in a CLOB(536870912) column.