multiple columns from single column in python - pandas
I am trying to split a column into multiple columns
The column has values like this:
message
------------
time=15:45:19 devname="FG3H0E3917903319" devid="FG3H0E3917903319"
logid="1059028705" type="utm" subtype="app-ctrl" eventtype="app-ctrl-all"
level="warning" vd="root" eventtime=1564226119 appid=16009
srcip=172.24.208.2 dstip=93.184.221.240 srcport=4832 dstport=80
srcintf="LAN-RahaNet" srcintfrole="lan" dstintf="WAN-RahaNet"
dstintfrole="lan" proto=6 service="HTTP" direction="outgoing" policyid=43
sessionid=493024483 applist="LanAppControl" appcat="Update"
app="MS.Windows.Update" action="block"
hostname="www.download.windowsupdate.com" incidentserialno=1522726002
url="/msdownload/update/v3/static/trustedr/en/authrootseq.txt" msg="Update:
MS.Windows.Update," apprisk="elevated"
Basically I need to split this column into:
time devname devid ...
--------------------------------------------------------------
15:45:19 FG3H0E3917903319 FG3H0E3917903319 ...
short answer:
split the message on space, to get a list of key value pairs.
split every key-value pair on = sign.
add corresponding keys to their respective columns.
Related
How to use JSON_MODIFY to change all of the keys in a column that has an array of JSON objects?
I have a column in my database that looks like this (3 separate rows of data)
Columns
[{"header":"C", "value":"A"},{"header":"D","value":"A2"},{"header":"E","value":"A3"}]
[{"header":"C", "value":"B"},{"header":"D","value":"B2"},{"header":"E","value":"B3"}]
[{"header":"C", "value":"C"},{"header":"D","value":"C2"},{"header":"E","value":"C3"}]
I want to null out all of the values of the "header" key and change the name to be test
I also want to change the name all of the "value"'s to be newHeader
I tried running a script like this to change all of the headers inside the array to be test but it doesn't expect the '*' character.
UPDATE Files
SET Columns = JSON_MODIFY(
JSON_MODIFY(Columns,'$.test', JSON_VALUE(Columns,'$[*].header')),
'$[*].header',
NULL
)
The end result I want to be like this:
Columns
[{"test":"", "newHeader":"A"},{"test":"","newHeader":"A2"},{"test":"","newHeader":"A3"}]
[{"test":"", "newHeader":"B"},{"test":"","newHeader":"B2"},{"test":"","newHeader":"B3"}]
[{"test":"", "newHeader":"C"},{"test":"","newHeader":"C2"},{"test":"","newHeader":"C3"}]
Ordering by sub string of a field with OPNQRYF
I have a requirement where I need to change the order in which records are printed in a Report. I need to order the records by a substring of a field of the records. There is an OPNQRYF as below before the call to the report RPG is made: OVRDBF FILE(MOHDL35) SHARE(*YES) BLDQRYSLT QRYSLT(&QRYSLT) + SELECT((CHARDT *GE &FRDATE F2) + (CHARDT *LE &TODATE F2) + (HDPLVL *EQ 'FS' F2) + (HDMPLT *EQ &PLANT F2)) OPNQRYF FILE((*LIBL/MOHDL35)) + QRYSLT(&QRYSLT) + KEYFLD(*FILE) + MAPFLD((ZONEDT HDAEDT *ZONED 8 0) + (CHARDT ZONEDT *CHAR 8)) One way I see how to do this is to do a RUNSQL to create a temp table in qtemp with the MOHDL35 records in the required order. The substr SQL function would help to achieve this much easier. This should have the same structure as that of MOHDL35 (FIELD NAMES, RECORD FORMAT) Then replace the use of this file in the RPG program with the newly created table name. I havent tried this yet, but would this work? does it sound like a good idea? Are there any better suggestions?
You can do that with OPNQRYF by using the MAPFLD parameter like this: OPNQRYF FILE((JCVCMP)) KEYFLD((*MAPFLD/PART)) MAPFLD((PART '%SST(VCOMN 2 5)')) The fields in JCVCOMN are now sorted like this: VENNO VCMTP VCMSQ VCOMN ----- ----- ----- ------------------------- 1,351 ICL 3 Let's see what wow 1,351 ICL 1 This is a test 1,351 NDA 2 another comment 1,351 NDA 1 more records Notice that the records are sorted by the substring of VCOMN starting with the second character. So here is your OPNQRYF with multiple key fields specified OPNQRYF FILE((*LIBL/MOHDL35)) QRYSLT(&QRYSLT) KEYFLD((*MAPFLD/CHARDT) (*MAPFLD/HDPROD)) MAPFLD((ZONEDT HDAEDT *ZONED 8 0) (CHARDT ZONEDT *CHAR 8) (HDPROD '%SST(HDPROD 1 2) *CAT %SST(HDPROD 10 12) *CAT %SST(HDPROD 13 16)')) Some notes: I am guessing that HDAEDT is a PACKED number. If so, you don't need to map it to a ZONED number just to get it to a character value. If you need the ZONED value, that is ok (but PACKED should work just as well). Otherwise, you can just use: MAPFLD((CHARDT HDAEDT *CHAR 8)) Also in your OVRDBF, you need to make sure you choose the correct Override Scope OVRSCOPE. The IBM default is OVRSCOPE(*ACTGRPDFN). OPNQRYF also has a scope OPNSCOPE. You need to make sure that the OVRSCOPE, the OPNSCOPE, and the program using the table all use the same activation group. There are a lot of different combinations. If you can't make it work, you can always change them all to *JOB, and that will work. But there is nothing intrinsic about OPNQRYF that prevents it from working from a CLP.
I would try creating a view with all the table fields plus the substring'd column, and then use OPNQRYF with that instead of the table, specifying the substring'd column as the KEYFLD. That would probably be simpler (& potentially quicker) than copying the whole lot into QTEMP every time.
Get all entries for a specific json tag only in postgresql
I have a database with a json field which has multiple parts including one called tags, there are other entries as below but I want to return only the fields with "{"tags":{"+good":true}}".
"{"tags":{"+good":true}}"
"{"has_temps":false,"tags":{"+good":true}}"
"{"tags":{"+good":true}}"
"{"has_temps":false,"too_long":true,"too_long_as_of":"2016-02-12T12:28:28.238+00:00","tags":{"+good":true}}"
I can get part of the way there with this statement in my where clause trips.metadata->'tags'->>'+good' = 'true' but that returns all instances where tags are good and true including all entries above. I want to return entries with the specific statement "{"tags":{"+good":true}}" only. So taking out the two entries that begin has_temps.
Any thoughts on how to do this?
With jsonb column the solution is obvious:
with trips(metadata) as (
values
('{"tags":{"+good":true}}'::jsonb),
('{"has_temps":false,"tags":{"+good":true}}'),
('{"tags":{"+good":true}}'),
('{"has_temps":false,"too_long":true,"too_long_as_of":"2016-02-12T12:28:28.238+00:00","tags":{"+good":true}}')
)
select *
from trips
where metadata = '{"tags":{"+good":true}}';
metadata
-------------------------
{"tags":{"+good":true}}
{"tags":{"+good":true}}
(2 rows)
If the column's type is json then you should cast it to jsonb:
...
where metadata::jsonb = '{"tags":{"+good":true}}';
If I get you right, you can check text value of the "tags" key, like here:
select true
where '{"has_temps":false,"too_long":true,"too_long_as_of":"2016-02-12T12:28:28.238+00:00","tags":{"+good":true}}'::json->>'tags'
= '{"+good":true}'
comparing rows in recarray
I have a csv file which looks like this
time,a1,a2,a3,a4,a5
0,0.0598729227761,0.0598729227761,0.0,-0.0598729227761
1,0.0598729227761,0.0598729227761,0.0,-0.0598729227761
2,0.0,-0.0598729227761,0.0,-0.0598729227761
3,0.0,-0.0598729227761,0.0,-0.0598729227761
4,0.0,-0.0598729227761,0.0,-0.0598729227761
5,0.0,-0.0598729227761,0.0,-0.0598729227761
6,0.0,-0.0598729227761,0.0,-0.0598729227761
7,0.0,-0.0598729227761,0.0,-0.0598729227761
8,0.0,-0.0598729227761,0.0,-0.0598729227761
9,0.0,-0.0598729227761,0.0,-0.0598729227761
10,0.0,-0.0598729227761,0.0,-0.0598729227761
11,0.0,-0.0598729227761,0.0,-0.0598729227761
12,0.0,0.179618768328,-0.0598729227761,-0.0598729227761
13,0.0,0.179618768328,-0.0598729227761,-0.0598729227761
14,0.0,0.179618768328,-0.0598729227761,-0.0598729227761
15,0.0,0.179618768328,-0.0598729227761,-0.0598729227761
16,0.0,0.179618768328,-0.0598729227761,-0.0598729227761
17,0.0,0.179618768328,-0.0598729227761,-0.0598729227761
18,0.0,0.179618768328,-0.0598729227761,-0.0598729227761
19,0.0,0.179618768328,-0.0598729227761,-0.0598729227761
20,0.0,0.179618768328,-0.0598729227761,-0.0598729227761
21,-0.119745845552,0.0,-0.0598729227761,-0.0598729227761
22,-0.119745845552,0.0,-0.0598729227761,-0.0598729227761
23,-0.119745845552,0.0,-0.0598729227761,-0.0598729227761
24,-0.119745845552,0.0,-0.0598729227761,-0.0598729227761
25,-0.119745845552,0.0,-0.0598729227761,-0.0598729227761
26,-0.119745845552,0.0,-0.0598729227761,-0.0598729227761
27,-0.119745845552,0.0,-0.0598729227761,-0.0598729227761
28,-0.119745845552,0.0,-0.0598729227761,-0.0598729227761
29,-0.119745845552,0.0,-0.0598729227761,-0.0598729227761
30,-0.119745845552,0.0598729227761,-0.0598729227761,-0.0598729227761
31,-0.119745845552,0.0598729227761,-0.0598729227761,-0.0598729227761
32,-0.119745845552,0.0598729227761,-0.0598729227761,-0.0598729227761
33,-0.119745845552,0.0598729227761,-0.0598729227761,-0.0598729227761
34,-0.119745845552,0.0598729227761,-0.0598729227761,-0.0598729227761
35,-0.119745845552,0.0598729227761,-0.0598729227761,-0.0598729227761
36,-0.119745845552,0.0598729227761,-0.0598729227761,-0.0598729227761
37,-0.119745845552,0.0598729227761,-0.0598729227761,-0.0598729227761
38,-0.119745845552,0.0598729227761,-0.0598729227761,-0.0598729227761
39,-0.119745845552,0.0598729227761,-0.0598729227761,-0.0598729227761
40,-0.0598729227761,0.0,0.0,-0.0598729227761
41,-0.0598729227761,0.0,0.0,-0.0598729227761
42,-0.0598729227761,0.0,0.0,-0.0598729227761
43,-0.0598729227761,0.0,0.0,-0.0598729227761
44,-0.0598729227761,0.0,0.0,-0.0598729227761
45,-0.0598729227761,0.0,0.0,-0.0598729227761
46,-0.0598729227761,0.0,0.0,-0.0598729227761
47,-0.0598729227761,0.0,0.0,-0.0598729227761
48,-0.0598729227761,0.0,0.0,-0.0598729227761
49,-0.0598729227761,0.0,0.0,-0.0598729227761
50,-0.0598729227761,0.0,0.0,-0.0598729227761
51,-0.0598729227761,0.0,0.0,-0.0598729227761
52,-0.0598729227761,0.0,0.0,-0.0598729227761
53,-0.0598729227761,0.0,0.0,-0.0598729227761
54,-0.0598729227761,0.0,0.0,-0.0598729227761
55,-0.0598729227761,0.0,0.0,-0.0598729227761
56,-0.0598729227761,0.0,0.0,-0.0598729227761
57,-0.0598729227761,0.0,0.0,-0.0598729227761
58,-0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
59,-0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
60,-0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
61,-0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
62,-0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
63,-0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
64,-0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
65,-0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
66,-0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
67,-0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
68,0.0,-0.0598729227761,0.0,-0.0598729227761
69,0.0,-0.0598729227761,0.0,-0.0598729227761
70,0.0,-0.0598729227761,0.0,-0.0598729227761
71,0.0,-0.0598729227761,0.0,-0.0598729227761
72,0.0,-0.0598729227761,0.0,-0.0598729227761
73,0.0,-0.0598729227761,0.0,-0.0598729227761
74,0.0,-0.0598729227761,0.0,-0.0598729227761
75,0.0,-0.0598729227761,0.0,-0.0598729227761
76,0.0,-0.0598729227761,0.0,-0.0598729227761
77,0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
78,0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
79,0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
80,0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
81,0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
82,0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
83,0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
84,0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
85,0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
86,0.0598729227761,0.0,-0.0598729227761,-0.0598729227761
87,-0.0598729227761,0.0598729227761,0.0,-0.0598729227761
88,-0.0598729227761,0.0598729227761,0.0,-0.0598729227761
89,-0.0598729227761,0.0598729227761,0.0,-0.0598729227761
90,-0.0598729227761,0.0598729227761,0.0,-0.0598729227761
91,-0.0598729227761,0.0598729227761,0.0,-0.0598729227761
92,-0.0598729227761,0.0598729227761,0.0,-0.0598729227761
93,-0.0598729227761,0.0598729227761,0.0,-0.0598729227761
94,-0.0598729227761,0.0598729227761,0.0,-0.0598729227761
95,-0.0598729227761,0.0598729227761,0.0,-0.0598729227761
96,0.0598729227761,-0.0598729227761,0.0598729227762,-0.0598729227761
97,0.0598729227761,-0.0598729227761,0.0598729227762,-0.0598729227761
98,0.0598729227761,-0.0598729227761,0.0598729227762,-0.0598729227761
99,0.0598729227761,-0.0598729227761,0.0598729227762,-0.0598729227761
100,0.0598729227761,-0.0598729227761,0.0598729227762,-0.0598729227761
101,0.0598729227761,-0.0598729227761,0.0598729227762,-0.0598729227761
102,0.0598729227761,-0.0598729227761,0.0598729227762,-0.0598729227761
103,0.0598729227761,-0.0598729227761,0.0598729227762,-0.0598729227761
104,0.0598729227761,-0.0598729227761,0.0598729227762,-0.0598729227761
the data is read using
acc = mlab.csv2rec('filename.csv')
and plotted like this
plt.plot((acc.time)/100.00,acc.a1,label='A1')
i wish to take only the unique values and then plot it. Is it possible to import a row only if it is different from the current row. and then use the data to plot.
There's the python set type, where you can only have one copy of each value; if you don't care about the row number, just drop it from your rows, and feed the rows into a set. If you need the row number, you can go a different route: generate a tuple from the values in your row, and use that tuple as the key in a dictionary, using the row number as value.
SQL Server - XQuery for XML
Just similar other post, I need to retrieve any rows from table applying criteria on Xml column, for instance, supposing you have an xml column like this:
<DynamicProfile xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.datacontract.org/2004/07/WinTest">
<AllData xmlns:d2p1="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
<d2p1:KeyValueOfstringstring>
<d2p1:Key>One</d2p1:Key>
<d2p1:Value>1</d2p1:Value>
</d2p1:KeyValueOfstringstring>
<d2p1:KeyValueOfstringstring>
<d2p1:Key>Two</d2p1:Key>
<d2p1:Value>2</d2p1:Value>
</d2p1:KeyValueOfstringstring>
</AllData>
</DynamicProfile>
My query would be able to return all rows where node value <d2p1:Key> = 'some key value' AND node value <d2p1Value = 'some value value'.
Imagine of that just as a dynamic table where KEY node represent the column name and Value node represent column's value.
The following query does not work because key and value nodes are not sequential:
select * from MyTable where
MyXmlField.exist('//d2p1:Key[.="One"]') = 1
AND MyXmlField.exist('//d2p1:Value[.="1"]') = 1
Instead of looking for //d2p1:key[.="One"] and //d2p1:Value[.="1"] as two separate searches, do a single query that looks for both at once, like so: //d2p1:KeyValueOfstringstring[./d2p1:Key="One"][./d2p1:Value=1]