Basically this data is in a report that I can view using 'Unicenter Outpout Management Document Viewer'..... Once I see the results I am able Export using the following parameters
Export Type = Worksheet
Worksheet Format = .XLS
I am also able to load a Column Rule Name file that includes the rules I would likle applied i.e 'COLRULE BLANKLINE DISCARD'
Below is the example of how the data is presented. I have changed some values due to privacy, I have made no changes to format or spacing.
XXX - DI tape DATE: 25SEP2017 Page: 0001
Header: File ID: CN15 Processing Date: 20170923
Detail
SL Agt Reason Cd Policy No. Prem Amt. Comm Rate Comm Amt. Agt Shr
Comm Yr Ins Name Paid To Date Cov Code GW Agt SL Agt Name Annual Prem Amt
001234 CSERF 0012345678 3.92 5.0000 0.20- 100.000
17 EVESO 20171028 00220 652348 ABCDE 47.04
001234 CSERF 0012345678 70.30 5.0000 3.52- 100.000
17 EVESO 20171028 30086 652348 ABCDE 843.60
001234 CSERF 0012345678 14.83 5.0000 0.74- 100.000
17 EVESO 20171028 30015 652348 ABCDE 177.96
001234 CSERF 0012345678 26.28 5.0000 1.31- 100.000
17 EVESO 20171028 30086 652348 ABCDE 315.36
Since the data seams to automatically go to the next row I have no idea what type of rule can be applied to have the headers go on row 1, and then read the data properly.
If anyone has an idea please help.
Thank you in advance.
Cheers
Related
I have an SSIS package that is to ingest a number of Excel files with similar structures but irregular names and import them into a SQL table. Along with the data from the excel files, I have a number of variables that are set and different with each file (User::ExcelFileName, User::VarMonth, User::VarProgram, User::VarYear, etc). All of the table data from the Excel files are going to the same destination table, but for each row of data alongside the Excel dataset I want to insert a column for each variable to pass through as well into SQL. An example of my dataset is below:
Excel
ID
Name
Foo
Bar
111
Bob
88yu
117
112
Jim
JKL
A TU
113
George
FTD
19900
SSIS Variables (set during execution)
User::ExcelFileName = c:\temp\excelfile1.xlsx
User::VarMonth = Jan
User::VarProgram = Daily
User::VarYear = 2023
Desired SQL Destination:
ExcelFileName
VarMonth
VarProgram
VarYear
ID
Name
Foo
Bar
c:\temp\excelfile1.xlsx
Jan
Daily
2023
111
Bob
88yu
117
c:\temp\excelfile1.xlsx
Jan
Daily
2023
112
Jim
JKL
A TU
c:\temp\excelfile1.xlsx
Jan
Daily
2023
113
George
FTD
19900
I've tried a few configurations and I've referenced this post for piping in variable data into SQL, but I haven't gotten a working model yet.
Worth noting, Excel COnnection is dynamic and set to run within a Foreach Loop container to iterate through my Excel sources. Any advice or guidance would be appreciated!
It sounds like you want a Derived Column task.
in the task, just add the new columns you want, and map the variables to the column.
Hi I have a CSV with this format
Headers: SKU, Product_Name, product_id
3735,[Freebies PC] - Holyshield! Sunscreen Comfort Corrector Serum SPF 50+ PA++++ 5 mL,154674
4568,"Consumables Mika furit 500 gr #250 (16x12x11) packaging grape, orange)",202737
2403,Laurier Active Day Super Maxi 30 Pcs,8992727002714
I want to be able to read as dataframe in csv, however the problem is that some product names uses "," which is not being able to be read as properly. I checked other sources trying to use sep, however some product names have that others don't. How can i read it properly?
I tried using
productList = pd.read_csv('products/products.csv',encoding='utf-8', engine'python)
It returns:
sku
Product_Name
product_id
3735
[Freebies PC] - Holyshield! Sunscreen Comfort Corrector Serum SPF 50+ PA++++ 5 mL
154674
4568,"Consumables Mika furit 500 gr #250 (16x12x11) packaging grape, orange)",202737
nan
nan
42403
Laurier Active Day Super Maxi 30 Pcs
8992727002714
expected output is
sku
Product_Name
product_id
3735
[Freebies PC] - Holyshield! Sunscreen Comfort Corrector Serum SPF 50+ PA++++ 5 mL
154674
4568
Consumables Mika furit 500 gr #250 (16x12x11) packaging grape, orange)
202737
42403
Laurier Active Day Super Maxi 30 Pcs
8992727002714
How can I do so?
Content of sample.csv file:
product_id,product_name,sku_number
2168,Sanjin Watermelon Frost Obat Sariawan Powder/Bubuk,6903193004029
3798,Common Grounds Cloak & Dagger Instant Coffee 1 Sachets,313166
3799,Common Grounds Ethiopia Guji Instant Coffee 1 Sachets,175744
3580,Emina Glossy Stain Lip Tint Autumn Bell 3gr,8993137707220
"3795,""Hansaplast Kasa Steril - 7,5 x 7,5cm"",8999777016043"
"2997,""Panda GP71 2,5mm"",616920"
It seems like output process from db generates error in exported data for some reason. If you are not able to correct the process possible solution is the following:
import pandas as pd
from io import StringIO
with open('sample.csv', 'r') as f:
data = f.read().replace(',""', '","').replace('"",', '","')
df = pd.read_csv(StringIO(data))
df
Returns
I'm running pgAdmin v5.2 over postgres v13.3
So I hit a wall with this one...
I'm running a query on a flight log table which is constantly updated with new flights data.
In it there is engn_hrs_contdwn column which calculates via window SUM function the accumulated hrs based on each individual flight time (hobs_total) which are then being deducted from 1200 which is a given value upon reaching an engine MUST be replaced.
This is the query that I run:
SELECT fleet.fleet_id,
(flt_log.date || ' '|| flt_log.tkof_01_time)::timestamp AS date,
flt_log.hobs_total,
1200-SUM (hobs_total)OVER (PARTITION BY fleet_id ORDER BY (flt_log.date || ' '|| flt_log.tkof_01_time)::timestamp
ROWS BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW) AS engn_hrs_contdwn
FROM flt_log,fleet
WHERE flt_log.aircraft_id = fleet.fleet_id
AND fleet_id = 2
;
fleet_id
date
hobs_total
engn_hrs_contdwn
2
2020-08-09 08:49:00
0.20
1199.80
2
2020-08-09 11:17:00
3.70
1196.10
2
2020-08-09 15:42:00
0.70
1195.40
2
2020-08-09 17:54:00
2.40
1193.00
2
2020-08-12 07:21:00
0.50
1192.50
2
2020-08-13 06:50:00
2.40
1190.10
2
2020-08-13 15:11:00
1.50
1188.60
2
2020-08-13 20:35:00
0.70
1187.90
2
2020-08-14 09:17:00
2.40
1185.50
This query works OK on calculating the remaining hrs but when reaching 0 it then returns negative values which are of course useless for calculating the countdown for a new engine.
My problem is how to reset the countdown initial value back to 1200 every time the engn_hrs_contdwn hit below 0 - so that the engn_hrs_contdwn column will start the countdown for the new engine and so on and so on.
Being novice at postgresql (and programming in general...) I researched this issue over the web and came across RECURSIVE QUERY and CASE FUNCTION which I think maybe the direction I should take for tackling this issue.
But quite honestly I got completely lost going over tutorials on these subjects and failed so far in my efforts.
Any guidance will be much appreciated.
I have 2 sheets in one excel file, the first one is :
Sheet: Person
Code date start end
2301 12/08/1993 08:02 08:17
4221 12/08/1993 09:04 09:25
2312 12/08/1993 10:02 10:28
1284 19/09/1994 11:02 11:21
2312 19/09/1994 15:57 16:20
1284 23/06/1995 17:12 17:35
2312 22/06/1996 13:14 13:32
4221 22/06/1996 15:53 16:13
4221 05/05/1999 08:06 08:22
2418 05/05/1999 08:10 08:33
2301 05/05/1999 09:12 09:37
2301 05/05/1999 09:28 10:28
2301 05/05/1999 13:28 13:38
Is a list of person of a company and anyone of them is identified by badge [row Code], what I hope is to Merge data by code to a costume sheet of a person, for example, for the person who have a number of badge 2301 he have his own sheet called B2301, so based on the first sheet "Person" I hope import data of a person like that grouped by code number of this person :
sheet B2301
date Period(min)
12/08/1987 12
.... ...
So Period will be calculated from start and end rows.
I tried by using this formula but it's not working for me :
=IFERROR(INDEX(Sheet1!A$2:A$14,SMALL(IF(Sheet1!$A$2:$A$14=INT(RIGHT(CELL("filename",A1),LEN(CELL("filename",A1))-FIND("]",CELL("filename",A1)))),ROW(Sheet1!A$2:A$14)-ROW(Sheet1!A$2)+1),ROWS(Sheet1!A$2:A2))),"")
Any Idea?
This will require a lot of research on your part. You'll need to:
create a VBA Macro
define variables and create a loop to look at your main sheet.
create a sheet name based on the code.
check if the sheet already exists, if not, create it.
copy the values from the first sheet to the "code" sheet.
once all values are processed, go through each sheet, loop through your values and calculate your periods.
This is not a trivial amount of code. Do research on these 6 items and write the code. When you have that, display it and we can give you more direction.
To populate the dates, in A2 put:
=IFERROR(INDEX(Sheet1!$B$2:$B$14,MATCH(SMALL(IF(--MID(MID(CELL("filename",A1),FIND("]",CELL("filename",A1))+1,255),2,999) = Sheet1!$A$2:$A$14,Sheet1!$B$2:$B$14),ROW()-1),IF(--MID(MID(CELL("filename",A1),FIND("]",CELL("filename",A1))+1,255),2,999) = Sheet1!$A$2:$A$14,Sheet1!$B$2:$B$14),0)),"")
To populate the period put this in B2:
=IFERROR(TEXT(INDEX(Sheet1!$D$2:$D$14,MATCH(SMALL(IF(--MID(MID(CELL("filename",A1),FIND("]",CELL("filename",A1))+1,255),2,999) = Sheet1!$A$2:$A$14,IF(Sheet1!$B$2:$B$14=A2,Sheet1!$C$2:$C$14)),COUNTIF($A$1:$A2,A2)),IF(--MID(MID(CELL("filename",A1),FIND("]",CELL("filename",A1))+1,255),2,999) = Sheet1!$A$2:$A$14,IF(Sheet1!$B$2:$B$14=A2,Sheet1!$C$2:$C$14)),0))-INDEX(Sheet1!$C$2:$C$14,MATCH(SMALL(IF(--MID(MID(CELL("filename",A1),FIND("]",CELL("filename",A1))+1,255),2,999) = Sheet1!$A$2:$A$14,IF(Sheet1!$B$2:$B$14=A2,Sheet1!$C$2:$C$14)),COUNTIF($A$1:$A2,A2)),IF(--MID(MID(CELL("filename",A1),FIND("]",CELL("filename",A1))+1,255),2,999) = Sheet1!$A$2:$A$14,IF(Sheet1!$B$2:$B$14=A2,Sheet1!$C$2:$C$14)),0)),"[m]"),"")
Both are array formulas and need to be confirmed with Ctrl-Shift-Enter. Then Copy both down to desired rows.
I'm trying to pull data stored as $24. I want to convert it from character to numeric. The input(variable-name,comma24.) function is not working for me. A sample of the data is given below.
5.35
5.78
413,000
3,280,000
5.97
6.72
5
6.53
6
4.59
4.25
5
6.38
6.41
4.1
6.56
5.45
6.07
4.28
5.54
5.87
3.88
5.53
5.65
6.47
207,000
4,935,000
4,400,000
6,765,000
2,856,000
53,690,000
You don't show your code, but for some reason I could get it work when the reading and conversion were in different data steps, but not when it was the same data step.
The following works just fine:
DATA one;
INPUT y: $24. ##;
DATALINES;
5.35 5.78 413,000 3,280,000 5.97
RUN;
DATA one;
SET one;
z = INPUT(y, comma24.);
RUN;
However if I put the calculation of z in the first data step, I was getting missing values without any error message. I have no explanation for this behavior, but hopefully the workaround will work for you as well.