Combine multiple csv files time based visual basic - vb.net

I have multiple csv files, with the same headers and in the same folder. My goal is to combine all the csv files using visual basic (vb.net) based on certain time.
I am using data logger that can log every 15 minutes. So, 1 file csv for 15 minutes logging, 2 files csv for 30 minutes logging, and so on.
I want to combine all of those csv files in one day (from 14.00 on day 1 to 13.59 on day 2) automatically with the file names based on the time.
For example :
Flowdata20170427220000.csv (27-04-2017 22:00:00)
Name,Attribute1,Attribute2,Attribute3
name1,111,abc,zzz
name2,222,def,yyy
Flowdata20170427221500.csv (27-04-2017 22:15:00)
Name,Attribute1,Attribute2,Attribute3
name3,333,ghi,xxx
name4,444,jkl,www
and so on until
Flowdata20170428214500.csv (28-04-2017 21:35:00)
Name,Attribute1,Attribute2,Attribute3
name5,555,mno,vvv
name6,666,pqr,uuu
and the final file is :
Flowdata20170427-20170428.csv
Name,Attribute1,Attribute2,Attribute3
name1,111,abc,zzz
name2,222,def,yyy
name3,333,ghi,xxx
name4,444,jkl,www
...,...,...,...
...,...,...,...
name5,555,mno,vvv
name6,666,pqr,uuu
Can you help me out please? Searching in google but nothing helps me.

Related

how to split large text files into smaller text files using vba?

I have a database textfile.
It is large text file about 387,480 KB. This file contains table name, headers of the table and values. I need to split this file into multiple files each containing table creation and insertion with a file name as table name.
Please can anyone help me??

I don't see how Excel will open a 347MB file. You can try to load it into Access, and do the split, using VBA. However, the process of importing a file that large may fragment enough to blow Access up to #GB, and then it's all over. SQL Server would handle this kind of job. Alternatively, you could use Python or R to do the work for you.
### Python:
import pandas as pd
for i,chunk in enumerate(pd.read_csv('C:/your_path/main.csv', chunksize=3)):
chunk.to_csv('chunk{}.csv'.format(i))
### R
setwd("C:/your_path/")
mydata = read.csv("annualsinglefile.csv")
# If you want 5 different chunks with same number of lines, lets say 30.
# Chunks = split(mydata,sample(rep(1:5,30))) ## 5 Chunks of 30 lines each
# If you want 100000 samples, put any range of 20 values within the range of number of rows
First_chunk <- sample(mydata[1:100000,]) ## this would contain first 100000 rows
# Or you can print any number of rows within the range
# Second_chunk <- sample(mydata[100:70,] ## this would contain last 30 rows in reverse order if your data had 100 rows.
# If you want to write these chunks out in a csv file:
write.csv(First_chunk,file="First_chunk.csv",quote=F,row.names=F,col.names=T)
# write.csv(Second_chunk,file="Second_chunk.csv",quote=F,row.names=F,col.names=T)

Qlikview does not upload what I ask for

I have this simple script to upload filenames:
Files:
LOAD
Distinct
FileName() as File
FROM [C:\Matias\Capacity Tracker\AllFiles\*];
And as a result while running the script, it happens the following:
Files << Analyst Time Sheet - Adam W - 0730-0805 0 lines fetched
Files << Analyst Time Sheet - Adam W - 0806-0812 0 lines fetched
Files << Analyst Time Sheet - Agnieszka J - 0702-0708 2 lines fetched
Files << Analyst Time Sheet - Agnieszka J - 0709-0715 3 lines fetched
Files << Analyst Time Sheet - Agnieszka J - 0716-0722 4 lines fetched
And so on...
So, the strange thing is that for the files from "Adam W", doesn't upload anything (no lines fetched). So then, I have the list of files except these ones. I find it very strange, because as I'm just asking for the filename, it can't be a thing of formatting (I think).
Any idea of what can be happening and how could I solve it?
Thank you in advance
Matias

Although QlikView offers that * option on the filename of the LOAD statment, the results are sometimes a little bit random. I would recomend that you try a different approach and see if it works.
For Each FILE in FileList('C:\Matias\Capacity Tracker\AllFiles\*')
Files:
LOAD
Distinct FileName() as File
FROM [$(FILE)];
next file
Hope this helps.

thanks for your idea. I tried that and unfortunately I had the same problem. Finally, I solved like this:
Files:
LOAD
Distinct
FileName() as File
FROM [C:\Matias\Capacity Tracker\AllFiles\*];
SET ErrorMode=0;
Files:
LOAD
Distinct
FileName() as File
FROM [C:\Matias\Capacity Tracker\AllFiles\*]
(ooxml, no labels, table is [Task Log])
Where Not Exists(File,FileName());
IF ScriptError <> 0 THEN
Files:
LOAD
FileName() as File
FROM [C:\Matias\Capacity Tracker\AllFiles\*]
(biff, no labels, table is [Task Log$])
Where Not Exists(File,FileName());
ENDIF
Despite they are all .xls files, it seems to be formatting differences between them. So the ones not uploaded at first, they were uploaded by the first statement after (ooxml), or if it failed, by the second one (biff files). Quite strange.
Maybe this is not the best and proper solution, but it was the only one that worked to upload all the filenames from the folder.

unable to load csv file from GCS into bigquery

I am unable to load 500mb csv file from google cloud storage to big query but i got this error
Errors:
Too many errors encountered. (error code: invalid)
Job ID xxxx-xxxx-xxxx:bquijob_59e9ec3a_155fe16096e
Start Time Jul 18, 2016, 6:28:27 PM
End Time Jul 18, 2016, 6:28:28 PM
Destination Table xxxx-xxxx-xxxx:DEV.VIS24_2014_TO_2017
Write Preference Write if empty
Source Format CSV
Delimiter ,
Skip Leading Rows 1
Source URI gs://xxxx-xxxx-xxxx-dev/VIS24 2014 to 2017.csv.gz
I have gzipped 500mb csv file to csv.gz to upload to GCS.Please help me to solve this issue

The internal details for your job show that there was an error reading the row #1 of your CSV file. You'll need to investigate further, but it could be that you have a header row that doesn't conform to the schema of the rest of the file, so we're trying to parse a string in the header as an integer or boolean or something like that. You can set the skipLeadingRows property to skip such a row.
Other than that, I'd check that the first row of your data matches the schema you're attempting to import with.
Also, the error message you received is unfortunately very unhelpful, so I've filed a bug internally to make the error you received in this case more helpful.

UniVerse - SQL LIST: View List of All Database Tables

I am trying to obtain a list of all the DB Tables that will give me visibility on what tables I may need to JOIN for running SQL scripts.
For example, in TCL when I run "LIST.DICT" it returns "Name of File:" for input. I then enter "PRODUCT" and it returns a list of all available fields.
However, Where can I get a list of all my available Tables or list of my options that I can enter after "Name of File:"?
Here is what I am trying to achieve. In the screen shot below, I would like to run a SQL script that gives me the latest Log File Activity, Date - Time - Description. I would like the script to return '8/13/14 08:40am BR: 3;BuyPkg'
Thank you in advance for your help.

From TCL within the database account containing your database files, type: LISTF
Sample output:
FILES in your vocabulary 03:21:38pm 29 Jun 2015 Page 1
Filename........................... Pathname...................... Type Modulo
File - Contains all logical device names
DICT &DEVICE& /u1/uv/D_&DEVICE& 2 1
DATA &DEVICE& /u1/uv/&DEVICE& 2 3
File - Used by MAKE.MAP.FILE
DICT &MAP& /u1/uv/D_&MAP& 2 1
DATA &MAP& /u1/uv/&MAP& 6 7
File - Contains all parts of Distributed Files
DICT &PARTFILES& /u1/uv/D_&PARTFILES& 2 1
DATA &PARTFILES& /u1/uv/&PARTFILES& 18 7
DICT &PH& D_&PH& 3 1
DATA &PH& &PH& 1
DICT &SAVEDLISTS& D_&SAVEDLISTS& 3 1
DATA &SAVEDLISTS& &SAVEDLISTS& 1
File - Used by uniVerse to access the current directory.
DICT &UFD& /u1/uv/D_UFD 2 1
DATA &UFD& . 19 1
DICT &XML& D_&XML& 18 3
DATA &XML& &XML& 19 1

Firstly, UniVerse has no Log File Activity Date and Time.
However, you can still obtain the table's modified/ accessed date from the file system however.
To do this,
You need to have a subroutine accepting a path of the table to return a date or a time.
e.g. SUBROUTINE GET.FILE.MOD.DATE(DAT.MOD, S.FILE.PATH)
Inside the subroutine, you can use EXECUTE to run shell command like istat for getting these info on a unix e.g.
Please beware that for a dynamic file e.g. there are Data and Overflow parts under a directory. You should compare the dates obtained and return only the latest one.
Globally catalog the subroutine
Create an I-Desc in VOC, e.g. I.FILE.MOD.DATE in the field definition of this I-Desc: SUBR("*GET.FILE.MOD.DATE",F2) and Conv Code as "D/MDY2"
Create another I-Desc e.g. I.FILE.MOD.TIME
Finally, you can
LIST VOC I.FILE.MOD.DATE I.FILE.MOD.TIME DESC WITH TYPE LIKE "F..."
alternatively in SQL,
SELECT I.FILE.MOD.DATE, I.FILE.MOD.TIME, VOC.DESC FROM VOC WHERE TYPE LIKE "F%";

JProfile jpexport csv output missing headings

I'm evaluating JProfiler to see if it can be used in a automated fashion. I'd like to run the tool in offline mode, save a snapshot, export the data, and parse it to see if there is performance degradation since the last run.
I was able to use the jpexport command to export the TelemetryHeap view from a snapshot file to a csv file (sample below). When I look at the csv file I see 10 to 13 columns of data but only 4 headings. Is there documentation that explains the output more fully?
Sample output:
"Time [s]","Committed size","Free size","Used size"
0.0,450,880,000,371,600,000,79,280,000
1.0,450,880,000,371,600,000,79,280,000
2.0,450,880,000,371,600,000,79,280,000
3.0,450,880,000,371,600,000,79,280,000
4.0,450,880,000,371,600,000,79,280,000
5.0,450,880,000,371,600,000,79,280,000
6.0,450,880,000,371,600,000,79,280,000
7.0,450,880,000,355,932,992,94,947,000
8.0,450,880,000,355,932,992,94,947,000
9.58,969,216,000,634,564,992,334,651,008
11.05,1,419,456,000,743,606,016,675,849,984
12.05,1,609,792,000,377,251,008,1,232,541,056
17.33,2,524,032,000,1,115,268,992,1,408,763,008
19.43,2,588,224,000,953,451,008,1,634,772,992
26.08,3,711,936,000,1,547,981,056,2,163,954,944
39.75,3,711,936,000,1,145,185,024,2,566,750,976
40.75,3,711,936,000,1,137,052,032,2,574,884,096
41.75,3,711,936,000,1,137,052,032,2,574,884,096
42.75,3,711,936,000,1,137,052,032,2,574,884,096
43.75,3,711,936,000,1,137,051,008,2,574,885,120
44.75,3,711,936,000,1,137,051,008,2,574,885,120
45.75,3,711,936,000,1,137,051,008,2,574,885,120
46.75,3,711,936,000,1,137,051,008,2,574,885,120
47.75,3,711,936,000,1,137,051,008,2,574,885,120
48.75,3,711,936,000,1,137,051,008,2,574,885,120
49.75,3,711,936,000,1,137,051,008,2,574,885,120
50.75,3,711,936,000,1,137,051,008,2,574,885,120
51.75,3,711,936,000,1,137,051,008,2,574,885,120
52.75,3,711,936,000,1,137,051,008,2,574,885,120
53.75,3,711,936,000,1,137,051,008,2,574,885,120
54.75,3,711,936,000,1,137,051,008,2,574,885,120
55.75,3,711,936,000,1,137,051,008,2,574,885,120
56.75,3,711,936,000,1,137,051,008,2,574,885,120
57.75,3,711,936,000,1,137,051,008,2,574,885,120
58.75,3,711,936,000,1,137,051,008,2,574,885,120
60.96,3,711,936,000,1,137,051,008,2,574,885,120
68.73,3,711,936,000,1,137,051,008,2,574,885,120
74.39,3,711,936,000,1,137,051,008,2,574,885,120