I'm converting excel data source to json by taking all data in a file and saving it. Now I'm looking for number of records in the output file, number of input, source path of excel file, etc. look at it if somebody can guide me thanks.
After tFileOutputJson put a tJava component .
Flow would be
tFileInputExcel->main->tFileOutputJson ->onComponentOK->tJava
In your tJava put this code :
System.out.println(((Integer)globalMap.get("tFileInputExcel_1_NB_LINE"))) ;
For explanation :
((Integer)globalMap.get("tFileInputExcel_1_NB_LINE")) is a global Variable that you can find in the outline of your studio
For othen information a tFileProperties would do the staff here is the link
https://help.talend.com/r/en-US/7.3/tfileproperties/tfileproperties-standard-properties
Related
I have written a query that uses a stream input and join it with static json reference data. I get the correct results when testing the query in "Test results" tab (uploading the same sample reference data). However, no output is generated when starting the job.
My query:
The stream input produces random results every second like :
And the json reference file is:
From the monitoring dashboard there are also no input events nor output.
I have confirmed that the output blob is created when no join with reference data is used in the query. I have uploaded the json reference data in a storage container and have provided the path pattern : ref/Atm.json in my example.
Thanks for any help.
I was just curious and tried to repro , I am using a very similar query as yours .
SELECT sum(A.amount),B.area_code
INTO Gen2
FROMEHInput A
JOIN JSONref B ON A.ATMCode=B.atm_code
group by B.area_code,TumblingWindow(minute,1)
This is the output which you have.( all junk data , but it confirms query works )
{"sum":63580.0,"area_code":20}
{"sum":73060.0,"area_code":30}
{"sum":68220.0,"area_code":20}
At this point , I thought to making some changes to check if I can repro your case of no output , I just updated the static file to something non existing file , ASA never complained about that but I never got the results either .
I think you may be hitting the same ( as you mentioned that without the join things just works fine ) .
In my case I have the static file location something like
https://blobaccount.blob.core.windows.net/container/**File2/Atm.json**
and when i create the Reference Input , I pass the file name as
I think for some reason its getting messed up here .
Other thing which you can try it increase the TW to a bigger window .
Let me know how it goes .
I managed to resolve this issue. The error as you suggested was the path pattern I had specified for my reference data. So I had created a container e.g. "mycontainer" and I just dropped the json files in it. The path pattern that worked for me was just the names of the json files inside that container. No "/" , just Atm.json in my example.
Now the stream job creates output files in my blob storage.
I have written a simple query and join it with a json reference data. I can see correct results when testing the query in "Test results" tab. However, no output is generated when starting the job.
I have confirmed that the output blob is created when no join with reference data is used in the query.
Any help is appreciated. The sample reference json follows:
[
{
"DeviceId":"DEV-021",
"Brand":"brand01",
"Model":"model01"
}
]
Use flat json structure instead of array. It should give you the output
Check the path you specified in the reference data, maybe it is not correct or you did not specify the file name. Does it contain something like {date}/{time}/filename.json?
If you forget to specify the file name, it does not work as well.
And if you are testing the job, usually you specify the file manually and that is why your query works.
I am currently trying to find an alternative to the Governance Dashboard that Rob Wunderlich (Qlik founder) created, since I am currently encountering errors when using it.
How do you differentiate between a data source (QVD, aka source) that is used by a QVW or a data file (QVD, aka target) that is generated by that QVW?
QVW:
LOAD
Lower(Discriminator) AS DataFile.Filepath
FROM C:\Sample_Transform_file.qvw (xmlSimple, Table is[DocumentSummary/LineageInfo])
Below is an example of what I found when parsing through the XML Metadata
(discriminator subtag within the lineageinfo tag) for one specific Transform QVW.
Sample Table Output
Are targets just identified by this?
STORE - [qvdName.qvd](qvd)
From what I have found, That appears to be the case, to a degree.
All of our QVW files that output a QVD utilize DIRECTORY statements rather than either hard-coded file location paths or variablized paths. Hence why all of the Targets are getting displayed as "STORE - qvdname.qvd", instead of displaying the filepath. In a sense, that is a flaw on QlikView's part, regarding its Governance Dashboard (or at the very least, they don't seem to recommend variablizing those paths as a standard in order to avoid breaking the lineage).
How do you Edit a (binary EBCDIC) Mainframe file in the RecordEditor with out a Cobol Copybook.
How do you generate Java code to read the file using the RecordEditor.
Note: This is an attempt to split a question that is far to broad to give meaningful answer to
into a series of simpler Question and Answer's.
Try and avoid editing a binary file with a Cobol Copybook if at all possible. This should only be attempted as a last resort !!!.
Try and get
that Cobol copybook (or some field layout document) for the file !!!
Some general advise:
It is feasible when dealing with 10 / 20 fields in a record but not if there a thousands of fields in a Record.
Take your time do not rush the process. Try and get each step correct before moving on
Finally upgrade to the latest version of the RecordEditor (currently 0.98.4)
This process will also work for normal Text file as well
RecordEditor Layout Wizard
To start the wizard select option Record Layouts >>> Layout Wizard.
File Structure screen
The file structure screen has 3 purposes:
Get the File structure - It could be Fixed Width, VB, Windows/Unix Text file
Get the Record-Length (if it is a fixed width file).
Get the font (character-set / encoding)
The RecordEditor will try and work this out for you
Field Selection Screen
The RecordEditor will try and work out where fields start and end but
it is not perfect. You need to carefully check and correct its choices
On this screen, the fields are displayed in alternating colors
you create/delete a field by clicking on
use the Clear Fields button clear all the fields
you can change what field-types to search for using the various check box's (e.g. Mainframe Zones Decimal)
The Add Fields will do another field search
Field Definition screen
On this screen you define the field names and Types. You may need to go back to the **Field Selection Screen* to adjust the fields
Editing the file
Once the Record Layout has been defined, it can be used on the open file screen
Generating Java code
When editing your file, you can generate java~JRecord code to read the file
by selecting Generate >>> Java >>> ....
You can the enter a package-id + generate options:
and finally your sample java code is generated to read / write the
file.
I am trying to import a file, source here and selections here (select all fields and select "Pilkkueroteltu (otsikollinen)" and then click Jatka to download), with two header rows, " as text qualifier, comma as a field separator and with UTF-8 format. I am unable to do this in Micsosoft Server SQL Management Studio. I will focus now only on the text qualifier where " does not work (only reading the first quote as text qualifier).
where I am unable to specify the column separator, no idea why this is occurring.
Update 1
Refresh/Reset buttons fixed the initial preview but I am getting the following preview error in the step Select Source Tables and Views later.
Update 2
I get the LocaleID error The LocaleID 11 is not installed on this system.
(SQL Server Import and Export Wizard). I am getting the same error despite Locale/Code page settings, what is causing this?
How to specify the text qualifier in the MSMM?
I tried to replicate your scene. Very first, I had to delete first heading entry eg : "Kuntien avainluvut 1987-2016"
Please see : sample image
Column delimited is: ,
Might not be accurate answer or different from something which you expect, but by applying above settings, I could import data through SSMS2012
edit : based on comments.
Here is the detailed steps :
next,
next, you need to change column width of first column as it gave me data truncate error
next,
I have also got a dtsx package for the same, but I don't know how can I share it with you here.