I have written a query that uses a stream input and join it with static json reference data. I get the correct results when testing the query in "Test results" tab (uploading the same sample reference data). However, no output is generated when starting the job.
My query:
The stream input produces random results every second like :
And the json reference file is:
From the monitoring dashboard there are also no input events nor output.
I have confirmed that the output blob is created when no join with reference data is used in the query. I have uploaded the json reference data in a storage container and have provided the path pattern : ref/Atm.json in my example.
Thanks for any help.
I was just curious and tried to repro , I am using a very similar query as yours .
SELECT sum(A.amount),B.area_code
INTO Gen2
FROMEHInput A
JOIN JSONref B ON A.ATMCode=B.atm_code
group by B.area_code,TumblingWindow(minute,1)
This is the output which you have.( all junk data , but it confirms query works )
{"sum":63580.0,"area_code":20}
{"sum":73060.0,"area_code":30}
{"sum":68220.0,"area_code":20}
At this point , I thought to making some changes to check if I can repro your case of no output , I just updated the static file to something non existing file , ASA never complained about that but I never got the results either .
I think you may be hitting the same ( as you mentioned that without the join things just works fine ) .
In my case I have the static file location something like
https://blobaccount.blob.core.windows.net/container/**File2/Atm.json**
and when i create the Reference Input , I pass the file name as
I think for some reason its getting messed up here .
Other thing which you can try it increase the TW to a bigger window .
Let me know how it goes .
I managed to resolve this issue. The error as you suggested was the path pattern I had specified for my reference data. So I had created a container e.g. "mycontainer" and I just dropped the json files in it. The path pattern that worked for me was just the names of the json files inside that container. No "/" , just Atm.json in my example.
Now the stream job creates output files in my blob storage.
Related
I'm converting excel data source to json by taking all data in a file and saving it. Now I'm looking for number of records in the output file, number of input, source path of excel file, etc. look at it if somebody can guide me thanks.
After tFileOutputJson put a tJava component .
Flow would be
tFileInputExcel->main->tFileOutputJson ->onComponentOK->tJava
In your tJava put this code :
System.out.println(((Integer)globalMap.get("tFileInputExcel_1_NB_LINE"))) ;
For explanation :
((Integer)globalMap.get("tFileInputExcel_1_NB_LINE")) is a global Variable that you can find in the outline of your studio
For othen information a tFileProperties would do the staff here is the link
https://help.talend.com/r/en-US/7.3/tfileproperties/tfileproperties-standard-properties
I have written a simple query and join it with a json reference data. I can see correct results when testing the query in "Test results" tab. However, no output is generated when starting the job.
I have confirmed that the output blob is created when no join with reference data is used in the query.
Any help is appreciated. The sample reference json follows:
[
{
"DeviceId":"DEV-021",
"Brand":"brand01",
"Model":"model01"
}
]
Use flat json structure instead of array. It should give you the output
Check the path you specified in the reference data, maybe it is not correct or you did not specify the file name. Does it contain something like {date}/{time}/filename.json?
If you forget to specify the file name, it does not work as well.
And if you are testing the job, usually you specify the file manually and that is why your query works.
I need a solution to the following or a link to a detailed parameter guide that describes a Mapserver MapFile DATA element so I can try and work out a solution.
I'm trying to pass a dynamic filter to the mapfile but can't work out the syntax to do so.
The context is having a web-application where I have a WMS layer in Openlayers that connects to a Mapserver Mapfile which in turn is reading from a PostGIS raster DB.
The vector layers version of this approach has the following base syntax which is fine and works for vector tables in the DB:
DATA "geom FROM some_table using unique id using srid=4326"
FILTER (id = '%id%')
In effect this generates an SQL statement where the FILTER is created as a where clause in the DATA SQL.
However, in a raster DB example the DATA syntax shown at this link [http://postgis.net/docs/RT_FAQ.html#idm28328] is as follows:
DATA "PG:host=localhost port=5432 dbname='some_db' user='some_user' password='some_password' schema='some_schema' table='some_table' where='id=12' mode='2' "
So I can get things to work with hardcoded where elements, i.e. id=12 but in the previous example I could set the id parameter dynamically in the Openlayers WMS query through the FILTER line.
Does anyone know of the syntax to achieve this in the raster DATA example or is it even possible?
you need to add a "validation" object to your id type.
Add something like:
VALIDATION
'id' '[0-9]+'
END
to your layer object.
You can see that page for more explanations:
https://mapserver.org/mapfile/validation.html
I am trying to show images for products inside a basic report. The image needs to be dynamic, meaning the image should change based on the SKU value.
Right now I am inserting an image into a table, setting to external, and i've tried:
=Fields!URL.Value
=http://externalwebservername/sku= & Fields!SKU.Value
="http://externalwebservername/sku=" & Fields!SKU.Value
I do not get any images in my table.
My stored proc has all the data, including a URL with the image I wan't to show. Here is a sample of what the URL looks like:
http://externalwebservername/sku=123456
If I enter the URL in the field without "=" it will show that ONE image only.
How should I set up the expression to properly show the external image based on a dynamic URL? Running SQL 2016
Alan's answer should work, but in our environment we have strict proxy/firewall rules, so the two servers could not contact each other.
Instead we are navigating to the file stored on our storage system.
We altered the URL column to point to file path in the stored procedure. Insert image, set Source to External and Value set to [URL].
URL= file://server\imagepath.jpg
As long as the account executing the report has permissions to access the URLs then your 3rd expression should have worked.
I put together a simple example as follows.
I created a new blank report then added a Data Source. It doesn't matter where this points, we won't use it directly.
Then I created a dataset (Dataset1) with the following SQL to give me list of image names.
SELECT '350x120' AS suffix
UNION SELECT '200x100'
UNION SELECT '500x500'
Actually, these are just parameters for the website http://placehold.it/ which will generate images based on the size you request, but that's not relevant for this exercise.
We'll be showing three images from the following URLs
http://placehold.it/350x120
http://placehold.it/200x100
http://placehold.it/500x500
Next, create a table, I used 3 columns to give me more testing options. Set the DataSetName to DataSet1 if it isn't already.
In the first column the expression is just =Fields!suffix.Value
In the second column I added an image, set it's source property to External and the Value to ="http://placehold.it/" & Fields!suffix.Value
I then added a 3rd column with the same expression as the image Value so I could see what was being used as the image URL. I also added an action that goes to the same URL, just to check the URL did not have any unprintable characters in it that might cause a problem.
The basic report design looks like this.
The rendered result looks like this.
I have a simple conversion of JSON to XML using MuleSoft. In "Transform Message" component, I provided JSON Schema as Input and XML Schema as Output. When I run the app, the conversion happens if the file matches with both schema but it generates an empty XML file if it doesn't match.
I want below conditions:
1) If the file matches with schema, the converted output file should be sent to converted folder and the original file should move to Success folder.
2) If the file doesn't match with schema, the original file should move to the Failure folder instead of conversion.
Hope, I explained it comprehensively as I am new to MuleSoft. Here is a sample diagram which may simplify my requirement. Provide me with a new one if I badly designed the process.
First thing you need to create a flowVar that will hold your original payload.
When your doing your evaluation, if its XML then use a simple XPath expression like //elementName[not(node())]
Lastly, on your success use scatter-gather for multi-threading write. Pull your original payload from flowVar and write to Success and Write your regular payload to your Converted folder