I have written a simple query and join it with a json reference data. I can see correct results when testing the query in "Test results" tab. However, no output is generated when starting the job.
I have confirmed that the output blob is created when no join with reference data is used in the query.
Any help is appreciated. The sample reference json follows:
[
{
"DeviceId":"DEV-021",
"Brand":"brand01",
"Model":"model01"
}
]
Use flat json structure instead of array. It should give you the output
Check the path you specified in the reference data, maybe it is not correct or you did not specify the file name. Does it contain something like {date}/{time}/filename.json?
If you forget to specify the file name, it does not work as well.
And if you are testing the job, usually you specify the file manually and that is why your query works.
Related
I have a job which processes files and then lands them as single CSVs on a blob storage container. The problem I face is that I also need to land empty files, which only contain the header. How can this be achieved when I use .saveSingleFile?
Example Code snipped:
df.coalesce(1)
.write
.options(configuration.readAndWriteOptions)
.partitionBy(INGESTION_TIME)
.format("csv")
.mode("append")
.saveSingleFile(path.toString)
Example readAndWriteOptions:
{"sep": ";", "header": "true"}
In other words:
In above case, if df.show() is only displaying a header, no CSV file is written. However, I want to output a csv file without data but column names. Is there an option which would allow this ? Both cases need to be possible, if data is available and if data is not available, therefore something like .take(1) will not be a sufficient solution.
Update:
Looks like this is related to a Spark API Bug and should have been resolved with Version 3.
I have written a query that uses a stream input and join it with static json reference data. I get the correct results when testing the query in "Test results" tab (uploading the same sample reference data). However, no output is generated when starting the job.
My query:
The stream input produces random results every second like :
And the json reference file is:
From the monitoring dashboard there are also no input events nor output.
I have confirmed that the output blob is created when no join with reference data is used in the query. I have uploaded the json reference data in a storage container and have provided the path pattern : ref/Atm.json in my example.
Thanks for any help.
I was just curious and tried to repro , I am using a very similar query as yours .
SELECT sum(A.amount),B.area_code
INTO Gen2
FROMEHInput A
JOIN JSONref B ON A.ATMCode=B.atm_code
group by B.area_code,TumblingWindow(minute,1)
This is the output which you have.( all junk data , but it confirms query works )
{"sum":63580.0,"area_code":20}
{"sum":73060.0,"area_code":30}
{"sum":68220.0,"area_code":20}
At this point , I thought to making some changes to check if I can repro your case of no output , I just updated the static file to something non existing file , ASA never complained about that but I never got the results either .
I think you may be hitting the same ( as you mentioned that without the join things just works fine ) .
In my case I have the static file location something like
https://blobaccount.blob.core.windows.net/container/**File2/Atm.json**
and when i create the Reference Input , I pass the file name as
I think for some reason its getting messed up here .
Other thing which you can try it increase the TW to a bigger window .
Let me know how it goes .
I managed to resolve this issue. The error as you suggested was the path pattern I had specified for my reference data. So I had created a container e.g. "mycontainer" and I just dropped the json files in it. The path pattern that worked for me was just the names of the json files inside that container. No "/" , just Atm.json in my example.
Now the stream job creates output files in my blob storage.
I need a solution to the following or a link to a detailed parameter guide that describes a Mapserver MapFile DATA element so I can try and work out a solution.
I'm trying to pass a dynamic filter to the mapfile but can't work out the syntax to do so.
The context is having a web-application where I have a WMS layer in Openlayers that connects to a Mapserver Mapfile which in turn is reading from a PostGIS raster DB.
The vector layers version of this approach has the following base syntax which is fine and works for vector tables in the DB:
DATA "geom FROM some_table using unique id using srid=4326"
FILTER (id = '%id%')
In effect this generates an SQL statement where the FILTER is created as a where clause in the DATA SQL.
However, in a raster DB example the DATA syntax shown at this link [http://postgis.net/docs/RT_FAQ.html#idm28328] is as follows:
DATA "PG:host=localhost port=5432 dbname='some_db' user='some_user' password='some_password' schema='some_schema' table='some_table' where='id=12' mode='2' "
So I can get things to work with hardcoded where elements, i.e. id=12 but in the previous example I could set the id parameter dynamically in the Openlayers WMS query through the FILTER line.
Does anyone know of the syntax to achieve this in the raster DATA example or is it even possible?
you need to add a "validation" object to your id type.
Add something like:
VALIDATION
'id' '[0-9]+'
END
to your layer object.
You can see that page for more explanations:
https://mapserver.org/mapfile/validation.html
When I do an extract from multiple files and include part of the filename in the fields list and in the FROM clause (e.g. FROM "/input/filename-{filedate:*}.nc"), the resulting output file only contains a header row. If I remove "filedate" from the fields list and the FROM clause, I get the correct output.
I noticed in the job graph that when including "filedate", an "Empty Input" and an "Extract Cross" step is added before the "PodAggregate" step, and in the "Extract Cross" no data is written. What is this step?
Also, if I run the original extract including "filedate" locally, I get the correct output, so it's only in ADLA this error occurs.
I use a custom extractor and I don't know if this has anything to do with it. I haven't tested with a built-in extractor.
We released the new "fast file set" option by default. Unfortunately, it introduced a regression for some plans. Until we fix it, please add the following statement to your script:
SET ##InternalDebug = "FileSetV2:off";
Our apologies for any inconvenience this may have caused.
I have a simple conversion of JSON to XML using MuleSoft. In "Transform Message" component, I provided JSON Schema as Input and XML Schema as Output. When I run the app, the conversion happens if the file matches with both schema but it generates an empty XML file if it doesn't match.
I want below conditions:
1) If the file matches with schema, the converted output file should be sent to converted folder and the original file should move to Success folder.
2) If the file doesn't match with schema, the original file should move to the Failure folder instead of conversion.
Hope, I explained it comprehensively as I am new to MuleSoft. Here is a sample diagram which may simplify my requirement. Provide me with a new one if I badly designed the process.
First thing you need to create a flowVar that will hold your original payload.
When your doing your evaluation, if its XML then use a simple XPath expression like //elementName[not(node())]
Lastly, on your success use scatter-gather for multi-threading write. Pull your original payload from flowVar and write to Success and Write your regular payload to your Converted folder