KDB: try-catch over a list and return list of fails - error-handling

I want to run the function
{`Security$x}
over a list
order`KDB_SEC_ID
and return the list of values that failed. I have the below, which works, but I'm wondering if there is a neater way to write this without the use of a do loop.
Example Code:
idx:0;
fails:();
do[count (order`KDB_SEC_ID);
error:#[{`Security$x};(order`KDB_SEC_ID)[idx];0Nj];
if[error=0Nj;fails:fails,(order`KDB_SEC_ID)[idx]];
idx:idx+1;
];
missingData:select from order where KDB_SEC_ID in distinct fails;

I agree that Terry's answer is the simplest method but here is a simpler way to do the method you were trying to help you see how achieve it without using do loops
q)SECURITY
`AAPL`GOOG`MSFT
q)order
KDB_SEC_ID val
--------------
AAPL 1
GOOG 2
AAPL 3
MSFT 4
IBM 5
q)order where #[{`SECURITY$x;0b};;1b] each order`KDB_SEC_ID
KDB_SEC_ID val
--------------
IBM 5
It outputs a 0b if it passes and 1b if it fails resulting in a boolean list. Using where on a boolean list returns the indices where the 1b's occur which you can use to index into order to return the failing rows.

If your test is to check which of the KDB_SEC_ID's can be enumerated against the Security list, couldn't you do
q)select from order where not KDB_SEC_ID in Security
Or am I missing something?
To answer your question in a more general case, you could achieve a try-catch over a list to return the list of fails using something like
q){x where #[{upper x;0b};;1b] each x}(2;`ab;"Er";1)
2 1

Related

How can I put several extracted values from a Json in an array in Kusto?

I'm trying to write a query that returns the vulnerabilities found by "Built-in Qualys vulnerability assessment" in log analytics.
It was all going smoothly I was getting the values from the properties Json and turning then into separated strings but I found out that some of the terms posses more than one value, and I need to get all of them in a single cell.
My query is like this right now
securityresources | where type =~ "microsoft.security/assessments/subassessments"
| extend assessmentKey=extract(#"(?i)providers/Microsoft.Security/assessments/([^/]*)", 1, id), IdAzure=tostring(properties.id)
| extend IdRecurso = tostring(properties.resourceDetails.id)
| extend NomeVulnerabilidade=tostring(properties.displayName),
Correcao=tostring(properties.remediation),
Categoria=tostring(properties.category),
Impacto=tostring(properties.impact),
Ameaca=tostring(properties.additionalData.threat),
severidade=tostring(properties.status.severity),
status=tostring(properties.status.code),
Referencia=tostring(properties.additionalData.vendorReferences[0].link),
CVE=tostring(properties.additionalData.cve[0].link)
| where assessmentKey == "1195afff-c881-495e-9bc5-1486211ae03f"
| where status == "Unhealthy"
| project IdRecurso, IdAzure, NomeVulnerabilidade, severidade, Categoria, CVE, Referencia, status, Impacto, Ameaca, Correcao
Ignore the awkward names of the columns, for they are in Portuguese.
As you can see in the "Referencia" and "CVE" columns, I'm able to extract the values from a specific index of the array, but I want all links of the whole array
Without sample input and expected output it's hard to understand what you need, so trying to guess here...
I think that summarize make_list(...) by ... will help you (see this to learn how to use make_list)
If this is not what you're looking for, please delete the question, and post a new one with minimal sample input (using datatable operator), and expected output, and we'll gladly help.

verify a method is called only once with a given parameter with Junit 5

I want to ensure that my test calls the userService.find(userName) only with the expected aTestName and only once. Currently, I am achieving this by the following code. Is there a way to combine the two BDDMockito.then... in one?
String aTestName = "aTestName";
BDDMockito.then(userService).should(times(1)).findByName(any());
BDDMockito.then(userService).should(times(1)).findByName(aTestName);
What you should use is only(). So like:
BDDMockito.then(userService).should(only()).findByName(aTestName);

Creating a string from a path

I have a chain of nodes, e.g.
{Name: 'X'}->{Name: 'Y'}->{Name: 'Z'}
and I'd like to create a string representing the path in cypher (I know I can do it client-side but I want to do it in a query) that would look like this:
'X->Y->Z'
is that feasible? I've investigated use of collect() and UNWIND, and have googled till I'm blue in the face
thoughts?
* Edit I *
as an addendum (and to make the problem more difficult) my query is going to return a collection of paths (a tree, a DAG), so I'll need to create a string for each of the paths in the tree
REDUCE is your friend here:
WITH reduce(s="",n in nodes(p) | s+n.name+"->") as str
RETURN substring(str,0,length(str)-2)
or if you want to save the extra operation
RETURN reduce(s=head(nodes(p)).name, n in tail(nodes(p)) | s+"->"+n.name)
or with APOC
RETURN apoc.text.join([n in nodes(p) | n.name],"->")
found an old posting (2014) from Michael Hunger with the answer:
http://grokbase.com/t/gg/neo4j/147ch1nj9b/concatenate-nodes-properties

MuleSoft: Sum line items and group by ID

I am new to integration and MuleSoft so I need your help. I have a flat file with different invoice line items per salesID, like this:
SalesOrderID OrderQty UnitPrice
43659 70 2024.994
43659 70 2024.994
43660 1 419.4589
43660 1 874.794
43661 1 809.76
I want to insert total invoice amount and quantity in another CSV file using Mule, something like this:
SalesOrderID OrderQty UnitPrice
43659 140 4049.988
43660 2 1294.4589
43661 1 809.76
I know how to do this in informatica,but im trying to figure out a way to do this in MuleSoft. How can I sum up all the line items and group them by SalesOrderID? Any help/clue will be really appreciated.
Thanks.
There are a number of ways to do this, let me explain my personal favourite assuming you are using Mule community edition.
Use the superb library: SuperCSV, it will allow you not only to parse the data (including those dot numbers) but also validate it and give back an exact report on why the file is broken in case it is.
This could be done in a transformer that transform the inbound stream and return either a map of maps (or even better a iterator that handles the whole thing but this is more difficult) or a custom exception with the error report.
Given that this requirement is one that faces Mule developers even today, it's useful to see a solution based on the Mule 4.x runtime and Dataweave 2.x.
If the data came from a file or is otherwise a monolithic stream of text, use the splitBy() function to get an array of text lines.
payload splitBy '\n'
remove the first line as the headers should not be calculated
payload[1 to -1] // this is my favorite way to do it
Now use the reduce() function to iterate over each of the lines in turn, updating the accumulator each time to account for quantity and price.
Hopefully that helps

DQL query to return all files in a Cabinet in Documentum?

I want to retrieve all the files from a cabinet (called 'Wombat Insurance Co'). Currently I am using this DQL query:
select r_object_id, object_name from dm_document(all)
where folder('/Wombat Insurance Co', descend);
This is ok except it only returns a maximum of 100 results. If there are 5000 files in the cabinet I want to get all 5000 results. Is there a way to use pagination to get all the results?
I have tried this query:
select r_object_id, object_name from dm_document(all)
where folder('/Wombat Insurance Co', descend)
ENABLE (RETURN_RANGE 0 100 'r_object_id DESC');
with the intention of getting results in 100 file increments, but this query gives me an error when I try to execute it. The error says this:
com.emc.documentum.fs.services.core.CoreServiceException: "QUERY" action failed.
java.lang.Exception: [DM_QUERY2_E_UNRECOGNIZED_HINT]error:
"RETURN_RANGE is an unknown hint or is being used incorrectly."
I think I am using the RETURN_RANGE hint correctly, but maybe I'm not. Any help would be appreciated!
I have also tried using the hint ENABLE(FETCH_ALL_RESULTS 0) but this still only returns a maximum of 100 results.
To clarify, my question is: how can I get all the files from a cabinet?
You have already accepted an answer which is using DFS.
Since your are playing with DFC, these information might help you.
DFS:
If you are using DFS, you have to aware about the number of concurrent sessions that you can consume with DFS.
I think it is 100 or 150.
DFC:
Actually there is a limit that you can fetch via DFC (I'm not sure with DFS).
Go to your DFC application(webtop or da or anything) and check the dfc.properties file.
# Maximum number of results to retrieve by a query search.
# min value: 1, max value: 10000000
#
dfc.search.max_results = 100
# Maximum number of results to retrieve per source by a query search.
# min value: 1, max value: 10000000
#
dfc.search.max_results_per_source = 400
dfc.properties.full or similar file is there and you can verify these values according to your system.
And I'm talking about the ContentServer side, not the client side dfc.properties file.
If you use ENABLE (RETURN_TOP) hint with DFC, there are 2 ways to fetch the results from the ContentServer.
Object based
Row based
You have to configure this by using the parameter return_top_results_row_based in the server.ini file.
All of these changes for the documentum server side, not for your DFC/DQL client.
Aha, I've figured it out. Using DFS with Java (an abstraction layer on top of DFC) you can set the starting index for query results:
String queryStr = "select r_object_id, object_name from dm_document(all)
where folder('/Wombat Insurance Co', descend);"
PassthroughQuery query = new PassthroughQuery();
query.setQueryString(queryStr);
query.addRepository(repositoryStr);
QueryExecution queryEx = new QueryExecution();
queryEx.setCacheStrategyType(CacheStrategyType.DEFAULT_CACHE_STRATEGY);
queryEx.setStartingIndex(currentIndex); // set start index here
OperationOptions operationOptions = null;
// will return 100 results starting from currentIndex
QueryResult queryResult = queryService.execute(query, queryEx, operationOptions);
You can just increment the currentIndex variable to get all results.
Well, the hint is being used incorrectly. Start with 1, not 0.
There is no built-in limit in DQL itself. All results are returned by default. The reason you get only 100 results must have something to do with the way you're using DFC (or whichever other client you are using). Using IDfCollection in the following way will surely return everything:
IDfQuery query = new DfQuery("SELECT r_object_id, object_name "
+ "FROM dm_document(all) WHERE FOLDER('/System', DESCEND)");
IDfCollection coll = query.execute(session, IDfQuery.DF_READ_QUERY);
int i = 0;
while (coll.next()) i++;
System.out.println("Number of results: " + i);
In a test environment (CS 6.7 SP1 x64, MS SQL), this outputs:
Number of results: 37162
Now, there's proof. Using paging is however a good idea if you want to improve the overall performance in your application. As mentioned, start counting with the number 1:
ENABLE(RETURN_RANGE 1 100 'r_object_id DESC')
This way of paging requires that sorting be specified in the hint rather than as a DQL statement. If all you want is the first 100 records, try this hint instead:
ENABLE(RETURN_TOP 100)
In this case sorting with ORDER BY will work as you'd expect.
Lastly, note that adding (all) will not only find all documents matching the specified qualification, but all versions of every document. If this was your intention, that's fine.
I've worked with DFC API (with Java) for a while but I don't remember any default limit on queries, IIRC we've always got all of the documents, there weren't any limit. Actually (according to my notes) we have to set the limit explicitly with, for example, enable (return_top 2000). (As far I know the syntax might be depend on the DBMS behind EMC Documentum.)
Just a guess: check your dfc.properties file.