I am trying to use AWS Config Advanced Query to generate a report against a specific rule I have created.
SELECT
configuration.targetResourceId,
configuration.targetResourceType,
configuration.complianceType,
configuration.configRuleList
WHERE
configuration.configRuleList.configRuleName = 'aws_config-requiredtags-rule'
AND configuration.complianceType = 'NON_COMPLIANT'
Results look similar to this:
[
0:{
"configRuleName":"aws_configrequiredtags-rule"
"configRuleArn":"arn:aws:config:us-east-2:123456789:config-rule/config-rule-dl6gsy"
"configRuleId":"config-rule-dl6gsy"
"complianceType":"COMPLIANT"
}
1:{
"configRuleName":"eaws_config-instanceinvpc-rule"
"configRuleArn":"arn:aws:config:us-east-2:123456789:config-rule/config-rule-dc4f1x"
"configRuleId":"config-rule-dc4f1x"
"complianceType":"NON-COMPLIANT"
}
While this query produces results, it separates my config rule and compliance type, so I am not only getting results where my config rule is ONLY Non-compliance for 'aws_config-requiredtags-rule' results.
I am pretty novice with SQL, but hope there is a way for me to specify that I only want to see Non-Compliant results against a specific rule.
thanks,
This is a limitation of the AWS Config Service - and a pretty big one IMO. When you filter on properties within arrays, those filters are treated like OR operations instead of AND. There doesn't seem to be a good way of performing meaningful queries for individual rules.
From the docs:
When querying against multiple properties within an array of objects, matches are computed against all the array elements
...
The first condition configuration.configRuleList.complianceType = 'non_compliant' is applied to ALL elements in R.configRuleList, because R has a rule (rule B) with complianceType = ‘non_compliant’, the condition is evaluated as true. The second condition configuration.configRuleList.configRuleName is applied to ALL elements in R.configRuleList, because R has a rule (rule A) with configRuleName = ‘A’, the condition is evaluated as true. As both conditions are true, R will be returned.
Related
I use some configuration logic to generate Sparql queries with RDF4j and the SparqlBuilder.
// prepare selectVariables, prefixes and whereCondition according to configuration
SelectQuery mainQuery = Queries.SELECT(selectVariables)
.prefix(prefixes)
.where(whereCondition)
Now I wish to allow for users to configure custom WHERE conditions to be used as SubSelects and composed with the rest of the query logic.
Since the configuration is YAML and the users are trained in Sparql, I wished to let users specify custom patterns as YAML multiline strings like this example
customQuery: |
?_ wdt:P31 wd:Q5;
wdt:P19/wdt:P131* wd:Q60.
This way I can let the users customize freely the different queries that I will generate based on the configured condition.
The problem
I already managed to parse the query fragment using RDFj SparqlParser:
SPARQLParserFactory PARSER_FACTORY = new SPARQLParserFactory();
QueryParser parser = PARSER_FACTORY.getParser();
ParsedQuery parsed = parser.parseQuery(query, null);
ProjectionVisitor projectionVisitor = new ProjectionVisitor();
parsed.getTupleExpr().visit(projectionVisitor);
TupleExpr parsedExpression = projectionVisitor.getProjectionArg();
but I can't use the parsedExpression into the SparqlBuilder methods, the nodes representation for the parser looks incompatible with the ones for the fluent builder.
Is there any way to use parsed expressions inside the SparqlBuilder?
No, it is not possible to use parsed expressions in the SparqlBuilder. What you could probably do instead though (freewheeling here) is use the SparqlBuilder to generate a query with a placeholder pattern of some sort, parse that, and then use a parse tree visitor to find that placeholder pattern and replace it with the custom parsed expression you got from the user.
New to SQL, I am writing as an exercise an API middleware that checks if the information contained in some headers match a database entry ("token-based authentication"). Database access is based on GORM.
To this, I have defined my ORM as follows:
type User struct {
ID uint
UserName string
Token string
}
In my middleware I retrieve the content of relevant headers and end up with the variables userHeader and tokenHeader. They are supposed to be matched to the database in order to do the authentication.
The user table has one single entry:
select * from users
// 1,admin,admintoken
The authentication code is
var auth User
res := db.Where(&User{UserName: userHeader, Token: tokenHeader}).Find(&auth)
if res.RowsAffected == 1 {
// authentication succeeded
}
When testing this, I end up with the following two incorrect results (other combinations are correct):
with only one header set to a correct value (and the other one not present) the authentication is successful (adding the other header with an incorrect value is OK (=auth fails))
no headers set → authentication goes though
I expected my query to mean (in the context of the incorrect results above)
select * from users where users.user_name = 'admin' and users.token = ''
select * from users where users.user_name = '' and users.token = ''
and this query is correct on the console, i.e. produces zero results (ran against the database).
The ORM one, however, seems to discard non-existing headers and assume they are fine (this is at least my understanding)
I also tried to chain the Where clauses via
db.Where(&User{UserName: userHeader}).Where(&User{Token: tokenHeader}).Find(&auth)
but the result is the same.
What should be the correct query?
The gorm.io documentation says the following on the use of structs in Where conditionals:
When querying with struct, GORM will only query with non-zero fields,
that means if your field’s value is 0, '', false or other zero
values, it won’t be used to build query conditions ...
The suggested solution to this is:
To include zero values in the query conditions, you can use a map,
which will include all key-values as query conditions ...
So, when the token header or both headers are empty, but you still want to include them in the WHERE clause of the generated query, you need to use a map instead of the struct as the argument to the Where method.
db.Where(map[string]interface{}{"user_name": userHeader, "token": tokenHeader}).Find(&auth)
You can use Debug() to check for the generated SQL (it gets printed into stderr); use it if you are unsure what SQL your code generates
Note: This is a learning exercise to learn how to implement a SQL-like relational database. This is just one thin slice of a question in the overall grand vision.
I have the following query, given a test database with a few hundred records:
select distinct "companies"."name"
from "companies"
inner join "projects" on "projects"."company_id" = "companies"."id"
inner join "posts" on "posts"."project_id" = "projects"."id"
inner join "comments" on "comments"."post_id" = "posts"."id"
inner join "addresses" on "addresses"."company_id" = "companies"."id"
where "addresses"."name" = 'Address Foo'
and "comments"."message" = 'Comment 3/3/2/1';
Here, the query is kind of unrealistic, but it demonstrates the point which I am trying to make. The point is to have a query with a few joins, so that I can figure out how to write this in sequential steps.
The first part of the question is (which I think I've partially figured out), is how do you write these joins as a sequence of independent steps, with the output of one fed into the input of the other? Also, is there more than one way to do it?
// step 1
let companies = select('companies')
// step 2
let projects = join(companies, select('projects'), 'id', 'company_id')
// step 3
let posts = join(projects, select('posts'), 'id', 'project_id')
// step 4
let comments = join(posts, select('comments'), 'id', 'post_id')
// step 5
let finalPosts = posts.filter(post => !!comments.find(comment => comment.post_id === post.id))
// step 6
let finalProjects = projects.filter(project => !!posts.find(post => post.project_id === project.id))
// step 7, could also be run in parallel to step 2 potentially
let addresses = join(companies, select('addresses'), 'id', 'company_id')
// step 8
let finalCompanies = companies.filter(company => {
return !!posts.find(post => post.company_id === company.id)
&& !!addresses.find(address => address.company_id === company.id)
})
These filters could probably be more optimized using indexes of some sort, but that is beside the point I think. This just demonstrates that there seem to be about 8 steps to find the companies we are looking for.
The main question is, how do you automatically figure out the steps from the SQL query?
I am not asking about how to parse the SQL query into an AST. Assume we have some sort of object structure we are dealing with, like an AST, to start.
How would you have to have the SQL query in structured object form, such that it would lead to these 8 steps? I would like to be able to specify a query (using a custom JSON-like syntax, not SQL), and then have it divide the query into these steps to divide and conquer so to speak and perform the queries in parts (for learning how to implement distributed databases). But I don't see how we go from SQL-like syntax, to 8 steps. Can you show how that might be done?
Here is the full code for the demo, which you can run with psql postgres -f test.sql. The result should be "Company 3".
Basically looking for a high level algorithm (doesn't even need to be code), which describes the key way you would break down some sort of AST-like object representation of a SQL query, into the actual planned steps of the query.
My algorithm looks like this in my head:
represent SQL query in object tree.
convert object tree to steps.
I am not really sure what (1) should be structured like, and even if we had some sort of structure, I'm not sure how to get that to complete (2). Looking for more details on the implementations of these steps, mainly step (2).
My "object structure" for step 1 would be something like this:
const query = {
select: {
distinct: true,
columns: ['companies.name'],
from: ['companies'],
},
joins: [
{
type: 'inner join',
table: 'projects',
left: 'projects.company_id',
right: 'companies.id',
},
...
],
conditions: [
{
left: 'addresses.name',
op: '=',
right: 'Address Foo'
},
...
]
}
I am not sure how useful that is, but it doesn't relate to steps at all. At a high level, what kind of code would I have to write to convert that object sort of structure into steps? Seems like one potential avenue is do a topological sort on the joins. But then you need to combine that with the select and conditions somehow, not sure how you would even begin to programmatically know what step should be before what other step, or even what the steps are. Maybe if I somehow could break it into known "chunks", then it would be simple to apply TOP sort to it after that, but then the question is still, how to get into chunks from the object structure / SQL?
Basically, I have been reading about the theory behind "query planning and optimization", but don't know how to apply it in this regard. How did this site do it?
One aspect is breaking at least the where conditions into CNF.
Implementing joins is a huge topic which is probably out of scope for a StackOverflow answer.
If you're looking for practical information about how joins are implemented, I would suggest...
The Join Operation section of Use The Index, Luke for different types of join implementation.
Section 7 of the The SQLite Query Optimizer Overview covers joins. And reading the SQLite source code. It is about as small a practical SQL implementation will get.
The output of explain in Postgresql gives very detailed information about how it has implemented the query. And they are explained in Operator Optimization Information
I have a subject like "accessTo" = ["123", "123-edit"]
and a resource like "interestedId" = "123"
Now I'm trying to write a condition - where it checks "interestedId" concatenated with "-edit" equals "123-edit" in "AccessTo".
Im trying to write rule like this
anyOfAny_xacml1(function[stringEqual], "accessTo", "interestedId"+"-edit")
It is not allowing to do this.
Any help is appreciated.
In addition to the answer from Keerthi S ...
If you know there should only be one value of interestedId then you can do this to prevent the indeterminate from happening:
stringBagSize(interestedId) == 1 && anyOfAny(function[stringEqual], accessTo, stringOneAndOnly(interestedId) + "-edit")
If more than value is present then evaluation stops prior to reaching the function that expects only one value. This condition would return false if more than one value is present.
On the other hand if interestedId can have multiple values then this would work:
anyOfAny(function[stringEqual], accessTo, map(function[stringConcatenate],interestedId, "-edit"))
The map function will apply the stringConcatenate function to all values in the bag.
Since Axiomatics products are compliant with XACML specification, all attributes by default are assumed to contain multiple values(called as 'bags').
So if you would like to append a string to an attribute use stringOneAndOnly XACML function for the attribute to indicate that the attribute can have only one value.
So assuming you mean accessTo has attribute ID as Attributes.access_subject.subject_id, interestedId has the attribute ID as Attributes.resource.resource_id and anyOfAny_xacml1 is equivalent to anyOfAny XACML function, the resulting condition would look like,
anyOfAny(function[stringEqual], Attributes.access_subject.subject_id, stringOneAndOnly(Attributes.resource.resource_id) + "-edit")
I'm currently trying to create an Endeca query using the Java API for a URLENEQuery. The current query is:
collection()/record[CONTACT_ID = "xxxxx" and SALES_OFFICE = "yyyy"]
I need it to be:
collection()/record[(CONTACT_ID = "xxxxx" or CONTACT_ID = "zzzzz") and
SALES_OFFICE = "yyyy"]
Currently this is being done with an ERecSearchList with CONTACT_ID and the string I'm trying to match in an ERecSearch object, but I'm having difficulty figuring out how to get the UrlENEQuery to generate the or in the correct fashion as I have above. Does anyone know how I can do this?
One of us is confused on multiple levels:
Let me try to explain why I am confused:
If Contact_ID and Sales_Office are different dimensions, where Contact_ID is a multi-or dimension, then you don't need to use EQL (the xpath like language) to do anything. Just select the appropriate dimension values and your navigation state will reflect the query you are trying to build with XPATH. IE CONTACT_IDs "ORed together" with SALES_OFFICE "ANDed".
If you do have to use EQL, then the only way to modify it (provided that you have to modify it from the returned results) is via string manipulation.
ERecSearchList gives you ability to use "Search Within" functionality which functions completely different from the EQL filtering, though you can achieve similar results by using tricks like searching only specified field (which would be separate from the generic search interface") I am still not sure what's the connection between ERecSearchList and the EQL expression above?
Having expressed my confusion, I think what you need to do is to use String manipulation to dynamically build the EQL expression and add it to the Query.
A code example of what you are doing would be extremely helpful as well.