I have created an Athena table which contains access logs (source s3).
Now I have a working query to check when account specific data was downloaded by a different account than the account itself. It looks like the query below (account id is used in s3 as prefix):
SELECT * FROM s3_access_logs_db.mybucket_logs WHERE requester LIKE '%account-id-A%' AND operation = 'REST.GET.OBJECT' AND key NOT LIKE '%account-id-A%';
Now I want to make this more dynamic. I don't want to hardcode account IDs as I'm doing now. Is there a way to make this more dynamic. I don't care what happens as long as the account ID doesn't match.
WHERE requester LIKE '%account-id-A%' AND key NOT LIKE '%account-id-A%';
How can I achieve this?
Related
I wonder if there is a way to count the actual number of accesses to a certain GCP service by analysing audit log stored in BigQ. In other words, I have audit tables sink to BigQ (no actual access to Stackdriver). I can see a number of rows were generated per single access, i.e. it was one physical access to the GCS, but about 10 rows generated due to different function calls. I'd like to be able to say how many attempts/accesses were made by the user account by looking at X number of rows. This is a data example.
Thank you
Assuming that the sink works well and the logs are being exported to BQ, then you would need to check the format on how audit logs are written to BQ and the fields that are exported/written to BQ [1].
Then filter the logs by resource_type and member, these documents [2][3] can help you with that.
[1] https://cloud.google.com/logging/docs/audit/understanding-audit-logs#sample
[2] https://cloud.google.com/logging/docs/audit/understanding-audit-logs#sample
[3] https://cloud.google.com/logging/docs/audit/understanding-audit-logs#interpreting_the_sample_audit_log_entry
I am new to SAP-MDG and I am trying to identify how can we store cross reference details in MDG.
I got to know about key mapping but that is not something that I understand.
My requirement is:
I have 10 applications sending data to me each having it's own app id and customer number, I want to store the mapping.
App -id --- Customer number from application ---- sap-id(at our end)
Is this something that can be done without a custom table?
Better place for get answers for your questions - sdn.sap.com (sap mdg community)
you can create this table like a data model (for example via reuse method) and save it in staging area sap mdg
you can use Z-table for store this data
How can I retrieve all blobs uploaded by 'opponent' (not just opponent.blobId)?
Is there any solution just like PagedRequest for currentUser's blob?
There is no base way to do this,
because all files can be private or something like this
I propose to do next: use Custom Objects API http://quickblox.com/developers/Custom_Objects
Create UserFiles or UserAlbum table and write all your files (blobx IDs i mean) to it which you want to open to users
Next, other users will be able to retrieve all files - just get all data from this table using filter user_id=
I have a table in Amazon dynamoDB with a record structure like
{"username" : "joe bloggs" , "products" : ["1","2"] , "expires1" : "01/01/2013" , "expires2" : "01/02/2013"}
where the products property is a list of products belonging to the user and the expires n properties relate to the products in the list, the list of products is dynamic and there are many. I need to transfer this data to S3 in a format like
joe bloggs|1|01/01/2013
joe bloggs|2|01/02/2013
Using hive external tables I can map the username and products columns in dynamoDB, however I am unable to map the dynamic columns. Is there a way that I could extend or adapt the org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler in order to interpret and structure the data retrieved from dynamo before hive ingests it? or is there an alternative solution to convert the dynamo data to first normal form?
One of my key requirements is that i maintain the throttling provided by the dynamodb.throughput.read.percent setting so that I do not compromise operational use of the table.
You could build a specific UDTF(User defined table-generating functions) for that case.
I'm not sure how Hive handles asterisk(probably for your case) as an argument for the function.
Something like what Explode (source) does.
I'm currently logging all actions of users and want to display their actions for the people following them to see - kind of like Facebook does it for friends.
I'm logging all these actions in a table with the following structure:
id - PK
userid - id of the user whose action gets logged
actiondate - when the action happened
actiontypeid - id of the type of action (actiontypes stored in a different table - i.e. following other users, writing on people's profiles, creating new content, commenting on existing content, etc.)
objectid - id of the object they just created (i.e. comment id)
onobjectid - id of the object they did the action to (i.e. id of the content that they commented on)
Now the problem is there are several types of actions that get logged (actiontypeid).
What would be the best way of retrieving the data to display to the user?
The easiest way out would be gabbing the people the user follows dataset and then just go from there and grab all other info from the other tables (i.e. the names of the users the people you're following just started following, names of the user profiles they wrote on, etc.). This however would create a a huge amount of small queries and trips to the database in a while loop. Not a good idea.
I could use joins to retrieve everything in one massive data set, but how would I know where to grab the data from in just one query? - there's different types of actions that require me to look into several different tables to retrieve data, based on the actiontypeid...
i.e. To get User X is now following User Y I'd have to get my data (User Y's username) from the followers table, whereas User X commented on content Y would need me to look in the content table to get the content's title and URL.
Any tips are welcome, thanks!
Consider creating several views for different actiontypeids. Union them to have one full history.