How to find who deletes a table in the logs - google-bigquery

I have a dataset ID, tables were created. Which were deleted. I need to check who deleted them and when.

The easiest way would be in the GCP console click in the "ACTIVITY" tab, filter the "Resource Type" by "Big Query" and look for the "Delete Table" entry. If you click the entry it'll expand and show info about the deletion, along with the account that performed the deletion.

You need to enable auditlogs export.
There are many types of deletes:
delete from UI / API
delete with QUERY (overwrite as destination table)
delete via CREATE OR REPLACE syntax
autodelete done via expiration
Common type of deletes are explained here:
Type 1: you issued a table deleted Query/API call etc..
Then you can run a query like:
SELECT * FROM `dataset.cloudaudit_googleapis_com_activity_20190919`
where resource.type='bigquery_resource'
and protopayload_auditlog.methodName='tableservice.delete'
you get back a large table which cannot be posted here as a nice image, but as simplified JSON it is here:
[
{
"logName": "projects/editedname/logs/cloudaudit.googleapis.com%2Factivity",
"resource": {
"type": "bigquery_resource",
"labels": {
"project_id": "editedname",
}
},
"protopayload_auditlog": {
"serviceName": "bigquery.googleapis.com",
"methodName": "tableservice.delete",
"resourceName": "projects/editedname/datasets/dataset/tables/industry2",
"authenticationInfo": {
"principalEmail": "something#domain.com",
"authoritySelector": null,
"serviceAccountKeyName": null,
"serviceAccountDelegationInfo": []
},
"authorizationInfo": [
{
"resource": "projects/editedname/datasets/dataset/tables/industry2",
"permission": "bigquery.tables.delete",
"granted": "true",
"resourceAttributes": null
}
],
"requestMetadata": {
"callerIp": "1.2.3.4",
"callerSuppliedUserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.75 Safari/537.36,gzip(gfe)",
"callerNetwork": null,
"requestAttributes": null,
"destinationAttributes": null
},
},
"timestamp": "2019-09-19 08:47:00.381 UTC",
"receiveTimestamp": "2019-09-19 08:47:00.590316 UTC",
"severity": "NOTICE",
}
]
As you can see you have the table that was deleted eg: industry2 in my example, also the user's or service account email address, and the date, even metadata of the caller by IP, and userAgent.
More about the available information here
Type 2: You issued a table override query or a CREATE OR REPLACE syntax
For these kind of "deletes" you won't find a separate deleted entry. But you can look for the "truncated" flag in the metadataJson

Related

Account list doesn't return account id for "grouped" accounts

I'm using the Xero API to do some integration and I'm using the accounts get feature to return a list of accounts. I'd like to join this to some of the reports, for example the profit and loss report which would allow me to then group the accounts returned by type. All looks good, other than some accounts have an extra "groupID" attribute. When an account has this extra attribute, the "account" attribute that usually contains a GUID which I can link to reports now contains what could be a GUID, only without the dashes, and doesn't link to any other report.
For example, this is an account that all works correctly...
{
"RowType": "Row",
"Cells": [
{
"Value": "General Expenses",
"Attributes": [
{
"Value": "8d631f87-7304-401f-a7cd-d1d42c1b458a",
"Id": "account"
}
]
},
{
"Value": "179.07",
"Attributes": [
{
"Value": "8d631f87-7304-401f-a7cd-d1d42c1b458a",
"Id": "account"
}
]
}
]
},
This one doesn't, and the GUID isn't formatted as a GUID anymore...
{
"RowType": "Row",
"Cells": [
{
"Value": "Directors' Remuneration",
"Attributes": [
{
"Value": "a02584cbb64e4c109b4355b292da0de5",
"Id": "account"
},
{
"Value": "a02584cbb64e4c109b4355b292da0de5",
"Id": "groupID"
}
]
},
{
"Value": "55737.84",
"Attributes": [
{
"Value": "a02584cbb64e4c109b4355b292da0de5",
"Id": "account"
},
{
"Value": "a02584cbb64e4c109b4355b292da0de5",
"Id": "groupID"
}
]
}
]
},
Any idea what would cause that? Is there a way I can list "grouped" accounts?
Thanks in advance.
At the moment, there is no current way to get the accountIDs in the Reports endpoints for accounts that have been grouped together.
But, there are 2 ways you can get that in the response. In the API request, if you send 'standardLayout=true', then that'll give you back the accountIDs.
Or on the Web app, in P&L edit layout mode, if you uncheck the 'Show Summary Only' box for the group of those accounts, that'll also work the same way and you should be able to see the accountIDs in the API response.
Thanks
Keertika
I believe that is due to the org having multi-currency for that account.
However I don't see much other documentation related to how to lookup the account from that. I will update this answer If I can find anything else out internally.
Multi-Currency System Accounts - FXGROUPID
For organisations in most regions, the standard layout of the profit and loss report will group multi currency system accounts into a single line with a Value of FXGROUPID (instead of an AccountID).
This is not the case for US organisations and Australian demo companies. Multi-currency system accounts will be displayed seperately with their respective AccountIDs.
Multi currency support could be seen here: https://go.xero.com/Setup/CurrencyRates.aspx
I think you can also just query the currencies endpoint to find out if an org may support mutiple.

GCP Bigquery: Can't query stackdriver access logs exported in cloudstorage because invalid json field "#type"

I store the access log of a pixel image in a cloudstorage bucket dev-access-log-bucket using the standard "sink"
so the files looks like this requests/2019/05/08/15:00:00_15:59:59_S1.json
and one line looks like this (I formatted the json, but it's on one line normmaly) :
{
"httpRequest": {
"cacheLookup": true,
"remoteIp": "93.24.25.190",
"requestMethod": "GET",
"requestSize": "224",
"requestUrl": "https://dev-snowplow.legalstart.fr/one_pixel_image.png?user_id=0&action=purchase&product_id=0&money=10",
"responseSize": "779",
"status": 200,
"userAgent": "python-requests/2.21.0"
},
"insertId": "w6wyz1g2jckjn6",
"jsonPayload": {
"#type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
"statusDetails": "response_sent_by_backend"
},
"logName": "projects/tracking-pixel-239909/logs/requests",
"receiveTimestamp": "2019-05-08T15:34:24.126095758Z",
"resource": {
"labels": {
"backend_service_name": "",
"forwarding_rule_name": "dev-yolaw-pixel-forwarding-rule",
"project_id": "tracking-pixel-239909",
"target_proxy_name": "dev-yolaw-pixel-proxy",
"url_map_name": "dev-urlmap",
"zone": "global"
},
"type": "http_load_balancer"
},
"severity": "INFO",
"spanId": "7d8823509c2dc94f",
"timestamp": "2019-05-08T15:34:23.140747307Z",
"trace": "projects/tracking-pixel-239909/traces/bb55577eedd5797db2867931f8de9162"
}
all of these once again are standard GCP things, I did not customize anything here.
So now I want to do some requests on it from Bigquery, I create a dataset and an external table configured like this :
External Data Configuration
Source URI(s) gs://dev-access-log-bucket/requests/*
Auto-detect schema true (note: I don't know why it puts true though i've manually defined it)
Ignore unknown values true
Source format NEWLINE_DELIMITED_JSON
Max bad records 0
and the following manual schema:
timestamp DATETIME REQUIRED
httpRequest RECORD REQUIRED
httpRequest. requestUrl STRING REQUIRED
and when I run a request
SELECT
timestamp
FROM
`path.to.my.table`
LIMIT
1000
I got
Invalid field name "#type". Fields must contain only letters, numbers, and underscores, start with a letter or underscore, and be at most 128 characters long.
How can I work around this without needing to pre-process the log to not have the "#type" field in it ?

Dropbox API V2 list_file_members/batch empty results

I'm currently trying to work with the Dropbox list_file_members API endpoint, as it appears to me to be the only place to find out who owns a file (
see follow example result taken from the documentation page )
{
"users": [
{
"access_type": {
".tag": "owner"
},
"user": {
"account_id": "dbid:AAH4f99T0taONIb-OurWxbNQ6ywGRopQngc",
"same_team": true,
"team_member_id": "dbmid:abcd1234"
},
"permissions": [],
"is_inherited": false
}
],
"groups":[...]
...
}
However, when I call the API on a single file I get the follow
{
"users": [],
"groups": [
{
"access_type": {
".tag": "editor"
},
"permissions": [],
"is_inherited": true,
"group": {
"group_name": "Everyone at TEAM_NAME_HERE",
"group_id": "g:GROUP_ID_HERE",
"member_count": 6,
"group_management_type": {
".tag": "company_managed"
},
"group_type": {
".tag": "team"
},
"is_owner": false,
"same_team": true
}
}
],
"invitees": []
}
This result contains no owner information, so I'm assuming this is because everyone has the same access levels ??
The problem worsens when I try to call files in batches using the sharing_list_file_members/batch endpoint, I get the following result
[
{
"file": "id:THIS_IS_MY_FILE_ID",
"result": {
".tag": "result",
"members": {
"users": [],
"groups": [],
"invitees": []
},
"member_count": 0
}
}
]
Obviously this is even less helpful, this is the same when I access the API via my own PHP, as well as the API explorer, could anyone tell me where I'm going wrong and why I'm getting no results from users and even groups when done in batches ?
The /2/sharing/list_file_members endpoint is documented as:
Use to obtain the members who have been invited to a file, both inherited and uninherited members.
The /2/sharing/list_file_members/batch endpoint is documented as:
Get members of multiple files at once. The arguments to this route are more limited, and the limit on query result size per file is more strict. To customize the results more, use the individual file endpoint.
Inherited users are not included in the result, and permissions are not returned for this endpoint.
It sounds like the file for your example is in a team folder, and so the group listed for your non-batch example is the team group, i.e., an inherited group. The documentation indicates that this group isn't expected when using the batch endpoint.

Accessing previous state of resource from gitlab webhooks

In implementing a custom webhook handler for Gitlab issues, I'm struggling to find out how to determine the previous value of a field before the hook fired.
For reference, this is the webhooks documention, and an issue-specific payload looks a bit like:
{
"object_attributes": {
"action": "update",
"url": "https://example.com/foo/bar/issues/10",
"iid": 10,
"state": "opened",
"milestone_id": null,
"description": "44",
"branch_name": null,
"position": 0,
"updated_at": "2015-11-07 17:39:47 +0000",
"created_at": "2015-11-07 17:06:07 +0000",
"project_id": 51,
"author_id": 6,
"assignee_id": null,
"title": "123",
"id": 180
},
"user": {
"avatar_url": "https://example.com/uploads/user/avatar/6.jpg",
"username": "shabble",
"name": "real name"
},
"object_kind": "issue"
}
when an issue is modified.
In this example, the assignee_id was changed from myself to null, but there appears to be no way to determine what the previous value (that is, me) was prior to update.
This is the same for almost all editable fields, including things like the entire issue description. So it's possible to tell what it is now, and maybe when it was last changed, but not see any of the previous changes.
Is there anywhere this information (previous state) can be found, perhaps via the gitlab issues API although I've not found anything plausible.
The only option seems to be a separate system to record the values from every hook, and then when a subsequent hook comes in that matches the same issue, diff the last record to determine changes. This would be less than ideal.
Any other ideas?

ARM - How can I get the access key from a storage account to use in AppSettings later in the template?

I'm creating an Azure Resource Manager template that instantiates multiple resources, including an Azure storage account and an Azure App Service with a Web App.
I'd like to be able to capture the primary access key (or the full connection string, either way is fine) from the newly-created storage account, and use that as a value for one of the AppSettings for the Web App.
Is that possible?
Use the listkeys helper function.
"appSettings": [
{
"name": "STORAGE_KEY",
"value": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]"
}
]
This quickstart does something similar:
https://azure.microsoft.com/en-us/documentation/articles/cache-web-app-arm-with-redis-cache-provision/
The syntax has changed since the other answer was accepted. The error you will now hit is 'Template language expression property 'key1' doesn't exist, available properties are 'keys'
Keys are now represented as an array of keys, and the syntax is now:
"StorageAccount": "[Concat('DefaultEndpointsProtocol=https;AccountName=',variables('StorageAccountName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('StorageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value)]",
See: http://samcogan.com/retrieve-azure-storage-key-in-arm-script/
I faced with this issue two times. First in the 2015 and last today in May of 2017.
I need to add connection strings to the WebApp - I want to add strings automatically from generated resources during deployment from the ARM template. It can help later to not add manually this values.
First time I used old version of the function listKeys (it looks like old version returns result not as object but as value):
"AzureWebJobsStorage": {
"type": "Custom",
"value": "[concat(variables('storageConnectionString'), listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2015-05-01-preview').key1)]"
},
Today last version of the working template is:
"resources": [
{
"apiVersion": "2015-08-01",
"type": "config",
"name": "connectionstrings",
"dependsOn": [
"[resourceId('Microsoft.Web/Sites/', parameters('webSiteName'))]"
],
"properties": {
"DefaultConnection": {
"value": "[concat('Data Source=tcp:', reference(resourceId('Microsoft.Sql/servers/', parameters('sqlserverName'))).fullyQualifiedDomainName, ',1433;Initial Catalog=', parameters('databaseName'), ';User Id=', parameters('administratorLogin'), '#', parameters('sqlserverName'), ';Password=', parameters('administratorLoginPassword'), ';')]",
"type": "SQLServer"
},
"AzureWebJobsStorage": {
"type": "Custom",
"value": "[concat(variables('storageConnectionString'), listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageName')), '2016-01-01').keys[0].value)]"
},
"AzureWebJobsDashboard": {
"type": "Custom",
"value": "[concat(variables('storageConnectionString'), listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageName')), '2016-01-01').keys[0].value)]"
}
}
},
Thanks.
below is example for adding storage account to ADLA
"storageAccounts": [
{
"name": "[parameters('DataLakeAnalyticsStorageAccountname')]",
"properties": {
"accessKey": "[listKeys(variables('storageAccountid'),'2015-05-01-preview').key1]"
}
}
],
in variable you can keep
"variables": {
"apiVersion": "[providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]]",
"storageAccountid": "[concat(resourceGroup().id,'/providers/','Microsoft.Storage/storageAccounts/', parameters('DataLakeAnalyticsStorageAccountname'))]"
},