Not able to access Bigquery dataset from .net app - google-bigquery

I am new to GCP and have created a Project in GCP which has a service account with a principal associated with it.The principal has role of editor , owner and viewer. I have created a dataset which contains a table which i am not able to access from .net application.
Steps used to connect to read data from table :
1.Created service account to authenticate Api requests
2.Created key,json file which will be used by .net application to connect to the datasets(gcloud iam service-accounts keys create ~/key.json)
3.
using Google.Apis.Auth.OAuth2;
using Google.Cloud.BigQuery.V2;
Console.WriteLine("Hello, World!");
string dir = Directory.GetParent(Directory.GetCurrentDirectory()).Parent.Parent.FullName + "\\"+"currency-342912-7c734b3fad06.json";
GoogleCredential credential = GoogleCredential.FromFile(dir);
var client = BigQueryClient.Create("currency-342912", credential);
var table = client.GetTable("bigquery-public-data", "austin_311", "311_service_requests");
var sql = $"SELECT * FROM {table} LIMIT 10";
var results = client.ExecuteQuery(sql,parameters:null);
getting the below error :
Google.GoogleApiException: 'Google.Apis.Requests.RequestError
Access Denied: Project bigquery-public-data: User does not have bigquery.jobs.create permission in project bigquery-public-data
But user has browser, editor , owner , viewer all the permissions

As per the error in the question above -"Access Denied: Project bigquery-public-data: User does not have bigquery.jobs.create permission in project bigquery-public-data"
it is mentioned to have the role "bigquery.jobs.create" so i applied this role at the project level and the only area left was the service account , i applied this role to the service account and this started working

Related

Upload file to Microsoft OneDrive using graph SDK using asp.net core

I am trying to upload a file to OneDrive using Graph SDK for .Net Core from worker service.
Basically, some files are created at random time and those files needs to be uploaded to specified path on OneDrive from worker service at midnight every day.
I have following information stored in appconfig.json file in application:
ClientID
ClientSecret
TenantID
I have checked samples on various sites but could not find how to upload files using above ID and Secret. I believe there must but some kind of authProvider that I could initialize using above ID and Secret.
I also checked miscrosoft's documentation but coudl not find any example on how to upload file using SDK with ID and Secret.
https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_put_content?view=odsp-graph-online
Additional Point
Upload new file
Overwrite if file already exists(hope that graph already supports this)
file size is < 4MB
path format /folder1/folder2/filename.pdf
Any help would be appreciated.
Remember to assign Application permission (Client credentials flow) to the app registration. See Application permission to Microsoft Graph.
You can use Client credentials provider.
IConfidentialClientApplication confidentialClientApplication = ConfidentialClientApplicationBuilder
.Create(clientId)
.WithTenantId(tenantID)
.WithClientSecret(clientSecret)
.Build();
ClientCredentialProvider authProvider = new ClientCredentialProvider(confidentialClientApplication);
Upload small file (<4M):
GraphServiceClient graphClient = new GraphServiceClient(authProvider);
using var stream = new System.IO.MemoryStream(Encoding.UTF8.GetBytes("The contents of the file goes here."));
await graphClient.Users[{upn or userID}].Drive.Items["{item-id}"].Content
.Request()
.PutAsync<DriveItem>(stream);
If you want to upload large file, see Upload large files with an upload session.
UPDATE:
Use Install-Package Microsoft.Graph.Auth -IncludePrerelease in Tools -> NuGet Package Manager -> Package Manager Console.
It supports both uploading a new file and replacing an existing item.
For how to get the items id, please refer to Get a file or folder.
For example, get the item id of /folder1/folder2 (have a quick test in Microsoft Graph Explorer):
GET https://graph.microsoft.com/v1.0/users/{userID}/drive/root:/folder1:/children
It will list all the children in folder1, including folder2. Then you can find item id of folder2.
UPDATE 2:
Client credentials provider is application identity while all the providers below are user identity.
Since you want to access personal OneDrive (including user identity), you could choose Authorization code provider or Interactive provider.
Authorization code provider: you need to implement interactively sign-in for your web app and use this provider.
Interactive provider: you can easily use this provider to implement interactively sign-in in a console app.
You can have a quick test with the second provider.
Please note that you should add the Delegated (personal Microsoft account) permissions into the app registration. See Delegated permission to Microsoft Graph.
And in this case, you should modify all graphClient.Users[{upn or userID}] to graphClient.Me in your code.
I'm afraid that you have to implement sign-in interactively auth flow to access personal OneDrive.

Trying to connect Polybase to Oracle leads to Login failed. The login is from an untrusted domain and cannot be used with Integrated authentication

I'm using SQL Server 2019 and try to connect to a Oracle DB via Polybase:
CREATE DATABASE SCOPED CREDENTIAL OracleUser WITH IDENTITY = 'username', Secret = 'password';
CREATE EXTERNAL DATA SOURCE [OrDB] WITH (LOCATION = N'oracle://192.168.1.5:1521'),
CREDENTIAL = OracleUser);
I'm getting the error:
Login failed. The login is from an untrusted domain and cannot be used with Integrated authentication.
However the user is not a domain user and oracle does not use integrated authentication.
As I mentioned in my comment above, you get this error if you are logged into your SQL Server instance with integrated security. This is due to a regression bug in SQL Server 2019.
Log in with SQL authentication instead, and it will work.
try this:
CREATE EXTERNAL DATA SOURCE [OrDB] WITH (LOCATION = N'oracle://192.168.1.5:1521'),
CREDENTIAL = OracleUser,
CONNECTION_OPTIONS = 'UseDefaultEncryptionOptions=false'
);

Insufficient Permissions with Access Token to Azure Data Catalog

I'm trying to issue an API call to Azure Data Catalog using a Client Secret, however I get a Permission denied error.
I'm able to create the token and I've given the client application the necessary permissions in AAD.
If I change the authentication method to use delegated access where the user signs in, it works but if i try to use a client secret the generated token does not have the proper permissions.
Here's some code that I've altered using various samples
The token from this code works
//get the token
authResult = await authContext.AcquireTokenAsync(resourceUri, clientId, new Uri(redirectUri), new PlatformParameters(PromptBehavior.Always));
//the token from this code doesn't work
IConfidentialClientApplication app;
app = ConfidentialClientApplicationBuilder.Create(clientId)
.WithClientSecret(clientSecret)
.WithAuthority(new Uri(authorityuri))
.Build();
string[] scopes = new string[] { "https://api.azuredatacatalog.com/.default" };
result = await app.AcquireTokenForClient(scopes);
I get this error The remote server returned an error:
(403) AccessDenied : Access denied..
Following is high level steps to configure the Service Principal configuration to support ADC REST API
Azure Active Directory | App Registration – Select Web app / API for Application Type and URL can be anything Example:  http://portal.azure.com
Select the Application | Required Permission | Add the Microsoft Azure Data Catalog
Navigate to http://www.azuredatacatalog.com | Settings - Add the Service Principal to Catalog User based on the business need you can add to glossary admin / catalog admin. The format is clientid#tenantid
****Clientid = Azure Active Directory | App Registration | Application ID**
**TenantID = Azure Active Directory | Properties | Directory ID****
Follow the Service Principal Authentication sample REST API code to build your solution https://github.com/Azure-Samples/data-catalog-dotnet-service-principal-get-started

Subscription-scope authorization for Azure Resource Manager API user

Trying to put together an Azure ARM integration, where my code uses the Azure API to retrieve the full list of VMs in the entire subscription.
Went successfully through all the (many!) steps here to get an API user ready with all relevant IDs needed for authentication.
Using the same set of instructions, we were also able to grant this user the Reader role, for listing VMs in specific groups (we did this through the Azure UI). However, we have been unsuccessful in implementing the instructions here for setting up Reader role for this user to the entire subscription (through the CLI).
Running this:
azure role assignment create --objectId app-oid --roleName Reader --scope /subscriptions/subscription-id
Or this:
azure role assignment create --objectId app-oid --roleName Reader --subscription subscription-id --scope /subscriptions/subscription-id
Yields this:
Principals of type Application cannot validly be used in role assignments.
So currently we have no way of programmatically browsing the full set of VMs without adding a specific authorization for each Resource Group.
Does anybody know of a way that actually works to assign this permission at the subscription level?
Found the answer in the comment area of the link in the question body:
In the new portal, edit the subscription and add the role, just like you would do with a resource group. Still curious as to why the CLI doesn't support this.
The steps to use the Azure CLI to create and authorize a service principal are documents here: https://azure.microsoft.com/en-us/documentation/articles/resource-group-authenticate-service-principal-cli/
Instead of using --objectId app-oid try using -ServicePrincipalName <appId>
https://github.com/Azure/azure-powershell/issues/4776
Had the same error for role assignment with terraform
Principals of type Application cannot validly be used in role assignments.
But as described here, using the Object Id that is displayed on the overview page of the app registration in the Azure portal resulting the above error.
The solution is to get the Object Id from azure cli:
az ad sp show --id [Application (client) Id] -o json | grep objectId
# and then using this objectId assign your role
az role assignment create --role contributor —-assignee-object-id [object id] —-resource-group [MyResourceGroup]

AWS Cognito finding right Parameters for CognitoSyncDemo

I'm going to setup this Example for Amazon Cognito Sync demo, using Eclipse + ADK + AWS libs V2.9.
I created an identity pool in the Cognito Console, so I got the Identity Pool ID which is used in CognitoSyncClientManager.java at:
private static final String IDENTITY_POOL_ID = "us-east-1:1a23b013-1abc-123-b123-123abc1fe5";
private static final Regions REGION = Regions.US_EAST_1;**
(perhaps not all regions seem to run in test mode)
I registered my (developing-)app in the Seller Central with Eclipse Help>Android>Bild>MD5 fingerprint and the package-name from AndroidManifest.xml, so got my api_key.txt Value.
From the AWS doku I got my Provider URL (code as sample), which is used in DeveloperAuthenticationProvider.java at:
private static final String developerProvider = "cognito-sync.us-east-1.amazonaws.com";
private static final String cognitoSampleDeveloperAuthenticationAppEndpoint = "arn:aws:iam::123456789123:role/Cognito_TestMyAppAuth_DefaultRole"
private static final String cognitoSampleDeveloperAuthenticationAppName = "Cognito sync demo";
After creating some Users which are attached to groups with permissions to IAM full access and (auto)generated roles for unautherized and autherized Roles, my app is running in this way:
Login with Amazon - is running, but where will their datasets be created and is it possible I see them in the AWS Console?
Simple Browse your data (without login) - is running, unauthenticated users and their datasets appear/are counted in the Identity console
Developer Authentication fails with Username or password do not match - combined with an "Unable to reach resource..." in the LogCat.
Now I think, the last missing thing in this game is the parameter cognitoSampleDeveloperAuthenticationAppEndpoint=...
Where can I find it and what else could I have missed to setup for a full running Example?
Thanks for using Cognito demo. Answers to your questions
Yes, you can see the datasets in the AWS console. Navigate to Cognito inside the AWS Console and go to your identity pool. On the left hand side menu go to identity browser and you will see the list of identities or you can search for an identity. Click on the identity and you can see the datasets owned by that identity.
Developer Authentication is a feature which helps to integrate your authentication system with Cognito. For this you should have a backend server serving user's authentication requests, and once you authenticate users you can request OpenId Connect tokens for them from Cognito using the GetOpenIdTokenForDeveloperIdentity API call. The developer provider name is the one which you setup for your identity pool in the AWS Cognito console. For using the sample you can setup a sample server application by following this blog and the readme file of the the server side application.
P.S.: Please avoid sharing your identity pool id.
Thanks,
Rachit