I have a locally running pyspark cluster and want to load data from big query.
I have generated the server-to-server json credentials file
Now how do I pass that to my code.
TIA
When I write code to talk to Google's APIs from a tool, I generate Service Account Credentials within the project I want to use, download them, and then initialize the client with those credentials.
From python, I would use oauth2client.client.SignedJwtAssertionCredentials to create the credentials, and then pass those credentials using the credentials keyword parameter to the apiclient constructor. Something like:
key_bytes = ... # read your downloaded key from somewhere
bigquery_service = discovery.build(
'bigquery', 'v2',
credentials=oauth2client.client.SignedJwtAssertionCredentials(
'1234567890#developer.gserviceaccount.com',
key_bytes,
['https://www.googleapis.com/auth/bigquery'],
private_key_password='notasecret',
user_agent=_CLIENT_USER_AGENT))
Then whenever you use bigquery-service, your credentials will be passed along.
Related
I need to create a way for anyone who visits my website to upload an image to an Azure Blob Container. The website will have input validations on the file.
I've considered using an Azure Function to write the validated file to the Blob Container, but I can't seem to find a way to do this without exposing the Function URL to the world (similar to this question.
I would use a System-Assigned Managed Identity (SAMI) to authenticate the Function to the Storage account, but because of this, anyone could take the Function URL and bypass the validations and upload.
How is this done in the real world?
If I understand correctly, the user uploads a file via an HTTP POST call to your server, which validates it. You would like to use an Azure Function to then upload the validated file to the Blob Storage.
In this case, you can restrict the access to the Azure Function; so that it can only be called from your server's IP. This way the users cannot reach that Function. This can be done via the networking settings, and is available on all Azure Function plans.
You could also consider implementing the validation logic within the Azure Function.
Finally (perhaps I should have started with this), if you are only considering writing an Azure Function to upload data to a Storage Account, you should perhaps first consider using the Blob Service REST API, specifically the PUT Blob endpoint. There are also official Storage Account SDKs for different languages/ecosystems that you could use to do this.
• Since, you are using an Azure function default generic URL on your website for uploading blobs with no authentication, I would suggest you to please create an ‘A’ host record for your function app. Considering that you have a website, you may be having a custom domain for your website to be unique and as you might be having a custom domain, the custom domain’s DNS records must be hosted on a public DNS server. Thus, similarly, on the same public DNS server, you will have to create an ‘A’ host record for the function app and assign it the same public IP address that is shown and assigned in Azure. This will ensure that your public DNS server has an active DNS resolver for the function app globally and then ensure to create a ‘CNAME’ record for your default generic Azure function app URL with the same URL as the alias in the DNS records and the ‘A’ host record as the assigned value in it.
In this way, whenever, any anonymous person visits your website and tries to upload an image, he will be shown the function app URL as ‘abc.xyz.com’ and not the generic Azure function app URL thus successfully ensuring that your objective is achieved.
• Once the above said has been done, then publish the new ‘CNAME’ record created in the public DNS server as your function app URL. This will not expose the generic Azure function app URL and mask it as well as ensure that it is secured since you will be uploading an SSL/TLS certificate for the website to be HTTPS protected in the function app workspace itself as shown below in the snapshot: -
For more information, kindly refer the below documentation link: -
https://learn.microsoft.com/en-us/azure/dns/dns-custom-domain
I use Nextcloud as a normal user to store and share files.
I decided to use it as a backend for a web application I am developing so that I can store the files in Nextcloud while the frontend is done by me.
I spent some hours on the API docs
https://docs.nextcloud.com/server/latest/developer_manual/client_apis/WebDAV/index.html
and, with some disappointment, unless I have not made a mistake, I realized that the only API that can be used from outside Nextcloud is the WebDav API.
This is a minimalistic API that allows doing basic things such as uploading a file by passing the full path like with this GET statement (authenticated by basic auth passing username and password in the headers:
GET https://nextcloud.example.com/remote.php/dav/files/username/FolderOne/SubFolderTwo/HelloWorld.txt
This will download the file located in /FolderOne/SubFolderTwo/HelloWorld.txt
with a PUT request, it is possible to overwrite the file by passing the file content in the raw body request
This is very effective but minimalistic.
I was expecting to have a full REST API to access more properties and perform complex operations.
Could you please tell me if I missed some important information?
There is the OCS API but it works only from inside Nextcloud.
Thanks.
A full REST API is avaiable - https://docs.nextcloud.com/server/22/developer_manual/client_apis/OCS/ocs-api-overview.html
Create a Share - https://docs.nextcloud.com/server/latest/developer_manual/client_apis/OCS/ocs-share-api.html
The OwnCloud documentation also offers more examples
https://doc.owncloud.com/server/10.8/developer_manual/core/apis/ocs-share-api.html
You can register an App id and use that to login or passthru a username and password in the authentication header.
I am trying to call a google cloud function on the In Contact Studio to fulfill a logic in my IVR Call, but I am unable to do that as I have closed public access to my cloud function, and now I am not getting a way how to authenticate the call.
I tried using the command gcloud auth print-identity-token to get a ID_TOKEN But this ID_TOKEN will be refreshed every time and I can't use this again and again, so is there any way that I can generate a ID_TOKEN every time I try to call this function using a simple API Call??
OR
Is there any other way to solve my problem?
Ways I have Tried :-
I have gone through this Documentation:- https://cloud.google.com/functions/docs/securing/authenticating#end-users
and I was using the access style of End-User But it is a way in which the access token was getting generated via login using browser, I want to do everything via code, cause it will be used as a backend code for IVR(call facility for assistance in various tasks), in this method also we get a access token and not a ID_TOKEN, whereas to invoke a function we need a ID_TOKEN and not a access token.
Secondly I tried the gcloud auth print-identity-token command on the google cloud shell where i was logged in with my google account so it generated the JWT token and I used it as a bearer token and the function worked, but how can I generate the token outside GCP or get the on frequent intervals via code.
I want a program way(NodeJS) of doing this and not a UI way, cause I need to attach this with backend of my program, and all the ways I have gone through on the internet have the only way is through UI, and none of them have a program way for outside GCP environment, so i need help on this scenario.
As John said, you can use the Google Doc code example to perform an URL call: The library generate a secure HTTP client to generate request to your endpoint.
If you want to generate the token and use it by yourselve, you can use this piece of code
const {GoogleAuth} = require('google-auth-library');
const auth = new GoogleAuth()
auth.getIdTokenClient("").then(client => {
client.idTokenProvider.fetchIdToken("audience").then(token => {
console.log(token)
})
})
I am attempting to create a dynamic responsive server which is able to:
read a request
store the token in an environment variable
use that environment variable in the response back to the source
I am unsure where to begin with this, I have had a look at some API tools such as PostMan and Insomnia but although I can set up a mock server which would receive the responses, the data I want to save is not being stored in the variable and thus is not being used in the response.
Some guidance would be appreciated.
Thank you
What language and platform you are using?
If you are using any of the major public cloud options, (AWS, Google, Azure), there are easy to deploy API Gateway Services that required minimal code to get started and supports environment variables.
If you are using a local development only,
then you can get started with ASP .NET Core Web API: https://learn.microsoft.com/en-us/aspnet/core/tutorials/first-web-api?view=aspnetcore-3.1&tabs=visual-studio
Then use environment variable:
https://learn.microsoft.com/en-us/aspnet/core/fundamentals/environments?view=aspnetcore-3.1
Or instead once you setup your API running locally, you can use Postman variables:
https://learning.postman.com/docs/sending-requests/variables/
For my application I want the user to be able to store files on Google Drive and for my service to have access to these same files created with the application.
I created a Client ID for web application and was able to upload/list/download files from JavaScript (client side) with drive.appfolder scope. This is good, this is half of what I want to do.
Now I want to access the same files from Node.js (server side). I am lost as to how to do this. Do I create a new Client ID for the server? (if so, how will the user authenticate?) Do I pass the AuthToken my user got client-side and try to use that on the server? I don't think this will work as the AuthToke is time-sensitive (and probably not intended to be used from multiple IPs).
Any direction or example server-side code will be helpful. Again, all I want is to access these same files the user created with my application, not any other files in the user's Google Drive.
CLARIFICATION: I think my question boils down to: "Is it possible to access the same Application Data on Google Drive both client-side and server-side?"
Do I create a new Client ID for the server?
Up to you. You don't need to, but you can. See below.
if so, how will the user authenticate?
Up to you. OAuth is about authorisation, not authentication.
Just in case you meant authorisation, the user authorises the Project, which may contain multiple client IDs.
Do I pass the AuthToken my user got client-side and try to use that on the server?
You can do, but not a good idea for the reason you state. The preferred approach is to have a separate server Client ID, and use that to request offline access, which returns (eventually) a Refresh Token, which you store in your server. You then use that Refresh Token to request Access Tokens whenever you need them.
AuthToken is ... (and probably not intended to be used from multiple IPs).
It is not bound to a specific IP address
Is it possible to access the same Application Data on Google Drive both client-side and server-side?"
Yes
Most of what you need is at https://developers.google.com/accounts/docs/OAuth2WebServer