A question about Google Storage:
Is it possible to give r/o access to a (not world-accessible) storage bucket to a user from another Google project?
If yes, how?
I want it to backup data to another Google project, for the case if somebody may incidentally delete all storage buckets from our project.
Yes. Access to Google Cloud Storage buckets and objects are controlled by ACLs that allow you to specify individual users, service accounts, groups, or project role.
You can add users to any existing object through the UI, the gsutil command-line utility, or via any of the APIs.
If you want to grant one specific user the ability to write objects into project X, you need only specify the user's email:
$> gsutil acl ch -u bob.smith#gmail.com:W gs://bucket-in-project-x
If you want to say that every member of the project my-project is permitted to write into some bucket in a different project, you can do that as well:
$> gsutil acl ch -p members-my-project:W gs://bucket-in-project-x
The "-u" means user, "-p" means 'project'. User names are just email addresses. Project names are the strings "owners-", "viewers-", or "editors-" and then the project's ID. The ":W" bit at the end means "WRITE" permission. You could also use O or R or OWNER or READ or WRITE instead.
You can find out more by reading the help page: $> gsutil help acl ch
Related
I want to use OS Login with GCP because we use IAM for scoping access to all other resources within GCP (storage buckets, SQL, Redis, etc.). I understand how to restrict users from accessing machines using service accounts and roles.
But, I don't understand how to restrict the possible usernames that someone can use to SSH into our Compute Engine machines. Assume we have a VM configured with OS Login. The problem is that everyone connects using a CLI string like
gcloud compute ssh $MACHINE_NAME which (possibly creates and then) logs in to a /home/$USER_DOMAIN_SUFFIX directory. So, the team's shell history, relevant home directory contents (downloaded files, created scripts, etc.), and running processes are all in a different scope (UID). We could soft-enforce that everyone does something like gcloud compute ssh $SPECIAL_USERNAME#$MACHINE_NAME where everyone uses the same $SPECIAL_USERNAME value. But, that doesn't prevent new home directories from being provisioned. It's a convention, not a software policy.
Is there a way to accomplish what I want, where I can freely choose the value of $SPECIAL_USERNAME? I don't want to be locked in to the generated usernames based on the user/service account email.
Using root for everything is unacceptable for a number of reasons (we want to use a non-root container runtime and we want to limit potential damage done by this $SPECIAL_USERNAME).
I want to publish a tutorial where a data from sample tsv file S3 is used by Redshift. Ideally I want it to be simple copy paste operation required to follow the exercises step by step, similar to what's in Load Sample Data from Amazon S3. The problem is with the first data import task using COPY command as it only supports S3, or EMR based load.
This seems like a simple requirement but no hassle-free way to really do it with Redshift COPY (I can make the file available for browser download without any problem but COPY requires CREDENTIALS parameter…)
Variety of options for Redshift COPY Authorization parameters is quite rich:
Should I ask user to Create an IAM Role for Amazon Redshift
himself?
Should I create it myself and publish the IAM role ARN? Sounds most hassle
free (copy paste) but security wise doesn't sound well…? Do I need to restrict S3 permissions to limit the access to only that particular file for that role?
Should I try temporary access instead?
You are correct:
Data can be imported into Amazon Redshift from Amazon S3 via the COPY command
The COPY command requires permission to access the data stored in Amazon S3. This can be granted either via:
Credentials (Access Key + Secret Key) associated with an IAM User, or
An IAM Role
You cannot create a Role for people and let them use it, because their Amazon Redshift cluster will be running in a different AWS Account than your IAM Role. You could possibly grant trust access so that other accounts can use the role, but this is not necessarily a wise thing to do.
As for credentials, they could either use their own or ones the you supply. They can access their own Access Key + Secret Key in the IAM console.
If you wish to supply credentials for them to use, you could create an IAM User that has permission only to access the Amazon S3 files they need. It is normally unwise to publish your AWS credentials because they might expose a security hole, so you should think carefully before doing this.
At the end of the day, it's probably best to show them the correct process so they understand how to obtain their own credentials. Security is very important in the cloud, so you would also be teaching them good security practice, in additional to Amazon Redshift itself.
I have a Compute VM that has storage permission of read-write. When I try to run the following command:
gsutil rsync -R local-dir gs://folder1/prod/www.domain.tld
I get insufficient permission error:
Building synchronization state...
Skipping cloud sub-directory placeholder object (gs://folder1/prod/www.domain.tld/) because such objects aren't needed in (and would interfere with) directories in the local file system
Starting synchronization
Copying file://local-dir/.gitignore [Content-Type=application/octet-stream]...
Uploading gs://folder1/prod/www.domain.tld/.gitignore: 174 B/174 B
AccessDeniedException: 403 Insufficient Permission
I am not sure what account needs to have specific permissions or how I would even assign the permissions. Can someone provide some direction as to what I need to look into? The only permissions I can think of is a service account but I am not sure how to create one or if that is what I even need to do. Also, once I grant an account access/permission, how would I use the account to authenticate?
Use gcloud auth list command to lookup the active account. The service account of Comoute Engine is similar to the following:
123845678986-compute#developer.gserviceaccount.com (active)
By default, this service account is a member of your project with Edit permission. Check ACLs of your GCS bucket and its folders and make sure the GCE service account or the group which it is belong to, has ownership or editing rights to them.
On this page:
https://developers.google.com/storage/docs/gsutil_install?hl=ja#install
The gsutil install recommends, right after install, running
gsutil update
which returns
CommandException: Invalid command "update".
Am I just seeing incorrect documentation? Is there some other way to update?
Checking on 'usage' doesn't mention there being any update command:
Usage: gsutil [-d][-D] [-h header]... [-m] [command [opts...] args...] [-q]
Available commands:
acl Get, set, or change bucket and/or object ACLs
cat Concatenate object content to stdout
compose Concatenate a sequence of objects into a new composite object.
config Obtain credentials and create configuration file
cors Set a CORS XML document for one or more buckets
cp Copy files and objects
defacl Get, set, or change default ACL on buckets
du Display object size usage
help Get help about commands and topics
lifecycle Get or set lifecycle configuration for a bucket
logging Configure or retrieve logging on buckets
ls List providers, buckets, or objects
mb Make buckets
mv Move/rename objects and/or subdirectories
notification Configure object change notification
perfdiag Run performance diagnostic
rb Remove buckets
rm Remove objects
setmeta Set metadata on already uploaded objects
stat Display object status
test Run gsutil tests
version Print version info about gsutil
versioning Enable or suspend versioning for one or more buckets
web Set a main page and/or error page for one or more buckets
Additional help topics:
acls Working With Access Control Lists
anon Accessing Public Data Without Credentials
crc32c CRC32C and Installing crcmod
creds Credential Types Supporting Various Use Cases
dev Contributing Code to gsutil
metadata Working With Object Metadata
naming Object and Bucket Naming
options Top-Level Command-Line Options
prod Scripting Production Transfers
projects Working With Projects
subdirs How Subdirectories Work
support Google Cloud Storage Support
versions Object Versioning and Concurrency Control
wildcards Wildcard Names
Use gsutil help for detailed help.
EDIT:
It is gsutil version 3.42
Maybe you have a very old version?
Try gsutil version to see yours.
you can check release notes here:
https://github.com/GoogleCloudPlatform/gsutil/blob/master/CHANGES.md
I'm storing all my content in AWS S3 and I would like to know which is the best approach to retrieve my images:
should I use the account access keys or should I create a user with the correct policies and then use the access keys for that "user"?
Always always always create users with their own IAM policies. You should never use the root account credentials to do anything if you can help it.
It's like permanently running commands on your local machine as the root user. The account-level access and secret access keys are the absolute keys to the kingdom. With them, a hacker, malicious employee, or well-intentioned-but-prone-to-accidents administrator could completely destroy every AWS resource you have, download anything off them, and in general cause chaos and discord. Even machines with pem files aren't safe. A root-level user could just cut an AMI off an existing machine.
Take a look at the IAM policy generator. Writing JSON policies is not fun and error prone, but tools like that one will help you get most of the way there.