Download NCDC weather dataset from s3 to local machine using S3cmd - amazon-s3

I would like to download the publicly available NCDC weather dataset on amazon public datasets. But am unable to find it. Could anyone tell me the exact bucket it is located in? Also could you please tell me how to download it onto my local machine with s3cmd?
Any help would be much appreciated.
http://aws.amazon.com/datasets/2759 is the link to the specific dataset I am looking for.

The dataset you have referenced is provided as an EBS Snapshot, not an S3 bucket. You will need to convert the snapshot into a new EBS Volume, which can be mounted on an Amazon EC2 instance.
To access it:
Access the Amazon EC2 Management Console
Set the region to US East
In the search bar, choose "Public Snapshot"
Enter snapshot ID in the search filter (from the page you referenced) and hit Enter. You should now see a single snapshot.
Optional: "Copy" the snapshot to another region if you desire
"Create Volume" to create an EBS Volume from the snapshot (be sure to select the same Availability Zone as your EC2 instance)
Go to "Volumes" in the left navigation pane
"Attach" the new EBS Volume onto an existing EC2 instance
You will then have access to the data.
Please note that the dataset is from 2009. It does not contain recent data.

Related

gcloud snapshot all disks attached to an instance

I'm using the gcloud cli api and I'm trying to write a script that creates a snapshot of all disks attached to an instance.
My problem is that the snapshot cli command requires the name of the disk however when I ask gcloud for a list of disks attached to an instance it doesn't (directly) give me the disk name.
If I run:
gcloud compute instances list --format=json
I get a json array of disks which includes the deviceName but not the name.
I do note that the array includes a 'source' attribute which contains a url which does end with the diskname.
However I'm concerned that parsing the diskname from the url won't provide a stable solution.
So what is the correct way to obtain the disk names?
Did you tired:
gcloud compute disks list --format=json ?
I don't know if it can be listed by instance, but it contains information by which instance particular disc is used (reference)
I hope it will help!

How to backup on premise data to AWS S3 bucket using tool/service?

Let me explain a little bit, we are keeping users data to a centralized Share folder(Configured on Domain Controller, permission set via NTFS & Security group), like M-Marketing, S-Sales & T-Trading, etc(These drives are mapped to windows login profile according to their work profile).
On-premise back is already configured. Now I want to back up some of the important drives (Like M, S, T) to AWS S3 to keep data safe & whenever is Source data is not available for whatever reason, I must be able to map those drives according to their work profile.

Can developer have access to limited S3 console

A developer of mine wants to be able to see the entire contents of the S3 bucket that I've given him to develop with. It seems the only way to do this is to give a limited version of the AWS console to see as objects enter the bucket.
Is this even possible? Is there any other way to allow him to see as objects populate the bucket?
You can use IAM roles to control access to resources at a granular level, even down to individual object contained in an S3 bucket.
You can read more about IAM here https://aws.amazon.com/iam/

Rename Google Compute Engine VM Instance

How do I rename a Google Compute Engine VM instance?
I created a new LAMP server and I'd like to rename it in the "VM Instances" dashboard.
I've tried renaming the Custom metadata, but that didn't seem to replicate to the dashboard.
I tried the solution provided by #Marius I . It works, but I lost my description, my metas, the tags and the permissions I've set on the old instance. I had to copy my metas, had to make sure the zone for the new instance was the same as the original, and had to check that the pricing was the same.
I think, it's best to just create a clone of your original instance, this way don't have to manually copy/set them on the new instance.
As #Marius said, create a snapshot of your disk ( DO NOT skip this part: you may lose all your files/configuration )
Make sure you completed the step 1.
Clone your instance (“Create similar” button)
Name your cloned instance the way you want.
Make sure to select the snapshop of your disk created at #1 ( make sure you select the same typeof disk as well: if your original disk was SSD for example, you have to select if you want the new disk to be SSD too )
Make sure your IPs are set correctly
You're done :)
Another way to do this is:
snapshot the disk of the existing instance
create a new disk from that snapshot
create a new instance with that disk and give it the name you would like
It sounds time-consuming, but in reality should take 5 minutes.
you can't ...! Once VM is created, you can’t change the Instance Name
There's now a "native" way to do this. The feature is currently in Beta and only available with gcloud and via the API. With gcloud you can run:
$ gcloud beta compute instances set-name CURRENT_NAME -—zone=ZONE -—new-name=NEW_NAME
Some caveats:
You'll need to shut down the VM first
The Developer Console UI won't be aware of the rename until you do a browser refresh
See the official documentation for more details.
Apart from the hacks above, it's not possible.
Yet, it has been requested on uservoice and has received 593 votes. (as of 2018) Currently, it's the topmost "planned" item.
I got lost in the instructions, so I thought I include screenshots because the navigation is confusing. I hope this helps you.
Stop your instance
Click on the stopped instance name
In VM Instance Details, scroll down and click on the disk
Click on Create snapshot
give it a name like snapshot-1 (or your new instance name)
click on Create button
click on newly created snapshot
Click on Create Instance
Give your instance the new name and configure the rest of the VM.
When dealing with a robust system, it's necessary to have a way to bring up a system quickly when it goes down. This could be via custom scripts, salt, ansible, etc.
So, if you want to change your instance name, delete the instance, create a new one with the correct name, and run your script again :)
To answer your question directly. You cannot edit the VM Instance name.
However, you can create New VM instance using your old disk. To meet the VM instance name that you want.
Kindly see below procedure:
Go to Compute Engine Page
Go to Disk Page
Select the disk of your VM instance that you want to create a snapshot
Click the three dot image same line of your disk
Select +Create Snapshot (You will be go to Create Snapshot page). Kindly name your snapshot (backup)
Just Click Create.
Then once you have created a snapshot for your VM instance disk, you may now proceed on creating your new instance from snapshot pointing to other region which you can consider such: us-central1, us-west1 and us-west2. Please see below procedure:
Go to Snapshot Page
Select snapshot "backup" (You should be on Snapshot details Page)
Click Create Instance (Choose best name for your new VM Instance)
Please select the region best fit for you (us-central1, us-west1 and us-west2) except us-east1.
Lastly, Click Create
Machine images are now in pre-GA!
This is currently the easiest way to clone an instance without losing your instance configurations, check this comparison table.
Detailed steps:
Go to Compute Engine > Virtual Machines > Machine Images
Click on create Machine Image
Select your current instance under Source VM instance and click create
Once the image becomes ready go to Machine image details and click on create instance
The form will be populated by your existing instance configuration and you'll be able to change them before creating the instance!
Sorry to resurrect this thread after so long, but when I searched for an answer I kept ending up in this article... :-)
The Cloud SDK now allows renaming an instance directly, provided it's stopped:
The command looks like this:
gcloud beta compute instances set-name INSTANCE_NAME --new-name=NEW_NAME [--zone=ZONE] [GCLOUD_WIDE_FLAG …]
This is not available yet in the UI.
The following worked for me:
gcloud beta compute instances set-name currentname --new-name=newname
Googler checking in. We're rolling out this feature (renaming a VM) to all users in the cloud console. Public documentation.
In the Google Cloud console:
Go to the VM instances page.
Go to VM instances
In the Name column, click the name of the VM.
Click Stop stop.
Click Edit edit.
In Basic information > Rename > VM instance name, enter a new name
for the VM.
Click Save.
Click Start / Resume play_arrow.
Using the gcloud Command Line Interface:
gcloud compute instances stop INSTANCE_NAME
gcloud beta compute instances set-name INSTANCE_NAME --new-name=NEW_INSTANCE_NAME
I also wanted to let you all know that we monitor these forums and use your feedback to influence our roadmap. Thank you for your engagement!
I am trying to do this 03/2019 and I saw a new option on panel
click Instance link
on top menu you will see "Create Similar"
could work if you need same machine without data. (solved my case)
if you need a full copy then you should create a snapshot and clone it.
This is now possible via the web console:

How to upload multiple files to google cloud storage bucket as a transaction

Use Case:
Upload multiple files into a cloud storage bucket, and then use that data as a source to a bigquery import. Use the name of the bucket as the metadata to drive which sharded table the data should go into.
Question:
In order to prevent partial import to the bigquery table, ideally, I would like to do the following,
Upload the files into a staging bucket
Verify all files have been uploaded correctly
Rename the staging bucket to its final name (for example, gs://20130112)
Trigger the bigquery import to load the bucket into a sharded table
Since gsutil does not seem to support bucket rename, what are the alternative ways to accomplish this?
Google Cloud Storage does not support renaming buckets, or more generally an atomic way to operate on more than one object at a time.
If your main concern is that all objects were uploaded correctly (as opposed to needing to ensure the bucket content is only visible once all objects are uploaded), gsutil cp supports that -- if any object fails to upload, it will report the number that failed to upload and exit with a non-zero status.
So, a possible implementation would be a script that runs gsutil cp to upload all your files, and then checks the gsutil exit status before creating the BigQuery table load job.
Mike Schwartz, Google Cloud Storage team
Object names are actually flat in Google Cloud Storage; from the service's perspective, '/' is just another character in the name. The folder abstraction is provided by clients, like gsutil and various GUI tools. Renaming a folder requires clients to request a sequence of copy and delete operations on each object in the folder. There is no atomic way to rename a folder.
Mike Schwartz, Google Cloud Storage team