I am trying to download zip data from a url (https://data.police.uk/data/archive/2019-10.zip) and place into a Storage Account (blob).
The PS need to run within an Azure automation account.
When using the below, I get an exception
There is not enough space on the disk
$url = "https://data.police.uk/data/archive/2019-10.zip"
$output = "\policedata.zip"
$wc = New-Object System.Net.WebClient
$download = $wc.DownloadFile($url, $output)
Is there a way to move the zip file to an Azure storage blob within downloading it locally first, or extracting the CSV's out of it?
Azure Automation is not the right solution here given the file you are downloading is 1.6 GB. The problem is the limits Azure Automation has:
Maximum amount of disk space allowed per sandbox: 1 GB
Maximum amount of memory given to a sandbox: 400 MB
You can neither download the file before uploading nor hold it in memory and then pipe it to the storage account.
You could try to setup a two way streaming where one stream downloads file and another stream uploads the file. But this will be an overkill for what you are trying to do. I would suggest using vanilla container or VM to automate this.
Related
I want to upload a 9.6 GB file with the extension .sql.gz to my SQL BigQuery Sandbox (free) account. I received a message that the file is too big and that I need to upload it from the cloud. When trying to upload it from the cloud, I am asked to create a bucket, and if I want to create a bucket, I get the message: "billing must be enabled". Is there any alternative, specifically for an sql.gz file?
As of now, there is no alternative but to upload .gz files files to a bucket in Cloud Storage and use the bq command-line tool to create a new table.
You may enable billing for your existing project to use Cloud Storage.
I have used blob storage as a file storage account in .NET Core Web application hosted on Azure app service(PaaS).
My requirement is to create .zip files and then attach to email where it requires UNC path for attachment.
Here I have one option to use app service local storage for temporary file creation and use in attachment.
I am searching other option to map blob storage to any virtual drive in cloud and get its UNC path or any other option?
Also, Can you please suggest what are the possible options to map Azure Blob storage drive in network? I know the following one - App service local storage, VM, Local machine network drive.
First of all, you need to know the concept of UNC path, and then azure webapp can be regarded as a virtual machine in essence, and azure blob storage can also be regarded as a machine. Therefore, it is not feasible to send mail directly through azure blob.
Suggestion:
1. I check the information, you can try azure files to store files and use them.
I think this should be the fastest way, without using other azure products.
Download the file to the project directory, you can create a temporary folder, such as: MailTempFolder, you can download the file from the blob to this folder, and then you can get the UNC path to send mail.
After the sending is successful, just delete the file, it will not occupy too much space of the azure webapp, even if the sending fails, you can still get the zip file without downloading it again.
I need to upload files in batch to azure storage securely, and the files will be processed by an ADF pipeline later.
While I tried to upload files from my angular front-end to the API, the time taken to upload a large file took time ad my request was timing out.
Any suggestions as to how I can approach this problem better?
I've got a bit of a problem in uploading a really large file into azure blob storage.
I have no problem uploading that file into the web site as a file
upload in an upload directory.
I have no problem either putting this into the blob storage, as chunking will be handled internally.
The problem I'm having is that the time it takes to move the large file from the upload directory to the blob storage takes longer than the browser timeout and the customer sees an error message.
As far as I know, the solution is to chunk-upload directly from the web browser.
But how do I deal with the block ids? Since the web service is supposed to be stateless, I don't think I can keep around a list of blocks already uploaded.
Also, can the blob storage deal with out-of-order blocks?
And do I have to deal with all the state manually?
Or is there an easier way, maybe just handing the blob service the httprequest input stream from the file upload post request (multipart form data)?
Lots of Greetings!
You could move from the web server to blobs asynchronously. So return success for the original request back once file is on web server, and then have javascript query your web server periodically to confirm file has made it to durable storage in blobs. This javascript doing the polling can then display success to the user once it gets a success response from web server, confirming that the file has made it to blob storage.
I would like to upload a form from a web page and directly save the file to S3 without first saving it to disk. This node.js app will be deployed to Heroku, where there is no local disk to save the file to.
The node-formidable library provides a great way to upload files and save them to disk. I am not sure how to turn off formidable (or connect-form) from saving file first. The Knox library on the other hand provides a way to read a file from the disk and save it on Amazon S3.
1) Is there a way to hook into formidable's events (on Data) to send the stream to Knox's events, so that I can directly save the uploaded file in my Amazon S3 bucket?
2) Are there any libraries or code snippets that can allow me to directly take the uploaded file and save it Amazon S3 using node.js?
There is a similar question here but the answers there do not address NOT saving the file to disk.
It looks like there is no good way to do it. One reason might be that the node-formidable library saves the uploaded file to disk. I could not find any options to do otherwise. The knox library takes the saved file on the disk and using your Amazon S3 credentials uploads it to Amazon.
Since on Heroku I cannot save files locally, I ended up using transloadit service. Though their authentication docs have some learning curve, I found the service useful.
For those who want to use transloadit using node.js, the following code sample may help (transloadit page had only Ruby and PHP examples)
var crypto, signature;
crypto = require('crypto');
signature = crypto.createHmac("sha1", 'auth secret').
update('some string').
digest("hex")
console.log(signature);
this is Andy, creator of AwsSum:
https://github.com/appsattic/node-awssum/
I just released v0.2.0 of this library. It uploads the files that were created by Express' bodyParser() though as you say, this won't work on Heroku:
https://github.com/appsattic/connect-stream-s3
However, I shall be looking at adding the ability to stream from formidable directly to S3 in the next (v0.3.0) version. For the moment though, take a look and see if it can help. :)