I am trying to learn spark SQL in databricks and want to work with the Yelp dataset; however, the file is too large to upload to DBFS from UI. Thanks, Filip
There are several approaches to that:
Use Databricks CLI's dbfs command to upload local data to DBFS.
Download dataset directly from notebook, for example by using %sh wget URL, and unpacking the archive to DBFS (either by using /dbfs/path/... as destination, or using dbutils.fs.cp command to copy files from driver node to DBFS)
Upload files to AWS S3, Azure Data Lake Storage, Google Storage or something like, and accessing data there.
Upload the file you want to load in Databricks to google drive
from urllib.request import urlopen
from shutil import copyfileobj
my_url = 'paste your url here'
my_filename = 'give your filename'
file_path = '/FileStore/tables' # location at which you want to move the downloaded file
# Downloading the file from google drive to Databrick
with urlopen(my_url) as in_stream, open(my_filename, 'wb') as out_file:
copyfileobj(in_stream, out_file)
# check where the file has download
# in my case it is
display(dbutils.fs.ls('file:/databricks/driver'))
# moving the file to desired location
# dbutils.fs.mv(downloaded_location, desired_location)
dbutils.fs.mv("file:/databricks/driver/my_file", file_path)
I hope this helps
Setting: lots of mp3 records of customer support conversations somewhere in a db. Each mp3 record has 2 channels, one is customer rep, another is customer's voice.
I need to extract embedding(tensor) of a customer's voice. It's a 3 step process:
get the channel, cut 10 secs, convert to embedding. I have all 3 functions for each step.
embedding is a vector tensor:
"tensor([[0.6540e+00, 0.8760e+00, 0.898e+00,
0.8789e+00, 0.1000e+00, 5.3733e+00]])
Tested with postman. Get embedding function:
I want to build a rest api that connects on 1 endpoint to the db of mp3 files and outputs embedding to another db.
I need to clarify important feature about docker.
When i run "python server.py" flask makes it available on my local pc - 127.0.1.01/9090:
def get_embedding(file):
#some code
#app.route('/health')
def check():
return jsonify({'response':'OK!'})
#app.route('/get_embedding')
def show_embedding():
return get_embedding(file1)
if __name__ == '__main__':
app.run(debug=True, port=9090)
when i do it with docker - where goes the server and files? where does it become available online, can docker upload all the files to default docker cloud?
You need to write a Dockerfile to build your Docker image and after that, Run a container from that image exposing on the port and then you can access it machineIP:PORT
Below is the example
Dockerfile
#FROM tells Docker which image you base your image on (in the example, Python 3).
FROM python:3
#WORKDIR tells which directory container has to word
WORKDIR /usr/app
# COPY files from your host to the image working directory
COPY my_script.py .
#RUN tells Docker which additional commands to execute.
RUN pip install pystrich
CMD [ "python", "./my_script.py" ]
Ref:- https://docs.docker.com/engine/reference/builder/
And then build the image,
docker build -t server .
Ref:- https://docs.docker.com/engine/reference/commandline/build/
Once, Image is built start a container and expose the port through which you can access your application.
E.g.
docker run -p 9090:9090 server
-p Publish a container's port(s) to the host
And access your application on localhost:9090 or 127.0.0.1:9090 or machineIP:ExposePort
In .ebextensions, I have a file (environmentvariables.config) that looks like this:
commands:
01_get_env_vars:
command: aws s3 cp s3://elasticbeanstalk-us-east-1-466049672149/ENVVAR.sh /home/ec2-user
02_export_vars:
command: source /home/ec2-user/ENVVAR.sh
The shell script is a series of simple export key=value commands.
The file is correctly placed on the server, but it seems like it isn't being called with source. If I manually log into the app and use source /home/ec2-user/ENVVAR.sh, it sets up all my environment variables, so I know the script works.
Is it possible to set up environment variables this way? I'd like to store my configuration in S3 and automate the setup so I don't need to commit any variables to source control (option_settings) or manually enter them into the console.
Answer:
Active load S3 variables in Rails app to bypass environment variable issue altogether.
Put a json in S3, download to server, read ENV VAR from there in environment.rb
I am using this
https://github.com/nuxusr/Node.js---Amazon-S3
for uploading files to s3 :
in test-s3-upload.js i had commented mostly tests because they was giving some error , as my goal is to upload the file to s3 so i keep only testUploadFileToBucket() test and while running node test.js gives ok.
but when i check in s3 fox the uploaded file is not being shown.
why file is not uploaded?
Use knox instead. https://github.com/learnboost/knox
Have a look at this project and especially the bin/amazon-s3-upload.js file so you can see how we're doing it using AwsSum:
https://github.com/appsattic/node-awssum-scripts/
https://github.com/appsattic/node-awssum/
It takes a bucket name and a filename and will stream the file up to S3 for you:
$ ./amazon-s3-upload.js -b your-bucket -f the-file.txt
Hope that helps. :)
When I try to upload a folder with subfolders to S3 through the AWS console, only the files are uploaded not the subfolders.
You also can't select a folder. It always requires opening the folder first before you can select anything.
Is this even possible?
I suggest you to use AWS CLI. As it is very easy using command line and awscli
aws s3 cp SOURCE_DIR s3://DEST_BUCKET/ --recursive
or you can use sync by
aws s3 sync SOURCE_DIR s3://DEST_BUCKET/
Remember that you have to install aws cli and configure it by using your Access Key ID and Secrect Access Key ID
pip install --upgrade --user awscli
aws configure
You don't need Enhanced Uploader (which I believe does not exist anymore) or any third-party software (that always has a risk that someone will steal your private data or access keys from the S3 bucket or even from all AWS resources).
Since the new AWS S3 Web Upload manager supports drag'n'drop for files and folders, just login to https://console.aws.amazon.com/s3/home and start the uploading process as usual, then just drag the folder from your desktop directly to the S3 page.
The Amazon S3 Console now supports uploading entire folder hierarchies. Enable the Ehanced Uploader in the Upload dialog and then add one or more folders to the upload queue.
http://console.aws.amazon.com/s3
Normally I use the Enhanced Uploader available via the AWS management console. However, since that requires Java it can cause problems. I found s3cmd to be a great command-line replacement. Here's how I used it:
s3cmd --configure # enter access keys, enable HTTPS, etc.
s3cmd sync <path-to-folder> s3://<path-to-s3-bucket>/
Execute something similar to the following command:
aws s3 cp local_folder_name s3://s3_bucket_name/local_folder_name/ --recursive
I was having problem with finding the enhanced uploader tool for uploading folder and subfolders inside it in S3. But rather than finding a tool I could upload the folders along with the subfolders inside it by simply dragging and dropping it in the S3 bucket.
Note: This drag and drop feature doesn't work in Safari. I've tested it in Chrome and it works just fine.
After you drag and drop the files and folders, this screen opens up finally to upload the content.
Solution 1:
var AWS = require('aws-sdk');
var path = require("path");
var fs = require('fs');
const uploadDir = function(s3Path, bucketName) {
let s3 = new AWS.S3({
accessKeyId: process.env.S3_ACCESS_KEY,
secretAccessKey: process.env.S3_SECRET_KEY
});
function walkSync(currentDirPath, callback) {
fs.readdirSync(currentDirPath).forEach(function (name) {
var filePath = path.join(currentDirPath, name);
var stat = fs.statSync(filePath);
if (stat.isFile()) {
callback(filePath, stat);
} else if (stat.isDirectory()) {
walkSync(filePath, callback);
}
});
}
walkSync(s3Path, function(filePath, stat) {
let bucketPath = filePath.substring(s3Path.length+1);
let params = {Bucket: bucketName, Key: bucketPath, Body: fs.readFileSync(filePath) };
s3.putObject(params, function(err, data) {
if (err) {
console.log(err)
} else {
console.log('Successfully uploaded '+ bucketPath +' to ' + bucketName);
}
});
});
};
uploadDir("path to your folder", "your bucket name");
Solution 2:
aws s3 cp SOURCE_DIR s3://DEST_BUCKET/ --recursive
Custom endpoint
if you have a custom endpoint implemented by your IT, try this
aws s3 cp <local-dir> s3://bucket-name/<destination-folder>/ --recursive --endpoint-url https://<s3-custom-endpoint.lan>
It's worth mentioning that if you are simply using S3 for backups, you should just zip the folder and then upload that. This Will save you upload time and costs.
If you are not sure how to do efficient zipping from the terminal have a look here for OSX.
And $ zip -r archive_name.zip folder_to_compress for Windows.
Alternatively a client such as 7-Zip would be sufficient for Windows users
I do not see Python answers here.
You can script folder upload using Python/boto3.
Here's how to recursively get all file names from directory tree:
def recursive_glob(treeroot, extention):
results = [os.path.join(dirpath, f)
for dirpath, dirnames, files in os.walk(treeroot)
for f in files if f.endswith(extention)]
return results
Here's how to upload a file to S3 using Python/boto:
k = Key(bucket)
k.key = s3_key_name
k.set_contents_from_file(file_handle, cb=progress, num_cb=20, reduced_redundancy=use_rr )
I used these ideas to write Directory-Uploader-For-S3
I ended up here when trying to figure this out. With the version that's up there right now you can drag and drop a folder into it and it works, even though it doesn't allow you to select a folder when you open the upload dialogue.
You can drag and drop those folders. Drag and drop functionality is supported only for the Chrome and Firefox browsers.
Please refer this link
https://docs.aws.amazon.com/AmazonS3/latest/user-guide/upload-objects.html
You can use Transfer Manager to upload multiple files, directories etc
More info on:
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3-transfermanager.html
You can upload files by dragging and dropping or by pointing and clicking. To upload folders, you must drag and drop them. Drag and drop functionality is supported only for the Chrome and Firefox browsers
Drag and drop is only usable for a relatively small set of files. If you need to upload thousands of them in one go, then the CLI is the way to go. I managed to upload 2,000,00+ files using 1 command...