My Rails3 application runs on multiple VPS servers on Rackspace.
We allow users to upload images on our application and place those files at /public directory.
What is the best way to sync it across the servers.
Setup: Ubuntu 12.x OS/Nginx/Phusion
Instead of replicating the data, why not side-load it to a standard location that is accessible by all of your VPSs?
That should reduce bandwidth overall, and storage costs, by only having one copy of the data.
Related
Here at work we use s3fs because our product uses Oracle and the legacy code load files uploaded thru web direct from Oracle pl/sql, so we need the Oracle Database to see the same file system the web server sees. And we cannot access the OS where Oracle is installed to mount Windows shares thru SAMBA.
The problem is, when a directory (prefix in s3) reaches rougly
40,000 files, the access to this directory thru samba become extremely slow, causing timeouts and even completely stop the application pool trying to access this directory.
Our web servers are EC2 instances with Windows 2019.
I wonder if someone know a solution to this.
S3 LIST API is slow when objects are lots. It should be better if you could re-organize files into multiple directory levels.
Need your help. I just want to locate the published files (physical files published) of my .NET Core 2.1 in the App Engine server. I used Google plugin tool to publish my site and everything is done automatically.
I'm using simple app.yaml file:
runtime: aspnetcore
env: flex
I tried to scan some folders of the App Engine server but I could not locate my site. I also wonder maybe because google uses docker (don't have experience with docker too) and those file are in the docker's container. Not really sure.
In your project's Cloud Storage Buckets list, you will find a Bucket named like artifacts.[project-id].appspot.com. There you will find the container images that were deployed to App Engine.
However, App Engine provides On Demand server provisioning and scaling. This means that App Engine instances will be created when requests start coming in and more instances will be created if traffic increases.
Each instance will load your app's image individually.
In this type of environment, you should not store any relevant information in the App's directory, because all of it will be erased when the instance is killed due to lack of traffic and the data stored in one instance will not be available in other instances. See how instances are managed for more info.
If you want your app to store data in a SQL database, you could have a look at Cloud SQL, or also, you may find that Cloud Firestore, which is a NoSQL database, can suit your needs. Here is a list of GCP databases
I've inherited a couple of web servers - one linux, one windows - with a few sites on them - nothing too essential and I'd like to test out setting up back-ups for the servers to both a local machine and a cloud server, and then also use the cloud server to access business documents and the local machine as a back-up for these business documents.
I'd like to be able to access all data wherever I am via an internet connection. I can imagine it running as follows,
My PC <--> Cloud server - access by desktop VPN or Web UI
My PC <--> Web Servers - via RDP, FTP, Web UI (control panels) or SSH
My PC <--> Local Back-up - via RDP, FTP, SSH or if I'm in the office, Local Network
Web servers --> Local Back-up - nightly via FTP or SSH
Cloud Server --> Local Back-up - nightly via FTP or SSH
Does that make sense? If so, what would everyone recommend for a cloud server and also how best to set up the back-up server?
I have a couple of spare PC's that could serve as local back-up machines - would that work? I'm thinking they'd have to be online 24/7.
Any help or advice given or pointed to would be really appreciated. Trying to understand this stuff to improve my skill set.
Thanks for reading!
Personally I think you should explore using AWS's S3. The better (S)FTP clients can all handle S3 (Cyberduck, Transmit, etc.), the API is friendly if you want to write a script, there is a great CLI suite that you could use in a cron job, and there are quite a few custom solutions to assist with the workflow you describe. s3tools being one of the better known ones. The web UI is fairly decent as well.
Automating the entire lifecycle like you described would be a fairly simple process. Here's one process for windows, another general tutorial, another windows, and a quick review of some other S3 tools.
I personally use a similar workflow with S3/Glacier that's full automated, versions backups, and migrates them to Glacier after a certain timeframe for long-term archival.
I would like to access my Amazon S3 buckets without third-party software, but simply through the WebDAV functionality available in most operating systems. Is there a way to do that ? It is important to me that no third-party software is required.
There's a number of ways to do this. I'm not sure about your situation, so here they are:
Option 1: Easiest: You can use a 3rd party "cloud gateway" provider, like http://storagemadeeasy.com/CloudDav/
Option 2: Set up your own "cloud gateway" server
Set up a dedicated server or virtual server to act as a gateway. Using Amazon's own EC2 would be a good choice.
Set up software that mounts S3 as a drive. Two I know of on Windows: (1) CloudBerry Drive http://www.cloudberrylab.com/ and (2) WebDrive (http://webdrive.com). For Linux, I have never done it, but you can try: https://github.com/s3fs-fuse/s3fs-fuse
Set up a webdav server like CrushFTP. (It comes to mind because it's stable and cheap and works on any OS.) Another option is IIS but I personally find it's harder to set up securely for webdav.
Set up a user in your WebDav server (ie CrushFTP or IIS) with access to the mapped S3 drive.
Possible snag: Assuming you're using Windows, to start your services automatically and have this work, you may need to set up both services to use the same Windows user account (Services->(Your Service)->[right-click]Properties->Log On tab). This is because the S3 mapping software might not map the S3 drive for all Windows users. Alternatively, you can use FireDaemon if you get stuck on this step to start the programs as a service all under the same username.
Other notes: I have experience using WebDrive under pretty heavy loads, and it seems to work well. Under tons of pounding (I'm talking thousands of files per hour being added to a 5 TB WebDrive) it started to crash Windows. But I'm not sure if you are going that far with it. Also, if you're using EC2, you may not have that issue since it was likely caused by a huge transfer queue in memory and EC2 will have faster transit to S3 and keep the queue smaller.
I finally gave up on this idea and today I use Rclone (https://rclone.org) to synchronize my files between AWS S3 and different computers. Rclone has the ability to mount remote storage on a local computer, but I don't use this feature. I simply use the copy and sync commands.
S3 does not support webdav, so you're out of luck!
Also, S3 does not support hierarchial name spaces, so you cant directly map a filesystem onto it
There is an example java project here for putting a webdav server over Amazon S3 - https://github.com/miltonio/milton-aws
I need you to solve big problem of mine. I've created an IIS smooth streaming application to deliver the media content.
I'm using azure windows virtual machine, small instance (CUP 1 Core), as a media server. I installed IIS Media Services on vm(Virtual Machine) and I'm creating publish points on it.
The number of users are too many and load on vm will be increase. So, I decided to go with the Load Balancing/Auto Scaling options. Well I'm doing this first time.
Here is my media server architecture:
I want to implement 3-tier architecture, like I'll create 3 virtual machines and want them communicate each other to balance the load. I mean if load increase on vm-1 then load will be balance with vm-2 and/or vm-3. Also I want to auto scaling of vms.
How can I do this?
Thanks in advance
Devendra
to do so you can create an availability set where you can join the 3 VMs , here are some resources I think they will be very beneficial
"managing the availability of the Virtual Machines" http://www.windowsazure.com/en-us/manage/windows/common-tasks/manage-vm-availability/
here is a second one for the load balancing I think it is a bit like the first one: http://www.windowsazure.com/en-us/manage/windows/common-tasks/how-to-load-balance-virtual-machines/
for the scaling I think it is still in the preview, you can test it by adding this feature to your account. after that you will be able to access it in your cloud service after creating the Virtual Machines required.