What information have you been able to gather regarding how do the amazon web services work?
What hardware do they use
What web server
What Operating System
What storage for AWS
What virtualization software for EC2/EBS
What software for they distributed firewall for EC2
Physical location of their data centers.
I like their services very much and use them a ton at work... just out of curiosity. If you know/heard/read and want to tell, if you saw something online and want to provide a link, very appreciated.
This might be interesting: http://highscalability.com/amazon-architecture
While this question can't be answered in precision, I'll try to shed some light on internal workings that Amazon announced publicly.
Below are some details for commonly used c and m instance types, as well as recently released bare metal instances. Also,
this can be starting point for further research as specifics are far behind single answer on SO.
Compute Hardware.
If you want to take a deep dive I suggest going through all previous generations and current generation instance types. Underlying hardware can be find on this pages.
Bare metal instances
Bare metal instances became GA in April 2018. One of the details - I3.metal instances are powered by 2.3 GHz Intel Xeon processors, offering 36 hyper-threaded cores (72 logical processors), 512 GiB of memory, and 15.2 TB of NVMe SSD-backed instance storage. More info
Compute optimized instances (C)
Latest c5 (late 2017) generation instances are using 3.0 GHz Intel Xeon
Platinum 8000-series. More info here
c4 (generation is using optimized for ec2 processor Intel Xeon E5-2666 v3 (code name Haswell) processor. More info here
c3 generation introduced SSD instance storage and used 2.8 GHz Intel Xeon E5-2680v2. More info here
General purpose ec2 instances (M)
m5 instances are based on Custom Intel® Xeon® Platinum 8175M series processors running at 2.5 GHz. Most likely running on Nitro hypervisor mentioned below. More info
m4's were released back in 2015 and have custom Intel Xeon E5-2676 v3 Haswell processor optimized specifically for EC2. More info
m3's were released in 2012 and for some who remember carried some price reduction with them, making use of AWS use more appealing to audience looking through budgeting lenses. They are/were using Intel Xeon E5-2670 processor and started using SSD instance memory.
What web server
I've seen couple of times error pages from their WebUI (AWS Console) rendered via Tomcat, so I would guess this is console server.
What virtualization software for EC2/EBS
AWS recently announced (with c5 instance type announcement) that they will start using KVM based hypervisor. Presentation linked here outlines hypervisor history very good (table below taken from same page)
Physical location of data centers
This is not (and due security reasons should not) be disclosed publicly. There are always rumors / some sources about it (look at related Quora thread
You can use linux instances or windows instances in Amazon aws. But first of all you shoul run an instance and then select it's operating system. For it's storage they have an instance that called it S3. it is a storage that you can save any kind of file format in it. They have many locations for their data centers. Depending on where you live, you must select the nearest data center to work with it's services so that you have to pay less for your billing payment.
You can go to console.amazon.com and find lots of documentation for each service in help menu.
Related
I was reading the Graphcore documentation and came across the V-IPU software. The documentation says I can download it from the Graphcore software download portal, however I don't see it anywhere among the downloadable software (link below):
https://downloads.graphcore.ai/
I checked whether the Poplar SDK contains it but it doesn't, and it's not already installed on the machine I'm using (DSS8440 IPU SERVER). On the other hand, I'm able to run PopTorch without it, so I'm guessing V-IPU doesn't exist anymore? If not, where can I find it?
Graphcore's Virtual-IPU (V-IPU) is a software layer for allocating and configuring Graphcore IPUs in IPU-Machine and IPU-POD (a collection of interconnected IPU-Machines) systems, as described e.g. in the Introduction section of the V-IPU user guide. As you are using a Dell DSS 8440 IPU server and not an IPU-M or an IPU-POD system, V-IPU is not needed to run your programs on the IPU, as you have already verified yourself, and it is not visible to you on the Downloads portal either. You only need Poplar SDK to get going on the DSS 8440 server.
I'll add a few more details about these different systems, to clarify why V-IPU software is not needed with the IPU server you're currently using. The Dell DSS 8440 IPU server features eight Graphcore dual-IPU C2 cards, which are connected via a high speed PCIe switched fabric. On the other hand, in IPU-Ms and IPU-POD systems the IPU devices are accessed using IPU over Fabric (IPUoF) network connectivity, based on 100G RDMA over converged Ethernet (RoCE). The V-IPU software provides the allocation, provisioning and monitoring of IPU chips and related infrastructure for machine-learning Poplar-based workloads in the IPU-POD. The IPU-specific part of the V-IPU software can run on a single IPU-Machine, when used in direct attach mode, or on a server in an IPU-POD.
You can read more details on IPU-POD systems in this section of Graphcore's documentation.
I hear a lot about "Hypervisors are not emulators. If you need to emulate another hardware specifications than you computer have, you need to use emulator, not hypervisor".
Well, but yesterday I saw this video on youtube - click here - which shows how to install Win 95 on modern macOS with VMware Fusion.
The strange thing for me is that on 17:39 you could see that Win 95 virtual machine is "Pentium Pro with 64 MB RAM".
Hmm! So, Fusion somehow faked processor and RAM, right? But it is not emulator, right? So, does it mean that any hypervisor can fake processor and RAM?
At the time of its release, Windows 95 only had code to recognize CPUIDs up to Pentium Pro. Any processor not lower than Pentium Pro is "called" Pentium Pro.
The main difference is the Hypervisor cannot emulate CPU code. All code must run on the original processor.
The hypervisor does emulate the BIOS, which in tells the OS the hardware specs available; including RAM, Boot order and peripherals attached.
When you are talking about VMWare Fusion the way this works depends on how virtualization is achieved. According to wikipedia VMWare Fusion utilizes hardware-assisted virtualization, dynamic binary translation, and para-virtualization.
In the hardware-assited virtualization case, #Strom is correct and guest instructions can be executed directly on the host CPU. Besides #Strom answer, you can fake the CPU type by trapping and emulating the cpuid instruction.
In the para-virtualization case you replace critical instructions by calls to the hypervisor which emulates the instruction on behalf of the guest. So again you emulate the cpuid instruction to "fake" the CPU type. Keep in mind that this requires a modified, hence para-virtualized, guest operating system.
Finally, dynamic binary translation scans the guest code for critical instructions during runtime and either replaces them by traps into the hypervisor achieving some kind of "live para-virtualization" or translating blocks of guest code into equivalent blocks of host code that modifies the VM state according to the original guest code (this is e.g. how the QEMU full system emulator works). As a result, again you are able to "fake" the CPU type by emulating the cpuid instruction. Notice that guest and host can be the same architecture in this case, but there is no need for this.
Of course a combination of above techniques is also feasible.
As for virtualization of main memory, the hypervisor is in full control of the hardware so you can simply configure a VM with just 64MB of main memory. The VM is not able to "see" more than this due to the techniques shortly discussed above.
Please keep in mind that this just gives a very short overview of virtualization and I tried to keep it short and informative, so I know my explanations are partially not very accurate. If you are really interested in virtualization I recommend reading "Virtual Machines: Versatile Platforms for Systems and Processes" or the papers on the topic by Popek & Goldberg and "Xen and the Art of Virtualization"
I am running Apache Guacamole on a Google Cloud Compute Engine f1-micro with CentOS 7 because it is free.
Guacamole runs fine for some time (an hour or so) then unexpectantly crashes. I get the ERR_CONNECTION_REFUSED error in Chrome and when running htop I can see that all of the tomcat processes have stopped. To get it running again I just have to restart tomcat.
I have a message saying "Instance "guac" is overutilized. Consider switching to the machine type: g1-small (1 vCPU, 1.7 GB memory)" in the compute engine console.
I have tried limiting the memory allocation to tomcat, but that didn't seem to work.
Any suggestions?
I think the reason for the ERR_CONNECTION_REFUSED is likely due to the VM instance falling short on resources and in order to keep the OS up, process manager shuts down some processes. SSH is one of those processes, and once you reboot the vm, resource will resume operation in full.
As per the "over-utilization" notification recommending g1-small (1 vCPU, 1.7 GB memory)", please note that, f1-micro is a shared-core micro machine type with 0.2 vCPU, 0.60 GB of memory, backed by a shared physical core and is only ideal for running smaller non-resource intensive applications..
Depending on your Tomcat configuration, also note that:
Connecting to a database is an intensive process.
Creating a Tomcat with Google Marketplace, the default VM setting is "VM instance: 1 vCPU + 3.75 GB memory (n1-standard-1) so upgrading to machine type: g1-small (1 vCPU, 1.7 GB memory) so should ideal in your case.
Why was g1 small machine type recommended. Please note that Compute Engine uses the same CPU utilization numbers reported on the Compute Engine dashboard to determine what recommendations to make. These numbers are based on the average utilization of your instances over 60-seconds intervals, so they do not capture short CPU usage spikes.
So, applications with short usage spikes might need to run on a larger machine type than the one recommended by Google, to accommodate these spikes"
In summary my suggestion would be to upgrade as recommended. Also note that, the rightsizing gives warnings when VM is underutilized or overutilized and in this case, it is recommending to increase your VM size due to overutilization and keep in mind that this is only a recommendation based on the available data.
Is there any possibility of running Orion ContextBroker on Raspberry Pi with Raspbian OS?
The requirements recommended in the Orion documentation are:
Although we haven't done yet a precise profiling on Orion Context
Broker, tests done in our development and testing environment show
that a host with 2 CPU cores and 4 GB RAM is fine to run the
ContextBroker and MongoDB server. In fact, this is a rather
conservative estimation, Orion Context Broker could run fine also in
systems with a lower resources profile. The critical resource here is
RAM memory, as MongoDB performance is related to the amount of
available RAM to map database files into memory.
Besides the board constrained resources, you will have to search about the equivalent required libraries for RaspbianOS.
There is a discussion about it here:
https://github.com/telefonicaid/fiware-orion/issues/15
Apologies if this is the wrong platform for this question.
If I want to migrate 100 VM's onto Azure VM's what all things I need to consider and how can I migrate?
This is not a comprehensive answer but some things to consider are:
- Start with a thorough inventory of the VMs to migrate. Issues to watch out for include..
- Any unsupported OS versions, including 32-bit.
- large numbers of attached drives.
- Disk drives >1TB.
- Gen 2 VHDs.
- Application and network interdependencies which need to be maintained.
- Specific performance requirements (i.e. any VMs that would need Azure premium storage, SSD drives etc.).
In developing a migration strategy some important considerations are:
- How much downtime can you tolerate? To minimize downtime look at solutions like Azure Site Recovery which supports rapid switchover. If downtime is more flexible there are more offline migration tools and scripts available.
- Understand whether to move to the new Azure Resource Manager or the Service Management deployment model. See https://azure.microsoft.com/en-us/documentation/articles/resource-group-overview/.
- Which machines to move first (pick the simplest, with fewest dependences).
- Consider cases where it may be easier to migrate the data or application to a new VM rather than migrate the VM itself).
A good forum to ask specific migration questions is: Microsoft Azure Site Recovery
Appending to sendmarsh's reply
The things you will have to consider are:
Version of virtual environment i.e VMWare or Hyper-V.
Os version, RAM Size, OS disk size, OS disk count, Number of disks, Capacity of each disk, format of hard disk, number of processor cores,number of NIC's, processor architecture, Network configurations such as IP address's, generation type if the environment is Hyper-V.
I could have missed a few more things... like checking if the VMWare tools are installed. Some of the configurations are not supported like having an iSCSI disk will not be supported. Microsoft supports not all naming conventions for the machines, so be careful in setting the name as that might affect things later.
A full length of pre-requisites list is over at:
[1]: https://azure.microsoft.com/en-us/documentation/articles/site-recovery-best-practices/#azure-virtual-machine-requirements
Update: Using Powershell to automate the migration would make your life easier.