Is Spring Cloud Stream suitable for development of microservices in highly resource-constrained environments (e.g. 2 GB RAM and 2 GB disk)? Is there a binder that is particularly well suited for such environments (assuming modest throughput requirements)?
With regard to RAM, 2GB is way more than enough. As far as disk, that is really up to the application, size of messages etc., as Spring Cloud Stream does not itself have any local storage features (other than the size of application and configuration files themselves).
Related
Is there any possibility of running Orion ContextBroker on Raspberry Pi with Raspbian OS?
The requirements recommended in the Orion documentation are:
Although we haven't done yet a precise profiling on Orion Context
Broker, tests done in our development and testing environment show
that a host with 2 CPU cores and 4 GB RAM is fine to run the
ContextBroker and MongoDB server. In fact, this is a rather
conservative estimation, Orion Context Broker could run fine also in
systems with a lower resources profile. The critical resource here is
RAM memory, as MongoDB performance is related to the amount of
available RAM to map database files into memory.
Besides the board constrained resources, you will have to search about the equivalent required libraries for RaspbianOS.
There is a discussion about it here:
https://github.com/telefonicaid/fiware-orion/issues/15
We have a galera cluster with 3 nodes, on 3 different physical machines but all located in the same datacenter.
From what I understood, the reason they deployed this in the past was to increase availability and reliability.
Each node is installed on a VM using 12 cores and 4Gb RAM.
We are asked to migrate this to Google Cloup Platform in order to get rid of the ops tasks.
I could create 3 compute engine instances and deploy the galera cluster, but I have difficulties to see the added value compared to Cloud SQL instances with replication and backup. I am not very familiar in scaling heavy load systems.
The db hosted in these nodes is kind of critical and should ensure maximum availability and reliability.
What strategy should I adopt in order to migrate this architecture to GCP ?
So I have a cloud virtual machine on google compute, does this mean by nature that it is highly available? If the VM is running on a single piece of hardware on GCE, if the piece of hardware breaks then the VM could go down. Is the VM running on some kind of RAID, but for servers? So if one of the machines goes down another machine will pick up and continue running the vm? Thanks.
The machine itself is not highly available. However, Google takes several steps to increase reliability:
Storage is replicated and independent of the physical machine the VM is running on (obviously not for local SSD). This means that even if the physical machine catches on fire, only the "runtime" state is lost but the attached disks are fine.
VMs can live-migrate. This is a setting you can control. If enabled, the VM will be migrated to a different physical machine on maintenance events. Live-migration can lead to brief performance degradation while memory etc. is synced to the other host but the machine is not shut down / restarted.
Even when the physical host suddenly dies, you can set your instance to restart automatically on a new machine. If you plan to use this mode, make sure your instance is able to cleanly boot to serving state without manual intervention.
If you need high availability, the best approach is to spread your instances among zones of the same region and using a network or HTTP(S) loadbalancer. These will automatically stop sending traffic to a machine in case it becomes unhealthy. Also see this short youtube video on Google's network architecture for more info.
For high availability of your application data, there are highly available options like Datastore for database-like usage and Cloud Storage for file-oriented data. Keep in mind that Cloud SQL also runs on a single instance/physical machine which means that you have to setup slaves/replicas to get high availability. However, you can also do that with your favorite DB system on plain Compute Engine instances if you are willing to maintain them yourself.
I was wondering if you guys had any tips which repository implementation has good clustering and horizontal scaling characteristics on common hardware?
The problem is that we have to implement a preservation system on top of a repository which is be able to ingest and manage LOTS of heterogeneous data (> 500 TB) with big files (>50GB).
Fedora Commons it seems can only be clustered by using a distributed filesystem. Apache Jackrabbit can be clustered but its DataStore (for large binary data) has to be the same for all nodes in a clustered environment. Do you guys have any tips which repository systems I should check out?
Give ModeShape a try. It is a JCR 2.0 implementation that can be configured to use an Infinispan data grid as its backing store, and ModeShape is also easily clustered (it uses JGroups, which is the same communication library used in the clustering features in Infinispan and JBoss Application Server, among many others).
What information have you been able to gather regarding how do the amazon web services work?
What hardware do they use
What web server
What Operating System
What storage for AWS
What virtualization software for EC2/EBS
What software for they distributed firewall for EC2
Physical location of their data centers.
I like their services very much and use them a ton at work... just out of curiosity. If you know/heard/read and want to tell, if you saw something online and want to provide a link, very appreciated.
This might be interesting: http://highscalability.com/amazon-architecture
While this question can't be answered in precision, I'll try to shed some light on internal workings that Amazon announced publicly.
Below are some details for commonly used c and m instance types, as well as recently released bare metal instances. Also,
this can be starting point for further research as specifics are far behind single answer on SO.
Compute Hardware.
If you want to take a deep dive I suggest going through all previous generations and current generation instance types. Underlying hardware can be find on this pages.
Bare metal instances
Bare metal instances became GA in April 2018. One of the details - I3.metal instances are powered by 2.3 GHz Intel Xeon processors, offering 36 hyper-threaded cores (72 logical processors), 512 GiB of memory, and 15.2 TB of NVMe SSD-backed instance storage. More info
Compute optimized instances (C)
Latest c5 (late 2017) generation instances are using 3.0 GHz Intel Xeon
Platinum 8000-series. More info here
c4 (generation is using optimized for ec2 processor Intel Xeon E5-2666 v3 (code name Haswell) processor. More info here
c3 generation introduced SSD instance storage and used 2.8 GHz Intel Xeon E5-2680v2. More info here
General purpose ec2 instances (M)
m5 instances are based on Custom Intel® Xeon® Platinum 8175M series processors running at 2.5 GHz. Most likely running on Nitro hypervisor mentioned below. More info
m4's were released back in 2015 and have custom Intel Xeon E5-2676 v3 Haswell processor optimized specifically for EC2. More info
m3's were released in 2012 and for some who remember carried some price reduction with them, making use of AWS use more appealing to audience looking through budgeting lenses. They are/were using Intel Xeon E5-2670 processor and started using SSD instance memory.
What web server
I've seen couple of times error pages from their WebUI (AWS Console) rendered via Tomcat, so I would guess this is console server.
What virtualization software for EC2/EBS
AWS recently announced (with c5 instance type announcement) that they will start using KVM based hypervisor. Presentation linked here outlines hypervisor history very good (table below taken from same page)
Physical location of data centers
This is not (and due security reasons should not) be disclosed publicly. There are always rumors / some sources about it (look at related Quora thread
You can use linux instances or windows instances in Amazon aws. But first of all you shoul run an instance and then select it's operating system. For it's storage they have an instance that called it S3. it is a storage that you can save any kind of file format in it. They have many locations for their data centers. Depending on where you live, you must select the nearest data center to work with it's services so that you have to pay less for your billing payment.
You can go to console.amazon.com and find lots of documentation for each service in help menu.