I am beginner with tensorflow and now in a project where I need to deploy distributed production platform for tensorflow. I appreciate if I could get some help to clarify my thought.
Reading the online doument, and youtube,
I understood that main components for distributed production are below.
TFX (Tensorflow extended) built with python 3.x
Pipeline: Apache Beam
Orchestrator: Apache Airflow or Kubeflow
However for orchestrator, I assume that there are pros and cons for both components but which one is the de facto standard for TFX ?
The guide mainly focus in Airflow so I thought this might be the one but kubeflow seems to be new so it might be the new challenger.
Note: The current revision of this user guide primarily discusses deployment on a bare-metal system using Apache Airflow for orchestration.
Thanks,
Yu
I think Kubernetes/Kubeflow is the best orchestrator, however, it brings a lot of upfront costs in setting up and managing your own cluster.
Google just released VertexAI pipelines which is a managed (serverless) service where GCP manages Kubernetes under the covers for you and you can just focus on writing pipeline code.
I highly recommend using it as if very affordable and straight forward to set up. https://cloud.google.com/vertex-ai/docs/pipelines/introduction
In my company we saved millions of dollars in operational and maintenance costs by using VertexAI pipelines.
Just to complete this answer, there are some cons associated with VertexAI. It is a Pre-GA offering so there are still some small issues here and there that I run into but I would say its 90% functional and we are using it to orchestrate our end-to-end machine learning workflows as well as automating some of our analytics and data validation workloads.
Related
Planning to Migrate the Websphere from 7.0 to 9 and 8.5 to 9.
Can anyone help me getting the detailed Process
Migration here is "In place". (Migration will be done on the same servers, where the old Installation are in)
if at all any migration tools need to be used, please provide the clear info on them.
any documental references, or any video references for the questioner is appreciated.
OS used : RHEL
CUrrent version: WAS 7x and 8.5
Migrating to : WAS 9.0
It sounds like you're in the very beginning stages of doing this migration. Therefore, I highly recommend you take some time to plan this out, especially to figure out the exact steps you'll be taking and how you'll handle something going wrong. For WebSphere, there is a collection of documents from IBM that discuss planning and executing the upgrade. Look around there for documentation on the tools and step by step guides for different kinds of topologies. The step by step guide for an in place migration of a cell is here.
You should make sure to take good backups before you start the process so you can restore back to before the migration if you need to.
In addition to doing the upgrade, an important part is to also make sure your applications are going to work on the new version if you haven't already. IBM provides this tool to scan applications and identify potential issues that developers will have to fix. There is documentation for the tool at that link as well.
If you are in the planning phase, I'd strongly suggest you to consider migrating to WebSphere Liberty instead of traditional WAS v9. All these migration tools (toolkit for binaries, Eclipse migration toolkit) support both migration scenarios.
Choosing Liberty might be a bit more work at the beginning, but you will gain more deployment flexibility and speed up future development. Liberty is also much better fitted for any cloud/containers environments, as it is much more lightweight, so in the future, if you would like to move to containers, it would be much easier.
Check this tutorial Migrate traditional WebSphere apps to WebSphere Liberty on IBM Cloud Private by using Kubernetes, although it shows the steps to migrate to Liberty on ICP, beginning is the same - analyzing of the application whether they are fit for Liberty and migrating. If you don't have access to IBM Cloud or ICP, you can use stand alone version of the Transformation Advisor that was released recently - IBM Cloud Transformation Advisor.
Having said all that, some apps include old or proprietary traditional WebSphere APIs and in that case it may be easier and cheaper to temporary migrate them to WAS v9, and modernize in the future.
Is there any way to automatically run regression/functional tests on Nifi flows using Jenkins pipeline ?
Searched for it, without any success.
Thanks for your help.
With the recent release of NiFI-1.5.0 and NiFi-Registry-0.1.0, the community has come together to produce a number of SDLC/CICD integration tools to make using things like Jenkins Pipeline easier.
There is both Python (NiPyAPI), and Java (NiFi-Toolkit-CLI) API wrappers being produced by a team of collaborators to allow scripted manipulation of NiFi Flows across different environments.
Common functions include interaction with integrated version control, import/export of flows as JSON documents, deployment between environments, start/stop of flows, etc.
So, we are working quickly towards supporting things like an integrated wrapper for declarative Jenkins Pipelines, and I would add it is being done fully in public codebase under the Apache license, so we (I am the lead NiPy author) would welcome your collaboration.
Is it practical or wise to use Lagom in production without ConductR? The commercial licensing is putting me off. This framework looks like it could be pretty arduous to deploy and custom tooling for that can take a lot of effort to get right.
(disclaimer: I'm a Lightbend employee, currently core member of the Lagom team)
Edit (Nov 2018): Please refer to https://www.lagomframework.com/documentation/current/java/ProductionOverview.html#Running-Lagom-in-production for up to date information on this topic.
(original answer, Aug 2017)
A lot has changed in the Lightbend stack since this question was added over a year ago. For example: ConductR is now free to use in production for up to three nodes.
Also, the team behind ConductR is also working on providing tools to deploy a Lagom application on Kubernetes. The efforts on that front are very advanced and some of our sample apps can be deployed in Kubernetes already.
Your question is rather open-ended and so let's start with "it depends".
If
you're comfortable managing your scaling within the configuration of your Akka cluster,
your usage doesn't violate the open-source licensing terms of lagom, play and Akka, AND
you don't have sufficient cash flow to justify leveraging Lightbend's production suite,
then you arguably can deploy with a minimum of effort and custom tooling.
If those conditions don't hold, your options are to go elsewhere (e.g., Spring Cloud) or retain Lightbend. You may find going elsewhere has it's own cognitive load and/or commercial expense.
Hope that helps even 7 months later.
The other answers are higher level, but I can essentially say "yes." I'm currently deploying a Lagom service to be hosted on a Kubernetes cluster, and I'm not using ConductR or any of the commerically licensed components.
You will need to dig a little bit into some Play internals to start the service properly, and if you want to hook in with some other service locator you may need to implement one yourself, but it certainly isn't impossible and I think it's less effort on net.
Being honest it really worth to pay, because you got much more, good reporting dashboards, automatic cluster formation and what really cool is split brain resolver....
But sometimes agree when you are working on a project that don't have a lot of money, you can do some small tricks and get it work and may be then do all what really lucks or buy enterprise subscription.
So Lagom can be very easy being used without service discovery at all if you will delegate all to Kubernetes DNS or without Kubernetes just put load balancer before each service and use its address.
How this can works
Each lagom service that you have should be external service
Production run configuration should be mixed with
ConfigurationServiceLocatorComponents
Service that should communicate with another one should be started with extra parameter that tell lagom that this service is external one and can be communicated directly(all can be in JAVA_OPTS as for play application)
-Dlagom.services.your_service_name=http://k8s_service_name.default.svc.cluster.local:9000
We're planning to develop a web based Healthcare Practice Management System. Due to HIPAA we're requested to deploy the app in our own premises. Our company is relatively small currently we have only software engineers and no devops engineers but still we want to develop the application to support horizontal scaling(adding more servers).
Planned to use:
Python3 (Django)
PostgreSQL
I'm looking for something like AppScale but with the freedom of choosing our own runtime, database and frameworks.
In other words from the software engineer's perspective:
Should provide an easy way to deploy django application
Should have web based dashboard to monitor and control(like AppScale)
Should make load balancing simple (app and database)
AppScale implements the Google App Engine APIs which, IMHO, make it super easy to develop web apps quickly and efficiently.
On top of that, you get auto-scaling, load balancing, and the ability to deploy on-premises and plug in any third-party library you need.
AppScale already comes with a dashboard and will soon be launching a new management service for your AppScale deployment(s).
If you're not particularly hung up on Python3 and PostgreSQL, all of the above seem to cover your requirements.
It's worth noting that opting for the GAE model means you opt for NoSQL and, so, postgres is probably not the best option.
Disclaimer: I'm part of the AppScale team and we're already helping companies develop and deliver their apps in the HIPAA compliance realm.
I chose Kubernetes which is a container orchestration technology specifically designed for Docker and also found that scaling is not just the responsibility of platform that the app is deployed on but also its depends on how the app is designed and coded. For that The Twelve-Factor App methodology is really helpful.
But I can't deploy database on Kubernetes because its not recommended by Kelsey Hightower(author of Kubernetes Up and Running) in his talk. So, for now I chose to deploy my database on a VM.
What tools or strategies are you using for automation of EC2 activities?
I need to be able to bring up a number of EC2 instances, provision various software to it (primarily Python packages), interact with S3 (primarily download data), and run various jobs. I'll be doing this both on-demand and on a scheduled basis.
I'm trying to decide if I should:
Create an AMI with all my software loaded on it
or
Launch a plain vanilla linux AMI instance and scp my software to it
For the provisioning and automation Boto looks pretty good. Or I could write something with Paramiko. Recommend either or anything else I should be looking it?
Basically I'm looking for advice / success stories, let me know what's working for you.
To answer your bullets about selecting AMIs, I would say that it depends on how much software you're installing.
I have been successful with a hybrid approach, where I build an AMI and load my heavyweight and more stable software. This is the stuff that needs to run an installer, or takes considerable time to install (remember that if you re-install a package every time as part of your startup process, you're paying for the install every time). Then, I upload the small and volatile software at provisioning/startup time. In this bucket goes most of the application code, data, etc. That way, I can change my app and not have to touch the AMI.
The benefits of this approach:
Don't have to pay for running the same software install thousands of times.
AMI can stay fairly stable over time.
Can use software that requires intervention or GUI interaction to install.
Major drawbacks:
Your AMI's OS version will become stale over time.
Your AMI may not be flexible as to the instance type/architecture it will run on. For instance, you may create it on a 32-bit OS and thereby prevent it from running on the High CPU instance types, or vice versa. So you may lock yourself into a pricing scheme.
I don't use Python, so I can't comment on either of the APIs you referenced.
AWS just released the Systems Manager suite, which includes an Automation service that will (among other things) handle your use cases around AMIs.
This question was asked some time ago now but I believe my answer could be useful to other users. I believe the best automations tools available on the market are provided by Cloud Management platforms. For example they offer auto-scaling, configuration software integration (Chef/Puppet), databases replications, dns management...
The most popular cloud management softwares are Scalr (disclaimer: I work there), RightScale and enStratus. Scalr is open-source and released under the Apache 2 license.
Regarding your specific question on AMIs, cloud Management platforms usually provide pre-configured AMIs (at Scalr, we call them roles). If you want to create your own AMI built on an existing instance, you'll be able to create snpashots and use them as a template for future instances.