I have a challenge from a customer. They'd like me to be able to support the creation, editing and deletion of custom fields against a Model within my application. This is so they can record any values that they see fit against a model and report against them.
I've tried a few methods of completing this however neither allowed for easy integration into our existing reporting engine and I'd quite like to keep that as untouched as possible.
I've scoured the web for similar answers to this question and most were from 2015 - 2018 and most were asking for dynamic models to be supported as well and frequently the answer was "No not yet but we will support this in future".
To support dynamic models I have seen solutions that reload waterline via sails.hooks.orm.reload() or even pro-grammatically restarting the sails app altogether and initialising again with the new model present. This could cause a few seconds of downtime for your application as the server restarted (for us it is about 30 as we have a huge app with nearly 50 models and 350 routes). We can't really go with this process as that down time is too much of an overhead and our clients would be a bit miffed.
Is potentially supporting just the dynamic handling of attributes possible without all the bootstrapping that takes place on sails lift or is it just not feasible? My initial feeling from all my reading is essentially no - as the processes in place during startup hook up and sync waterline/your db/your files.
Any discussion or examples of similar work welcomed.
Related
We are 3 teams:
Website front-end (React)
Website back-end (Node.js)
Native app (React Native, Node.js)
We want to share logic (e.g. Validations).
As of now I found articles on 3 ways to do so:
A NPM Package we will create for our own needs
A micro-service with endpoints who carry relevant logic
Serverless functions who carry relevant logic
Any other real-life, production suggestions?
Any other real-life, production suggestions?
Kind of - in no specific order:
You could specify the rules in a language/technology agnostic way, and then have your app load them at runtime (or be compiled in during build). The rules could then exist as a config file, or even be fetched from a remote location (a variation on your options 2 & 3).
Of course, designing a language agnostic rules engine / approach is non-trivial, and depends on what you need the rules to do (how complex, etc). You might find a pre-built open source solution that does that.
I have seen people try this, but the projects never succeeded (for unrelated reasons). One team specified the rules in an Excel sheet.
But there are trade-offs:
Performance hit - how to take language agnostic rules and be able to execute them? This will probably take some translation. Native code is almost always going to be faster and more efficient.
Higher development effort.
Added complexity - harder to debug (even if you compensate by developing more mechanisms to assist you do that - which is more development effort).
Regarding Your Options
For what it's worth, code / design-time sharing is an obvious approach, which I guess is sufficiently covered by NPM. I don't know enough about React and Node to know if they have any better ways of doing that. Normally if I have logic I want to share I'll use a component which is purpose built (lean as possible, minimal dependencies, intended to be re-used across multiple projects), and ingested in (C# / .Net) at compile/design time.
As an alternative to NPM you could look at dependency injection. This would allow you to do things like update the logic even after the app was deployed, as long as it can access where ever a newer set of rules are. So it's a bit like your option 1 (NPM, code level loading) but at runtime, and just once, and your options 2 & 3 - fetched remotely at runtime - the difference being that you're ingesting the logic not firing off questions and receiving answers (less chatty).
Service base rules are good in that they are totally separated, but the obvious trade-offs are availability and performance at runtime.
I don't see a difference in your options 2 & 3 from the stand-point of creating, managing and sharing logic. The only material impact is on whomever implements and supports that service system.
I wish to have an ability to write a dynamic plugin (*.so) which will allow the server to use data from the Amazon queue. Working data to make predictions based on an already taught model.
I have read the manual about the server extension points
https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/architecture.md
It says it is allowed to have custom sources, including the cloud ones. The problems are, I was unable to find a good complete example for the beginners (like I am), the help is a tiny one, and (if I understand correctly), those custom modules should be statically linked in the server code (which is, for some serious reasons uncontrollable by me, is not acceptable).
I am recently learning Angular 6 with #ngrx/store while one of the tutorial is to use #ngrx/store for state management, however I don't understand the benefit of using #ngrx/store behind the scene.
For example, for a simple login and signup action, previously by using the service(Let's call it AuthService) we might use it to call the backend api, store "userInfo" or "token" in the AuthService, redirect user to "HOME" page and we can inject AuthService in any component where we need to get the userInfo by using DI, which simply that one file AuthService handles everything.
Now if we are using #ngrx/store, we need to define the Action/State/Reducer/Effects/Selector which probably need to write in 4 or 5 files to handle above action or event, then sometimes still we need to call backend api using service, which seems much much more complicated and redundant...
In some other scenario, I even see some page uses #ngrx/store to store the object or list of object like grid data., is that for some kind of in-memory store usage?
So back to the question, why are we using #ngrx/store over service registration store here in Angular project? I know it's for "STATE MANAGEMENT" usage, but what exactly is the "STATE MANAGEMENT"? Is that something like transaction log and When do we need it? Why would we manage it on the front end? Please feel free to share your suggestion or experience in the #ngrx/store area!
I think you should read those two posts about Ngrx store:
Angular Service Layers: Redux, RxJs and Ngrx Store - When to Use a Store And Why?
Ngrx Store - An Architecture Guide
If the first one explains the main issues solved by Ngrx Store, it also quote this statement from the React How-To "that seems to apply equally to original Flux, Redux, Ngrx Store or any store solution in general":
You’ll know when you need Flux. If you aren’t sure if you need it, you don’t need it.
To me Ngrx store solves multiple issues. For example when you have to deal with observables and when responsability for some observable data is shared between different components. In this case store actions and reducer ensure that data modifications will always be performed "the right way".
It also provides a reliable solution for http requests caching. You will be able to store the requests and their responses, so that you could verify that the request you're making has not a stored response yet.
The second post is about what made such solutions appear in the React world with Facebook's unread message counter issue.
Concerning your solution of storing non-obvervable data in services. It works fine when you're dealing with constant data. But when several components will have to update this data you will probably encounter change detection issues and improper update issues, that you could solve with:
observer pattern with private Subject public Observable and next function
Ngrx Store
I'm almost only reading about the benefits of Ngrx and other Redux like store libraries, while the (in my opinion) costly tradeoffs seem to be brushed off with far too much ease. This is often the only reason that I see given: "The only reason not to use Ngrx is if your app is small and simple". This (I would say) is simply incomplete reasoning and not good enough.
Here are my complaints about Ngrx:
You have logic split out into several different files, which makes the code hard to read and understand. This goes against basic code cohesion and locality principles. Having to jump around to different places to read how an operation is performed is mentally taxing and can lead to cognitive overload and exhaustion.
With Ngrx you have to write a lot more code, which increases the chances of bugs. More code -> more places for bugs to appear.
An Ngrx store can become a dumping ground for all things, with no rhyme or reason. It can become a global hodge podge of stuff that no one can get a coherent overview of. It can grow and grow until no one understands it any more.
I've seen a lot of unnecessary deep object cloning in Ngrx apps, which has caused real performance issues. A particular app I was assigned to work on was taking 40 ms to persist data in the store because of deep cloning of a huge store object. This is over two lost render frames if you are trying to hit a smooth 60 fps. Every interaction felt janky because of it.
Most things that Ngrx does can be done much simpler using a basic service/facade pattern that expose observables from rxjs subjects.
Just put methods on services/facades that return observables - such a method replaces the reducer, store, and selector from Ngrx. And then put other methods on the service/facade to trigger data to be pushed on these observables - these methods replace your actions and effects from Ngrx. So instead of reducers+stores+selectors you have methods that return observables. Instead of actions+effects you have methods that produce data the the observables. Where the data comes from is up to you, you can fetch something or compute something, and then just call subject.next() with the data you want to push.
The rxjs knowledge you need in order to use ngrx will already cause you to be competent in using bare rxjs yourself anyways.
If you have several components that depend on some common data, then you still don't need ngrx, as the basic service/facade pattern explicitly handles this already.
If several services depend on common data between them, then you just make a common service between these services. You still don't need ngrx. It's services all the way down, just like it is components all the way down.
For me Ngrx doesn't look so good on the bottom line.
It is essentially a bloated and over engineered Enterprise™🏢👨💼🤮 grade Rxjs Subject, when you could have just used the good old and trusty Rxjs Subject. Listen to me kids, life is too short for unnecessary complexity. Stick to the bare necessities. The simple bare necessities. Forget about your worries and your strife.
I've been working with NgRx for over three years now. I used it on small projects, where it was handy but unnecessary and I used it in applications where it was perfect fit. And meanwhile I had a chance to work on the project which did not use it and I must say it would profit from it.
On the current project I was responsible for designing the architecture of new FE app. I was tasked to completely refactor the existing application which for the same requirements used non NgRx way and it was buggy, difficult to understand and maintain and there was no documentation. I decided to use NgRx there and I did it because of following reasons:
The application has more than one actor over the data. Server uses
the SSE to push state updates which are independent from user
actions.
At the application start we load most of available data which are
then partially updated with SSE.
Various UI elements are enabled/disabled depending on multiple
conditions which come from BE and from user decisions.
UI has multiple variations. Events from BE can change currently
visible UI elements (texts in dialogs) and even user actions might
change how UI looks and works (recurring dialog can be replaced by
snack if user clicked some button).
State of multiple UI elements must be preserved so when user leaves
the page and goes back the same content (or updated via SSE) is
visible.
As you can see the requirements does not meet the standard CRUD operations web page. Doing it the "Angular" way brought such complexity to the code that it became super hard to maintain and what's worst by the time I joined the team the last two original members were leaving without any documentation of that custom made, non NgRx solution.
Now after the year since refactoring the app to use NgRx I think I can sum up the pros and cons.
Pros:
The app is more organized. State representation is easy to read,
grouped by purpose or data origin and is simple to extend.
We got rid of many factories, facades and abstract classes which lost
their purpose. The code is lighter, and components are 'dumber', with
less hidden tricks coming from somewhere else.
Complicated state calculations are made simple using effects and
selectors and most components can be now fully functional just by
injecting the store and dispatching the actions or selecting the
needed slice of the state while handling multiple actions at once.
Because of updated app requirements we were forced to refactor the
store already and it was mostly Ctrl + C, Ctrl + V and some renaming.
Thanks to Redux Dev Tools it is easier to debug and optimize (yep
really)
This is most important - even thought our state itself is unique the
store management we are using is not. It has support, it has
documentation and it's not impossible to find solutions to some
difficult problems on the internet.
Small perk, NgRx is another technology you can put to your CV :)
Cons:
My colleagues were new to the NgRx and it took some time for them to
adapt and fully understand it.
On some occasions we introduced the issue where some actions were
dispatched multiple times and it was difficult to find the cause of
it and fix it
Our Effects are huge, that's true. They can get messy but that's what
we have pull requests for. And if this code wasn't there it would
still end up somewhere else :)
Biggest issue? Actions are differentiated by their string type. Copy
an action, forget to rename it and boom, something different is
happening than you expect, and you have no clue why.
As a conclusion I would say that in our case the NgRx was a great choice. It is demanding at first but later everything feels natural and logical. Also, when you check the requirements, you'll notice that this is a special case. I understand the voices against NgRx and in some cases I would support them but not on this project. Could we have done it using 'Angular' way? Of course, it was done this way before, but it was a mess. It was still full of boiler plate code, things happening in different places without obvious reasons and more.
Anyone who would have the chance to compare those two versions would say the NgRx version is better.
There is also a 3rd option, having data in service and using service directly in html, for instance *ngFor="let item of userService.users". So when you update userService.users in service after add or update action is automatically rendered in html, no need for any observables or events or store.
If the data in your app is used in multiple components, then some kind of service to share the data is required. There are many ways to do this.
A moderately complex app will eventually look like a front end back end structure, with the data handling done in services, exposing the data via observables to the components.
At one point you will need to write some kind of api to your data services, how to get data in and out, queries, etc. Lots of rules like immutability of the data, and well defined single paths to modify the data. Not unlike the server backend, but much quicker and responsive than the api calls.
Your api will end up looking like one of the many state management libraries that already exist. They exist to solve difficult problems. You may not need them if your app is simple.
NGRX sometimes has a lot of files and a lot of duplicate code. Currently working on a fix for this. To make generic type classes for certain NGRX state management situations that are very common inside an Angular project like pagers and object loading from back-ends
Looking into Meteor to create a collaborative document editing app, because it’s great that Meteor synchronizes data between multiple clients by default.
But when using a text-area, like in Sameer Kalburgi’s example
http://www.skalb.com/2012/04/16/creating-a-document-sharing-site-with-meteor-js/
http://docshare-tutorial.meteor.com/
the experience is sub-optimal.
I tried to type at the same time with a colleague and my changes would be overwritten when she typed and vice versa. So in the conflict resolution there is no merge algorithm yet, I think?
Is this planned for the feature? Are there ways to implement this currently? Etherpad seems to handle this problem rather well. Having this in Meteor would make creating collaborative document editing apps way more accessible.
So I looked into it some more, the algorithm used in Etherpad is known as Operational Transformation:
The solution is Operational Transformation (OT). If you haven’t heard of it, OT is a class of algorithms that do multi-site realtime concurrency. OT is like realtime git. It works with any amount of lag (from zero to an extended holiday). It lets users make live, concurrent edits with low bandwidth. OT gives you eventual consistency between multiple users without retries, without errors and without any data being overwritten.
Unfortunately, implementing OT sucks. There's a million algorithms with different tradeoffs, mostly trapped in academic papers. The algorithms are really hard and time consuming to implement correctly. We need some good libraries, so any project can just plug in OT if they need it.
Thats’s from the site of sharejs. A node.js based ot server-client that you can hook into your existing client.
OT is also implemented in the Racer model synchronization engine for Node.js, that forms the underpinnings for Derby. At the moment, derby.js doesn’t transparently provide it yet, but they plan too, from the Derby docs:
Currently, Racer defaults to applying all transactions in the order received, i.e. last-writer-wins. (…) Racer [also] supports conflict resolution via a combination of Software Transactional Memory (STM), Operational Transformation (OT), and Diff-match-patch techniques.
These features are not fully implemented yet, but the Racer demos show preliminary examples of STM and OT.
Coincidentally, both the sharejs and derbyjs teams have an ex Google-waver on board. Meteor has an ex etherpad/Google Waver in their core team. Since Etherpad is one of the best known implementations of OT I was imagining Meteor would surely want to support it at some point as well…
I've created a Meteor smart package that integrates ShareJS:
https://github.com/mizzao/meteor-sharejs
It's quite preliminary right now, but you can import it into your app, drop in textareas, and it "just works". Please try it out and submit some new features via pull requests :)
Check out a demo here:
http://documents.meteor.com
What you describe seems out of Meteors scope for me. Its not a tool to set up collaboration possibilities!
What it provides is a way to transparently work against a subset of a servers database. But the implementation of use-case specific merging functionality is the job of the application, not the framework.
I am new to Magento and impressed by the MVC framework that powers it, making module development a well thought out solution. I am strong CakePHP developer.
I am working on a project that uses a dropshipper for the physical products. As a result, every day at 4am a feed needs to be parsed and the products / categories modified, plus stock information. A CRON will be setup to do this.
Additional requirements are:
Upon a sucessful order, the system must upload a CSV feed to the Dropshipper via FTP with the order details for distribution.
Realtime stock checks, either every hour by CRON or a lookup on the product page
I can think of 2 approaches:
Write everything natively into Magento. As a newbie, this is going to be a big learning curve, but it is the right solution?
Write a simple CakePHP app that runs as a shell. This will use the Magento API to manage all dropshipper processes. This solution will be easier to rollout but introduces an additional system to support.
Does anyone have an advice relating to dropshipping in Magento?
First, with respect to the product import (product, stock data), make sure to do the real data saving inside of Magento. There have been changes to the catalog implementation in the past, and it's likely with a framework like Magento that there will be more. Keeping it inside the framework will reduce the likelihood of it simply no longer operating and you getting a very unpleasant phone call.
Another advantage to this approach is that, in contrast to the API approach, the native code will not try to spin up the entire framework for every request. This is expensive and to be avoided. Depending on how many products there are, you may need to break the script into multiple executions due to memory leaks when saving catalog products.
Don't tie the stock checks to a catalog page view. Some web crawler will come eat your lunch.
Finally, there's no easy FTP library built into Magento, but throwing that on another cronjob and using system calls to perform the actual (S)FTP call is possibly your easiest option.
Hope that helps!
Thanks,
Joe
I think the answer to this question is simple. Write it in what you know. The biggest reason is "UPGRADES"... with Magento being as high profile, the possibility of being hacked with older versions increase every day. Therefore, when they release new versions, you are going to want to upgrade. With that in mind, are you going to want to add all of your changes into each new version as it is released? Probably not. If there is a solution to write this as a separate tool, that is what you should do.
PROS TO BUILDING OUTSIDE OF MAGENTO
No need to re-integrate the upgrades
every time a new version of Magento
is released.
Code is easier to maintain.
Tool is easier to write in something
you are familiar with.
No learning curve.
Integration speed will be much
quicker.
More flexibility since you do not
have to fit inside Magento code
limitations.