Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
What are some common best practices one needs to consider while developing Apps in MuleSoft for clients, focusing Anypoint Studio 7.x.x and Mule 4.
List down your recommendations, which you have followed with any clients.
Please Note: I asked this general question to have a dedicated section of MuleSoft Development Best Practices on SO, rather than having similar titles where users having their personal agendas\user driven case scenario.
Mule Developers must considered this to be a critical topic.
Given below are the wide range of MuleSoft best practices which are involved during App development phase.
Development best practices are commonly divided into three categories:
Mule General Best Practices
Mule Munits Best Practices
Mule Error Handling and Exception Scenario Best Practices
Mule General Best Practices
Note: Suggestions are placed in <>. These are just best practices, not a compulsion.
Naming Conventions
Flow and SubFlow Names. <Must use camelCase>
XML files and properties files <Must be all lower case with '-' in between words>
Other common files (Examples, JSON files, Scripts) <Must use camelCase>
All the rest (Components, Transformers, Scopes, Flow Controls). <First letters of each word must be capitalized. Spaces must be used between words>
Property Parameterization
Configuration Properties. <All configurations must be declared as *key=value* in the property files>
Environment Based Properties. <Configuration properties must be segregated into files based on the *Environment* we deploy the app. Example "config-dev.properties", "config-qa.properties", "config-prod.properties">
Runtime Property Variables. <Should be used to fetch appropriate ".properties" files needed for the environment we deploy. Example, name your environment files as "/config-${env}.properties" using Configuration properties in global elements and pass 'env=dev' or 'env=qa' as a Runtime variables in Run Configurations. We can also pass global arguments like 'crypto.key=w4ref$%wrfw3', used to decrypt encrypted values>
Externalizing Transform Message Code to dw Script files. <A common rule of thumb is to use script files when the lines of code is greater than 10>
RAML and Project Files Folder Structuring
Place all the .properties files in src/main/resources/properties
Place all dw script files in src/main/resources/scripts.
Place the RAML examples, libraries, dataTypes, securitySchemes and Traits in dedicated folders while designing in Design Center.
Keep a separate file for API Kit Router and all its generated flow.
Error handling must have its own separate file. Error flows must not be seen anywhere else apart from this file.
Move all Connector Configurations/Global Elements into a separate 'global-config.xml' file. <This keep the rest of mule xml files clean and tidy>
Hard coded values
Must be aware of which code values you must 'Hard code' and which ones not.
Most Global elemental configurations must be property parameterized rather hardcoded. Example, 'Reconnection Strategy, 'Connection Idle Timeouts', 'Max Idle Timeouts', 'host', 'port', 'usernames', 'passwords' and more.
Property Value Encryption. Critical Information must be encrypted. <Using secure-properties-tools.jar or Mule property editor>
Autohide sensitive Property Values passed in Runtime Manager Tab of cloud hub. <Achieved using 'mule-artifact.json'>
Using functions, local/global variables in Transform messages to enforce DRY
Add detailed inline XML comments for flows, choice, etc.
Add descriptive multiline code comments for any complex transformations in Transform messages.
Replace long repeating 'if/else' statements 'with match/case' in data weave.
If flows are getting big using more choice routers. Refactor each choice scope into its own subflow.
A good rule of thumb is that if you have to scroll the Mule canvas back and forth to see the whole flow, it's too complex and should be rewritten.
Avoid Mule Async Scope Calls as much as you can. It caused data integrity issue, based on several developer complaints.
Do not use mule-objectstores for fast-long repeated operations. Know your TPS. Always clear your object stores in your mule cyclic execution in relevance to the requirement from time to time.
Keep a track of each 'variable' initialized in the flow executions. Always make sure to clear or remove variables once finished using them. <Helps you to have a clean process, removing unnecessary code manipulation and heap limitations>
Change your mule loggers from 'INFO' to 'DEBUG' after done development. <Helps you by not over-burdening the Mule APP when deployed in cloudhub. Keeps the mule health monitor in check, so that the APP does not auto restart>
Make sure to never cross an App's 70 percentage CPU usage shown in dashboards. Create Apps accordingly.
18. Be always aware of Data losses caused by Fatal Errors\Application Restarts. <Always use a backup data centers like AWS, Database, Object stores, Mule Load Balancers etc>
Mule Munits Best Practices
Never forget to use Spy and Asserts.
Scenarios based test cases.
Success Scenarios. <Have one major test case to run through the entire API once>
Failure Scenarios. <Have multiple test cases for each flow or subflows, testing for all possible failure situations, like testing mapping, choice routing etc>
Always mock all external service calls, like HTTP, DB, SQS Connectors. <Never call your actual endpoints in Munits>
Consider to put your test payloads in 'src/test/resources/testExample.json' but not directly in the Mocks or Events. <use #[MunitTools::getResourceAsString('testExample.json')]>
Include the files needed under 'src/test/resources' for Munit Test Runs, similar to having 'src/main/resources'.
Mule Error Handling and Exception Scenario Best Practices
All Error status codes must be included appropriately as per the requirement.
Errors must be separately specified in a 'global-error-handling.xml' file.
All Exceptions\Errors must be properly branched as given below
System Exceptions <Source related data exceptions>
Business Exceptions <Target\End System exceptions (Not to be bothered by the Mule APP, but must be handled)>
System\Application Errors
Admiring the usage of object stores and Data Queues for failed messages and record reprocessing.
Having a Retry mechanism for all HTTP based errors.
Can you imagine all the hours of pain we can avoid by following some simple recommendations.
Hope you this helps !
Related
The Scenario
I've been using templates in Cro (documented at https://cro.services/docs/reference/cro-webapp-template), and have enjoyed that there are subs in them.
I currently have a 'main' template, and some reports, let's say report1, report2, and report3.
Let's say that, from report3, I want to include an array of report1.
Now, let's say that the reports each have the following subs:
init: Some Javascript initialisation code (that should only be included once, no matter how many instances of the report are used)
HTML: Some HTML code that should be included for each instance of the report (with a few parameters to differentiate it, but that, due to the restriction of the Javascript framework, may not contain any <script> or <style> tags
data: A snippet of Javascript that again has to be repeated for each time the report is included
Currently I have each of the above in a separate sub in the template.
The Problem
Redeclaration of symbol '&__TEMPLATE_SUB__report-initial'.
The Question
While I can pass a report name (eg. "report1") to the main template, what I'm lacking is a way to have the main template call the subs on the report name that has been passed in, since there may be multiple reports involved.
Ideas I've tried
What would be ideal is if I could somehow create a "report" class that inherits from the template, and pass instances of the template class into the main report, and then call the subs as methods on the report class. However, I've been unable to figure out a way to do this.
I can see three likely options here:
My difficulty may be that I'm not thinking "The Cro Way". If there's a better way to achieve what I'm trying to do, please let me know
There may be a way to achieve what I want, and I've just been unable to understand the documentation (or it may be missing)
While unlikely, it's possible that Cro hasn't been designed with this kind of possibility in mind.
Any help anyone can provide would be appreciated.
Thanks!
Edit: I think a macro that can have multiple (named) "bodies" would solve the problem.
It looks like &__TEMPLATE_SUB__report1-initial is a global that is redeclared when you import report1 into report3. May I suggest to try and use template fragments instead of the whole template?
my initial response to your question is "please can you provide a minimal reproducible example of your code so that we can get a deeper view of the context and have something that we can experiment with"
my current understanding of what you need is "to use raku style classes & objects (with callbacks) in a Cro template setting" - and that the standard ways of doing this such as associative access to a nested topic variable are too limited
in itself, this is not necessarily a weakness of raku / Cro in that the power of a template slang needs to be limited to avoid potential security issues and, as with most template systems, is a bit more prosaic than a full blown coding language
my guess is that Cro template-parts which can chunk up web parts and steps in and out of the (real raku) root block can, depending on how you chunk things up, handle the report data structure that you describe - have you tried this?
if this is still not tenable, there are a couple of ways to expand the options such as dependency injection and route handlers
Just implemented a custom cache store based on the official existing RocksDb one for a different backend store.
That leads me to a number of concerns/questions:
Found out the hard way that PersistenceContextInitializerImpl is auto-generated and had added an import from Eclipse to resolve the issue. Now I have to leave it non-imported and showing as an error in Eclipse, is there a best practice way to handle this?
Why is RocksDBDbStoreTest#testSegmentsRemovedAndAdded call when segmented is false, since this calls removeSegments that contractually should not be called if not segmented?
Same class, why is buildConfig numSegments set or larger than 1 for non-segmented test cases?
Any example of store implementing the NonBlockingStore transactional methods? Mostly wondering to make sure that all calls are from the same thread?
Wanted to disable the compatibility test, since not supported in prior versions. Changed group to unstable or manual and would always still get called, which doesn't seem to match documentation. What is the right way to disable it from build time run?
Are there any kind of performance/stress tests for persistence store that can be executed or adapted?
Found out the hard way that PersistenceContextInitializerImpl is auto-generated and had added an import from Eclipse to resolve the issue. Now I have to leave it non-imported and showing as an error in Eclipse, is there a best practice way to handle this?
There should be a way to have it run the annotation processor. We use IntelliJ and it works fine OOTB.
Why is RocksDBDbStoreTest#testSegmentsRemovedAndAdded call when segmented is false, since this calls removeSegments that contractually should not be called if not segmented?
This is just a side effect of having a parameterized test like that. In actual runtime it won't be invoked. You can just have the test ignore it if it is segmented.
if (segmented) return;
Same class, why is buildConfig numSegments set or larger than 1 for non-segmented test cases?
Infinispan data container is always segmented when running any of the cluster modes. The store however is not required to be segmented in those cases. If the store is not segmented you can ignore any segment parameters as documented.
Any example of store implementing the NonBlockingStore transactional methods? Mostly wondering to make sure that all calls are from the same thread?
You can see some at https://github.com/infinispan/infinispan/blob/main/persistence/jdbc/src/main/java/org/infinispan/persistence/jdbc/stringbased/JdbcStringBasedStore.java#L724. The methods can and will be invoked from different metehods, thus why it stores a Map keyed by the Transaction.
Wanted to disable the compatibility test, since not supported in prior versions. Changed group to unstable or manual and would always still get called, which doesn't seem to match documentation. What is the right way to disable it from build time run?
Not sure I understand the question. For your store you shouldn't need any compatibility test, so just don't copy that test file.
Are there any kind of performance/stress tests for persistence store that can be executed or adapted?
We have https://github.com/infinispan/infinispan-benchmarks/blob/main/cachestores/src/main/java/org/infinispan/jmhbenchmarks/InfinispanHolder.java that should work.
I'm developing a simple web service which is going to add user provided facts to my Prolog database (using assert). I thought it's better to keep these dynamic facts ("data") separate from my service rules that operate on these facts ("code"), hence split them into two different modules. Main reason was that I wanted to persist the dynamic facts to disk periodically, while being able to develop the code with no issues and independently of user data.
I've been using assert(my_store:fact(...)) to add user data to the my_store module and then in the code module I started coding rules like
:- module (my_code, [a_rule/1, ...]).
a_rule(Term) :-
my_store:fact(...), ...
All seems ok but with this approach my_store is hard-coded in the code module, which is a little worrying. For example, what if after some time I decide to change data module name or, perhaps, I'll need two separate data modules one with frequent persistence, another with persistence done only sporadically?
Can anyone advise what are the best practices in code and data organisation? Perhaps separation of code and data is against "the Prolog way"? Are there any good books that cover these issues in depth?
Cheers,
Jacek
That's a good question, touching on several very important topics.
I hope that the following comments help you to sort out most of your questions, possibly more so if you follow up on the points that interest you most with new questions that address the specific question in isolation.
First, when accepting user code as input, make sure you only allow safe code to be added to your programs. In SWI-Prolog, there is safe_goal/1, which helps you to ensure some safety properties. It's not perfect, but better than nothing.
Again in SWI-Prolog, there is library(persistency). Please carefully study the documentation to see how it works, and how you can use it to store dynamic data on disk.
Regarding the module names, I have two comments:
Explicit module qualifications of goals are very rare. Simply load the module and use its clauses.
Note that you can load modules dynamically. That is, nothing prevents you from using use_module/1 with a parameter that was obtained from somewhere else. This allows you more flexibility to specify the module from which you want to fetch your definitions, without having to change and recompile your code.
Regarding separation of code and data: In Prolog, there is no such separation.
All Prolog clauses are Prolog terms.
Terms! TERMS ALL THE WAY DOWN!
Thanks to #mat for his suggestions which made me to read and think a little more. Now, I can post a potential solution to my issue; not ideal, not using the persistency library but a simple, first attempt.
As mentioned, user data are stored with assert(my_store:fact(...)). That means module my_store is created dynamically and there's no file which would allow use_module to be used. There's, however, the import/1 predicate which I can use to import dynamically asserted facts, and so my solution looks like this:
:- module(my_code, [a_rule/1, ...]).
:- initialization import_my_store.
import_my_store :-
import(my_store:fact/1),
import(my_store:another_fact/1),
...
a_rule(Term) :-
fact(...), ...
Note that I can use fact/1 without explicit specification of the my_store module. And I can also easily dump the user data to a file.
save_db(File) :-
tell(File),
my_store:listing,
told.
The downside is that on initialization the import/1 calls generate warnings such as: import/1: my_store:fact/1 is not exported (still imported into my_code). But that's not a big issue because they are still imported into my_code and I can use the user facts without explicit module specification.
Looking forward to hearing any comments. Cheers,
A solution using Logtalk, which provides an alternative to modules. First define an object with your code:
:- object(my_code).
:- public([a_rule/1, ...]).
:- private([fact/1, another_fact/1, ...]).
:- dynamic([fact/1, another_fact/1, ...]).
a_rule(Term) :-
::fact(...), ...
...
:- end_object.
Then, dynamically create any number of data stores as necessary as extensions (derived prototypes) of the my_code object:
?- create_object(my_store, [extends(my_code)], [], []).
To query a data store, simply send it a message:
?- my_store::a_rule(Term).
The create_object/4 built-in predicate can load the persistency file for the store if necessary (so that you resume where you left):
?- create_object(my_store, [extends(my_code)], [include('my_store.pl'))], []).
User data can by saved in a data store by asserting it as expected:
?- my_store::assertz(fact(...)).
You will need a predicate to dump a data store to a file as a public predicate in the my_code object. For example:
:- public(dump/0).
dump :-
self(Self),
atom_concat(Self, '.pl', File),
tell(File),
dump_dynamic_predicates,
told.
dump_dynamic_predicates :-
current_predicate(Functor/Arity),
functor(Template, Functor, Arity),
predicate_property(Template, (dynamic)),
::Template,
write_canonical(Template), write('.\n'),
fail.
dump_dynamic_predicates.
now you can dump a data store, by typing:
?- my_store::dump.
Note that with this solution is trivial to have concurrently any number of data stores. If a data store requires a specialized version of the code, then you can simply extend the code object and then create the specialized data store as an extension of that specialized object.
I am attempting to use jsonix to unmarshal xml response from an SOS DescribeSensor request. In the broader scope I am going to be using jsonix to unmarshal all responses from SOS, particularly 2.0. I noticed that the response uses SML or SensorML namespace and so I added the extra module dependencies and sub-dependencies (namely GML_3_1_1, SWE_1_0_1, IC_2_0, SMIL_2_0, SMIL_2_0_Language, and of course SensorML_1_0_1). Before I added these I noticed the return was a generic json (see first screenshot, particularly near sml:physicalsystem). After I added the dependencies I got an error in my console during part of the unmarshaling process which I do not understand (see second screenshot). Here is a link to the xml response from the server for reference. https://drive.google.com/file/d/0B8LdnPVJpHz7M3VGb0FZc2lQcjQ/view?usp=sharing. I would really like to understand if this has anything to do with the ordering of the modules when I create the context though I believe it is fine. Once the solution to this is discovered I have two follow up questions.
Is it reasonable to expect (in general) that using the modules built from the ogc-schemas on the highsource github page should allow me to handle all responses via jsonix? i.e. every element will always be mapped to a defined type. I know these schemas/mappings are very complicated.
Are there any other tools I can use to verify the modules or validate them against schemas to make life easier rather than tracking down elements on an individual basis or tracing through various module files when jsonix seems to parse incorrectly?
Thanks in advance - Richard3d
var context = new Jsonix.Context([XLink_1_0, GML_3_2_1, IC_2_0, SMIL_2_0, SMIL_2_0_Language, GML_3_1_1, SWE_1_0_1, SensorML_1_0_1, OWS_1_1_0, SWE_2_0, SWES_2_0, WSN_T_1, WS_Addr_1_0_Core, OM_2_0, ISO19139_GMD_20070417, ISO19139_GCO_20070417, ISO19139_GSS_20070417, ISO19139_GTS_20070417, ISO19139_GSR_20070417, Filter_2_0, SOS_2_0]);
Disclaimer: I am the author of jsonix and main dev of ogc-schemas.
First of all, you're on the right track, stay on it.
Yes, if you have all the required mappings then you should get a "nice" JSON with all the properties with specific types, cardinatilities etc.
The goal of Jsonix is to provide bi-directional XML<->JSON conversion with deterministic structure, types and cardinalities.
The goal of OGC Schemas is to provide JAXB and Jsonix mappings for all of the OGC Schemas.
So togethere these two should allow to transform any OGC XMLs from/to JSON.
"Generic JSON" was actually just DOM. If a property allows DOM and Jsonix does not have mapping for certain element, it is just taken as DOM. You were just missing SensorML mappings.
You're right the structure of schema dependencies is very complex. But this is something we should take to OGC. :) It's a bit crazy that you need, like, a dozen of schemas to read sensor data. I was actually intending to build automatic loading of dependencies but did not yet implement this feature.
The next GML_3_1_1.AbstractFeatureType problem is probably this issue. Try changing the order of mappings (move GML_3_1_1 to the earlier places). Actually the order of mappings should not be significant, but, well, there's a bug.
Tools to cross-check - no, probably not. My approach is to do roundtrip tests (unmarshal-marshal-unmarshal-check equality). From experience, there are normally a couple of caveats at the start, but then it works by design. Of course there are bugs in Jsonix and there may be problems with mappings, but this gets sorted out.
Also feel to create a support project here:
https://github.com/highsource/jsonix-support
For instance https://github.com/highsource/jsonix-support/s/sos.
Here's an example of such a support project:
https://github.com/highsource/jsonix-support/tree/master/l/lightstalker89
I need this because just downloading XML from Google Drive (a) takes me effort to set up the support project (b) legally dangerous as I have not idea where this XML comes from and if I have rights/license to add these files to my test suites.
Using the dojo toolkit, what is the proper way of locally testing code that will be executed as cross-domain, without making the actual build?
As it appears, there are three possible options (each, with their own drawbacks):
Using local (non xd) XMLHttpRequest dojo.require
This option does not really test the xd behavior, since it dojo.require[s] the js synchronously via XHR.
djConfig.debugAtAllCosts = true;
Although this option does load the required code asynchronously (via the 'script' tag), it also pulls the code in via XHR, parses the dojo.require[s] inside that, and pulls them in. This (using the loader_debug), again, is not what the loader_xd is doing. More info on this topic in a different question.
Creating a cross-domain build
This approach requires a build, which is not possible in the environment which I'm running the code in (We're using our own on-the-fly build process, which includes only the js that is necessary for a particular page. This process is not suitable for development).
Thus, my question: is there a way to use the loader_xd, which does not require an xd build (which adds the xd prefix / suffix to every file)?
The 2nd way (using the debugAtAllCosts) also makes me question the motivation for pre-parsing the dojo.require[s]. If the loader_xd will not (or rather can not) pre-parse, why is the method that was created for testing/debugging doing so?
peller has described the situation. If you wanted to just generate .xd.js file for your modules, you could look at util/buildscripts/jslib/buildUtilXd.js and its buildUtilXd.xdgen() function.
It would take a bit of work to make your own script, but you could look at util/buildscripts/build.js for pointers.
I am hoping in the future for Dojo (maybe Dojo 2.x timeframe) we can switch to a loader that just uses script tags with a module format that has a function wrapper around the module, something that is coded by the developer. This would allow the same module format to work in the local and xd cases.
I don't think there's any way to do XD loading without building and deploying it. Your analysis of the various options seems about right.
debugAtAllCosts is there specifically to solve a debugging problem, where most browsers, until recently, could not do anything intelligent with code brought in through eval. Still today, Firefox will report exception in the console as appearing at the eval site (bootstrap.js) with a line number offset from the eval, rather than from the actual eval buffer, and normally that eval buffer is anonymous. Firebug was the first debugger to jump through some hoops to enhance the debugging experience and permitted special metadata that Dojo's loader injects between the XHR and the eval to determine a filepath to the source. Webkit/Safari have recently implemented this also. I believe debugAtAllCosts pre-dates the XD loader.