I am trying to learn, how to create a custom NiFi controller service. To start off, I thought of mimicking the DBCPConnectionPool controller service by simply copying the original source code of DBCPConnectionPool service. To implement the same, I generated a maven archetype from "nifi-service-bundle-archetype" and got the following project structure
However, when i generated the archetype from 'nifi-processor-bundle-archetype , I got the following structure: -
I understand that in case of processor I simply need to write my code in MyProceesor.java present under nifi-ListDbTableDemo-processors folder and then create a nar file out of it. But in case of controller service, I have 4 folders generated. I can see two java files i.e.
StandardMyService.java present under nifi-DbcpServiceDemo folder
MyService.java present under nifi-DbcpServiceDemo-apifolder
Now, why is there two java files generated in case of custom controller service, while there was only one java file generated in case of custom processor. Also, Since I am trying to mimick the DBCPConnectionPool service, in which java file out of two should I copy the original source code of DBCPConnectionPool service.
Please guide me from scratch, the steps that I need to follow to create a custom service equivalent to that of DBCPConnectionPool service.
MyService.java under nifi-DbcpServiceDemo-api is an interface which be implemented by the StandardMyService.java under nifi-DbcpServiceDemo. Once the implementation is done, you have to use nifi-DbcpServiceDemo-api as dependency in the processor bundle which needs to work with this custom controller Service.
The reason why controller services are implemented this way is:
We will be hiding the actual implementation from the processor bundle because it need not depend on the implementation.
Tomorrow you write a new controller service implementation, say StandardMyServiceTwo which again implements MyService because only the implementation varies from StandardMyService and other members remains the same and can be shared. This new controller service can be introduced transparently without making any changes on the processor bundle.
Example:
The best example is the record reader/writer controller services. If you look at the nifi-record-serialization-services-bundle in nifi, they have different implementation for serializing records of JSON, Grok, avro, CSV data formats but they all are actually implementing one API - nifi-record-serialization-service-api And hence for the processors which want to use the Record Reader or Record Writer, instead of having the actual implementations as its dependency, they rather can have the api as its dependency.
So tomorrow you can add add a new implementation in the record-serialization-services-bundle for a new data format without touching anything on the processors bundle.
For you references, please take a look at the following links which would help you in writing the custom controller service from scratch
http://www.nifi.rocks/developing-a-custom-apache-nifi-controller-service/
https://github.com/bbende/nifi-dependency-example
Related
I'm creating a new application in Zend Framework 3 and i have a question about a design pattern
Without entering in much details this application will have several Services, as in, will be connecting to external APIs and even in multiple databases, the workflow is also very complex, a single will action can have multiple flows depending on several external information (wich user logged in, configs, etc).
I know about dependency injections and Zend Framework 3 Service Manager, however i am worried about instanciating sereval services when the flow will actually use only a few of them in certain cases, also we will have services depending on other services aswell, for this, i was thinking about using singletons.
Is singleton really a solution here? I was looking a way to user singletons in Zend Framework 3 and haven't figured out a easy way since i can't find a way to user the Service Manager inside a service, as I can't retrive the instance of the Service Manager outside of the Factory system.
What is an easy way to implement singletons in Zend Framework 3?
Why use singletons?
You don't need to worry about too many services in your service manager since they are started only when you get them from the service manager.
Also don't use the service manager inside another class except a factory. In ZF3 it's removed from the controllers for a reason. One of them is testability. If all services are inject with a factory, you can easily write tests. Also if you read your code next year, you can easily see what dependencies are needed inside a class.
If you find there are too many services being injected inside a class which are not always needed you can:
Use the ProxyManager. This lazy loads a service but doesn't start it until a method is called.
Split the service: Move some parts from a service into a new service. e.g. You don't need to place everything in an UserService. You can also have an UserRegisterService, UserEmailService, UserAuthService and UserNotificationsService.
In stead of ZF3, you can also think about zend-expressive. Without getting into too much detail, it is a lightweight middleware framework. You can use middleware to detect what is needed for a request and route to the required action to process the request. Something like this can probably also done in ZF3 but maybe someone else can explain how to do it there.
We use web service Consumers\ Providors quite a bit in our application, both LS and java design elements.
I have a web service (wcf) wsdl that I am trying to import into a LotusScript Web Consumer design element and when saving I am getting the following error:
Not a member: INVOKE
...on line indicated below.
Function GetCopyDestinations(service As XSD_STRING, ocsCode As XSD_STRING, Fault1 As InteropErrorInfoStruct_n1) As ArrayOfInteropMessageDestination_n1
Set GetCopyDestinations = Service.Invoke("GetCopyDestinations", service, ocsCode, Fault1) <<<<=====ERROR ON THIS LINE
End Function
I take this to be that the wsdl --> Web service classes transformer does not like the structure of the wsdl...? I work with the developer who is creating the service we are consuming, so I can get the structure changed, but, from error above, I am not really sure where to start. I assume this means it does not think the Object Service has the method .Invoke().
I've had a look in the file: lsxsd.lss, which LS web service elements include. The "stub" class in the LS web service inherits from PortTypeBase which is in the lsxsd.lss file, PortTypeBase does not have an invoke() method, so I am not quite sure how\ where this method is inherited from. Other LS web services we use successfully, have exactly the same structure, so it would appear this is correct.
The only thing I can think of is to get my colleague to strip the wsdl back to bare bones, then add 1 method, see if it imports, add another, see if it imports, etc.
As a side note, I tried to import the ws into a java consumer, it imports fine. I then construct the stub in exactly the same way I do for all our other java consumer services, which it appears to do successfully. When I run my test harness agent, the stub is created, but it errors when trying to run any of the actions:
java.rmi.RemoteException: No operation description was found for the service
I take this to mean, it can't find the action in the wsdl file, or something similar...? The end point url and PortName appear correct if I output as it is running.
As a side note to this...the service I am attempting to consume above, we have a version working in production now, it is in a LS web consumer, but new one has had new actions added. For a sanity check, I then imported the "live" wsdl...which we know works...into a LS web consumer design element and run a test harness agent, works as expected...good. Then I imported "live" wsdl into a java consumer design element. When running the service, we get the same error...!?
java.rmi.RemoteException: No operation description was found for the service
So now I'm really, really confused. I would like to post the wsdl here, but I think NDA etc., will not allow me, but if it's any help, I am constructing the stub as follows:
ITKInteropV10Stub stub = (ITKInteropV10Stub) new ITKInteropServiceV10_ServiceLocator().getITKInteropV10();
This is a similar stub constructor pattern to all other java web services I consume.
I have been googling\ searching for a couple of days now, searching for errors listed above, I'm really struggling to find a way forward.
Ideally, solving the import in the LS web consumer would work best, since back end logic is currently in LS Script libs which will access this service. If I get it working in java, that's fine as well, it just makes the retrieval of data from the Web Service into the LS libs a bit more complex.
Any comments\ suggestion would be welcomed with open arms!
Regards
Nick
LotusScript is not case sensitive. You have a variable called "service" of XSD_STRING and another object called "Service" which looks like it might be a NotesWebServiceEngine object? (internal LS object).
I want to download xsd specifications from a web service and automatic converting (serialize) these schemas to classes (visual studio - vb.net). If the organization that is responsible for the xsd schemas alter them in a way that only my class corresponding to the xsd have to be altered (not the rest of my code) I would like to automatic update my xsd corresponding class. Is this possible? If so, can somebody tell me how to do it?
Thanks!
I use vs2010. What I want to do is: call a web service where I can send in an input parameter to the service which specifies the xsd I want to retrieve (the service is GetShemaDefenition and returns an object with the schema specification in a string property of the object). I den have to read the xsd string from the string property and convert this to a class representation of this xsd specification. Is it possible to do this automatically? I have done this manually by using xsd.exe. If the owner organization of the xsd has altered the xsd specification, I have to test if there is a new specification, and if there is I have to build a new class representation of this xsd? Is it possible to do what I want? And how would I know if it has been a big change in the xsd which also affect other parts of my code, not just the class representation of the xsd?
Tanks a lot for your reply! So what you are saying, if I understand you correct, is that there is not a good solution for automating this functionality because if the xsd change I most likely (in some occasions’) have to change my code manually? So I have to choose, either in my application or in my intermediate service? But what is the purpose for providing the xsd in a web service? What can I use the web service for? I just wondering, maybe it is clear but I am new to web services and is eager to learn more.
Update:
Thanks! But can you explain a little bit more. What I have to do is: I use one web service where one of the properties is a string. The string is an XML inside a CDATA block. The organization which provides the web service will not pares the xml inside the CDATA block but instead forward this to another organization that will use the xml data. The organization which uses the xml data specifies the xsd schem that I have to follow to generate my xml correct. This is the xsd schema I can get from another web service. I don’t really understand what I can do with this xsd file from the web service. What can I do with it and why do I want to download it from the web service, when I can’t use it automatically? Because I have to manually do the changes when the xsd changes I can easily download the xsd schema from the organization’s home page and make the new class with xsd.exe. I understand there is something I don’t understand :o), can you pleas clarify?
What visual studio version you are using?, Normally you can click on the project's references and Add Web service. In this case Visual studio creates automatically the objects required to consume the service. you can update it any time by a right click on the reference.
However if it is very likely to change often, One solution is to implement an adapter class. use create an interface that provides the same functionality and call the actual web service. In your application you use only the proxy class and not the Web Service. Later when the web service interface changes all you have to do is to change the internals of this intermediate class.
Update:
you can use this tool to create you object model in code. Then you can compile your new object model and use it in you application. There are many complications in what you want to do and the bottom line is; when the object model changes, your code will fail. There is absolutely no way to imagine how the interface will change so while you can do all that automatically there is nothing to do if the name of a function changes.
However the answer to your situation is indirection. If you can't guaranty the stability of a external service. Why not create a stable intermediate service that calls the actual one? this way in future you don't need to touch you application. All you have to do is to modify the intermediate service while keeping it's interface compatible.
We have a dynamically composed application, in which user can add services and operations. This application is installed on a server cluster.
Since adding services to application involves so much writing to web.config, i was wondering if its possible to read system.servicemodel section from a database instead of web.config.
Seems like microsoft's implementation of configuration is very tightly coupled with where its stored.
There is no "out-of-the-box" way to do that. However, it is possible.
Few feet below, Configuration class uses FileStream instance where it actually can use any Stream. That particular step can be replaced with custom implementation of IInternalConfigHost interface (a lot of properties and methods to implement there).
Particularly interesting are OpenStreamForRead and OpenStreamForWrite, both are returning Stream instances. There you can put logic to pull XML of configuration sections from database into ConfigurationSection instances and to put ConfigurationSection instances as XML into database.
The next step is to create an instance of Configuration class. However, here we must get dirty because its constructor never leaves the System.Configuration kingdom. The need to use reflection to reach and use it. I suggest implementation of IInternalConfigConfigurationFactory to wrap the reflection magic.
Configuration Create( Type typeConfigHost,
params object[] hostInitConfigurationParams );
As first parameter pass the type of implemented configuration host.
After we have Configuration instance, we can use it a custom ServiceHost, ChannelFactory<T> and DuplexChannelFactory<T>.
I'm just starting to play with MEF and have a couple questions.
1) I wrote a WCF service that takes in some xml and passes the xml off to a parser. The parsers are composed using MEF. (metadata in the xml lets me determine which parser to use). I can add a new parser, and support new XML, by just dropping the dll in a directory. That part all works. But, WCF services can be instantiated multiple times, I want my parser catalog to be static, that is, if multiple instances of my service are spun up, and they get the same XML, I only need one instance of the parser running, they are written to be thread safe. I can't seem to configure MEF to do this. Anyone know how?
2) I can drop in a new parser into the directory and a catalog refresh will automatically discover it, that works great. But if I try to drop a modified dll into the directory, and that parser has been activated in the service, I get an error saying the file is in use. Is there a way to override this?
1) It sounds like you should make your MEF container and catalogs static so they only get created once. Make sure you specify that the CompositionContainer should be thread safe by using the constructor with the isThreadSafe parameter and setting it to true.
2) You can enable shadow copying which will prevent the file from being locked when the DLL is loaded. However, you can't unload DLLs from an AppDomain in .NET, and furthermore it is not safe to recompose a CompositionContainer that can be used on multiple threads. In other words, using the isThreadSafe parameter only makes the container thread-safe for "reading"/pulling exports from the container, not modifying it via composition/recomposition.
So if you want to add a new parser it's probably best to restart the service.