This is for nifi 1.3.
I have an executeScript Processor which is connected to a putSplunk Processor through a failure flow. I am doing some testing so currently I make the executeScript Processor fail which causes a flow file to go into the failure flow. The flow file seems to sit in the queue and is not processed or at least not processed fully by the putSplunk processor. If I stop the processor group containing these processors and then start again while a
flow file is stuck in the failure flow queue then the flow file is pushed through and processed in the putSplunk Processor. I have this already running in nifi 1.1 and I do not get this issue. What is going on?
I got it working, just in case anyone runs into this. Seems like it has something to do with the scheduler while I could set the putSplunk Timer to high number of seconds in 1.1 it seems that in 1.3 I need to set the number of seconds to 0 or a low amount of time and it seems to work now. Maybe it does not run for the first time until the number of seconds specified or there are multiple flow files queued. Success and Failure are terminated so that does not matter.
Related
Can someone of you help me, how to make the following service selected in the image get into wait mode after starting the server.
Please let me know if developer trace is required to be posted for resolving this issue.
that particular process is a BATCH process, a process that runs scheduled background tasks (maintained by transaction SM36/SM37). If the process is busy right after starting the server, that means there were scheduled tasks with status released waiting for execution, and as soon as the server was up, it started those tasks.
If you want to make sure the system doesn't immediately start released background tasks, you'll have to set the status back to scheduled (which, thanks to a bit of weird translation, means they won't be executed because they are not released).
if you want to start the server without having a chance to first change the job status in SM37, you would either have to reset the status on database level (likely not officially supported by SAP) or first start the server without any BATCH processes (which would give you a number of great big warning messages upon login) and change the job status before then restarting the server with the BATCH processes. You can set the number of processes for each type in the profile of your instance (parameter rdisp/wp_no_btc).
I am facing a problem while working with Apache Nifi. Is there a way to stop ExecuteSQL processor once it is completed fetching all the data in the table, instead of fetching repeatedly until I stop it manually?
Generally processors are meant to be scheduled on some frequency through their scheduling tab. Processors in the middle of the graph with incoming relationships usually leave their scheduling at 0 seconds, which means run as fast as possible when data is queue. Source processors typically run on some interval based on Timer Driver or Cron Driven scheduling.
That being said... ExecuteSQL supports being triggered by incoming flow files, so you might be able to do something like put a ListenHTTP processor in front of ExecuteSQL and whenever you want to trigger it you would invoke the http end-point for ListenHTTP. This way you can leave it running, but it will only be triggered when you want it to be.
I am running NSB 5 and I am using NHibernate Persistence and have MaximumConcurrencyLevel set to 10.
I have a handler that calls a stored proc that executes an SSIS package. This package takes a non trivial amount of time to run. I started to notice that whenever this particular message type is handled all other message handling stops. I noticed via SQL Profiler that the constant querying of the queue table that NSB does in the background stops and that any extra messages put into the queue are not handled even though NSB is only handling one message.
Is there any guidelines or known issues for dealing with handlers that block the queue because database commands take a long time to complete?
Is sounds like 10 threads are busy, so the endpoint is blocked, can you test this?
I would recommend hosting this message handler in its own process
Make sense?
I do have bunch of xml files say hundreds in my source directory. I have made my flow processing strategy to be synchronous to execute only 1 xml file at a time as performance
is not much priority to me. But i do have batch processing in my flow. So what i under stand is flow thread is creating a child thread to execute my Batch processing and control moves forward. My whole transformation code lies in batch processing which takes 30secs to execute a xml. So nothing much logic in my main flow except file inbound EP and batch execute component(to trigger batch job). So file inbound endpoint is keep on pollingfiles and whole bunch xmls getting picked in very less time make my mule memory out and unexpected behavior occurs.
Came to know fork-join pattern very late and it may or not fit into my req.
So is there any configuration to make my batch process completely and
execute and pick the next files. Help me out. I already made processing strategy synchronous!!
Shouldn't you in this case just adjust the polling frequency at the file inbound endpoint?
https://docs.mulesoft.com/mule-user-guide/v/3.7/file-connector
Polling Frequency
(Applies to inbound File endpoints only.)
Specify how often the endpoint should check for incoming messages. The default value is 1000 ms.
Set maxThreadsActive and maxBufferSize
https://docs.mulesoft.com/mule-user-guide/v/3.6/tuning-performance#calculating-threads
I have a process which reads a file and uploads it to a database. The flow goes as below.
File connector
Processing within a for-each loop (Update to database)
The problem with the above approach was that at any time when an exception occurs, the processing stops at that record and the rest of the records are not processed. As a work around, I have changed the flow as below:
File connector
For each - Within the for-each a flow-ref is placed to call a separate flow which does the processing.
The thing that I noticed is that, at the point of calling the new flow, separate threads are used for processing, due to which an exception does not cause all the records to fail. Now I am facing another difficulty, which is that after the processing completes, I need to initiate a report with the complete processing details (no of records processed, rejected, etc). Since all the records are processed asynchronously in different threads, I am not able to figure out when the processing is completed. Is there a way to monitor whether the processing is complete from another mule flow, so that I can generate the report when it is complete?
at the point of calling the new flow, separate threads are used for processing
A flow-ref doesn't necessarily imply a new thread: you can tune the processing-strategy of the ref-ed flow to force synchronous processing.
With this, you'll be able to remain synchronous, have a custom expression strategy in the ref-ed flow and achieve your goal of not breaking processing when an error occurs.