I'm using RxJava to create a background job syncronizing my db.
It connects to an external source and start to process entries, map them and insert in the db.
When it ends I need the list with all the elements processed, I can get it when everything goes right, but how can I collect all the elements processed if during the flow something fail?
final List<String> res = Observable.create(onSubscribe)
.buffer(4)
.flatMap(TestRx::doStuff)
.buffer(8)
.map(TestRx::calculateList)
.toList()
.toBlocking()
.single();
System.out.println("strings = " + res);
What I would like to have is a way that if doStuff or calculateList throw exceptions, the flow stop an returns the list with everything it processed until the error.
List<String> res = Observable.create(onSubscribe)
.buffer(4)
.flatMap(TestRx::doStuff)
.onErrorResumeNext(Observable.empty()) // turn error into completion
.buffer(8)
.map(TestRx::calculateList)
.onErrorResumeNext(Observable.empty()) // turn error into completion
.toList()
.toBlocking()
.single();
System.out.println("strings = " + res);
Related
I'm actually developing a project that read data from 19 PLCs Siemens S1500 and 1 modicon. I have used the scraper tool following this tutorial:
PLC4x scraper tutorial
but when the scraper is working for a little amount of time I get the following exception:
I have changed the scheduled time between 1 to 100 and I always get the same exception when the scraper reach the same number of received messages.
I have tested if using PlcDriverManager instead of PooledPlcDriverManager could be a solution but the same problem persists.
In my pom.xml I use the following dependency:
<dependency>
<groupId>org.apache.plc4x</groupId>
<artifactId>plc4j-scraper</artifactId>
<version>0.7.0</version>
</dependency>
I have tried to change the version to an older one like 0.6.0 or 0.5.0 but the problem still persists.
If I use the modicon (Modbus TCP) I also get this exception after a little amount of time.
Anyone knows why is happening this error? Thanks in advance.
Edit: With the scraper version 0.8.0-SNAPSHOT I continue having this problem.
Edit2: This is my code, I think the problem can be that in my scraper I am opening a lot of connections and when it reaches 65526 messages it fails. But since all the processing is happenning inside the lambda function and I'm using a PooledPlcDriverManager, I think the scraper is using only one connection so I dont know where is the mistake.
try {
// Create a new PooledPlcDriverManager
PlcDriverManager S7_plcDriverManager = new PooledPlcDriverManager();
// Trigger Collector
TriggerCollector S7_triggerCollector = new TriggerCollectorImpl(S7_plcDriverManager);
// Messages counter
AtomicInteger messagesCounter = new AtomicInteger();
// Configure the scraper, by binding a Scraper Configuration, a ResultHandler and a TriggerCollector together
TriggeredScraperImpl S7_scraper = new TriggeredScraperImpl(S7_scraperConfig, (jobName, sourceName, results) -> {
LinkedList<Object> S7_results = new LinkedList<>();
messagesCounter.getAndIncrement();
S7_results.add(jobName);
S7_results.add(sourceName);
S7_results.add(results);
logger.info("Array: " + String.valueOf(S7_results));
logger.info("MESSAGE number: " + messagesCounter);
// Producer topics routing
String topic = "s7" + S7_results.get(1).toString().substring(S7_results.get(1).toString().indexOf("S7_SourcePLC") + 9 , S7_results.get(1).toString().length());
String key = parseKey_S7("s7");
String value = parseValue_S7(S7_results.getLast().toString(),S7_results.get(1).toString());
logger.info("------- PARSED VALUE -------------------------------- " + value);
// Create my own Kafka Producer
ProducerRecord<String, String> record = new ProducerRecord<String, String>(topic, key, value);
// Send Data to Kafka - asynchronous
producer.send(record, new Callback() {
public void onCompletion(RecordMetadata recordMetadata, Exception e) {
// executes every time a record is successfully sent or an exception is thrown
if (e == null) {
// the record was successfully sent
logger.info("Received new metadata. \n" +
"Topic:" + recordMetadata.topic() + "\n" +
"Partition: " + recordMetadata.partition() + "\n" +
"Offset: " + recordMetadata.offset() + "\n" +
"Timestamp: " + recordMetadata.timestamp());
} else {
logger.error("Error while producing", e);
}
}
});
}, S7_triggerCollector);
S7_scraper.start();
S7_triggerCollector.start();
} catch (ScraperException e) {
logger.error("Error starting the scraper (S7_scrapper)", e);
}
So in the end indeed it was the PLC that was simply hanging up the connection randomly. However the NiFi integration should have handled this situation more gracefully. I implemented a fix for this particular error ... could you please give version 0.8.0-SNAPSHOT a try (or use 0.8.0 if we happen to have released it already)
Is there an option to have variant of timeout that does not emit Throwable?
I would like to have complete event emited.
You don't need to map errors with onErrorResumeNext. You can just provide a backup observable using:
timeout(long,TimeUnit,Observable)
It would be something like:
.timeout(500, TimeUnit.MILLISECONDS, Observable.empty())
You can resume from an error with another Observable, for example :
Observable<String> data = ...
data.timeout(1, TimeUnit.SECONDS)
.onErrorResumeNext(Observable.empty())
.subscribe(...);
A simpler solution that does not use Observable.timeout (thus it does not generate an error with the risk of catching unwanted exceptions) might be to simply take until a timer completes:
Observable<String> data = ...
data.takeUntil(Observable.timer(1, TimeUnit.SECOND))
.subscribe(...);
You can always use onErrorResumeNext which will get the error and you can emit whatever item you want-
/**
* Here we can see how onErrorResumeNext works and emit an item in case that an error occur in the pipeline and an exception is propagated
*/
#Test
public void observableOnErrorResumeNext() {
Subscription subscription = Observable.just(null)
.map(Object::toString)
.doOnError(failure -> System.out.println("Error:" + failure.getCause()))
.retryWhen(errors -> errors.doOnNext(o -> count++)
.flatMap(t -> count > 3 ? Observable.error(t) : Observable.just(null)),
Schedulers.newThread())
.onErrorResumeNext(t -> {
System.out.println("Error after all retries:" + t.getCause());
return Observable.just("I save the world for extinction!");
})
.subscribe(s -> System.out.println(s));
new TestSubscriber((Observer) subscription).awaitTerminalEvent(500, TimeUnit.MILLISECONDS);
}
template.setEnableTransactionSupport(true);
template.multi();
template.opsForValue().set("mykey", "Hello World");
List<String> dataList = template.opsForList().range("mylist", 0, -1);
template.exec();
Hi guys.
I have a list called "mylist' in my redis and its size is 50.
But when I run this code, I can't get what I want.
The field "dataList" is null, however, "mykey" with the value "Hello World" have persisted in my redis.
So how can I get my list data in a spring-data-redis transaction? many thanks.
The transaction suppport in SD-Redis helps participating in ongoing transactions and allows automatic commit (exec) / rollback (discard), so it helps wrapping commands into thread bound multi exec blocks using the same connection.
More generally redis transactions and the commands within a transaction get queued on server side and return a list of results on exec.
template.multi();
// queue set command
template.opsForValue().set("mykey", "Hello World");
// queue range command
List<String> dataList = template.opsForList().range("mylist", 0, -1);
// execute queued commands
// result[0] = OK
// result[1] = {"item-1", "item-2", "item-", ...}
List<Object> result = template.exec();
I'm performing API testing of basic CRUD functionality. For the creation of each record type in each thread group, I am using the same BeanShell Assertion template customized to each thread group.
import org.apache.jmeter.services.FileServer;
if (ResponseCode != null && ResponseCode.equals("200") == true) {
SampleResult.setResponseOK();
}
else if (!ResponseCode.equals ("200") == true ) {
Failure = true;
FailureMessage ="Creation of a new {insert record type} record failed. Response code " + ResponseCode + "." ; // displays in Results Tree
print ("Creation of a new {insert record type} record failed: Response code " + ResponseCode + "."); // goes to stdout
log.warn("Creation of a new {insert record type} record failed: Response code " + ResponseCode); // this goes to the JMeter log file
// Static elements or calculations
part1 = "\n FAILED TO CREATE NEW {insert record type} RECORD via POST. The response code is: \"";
part2 = "\". \n\n - For \'Non-HTTP response code - org.apache.jorphan.util.JMeterStopThreadException\' is received,verify the payload file still exists. \n - For response code = 409, \n\t a) check the payload for validity.\n\t b) verify the same {insert record type} name doesn't already exist in the {insert table name} table. If found, delete record and re-run the test. \n - For response code = 500, verify the database and its host server are reachable. \n";
// Open File(s)
FileOutputStream f = new FileOutputStream(FileServer.getFileServer().getBaseDir() + "\\error.log", true);
//FileOutputStream f = new FileOutputStream("c:\\error.log", true);
PrintStream p = new PrintStream(f);
// Write data to file
p.println( part1 + ResponseCode + part2 );
// Close File(s)
p.close();
f.close();
}
Is there a way to make this re-callable as opposed to having to repeat it in each thread group? Right now I'm up to 20 thread groups, thus 20 versions of this same assertion.
I've looked at multiple pages on this site and also at How to Use BeanShell: JMeter's Favorite Built-in Component, but I'm not finding a solution to this. Any feedback is appreciated.
If you place your Assertion (any Assertion) at the same level as Thread Groups like:
It will be applied to both "Sampler 1" and "Sampler 2". Moreover, the assertion will be applied to each sampler in each thread group on each iteration.
See How to Use JMeter Assertions in Three Easy Steps guide which clarifies assertions scope, cost and best practices.
I am learning to use the Rx extensions for a Silverlight 4 app I am working on. I created a sample app to nail down the process and I cannot get it to return anything.
Here is the main code:
private IObservable<Location> GetGPSCoordinates(string Address1)
{
var gsc = new GeocodeServiceClient("BasicHttpBinding_IGeocodeService") as IGeocodeService;
Location returnLocation = new Location();
GeocodeResponse gcResp = new GeocodeResponse();
GeocodeRequest gcr = new GeocodeRequest();
gcr.Credentials = new Credentials();
gcr.Credentials.ApplicationId = APP_ID2;
gcr.Query = Address1;
var myFunc = Observable.FromAsyncPattern<GeocodeRequest, GeocodeResponse>(gsc.BeginGeocode, gsc.EndGeocode);
gcResp = myFunc(gcr) as GeocodeResponse;
if (gcResp.Results.Count > 0 && gcResp.Results[0].Locations.Count > 0)
{
returnLocation = gcResp.Results[0].Locations[0];
}
return returnLocation as IObservable<Location>;
}
gcResp comes back as null. Any thoughts or suggestions would be greatly appreciated.
The observable source you are subscribing to is asynchronous, so you can't access the result immediately after subscribing. You need to access the result in the subscription.
Better yet, don't subscribe at all and simply compose the response:
private IObservable<Location> GetGPSCoordinates(string Address1)
{
IGeocodeService gsc =
new GeocodeServiceClient("BasicHttpBinding_IGeocodeService");
Location returnLocation = new Location();
GeocodeResponse gcResp = new GeocodeResponse();
GeocodeRequest gcr = new GeocodeRequest();
gcr.Credentials = new Credentials();
gcr.Credentials.ApplicationId = APP_ID2;
gcr.Query = Address1;
var factory = Observable.FromAsyncPattern<GeocodeRequest, GeocodeResponse>(
gsc.BeginGeocode, gsc.EndGeocode);
return factory(gcr)
.Where(response => response.Results.Count > 0 &&
response.Results[0].Locations.Count > 0)
.Select(response => response.Results[0].Locations[0]);
}
If you only need the first valid value (the location of the address is unlikely to change), then add a .Take(1) between the Where and Select.
Edit: If you want to specifically handle the address not being found, you can either return results and have the consumer deal with it or you can return an Exception and provide an OnError handler when subscribing. If you're thinking of doing the latter, you would use SelectMany:
return factory(gcr)
.SelectMany(response => (response.Results.Count > 0 &&
response.Results[0].Locations.Count > 0)
? Observable.Return(response.Results[0].Locations[0])
: Observable.Throw<Location>(new AddressNotFoundException())
);
If you expand out the type of myFunc you'll see that it is Func<GeocodeRequest, IObservable<GeocodeResponse>>.
Func<GeocodeRequest, IObservable<GeocodeResponse>> myFunc =
Observable.FromAsyncPattern<GeocodeRequest, GeocodeResponse>
(gsc.BeginGeocode, gsc.EndGeocode);
So when you call myFunc(gcr) you have an IObservable<GeocodeResponse> and not a GeocodeResponse. Your code myFunc(gcr) as GeocodeResponse returns null because the cast is invalid.
What you need to do is either get the last value of the observable or just do a subscribe. Calling .Last() will block. If you call .Subscribe(...) your response will come thru on the call back thread.
Try this:
gcResp = myFunc(gcr).Last();
Let me know how you go.
Richard (and others),
So I have the code returning the location and I have the calling code subscribing. Here is (hopefully) the final issue. When I call GetGPSCoordinates, the next statement gets executed immediately without waiting for the subscribe to finish. Here's an example in a button OnClick event handler.
Location newLoc = new Location();
GetGPSCoordinates(this.Input.Text).ObserveOnDispatcher().Subscribe(x =>
{
if (x.Results.Count > 0 && x.Results[0].Locations.Count > 0)
{
newLoc = x.Results[0].Locations[0];
Output.Text = "Latitude: " + newLoc.Latitude.ToString() +
", Longtude: " + newLoc.Longitude.ToString();
}
else
{
Output.Text = "Invalid address";
}
});
Output.Text = " Outside of subscribe --- Latitude: " + newLoc.Latitude.ToString() +
", Longtude: " + newLoc.Longitude.ToString();
The Output.Text assignment that takes place outside of Subscribe executes before the Subscribe has finished and displays zeros and then the one inside the subscribe displays the new location info.
The purpose of this process is to get location info that will then be saved in a database record and I am processing multiple addresses sequentially in a Foreach loop. I chose Rx Extensions as a solution to avoid the problem of the async callback as a coding trap. But it seems I have exchanged one trap for another.
Thoughts, comments, suggestions?