Wait for multiple statements to execute with R2DBC - spring-webflux

I need to execute multiple statement with R2DBC but couldn't find useful DatabaseClient#inConnection* examples... So my function keeps getting out too early:
public Publisher<Person> groupStatements(DatabaseClient client, Person person) {
// yes, I know that's harsh, but hey! so is JPA's #ElementCollection
return client.sql("DELETE FROM persons_address WHERE person = :id")
.bind("id", person.getId())
.fetch().rowsUpdated()
.map(deleted -> {
// now recreate every relationship
GenericExecuteSpec statement = client.sql("INSERT INTO persons_address (person, address) VALUES (:person, :address)");
person.getOfficePlaces().forEach(address -> {
statement
.bind("person", person.getId()).bind("address", address.getId())
.fetch().rowsUpdated() // there we go AWOL
.subscribe(inserted -> {
// logging here
});
});
return person; //FIXME wait! need above grouped statements to complete
});
}
NB: I'm using H2 as a backend.
Thanks for any information!

I found a proper batch processing technic (here replacing the map/deleted section), but even got stuck because Statement#execute is returning a Publisher with only #subscribe method and I couldn't return from the chain. So I fed the beast with a few gearing
//DEBUG I couldn't figure out how to use labels! good enough
private static final String SQL_INSERT = "INSERT INTO persons_address (person, address) VALUES ($1, $2)";
...
.flatMap(deleted -> {
if (person.getOfficePlaces().isEmpty()) {
return Mono.just(person);
} else {
return client.inConnection(cnx -> {
Statement stmt = cnx.createStatement(SQL_INSERT);
person.getOfficePlaces().forEach(address -> {
stmt.bind(0, person.getId()).bind(1, address.getId()).add();
});
return Flux.from(stmt.execute()).last().map(dontcare -> person);
});
}

Related

How to extract value from MonoNext (and convert to byte[])

Map<String,Mono<byte[]>> map = new HashMap<>();
List<User> userList = new ArrayList<>();
map.entrySet().stream().forEach(entry -> {
if (entry.getValue() == null) {
log.info("Data not found for key {} ", entry.getKey());
} else {
entry.getValue().log().map(value -> {
try {
return User.parseFrom(value);
} catch (InvalidProtocolBufferException e) {
e.printStackTrace();
}
return null;
}).log().subscribe(p -> userList.add(p));
}
here entry.getValue() => MonoNext
parseFrom(accepts byte[])
I am new to reactive programming world, How to resolve this MonoNext to values it actually have, tried using flatMap instead but that also didnot work
Any suggestion appreciated !! Thanks in advance !!
MonoNext (an internal Reactor implementation of Mono) emits the value asynchronously, which means that it might not have yet the value when evaluated in your code. The only way to retrieve the value is to subscribe to it (either manually or as part of a Reactor pipeline using flatMap and others) and wait until the Mono emits its item.
Here is what your code would look like if placed in a Reactor pipeline using flatMap:
Map<String, Mono<byte[]>> map = new HashMap<>();
List<User> userList = Flux.fromIterable(map.entrySet())
.filter(entry -> entry.getValue() != null)
.doOnDiscard(Map.Entry.class, entry -> log.info("Data not found for key {} ", entry.getKey()))
.flatMap(entry -> entry.getValue()
.log()
.map(User::parseFrom)
.onErrorResume(error -> Mono.fromRunnable(error::printStackTrace)))
.collectList()
.block();
Note that the block operator will wait until all items are retrieved. If you want to stay asynchronous, you can remove the block and return a Mono<List<User>>, or also remove the collectList to return a Flux<User>.

RxJava2 Flowable that emits results of multiple network calls without using create?

I have a generic screen that subscribes to an RxJava2 flowable that returns a List. It then displays the content in the list.
I have a use case now though where I need to collect data from multiple endpoints, and emit data once some complete, and then emit data again once the remaining ones complete.
I'm doing this using Flowable.create() but I've seen a lot of posts saying that there's usually a better and safer way to do so than using create? I seem to believe that is the case since I need to subscribe to an observable within the observable which ideally I wouldn't want to do?
Because I subscribe within, I know the emitter can become cancelled within the observable while other network calls are completing so I've added checks to ensure it doesn't throw an error after its disposed which do work (at least in testing...) [I also just remembered I have the code available to dispose of the inner subscription if I kept it like this, when the outer is disposed]
The first 2 calls may be incredibly fast (or instant) which is why i want to emit the first result right away, and then the following 4 network calls which rely on that data may take time to process.
It looks roughly like this right now...
return Flowable.create<List<Object>>({ activeEmitter ->
Single.zip(
single1(),
single2(),
BiFunction { single1Result: Object, single2result: Object ->
if (single1result.something || single2Result.somethingElse) {
activeEmitter.onNext(function(single1result, single2result) //returns list
}
Single.zip(
single3(single1result),
single4(single2result),
single5(single1result),
single6(single2result),
Function4 { single3Result: Object,
single4Result: Object,
single5Result: Object,
single6Result: Object ->
ObjectHolder(single1Result, single2Result, single3Result, single4Result, single5Result, single6Result)
}
)
}
).flatMap { objectHolder ->
objects.flatMap { objectHolder ->
Single.just(parseObjects(objectHolder))
}
}.subscribeBy(
onError = { error ->
if (!activeEmitter.isCancelled) {
activeEmitter.onError(error)
}
},
onSuccess = { results ->
if (!activeEmitter.isCancelled) {
activeEmitter.onNext(results)
activeEmitter.onComplete()
}
}
)
}, BackpressureStrategy.BUFFER)
I can't figure out another way to return a Flowable that emits the results of multiple different network calls without doing it like this?
Is there a different/better way I can't find?
I worked this out given ctranxuan response. Posting so he can tweak/optimize and then I accept his answer
return Single.zip(single1(), single2(),
BiFunction { single1result: Object, single2result: Object ->
Pair(single1result, single2result)
}
).toFlowable()
.flatMap { single1AndSingle2 ->
if (isFirstLoad) {
createItemOrNull(single1AndSingle2.first, single1AndSingle2.second)?.let { result ->
Single.just(listOf(result)).mergeWith(proceedWithFinalNetworkCalls(single1AndSingle2))
}.orElse {
proceedWithFinalNetworkCalls(single1AndSingle2).toFlowable()
}
} else {
proceedWithFinalNetworkCalls(single1AndSingle2).toFlowable()
}
}.doOnComplete {
isFirstLoad = false
}
fun proceedWithFinalNetworkCalls(): Flowable<List> {
return Single.zip(
single3(single1result),
single4(single2result),
single5(single1result),
single6(single2result),
Function4 { single3Result: Object,
single4Result: Object,
single5Result: Object,
single6Result: Object ->
ObjectHolder(single1Result, single2Result, single3Result, single4Result, single5Result, single6Result)
}
)
Sorry, it's in Java but from what I've understood, something like that may be a possible solution?
public static void main(String[] args) {
final Single<String> single1 = single1().cache();
single1.map(List::of)
.mergeWith(single1.zipWith(single2(), Map::entry)
.flatMap(entry -> Single.zip(
single3(entry.getKey()),
single4(entry.getValue()),
single5(entry.getKey()),
single6(entry.getValue()),
(el3, el4, el5, el6) -> objectHolder(entry.getKey(), entry.getValue(), el3, el4, el5, el6))))
.subscribe(System.out::println,
System.err::println);
Flowable.timer(1, MINUTES) // Just to block the main thread for a while
.blockingSubscribe();
}
private static List<String> objectHolder(final String el1,
final String el2,
final String el3,
final String el4,
final String el5,
final String el6) {
return List.of(el1, el2, el3, el4, el5, el6);
}
static Single<String> single1() {
return Single.just("s1");
}
static Single<String> single2() {
return Single.just("s2");
}
static Single<String> single3(String value) {
return single("s3", value);
}
static Single<String> single4(String value) {
return single("s4", value);
}
static Single<String> single5(String value) {
return single("s5", value);
}
static Single<String> single6(String value) {
return single("s6", value);
}
static Single<String> single(String value1, String value2) {
return Single.just(value1).map(l -> l + "_" + value2);
}
This outputs:
[s1]
[s1, s2, s3_s1, s4_s2, s5_s1, s6_s2]

500 The operation couldn’t be completed. (PostgreSQL.DatabaseError error 1.)

While doing Ray Wenderlich tutorial "Server Side Swift with Vapor: Persisting Models" I tried to add one more parameter(param) to the class Acronyms.
import Vapor
final class Acronym: Model {
var id: Node?
var exists: Bool = false
var short: String
var long: String
var param: String
init(short: String, long: String, param: String) {
self.id = nil
self.short = short
self.long = long
self.param = param
}
init(node: Node, in context: Context) throws {
id = try node.extract("id")
short = try node.extract("short")
long = try node.extract("long")
param = try node.extract("param")
}
func makeNode(context: Context) throws -> Node {
return try Node(node: [
"id": id,
"short": short,
"long": long,
"param": param
])
}
static func prepare(_ database: Database) throws {
try database.create("acronyms") { users in
users.id()
users.string("short")
users.string("long")
users.string("param")
}
}
static func revert(_ database: Database) throws {
try database.delete("acronyms")
}
}
At first I run this code without one more parameter. And it works. But when i added one it fails.
Error: 500The operation couldn’t be completed. (PostgreSQL.DatabaseError error 1.)
My main.swift:
import Vapor
import VaporPostgreSQL
let drop = Droplet(
preparations: [Acronym.self],
providers: [VaporPostgreSQL.Provider.self]
)
drop.get("hello") { request in
return "Hello, world!"
}
drop.get("version") { req in
if let db = drop.database?.driver as? PostgreSQLDriver {
let version = try db.raw("SELECT version()")
return try JSON(node: version)
} else {
return "No db connection"
}
}
drop.get("test") { request in
var acronym = Acronym(short: "AFK", long: "Away From Keyboard", param: "One More Parametr")
try acronym.save()
return try JSON(node: Acronym.all().makeNode())
}
drop.run()
I assume you didn't revert the database. You changed the model's properties, so just write in terminal vapor run prepare --revert . That will revert your database and vapor will be able to create new parameter.
Another case when you done this
vapor run prepare --revert
and the error is still there.
You should to check table name that you create in your prepare method in your model.
static func prepare(_ database: Database) throws {
try database.create(entity) { users in
...
}
}
entity is the name of the table, as far as Vapor/Fluent thinks. By default, it's the name of the model, with an -s on the end.
For example: if you create Car model you should to name your table "cars". So Car+s
static var entity = "cars"
Another example: You have model Carwash that becomes the grammatically incorrect carwashs. So you should to name is carwashs or use entity like this.
static var entity = "carwashes"
I run into the exactly same error, in my prepare method:
public static func prepare(_ database: Database) throws {
try database.create(self.entity) { tasks in
tasks.id()
tasks.string(Identifiers.title)
}
}
I was using self to refer to my database, like this:
public static func prepare(_ database: Database) throws {
try self.database.create(self.entity) { tasks in
tasks.id()
tasks.string(Identifiers.title)
}
}
Apparently, you must access the Database instance passed in as parameter, rather than the static Database variable. Hope this will help someone.

how to write new entity in postCalculateUnitOfWorkChangeSet or preCommitTransaction

I am using Eclipselink and want to write changlog (an entity) for all insert/change/delete business enitities. I found the hooks postCalculateUnitOfWorkChangeSet or preCommitTransaction of SessionEventAdapter where I can build and insert changelog basing on the changeset.
How can I get the changeset for insert/update/delete object in the two methods?
How can I insert new changelog entities with the same unit of work because I need they happen in the same transaction.
What I tried is as below:
#Override
public void postCalculateUnitOfWorkChangeSet(SessionEvent event) {
Object source = event.getSource();
if (source instanceof UnitOfWork) {
UnitOfWork unitOfWork = (UnitOfWork) source;
UnitOfWorkChangeSet unitOfWorkChangeSet = unitOfWork.getUnitOfWorkChangeSet();
if (unitOfWorkChangeSet != null && unitOfWorkChangeSet.hasChanges()) {
Map allChangeSets = unitOfWorkChangeSet.getAllChangeSets();
allChangeSets.forEach((k, v) -> {
if (v instanceof ObjectChangeSet) {
ObjectChangeSet ocs = (ObjectChangeSet) v;
if (ocs.isNew()) {
Object unitOfWorkClone = ocs.getUnitOfWorkClone();
if (unitOfWorkClone instanceof Traceable) {
ChangeLog changeLog = getChangeLog(Operation.INSERT, (Traceable) unitOfWorkClone);
unitOfWork.registerNewObject(changeLog);
unitOfWork.getParent().insertObject(changeLog);
}
} else {
List<ChangeRecord> changes = ((ObjectChangeSet) v).getChanges();
changes.forEach(c -> {
Object unitOfWorkClone = ((ObjectChangeSet) (c.getOwner())).getUnitOfWorkClone();
if (unitOfWorkClone instanceof Traceable) {
ChangeLog changeLog = getChangeLog(Operation.UPDATE, (Traceable) unitOfWorkClone);
unitOfWork.getParent().insertObject(changeLog);
}
});
}
}
}
);
}
}
}
This can work but :
1. I am not sure if it is right way to get the changeset because I did not find document on this.
2. It cannot batch write. When I batch insert business objects, the business obejcts can be written in batch. But the changelogs are inserted one by one.
I also tried history policy. but it cannot work well with batch write.
eclipselink batch write is disabled when use history policy or DescriptorEventAdapter
Thanks a lot for your any comments.

StackExchange.Redis - LockTake / LockRelease Usage

I am using Redis with StackExchange.Redis. I have multiple threads that will at some point access and edit the value of the same key, so I need to synchronize the manipulation of the data.
Looking at the available functions, I see that there are two functions, TakeLock and ReleaseLock. However, these functions take both a key and a value parameter rather than the expected single key to be locked. The intellisene documentation and source on GitHub don't explain how to use the LockTake and LockRelease functions or what to pass in for the key and value parameters.
Q: What is the correct usage of LockTake and LockRelease in StackExchange.Redis?
Pseudocode example of what I'm aiming to do:
//Add Items Before Parallel Execution
redis.StringSet("myJSONKey", myJSON);
//Parallel Execution
Parallel.For(0, 100, i =>
{
//Some work here
//....
//Lock
redis.LockTake("myJSONKey");
//Manipulate
var myJSONObject = redis.StringGet("myJSONKey");
myJSONObject.Total++;
Console.WriteLine(myJSONObject.Total);
redis.StringSet("myJSONKey", myNewJSON);
//Unlock
redis.LockRelease("myJSONKey");
//More work here
//...
});
There are 3 parts to a lock:
the key (the unique name of the lock in the database)
the value (a caller-defined token which can be used both to indicate who "owns" the lock, and to check that releasing and extending the lock is being done correctly)
the duration (a lock intentionally is a finite duration thing)
If no other value comes to mind, a guid might make a suitable "value". We tend to use the machine-name (or a munged version of the machine name if multiple processes could be competing on the same machine).
Also, note that taking a lock is speculative, not blocking. It is entirely possible that you fail to obtain the lock, and hence you may need to test for this and perhaps add some retry logic.
A typical example might be:
RedisValue token = Environment.MachineName;
if(db.LockTake(key, token, duration)) {
try {
// you have the lock do work
} finally {
db.LockRelease(key, token);
}
}
Note that if the work is lengthy (a loop, in particular), you may want to add some occasional LockExtend calls in the middle - again remembering to check for success (in case it timed out).
Note also that all individual redis commands are atomic, so you don't need to worry about two discreet operations competing. For more complexing multi-operation units, transactions and scripting are options.
There is my part of code for lock->get->modify(if required)->unlock actions with comments.
public static T GetCachedAndModifyWithLock<T>(string key, Func<T> retrieveDataFunc, TimeSpan timeExpiration, Func<T, bool> modifyEntityFunc,
TimeSpan? lockTimeout = null, bool isSlidingExpiration=false) where T : class
{
int lockCounter = 0;//for logging in case when too many locks per key
Exception logException = null;
var cache = Connection.GetDatabase();
var lockToken = Guid.NewGuid().ToString(); //unique token for current part of code
var lockName = key + "_lock"; //unique lock name. key-relative.
T tResult = null;
while ( lockCounter < 20)
{
//check for access to cache object, trying to lock it
if (!cache.LockTake(lockName, lockToken, lockTimeout ?? TimeSpan.FromSeconds(10)))
{
lockCounter++;
Thread.Sleep(100); //sleep for 100 milliseconds for next lock try. you can play with that
continue;
}
try
{
RedisValue result = RedisValue.Null;
if (isSlidingExpiration)
{
//in case of sliding expiration - get object with expiry time
var exp = cache.StringGetWithExpiry(key);
//check ttl.
if (exp.Expiry.HasValue && exp.Expiry.Value.TotalSeconds >= 0)
{
//get only if not expired
result = exp.Value;
}
}
else //in absolute expiration case simply get
{
result = cache.StringGet(key);
}
//"REDIS_NULL" is for cases when our retrieveDataFunc function returning null (we cannot store null in redis, but can store pre-defined string :) )
if (result.HasValue && result == "REDIS_NULL") return null;
//in case when cache is epmty
if (!result.HasValue)
{
//retrieving data from caller function (from db from example)
tResult = retrieveDataFunc();
if (tResult != null)
{
//trying to modify that entity. if caller modifyEntityFunc returns true, it means that caller wants to resave modified entity.
if (modifyEntityFunc(tResult))
{
//json serialization
var json = JsonConvert.SerializeObject(tResult);
cache.StringSet(key, json, timeExpiration);
}
}
else
{
//save pre-defined string in case if source-value is null.
cache.StringSet(key, "REDIS_NULL", timeExpiration);
}
}
else
{
//retrieve from cache and serialize to required object
tResult = JsonConvert.DeserializeObject<T>(result);
//trying to modify
if (modifyEntityFunc(tResult))
{
//and save if required
var json = JsonConvert.SerializeObject(tResult);
cache.StringSet(key, json, timeExpiration);
}
}
//refresh exiration in case of sliding expiration flag
if(isSlidingExpiration)
cache.KeyExpire(key, timeExpiration);
}
catch (Exception ex)
{
logException = ex;
}
finally
{
cache.LockRelease(lockName, lockToken);
}
break;
}
if (lockCounter >= 20 || logException!=null)
{
//log it
}
return tResult;
}
and usage :
public class User
{
public int ViewCount { get; set; }
}
var cachedAndModifiedItem = GetCachedAndModifyWithLock<User>(
"MyAwesomeKey", //your redis key
() => // callback to get data from source in case if redis's store is empty
{
//return from db or kind of that
return new User() { ViewCount = 0 };
},
TimeSpan.FromMinutes(10), //object expiration time to pass in Redis
user=> //modify object callback. return true if you need to save it back to redis
{
if (user.ViewCount< 3)
{
user.ViewCount++;
return true; //save it to cache
}
return false; //do not update it in cache
},
TimeSpan.FromSeconds(10), //lock redis timeout. if you will have race condition situation - it will be locked for 10 seconds and wait "get_from_db"/redis read/modify operations done.
true //is expiration should be sliding.
);
That code can be improved (for example, you can add transactions for less count call to cache and etc), but i glad it will be helpfull for you.