I'm using project reactor and I'd like to perform the following:
#Override
public void run(ApplicationArguments args) {
Flux.from(KafkaReceiver.create(receiverOptions)
.receive()
.map(this::getObject)
.flatMap(this::iterateElasticWrites)
.flatMap(this::writeTheWholeObjectToS3)
).subscribe();
}
// What I'd like to do - but a non reactive code
private Publisher<MyObj> iterateElasticWrites(MyObj message) {
for (MyDoc file: message.getDocs()) {
writeElasticDoc(file.getText());
}
return Mono.just(message);
}
I'm struggling in finding out the equivalent of the iterateElasticWrites in Project Reactor. I'd like to perform an iteration of an object of mine (MyObj), and write each of its documents list's element into elasticsearch reactively.
In Reactor you always need to construct a reactive flow using different operators and all reactive/async code should return Mono or Flux.
Looking at your example it could look like
private Mono<MyObj> iterateElasticWrites(MyObj message) {
return Flux.fromIterable(message.getDocs())
.flatMap(doc -> writeElasticDoc(doc.getText()))
.then(Mono.just(message));
}
where writeElasticDoc could be defined as
private Mono<Void> writeElasticDoc(String text) {
...
}
Related
A custom object that takes a parameter of (DocumentSnapShot documentsnapShot). also is an inner object from Firebase that retrieves a snapshot and set the values to my custom model also have its argument (DocumentSnapShot documentsnapShot). However, I wish to get the data from Firebase and pass it to my custom argument because mine takes multiple data not only Firebase. And it's not possible to iterate Firestore without an override.
Here's the code:
public UserSettings getUserSettings(DocumentSnapshot documentSnapshot){
Log.d(TAG, "getUserSettings: retrieving user account settings from firestore");
DocumentReference mSettings = mFirebaseFirestore.collection("user_account_settings").document(userID);
mSettings.get().addOnSuccessListener(new OnSuccessListener<DocumentSnapshot>() {
#Override
public void onSuccess(DocumentSnapshot documentSnapshot) {
UserAccountSettings settings = documentSnapshot.toObject(UserAccountSettings.class);
settings.setDisplay_name(documentSnapshot.getString("display_name"));
settings.setUsername(documentSnapshot.getString("username"));
settings.setWebsite(documentSnapshot.getString("website"));
settings.setProfile_photo(documentSnapshot.getString("profile_photo"));
settings.setPosts(documentSnapshot.getLong("posts"));
settings.setFollowers(documentSnapshot.getLong("followers"));
settings.setFollowing(documentSnapshot.getLong("following"));
}
});
}
You cannot return something now that hasn't been loaded yet. Firestore loads data asynchronously, since it may take some time for this. Depending on your connection speed and the state, it may take from a few hundred milliseconds to a few seconds before that data is available. If you want to pass settings object to another method, just call that method inside onSuccess() method and pass that object as an argument. So a quick fix would be this:
#Override
public void onSuccess(DocumentSnapshot documentSnapshot) {
UserAccountSettings settings = documentSnapshot.toObject(UserAccountSettings.class);
yourMethod(settings);
}
One more thing to mention is that you don't need to set the those values to object that already have them. You are already getting the data from the database as an object.
So remember, onSuccess() method has an asynchronous behaviour, which means that is called even before you are getting the data from your database. If you want to use the settings object outside that method, you need to create your own callback. To achieve this, first you need to create an interface like this:
public interface MyCallback {
void onCallback(UserAccountSettings settings);
}
Then you need to create a method that is actually getting the data from the database. This method should look like this:
public void readData(MyCallback myCallback) {
DocumentReference mSettings = mFirebaseFirestore.collection("user_account_settings").document(userID);
mSettings.get().addOnSuccessListener(new OnSuccessListener<DocumentSnapshot>() {
#Override
public void onSuccess(DocumentSnapshot documentSnapshot) {
UserAccountSettings settings = documentSnapshot.toObject(UserAccountSettings.class);
myCallback.onCallback(settings);
}
});
}
In the end just simply call readData() method and pass an instance of the MyCallback interface as an argument wherever you need it like this:
readData(new MyCallback() {
#Override
public void onCallback(UserAccountSettings settings) {
Log.d("TAG", settings.getDisplay_name());
}
});
This is the only way in which you can use that object of UserAccountSettings class outside onSuccess() method. For more informations, you can take also a look at this video.
Use LiveData as return type and observe the changes of it's value to execute desired operation.
private MutableLiveData<UserAccountSettings> userSettingsMutableLiveData = new MutableLiveData<>();
public MutableLiveData<UserAccountSettings> getUserSettings(DocumentSnapshot documentSnapshot){
DocumentReference mSettings = mFirebaseFirestore.collection("user_account_settings").document(userID);
mSettings.get().addOnSuccessListener(new OnSuccessListener<DocumentSnapshot>() {
#Override
public void onSuccess(DocumentSnapshot documentSnapshot) {
UserAccountSettings settings = documentSnapshot.toObject(UserAccountSettings.class);
settings.setDisplay_name(documentSnapshot.getString("display_name"));
settings.setUsername(documentSnapshot.getString("username"));
settings.setWebsite(documentSnapshot.getString("website"));
settings.setProfile_photo(documentSnapshot.getString("profile_photo"));
settings.setPosts(documentSnapshot.getLong("posts"));
settings.setFollowers(documentSnapshot.getLong("followers"));
settings.setFollowing(documentSnapshot.getLong("following"));
userSettingsMutableLiveData.setValue(settings);
}
});
return userSettingsMutableLiveData;
}
Then from your Activity/Fragment observe the LiveData and inside onChanged do your desired operation.
getUserSettings().observe(this, new Observer<UserAccountSettings>() {
#Override
public void onChanged(UserAccountSettings userAccountSettings) {
//here, do whatever you want on `userAccountSettings`
}
});
I have a repository that returns a flux and wanted to set the result to another object which is expecting a list. Is there any other way to get the results as a list without blocking?
The block is working but it is taking long time.
public class FluxToListTest {
#Autowired PostRepository postRepository;
public void setUserPosts(User user) {
user.setPostList(postRepository.findAllByOrderId(user.getId()).collectList().block());
}
}
interface PostRepository {
Flux<Post> findAllByOrderId(final UUID userId);
}
#Data
class User {
UUID id;
List<Post> postList;
}
class Post {
UUID id;
String content;
}
In short - NO.
You don't need to extract List from Flux.
If you've started use Reactor Streams - stay with it.
Try this code:
public void setUserPosts(User user) {
postRepository.findAllByOrderId(user.getId())
.collectList()
.doOnNext(user::setPostList)// (1)
.subscribe(); // (2)
}
if you set operation is blocking please use publishOn/subscribeOn to avoid blocking for the all stream.
it starts your stream performing
I am trying to build a kind of hub service that can emit through a hot Flux (output) but you can also register/unregister Flux producers/publishers (input)
I know I can do something like:
class Hub<T> {
/**
* #return unregister function
*/
Function<Void, Void> registerProducer(final Flux<T> flux) { ... }
Disposable subscribe(Consumer<? super T> consumer) {
if (out == null) {
// obviously this will not work!
out = Flux.merge(producer1, producer2, ...).share();
}
return out;
}
}
... but as these "producers" are registered and unregistered, how do I add a new flux source to the existing subscribed to flux? or remove a unregistered source from it?
TIA!
Flux is immutable by design, so as you've implied in the question, there's no way to just "update" an existing Flux in situ.
Usually I'd recommend avoiding using a Processor directly. However, this is one of the (rare-ish) cases where a Processor is probably the only sane option, since you essentially want to be publishing elements dynamically based on the producers that you're registering. Something similar to:
class Hub<T> {
private final FluxProcessor<T, T> processor;
private final FluxSink<T> sink;
public Hub() {
this.processor = DirectProcessor.<T>create().serialize();
this.sink = processor.sink();
}
public Disposable registerProducer(Flux<T> flux) {
return flux.subscribe(sink::next);
}
public Flux<T> read() {
return processor;
}
}
If you want to remove a producer, then you can keep track of the Disposable returned from registerProducer() and call dispose() on it when done.
I am new to Reactive programming and Spring WebFlux. I want to make my App 1 publish Server Sent event through Flux and my App 2 listen on it continuously.
I want Flux publish on-demand (e.g. when something happens). All the example I found is to use Flux.interval to periodically publish event, and there seems no way to append/modify the content in Flux once it is created.
How can I achieve my goal? Or I am totally wrong conceptually.
Publish "dynamically" using FluxProcessor and FluxSink
One of the techniques to supply data manually to the Flux is using FluxProcessor#sink method as in the following example
#SpringBootApplication
#RestController
public class DemoApplication {
final FluxProcessor processor;
final FluxSink sink;
final AtomicLong counter;
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
public DemoApplication() {
this.processor = DirectProcessor.create().serialize();
this.sink = processor.sink();
this.counter = new AtomicLong();
}
#GetMapping("/send")
public void test() {
sink.next("Hello World #" + counter.getAndIncrement());
}
#RequestMapping(produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ServerSentEvent> sse() {
return processor.map(e -> ServerSentEvent.builder(e).build());
}
}
Here, I created DirectProcessor in order to support multiple subscribers, that will listen to the data stream. Also, I provided additional FluxProcessor#serialize which provide safe support for multiproducer (invocation from different threads without violation of Reactive Streams spec rules, especially rule 1.3). Finally, by calling "http://localhost:8080/send" we will see the message Hello World #1 (of course, only in case if you connected to the "http://localhost:8080" previously)
Update For Reactor 3.4
With Reactor 3.4 you have a new API called reactor.core.publisher.Sinks. Sinks API offers a fluent builder for manual data-sending which lets you specify things like the number of elements in the stream and backpressure behavior, number of supported subscribers, and replay capabilities:
#SpringBootApplication
#RestController
public class DemoApplication {
final Sinks.Many sink;
final AtomicLong counter;
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
public DemoApplication() {
this.sink = Sinks.many().multicast().onBackpressureBuffer();
this.counter = new AtomicLong();
}
#GetMapping("/send")
public void test() {
EmitResult result = sink.tryEmitNext("Hello World #" + counter.getAndIncrement());
if (result.isFailure()) {
// do something here, since emission failed
}
}
#RequestMapping(produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ServerSentEvent> sse() {
return sink.asFlux().map(e -> ServerSentEvent.builder(e).build());
}
}
Note, message sending via Sinks API introduces a new concept of emission and its result. The reason for such API is the fact that the Reactor extends Reactive-Streams and has to follow the backpressure control. That said if you emit more signals than was requested, and the underlying implementation does not support buffering, your message will not be delivered. Therefore, the result of tryEmitNext returns the EmitResult which indicates if the message was sent or not.
Also, note, that by default Sinsk API gives a serialized version of Sink, which means you don't have to care about concurrency. However, if you know in advance that the emission of the message is serial, you may build a Sinks.unsafe() version which does not serialize given messages
Just another idea, using EmitterProcessor as a gateway to flux
import reactor.core.publisher.EmitterProcessor;
import reactor.core.publisher.Flux;
public class MyEmitterProcessor {
EmitterProcessor<String> emitterProcessor;
public static void main(String args[]) {
MyEmitterProcessor myEmitterProcessor = new MyEmitterProcessor();
Flux<String> publisher = myEmitterProcessor.getPublisher();
myEmitterProcessor.onNext("A");
myEmitterProcessor.onNext("B");
myEmitterProcessor.onNext("C");
myEmitterProcessor.complete();
publisher.subscribe(x -> System.out.println(x));
}
public Flux<String> getPublisher() {
emitterProcessor = EmitterProcessor.create();
return emitterProcessor.map(x -> "consume: " + x);
}
public void onNext(String nextString) {
emitterProcessor.onNext(nextString);
}
public void complete() {
emitterProcessor.onComplete();
}
}
More info, see here from Reactor doc. There is a recommendation from the document itself that "Most of the time, you should try to avoid using a Processor. They are harder to use correctly and prone to some corner cases." BUT I don't know which kind of corner case.
Given a bean like this:
public class MyBean {
private List<Something> things;
private List<Something> internalGetThings() {
if (things == null) {
things = new ArrayList<Something>();
}
return things;
}
public Iterable<Something> getThings() {
return <an immutable copy of internalGetThings()>;
}
public void setThings(List<Something> someThings) {
things.clear();
for (Something aThing : someThings) {
addThing(aThing);
}
}
public void addThing(Something aThing) {
things.add(aThing);
// Do some special stuff to aThing
}
}
Using external mapping file, when I map like this:
<xml-element java-attribute="things" name="thing" type="com.myco.Something" container-type="java.util.ArrayList" />
It seems that each individual Something is being added to the MyBean by calling getThings().add(). That's a problem because getThings() returns an immutable copy of the list, which is lazily initialized. How can I configure mapping (I'm using an external mapping file, not annotations) so that MOXy uses setThings() or addThing() instead?
Why Does JAXB/MOXy Check the Get Method for Collection First?
JAXB (JSR-222) implementations give you a chance to have your property be the List interface and still leverage the underlying List implementation that you choose to use. To accomplish this a JAXB implementation will call the get method to see if the List implementation has been initialized. It it has the List will be populated using the add method.
public List<String> getThings() {
if(null == things) {
things = new ArrayList<String>();
}
return things;
}
public List<String> getThings() {
if(null == things) {
things = new LinkedList<String>();
}
return things;
}
If you don't pre-initialize the List property then MOXy/JAXB will build an instance of the List (default is ArrayList) and set it on the object using the set method.
private List<Something> things; // Don't Initialize
public List<String> getThings() {
return things;
}
public void setThings(List<String> things) {
this.things = things;
}
Given the reason in #Blaise's answer, it doesn't seem possible to have MOXy (or any JAXB implementation in general?) populate a lazily-initialized collection via a setter method on the collection. However, a combination of xml-accessor-type="FIELD" (or #XmlAccessorType if using annotations) and defining a JAXB unmarshal event callback will get the job done. In my afterUnmarshal() implementation I do the special work on Something instances that is done in addSomething().
private void afterUnmarshal(Unmarshaller, Object parent) {
for (Something aThing : getSomethings()) {
// Do special stuff on aThing
}
}
Using FIELD access type gets JAXB/MOXy to directly inject the collection into the field, bypassing the getter. Then the call back cleans things up properly.