We have four specific products with a massive amount of variants. When running the Product Indexer we run out of memory because of these products.
So we want to exclude these specific products from the Product Indexer Job.
My first approach was to use the ProductIndexerEvent, but the event is dispatched at the end of the handle() method :
(vendor/shopware/core/Content/Product/DataAbstractionLayer/ProductIndexer.php:187),
which is probably too late.
What is the best approach to implement that behaviour?
I would advise against excluding products from being indexed. There's business logic relying on the data being indexed.
If you're confident in what you're doing and know about the consequences, you could decorate the ProductIndexer service.
<service id="Foo\MyPlugin\ProductIndexerDecorator" decorates="Shopware\Core\Content\Product\DataAbstractionLayer\ProductIndexer">
<argument type="service" id="Foo\MyPlugin\ProductIndexerDecorator.inner"/>
</service>
In the decorator you would have to deconstruct the original event, filter the WriteResult instances by excluded IDs and then pass the reconstructed event to the decorated service.
class ProductIndexerDecorator extends EntityIndexer
{
const FILTERED_IDS = ['9b180c61ddef4dad89e9f3b9fa13f3be'];
private EntityIndexer $decorated;
public function __construct(EntityIndexer $decorated)
{
$this->decorated = $decorated;
}
public function getDecorated(): EntityIndexer
{
return $this->decorated;
}
public function getName(): string
{
return $this->getDecorated()->getName();
}
public function iterate($offset): ?EntityIndexingMessage
{
return $this->getDecorated()->iterate($offset);
}
public function update(EntityWrittenContainerEvent $event): ?EntityIndexingMessage
{
$originalEvents = clone $event->getEvents();
if (!$originalEvents) {
return $this->getDecorated()->update($event);
}
$event->getEvents()->clear();
/** #var EntityWrittenEvent $writtenEvent */
foreach ($originalEvents as $writtenEvent) {
if ($writtenEvent->getEntityName() !== 'product') {
$event->getEvents()->add($writtenEvent);
continue;
}
$results = [];
foreach ($writtenEvent->getWriteResults() as $result) {
if (\in_array($result->getPrimaryKey(), self::FILTERED_IDS, true)) {
continue;
}
$results[] = $result;
}
$event->getEvents()->add(new EntityWrittenEvent('product', $results, $event->getContext()));
}
return $this->getDecorated()->update($event);
}
public function handle(EntityIndexingMessage $message): void
{
$data = array_diff($message->getData(), self::FILTERED_IDS);
$newMessage = new ProductIndexingMessage($data, $message->getOffset(), $message->getContext(), $message->forceQueue());
$this->getDecorated()->handle($newMessage);
}
public function getTotal(): int
{
return $this->getDecorated()->getTotal();
}
public function getOptions(): array
{
return $this->getDecorated()->getOptions();
}
}
I'm trying to translate a ParameterizedTest from JUnit4 to JUnit5 (sadly I'm not particularly skilled in testing).
In JUnit4 I have the following class:
#RunWith(Parameterized.class)
public class AssertionTestCase {
private final TestInput testInput;
public AssertionTestCase(TestInput testInput) {
this.testInput = testInput;
}
#Parameterized.Parameters
public static Collection<Object[]> data() {
return AssertionTestCaseDataProvider.createDataCase();
}
#Test(timeout = 15 * 60 * 1000L)
public void testDailyAssertion() {
LOG.info("Testing input {}/{}", testInput.getTestCase(), testInput.getTestName());
//assert stuffs
}
}
in the AssertionTestCaseDataProvider class I have a simple method generating a collection of Object[]:
class AssertionTestCaseDataProvider {
static Collection<Object[]> createDataCase() {
final List<TestInput> testInputs = new ArrayList<>();
//create and populate testInputs
return testInputs.stream()
.map(testInput -> new Object[]{testInput})
.collect(Collectors.toList());
}
}
I've been trying to translate it using JUnit5 and obtained this:
class AssertionTestCase {
private final TestInput testInput;
public AssertionTestCase(TestInput testInput) {
this.testInput = testInput;
}
public static Collection<Object[]> data() {
return AssertionTestCaseDataProvider.createDataCase();
}
#ParameterizedTest
#MethodSource("data")
void testDailyAssertion() {
LOG.info("Testing input {}/{}", testInput.getTestCase(), testInput.getTestName());
// assert stuffs
}
}
I did not apply any change to the AssertionTestCaseDataProvider class.
Nevertheless, I'm getting the following error:
No ParameterResolver registered for parameter [com.xxx.xx.xxx.xxx.testinput.TestInput arg0] in constructor [public `com.xxx.xxx.xxx.xxx.xxx.AssertionTestCase(com.xxx.xxx.xxx.xxx.testinput.TestInput)]. org.junit.jupiter.api.extension.ParameterResolutionException: No ParameterResolver registered for parameter [com.xxx.xx.xxx.xxx.testinput.TestInput arg0] in constructor [public com.xxx.xxx.xxx.xxx.xxx.AssertionTestCase(com.xxx.xxx.xxx.xxx.testinput.TestInput)].`
I understand I'm probably not applying correctly JUnit5 when initializing the input collection for the test. Am I missing some annotations?
I've also tried to use #ArgumentSource instead of #MethodSource and implementing Argument for AssertionTestCaseDataProvider, with the same failing results.
It works in a bit another way in Junit5.
Test Method should have parameters, and provider method should return a Stream.
static Stream<Arguments> data(){
return Stream.of(
Arguments.of("a", 1),
Arguments.of("d", 2)
);
}
#ParameterizedTest
#MethodSource("data")
void testDailyAssertion(String a, int b) {
Assertions.assertAll(
() -> Assertions.assertEquals("a", a),
() -> Assertions.assertEquals(1, b)
);
}
In your case you can just return a Stream<TestInput>:
static Stream<TestInput> createDataCase() {
final List<TestInput> testInputs = new ArrayList<>();
//create and populate testInputs
return testInputs.stream();
}
and then in your testMethod:
#ParameterizedTest
#MethodSource("createDataCase")
void testDailyAssertion(TestInput testInput) {
{your assertions}
}
I have a generic screen that subscribes to an RxJava2 flowable that returns a List. It then displays the content in the list.
I have a use case now though where I need to collect data from multiple endpoints, and emit data once some complete, and then emit data again once the remaining ones complete.
I'm doing this using Flowable.create() but I've seen a lot of posts saying that there's usually a better and safer way to do so than using create? I seem to believe that is the case since I need to subscribe to an observable within the observable which ideally I wouldn't want to do?
Because I subscribe within, I know the emitter can become cancelled within the observable while other network calls are completing so I've added checks to ensure it doesn't throw an error after its disposed which do work (at least in testing...) [I also just remembered I have the code available to dispose of the inner subscription if I kept it like this, when the outer is disposed]
The first 2 calls may be incredibly fast (or instant) which is why i want to emit the first result right away, and then the following 4 network calls which rely on that data may take time to process.
It looks roughly like this right now...
return Flowable.create<List<Object>>({ activeEmitter ->
Single.zip(
single1(),
single2(),
BiFunction { single1Result: Object, single2result: Object ->
if (single1result.something || single2Result.somethingElse) {
activeEmitter.onNext(function(single1result, single2result) //returns list
}
Single.zip(
single3(single1result),
single4(single2result),
single5(single1result),
single6(single2result),
Function4 { single3Result: Object,
single4Result: Object,
single5Result: Object,
single6Result: Object ->
ObjectHolder(single1Result, single2Result, single3Result, single4Result, single5Result, single6Result)
}
)
}
).flatMap { objectHolder ->
objects.flatMap { objectHolder ->
Single.just(parseObjects(objectHolder))
}
}.subscribeBy(
onError = { error ->
if (!activeEmitter.isCancelled) {
activeEmitter.onError(error)
}
},
onSuccess = { results ->
if (!activeEmitter.isCancelled) {
activeEmitter.onNext(results)
activeEmitter.onComplete()
}
}
)
}, BackpressureStrategy.BUFFER)
I can't figure out another way to return a Flowable that emits the results of multiple different network calls without doing it like this?
Is there a different/better way I can't find?
I worked this out given ctranxuan response. Posting so he can tweak/optimize and then I accept his answer
return Single.zip(single1(), single2(),
BiFunction { single1result: Object, single2result: Object ->
Pair(single1result, single2result)
}
).toFlowable()
.flatMap { single1AndSingle2 ->
if (isFirstLoad) {
createItemOrNull(single1AndSingle2.first, single1AndSingle2.second)?.let { result ->
Single.just(listOf(result)).mergeWith(proceedWithFinalNetworkCalls(single1AndSingle2))
}.orElse {
proceedWithFinalNetworkCalls(single1AndSingle2).toFlowable()
}
} else {
proceedWithFinalNetworkCalls(single1AndSingle2).toFlowable()
}
}.doOnComplete {
isFirstLoad = false
}
fun proceedWithFinalNetworkCalls(): Flowable<List> {
return Single.zip(
single3(single1result),
single4(single2result),
single5(single1result),
single6(single2result),
Function4 { single3Result: Object,
single4Result: Object,
single5Result: Object,
single6Result: Object ->
ObjectHolder(single1Result, single2Result, single3Result, single4Result, single5Result, single6Result)
}
)
Sorry, it's in Java but from what I've understood, something like that may be a possible solution?
public static void main(String[] args) {
final Single<String> single1 = single1().cache();
single1.map(List::of)
.mergeWith(single1.zipWith(single2(), Map::entry)
.flatMap(entry -> Single.zip(
single3(entry.getKey()),
single4(entry.getValue()),
single5(entry.getKey()),
single6(entry.getValue()),
(el3, el4, el5, el6) -> objectHolder(entry.getKey(), entry.getValue(), el3, el4, el5, el6))))
.subscribe(System.out::println,
System.err::println);
Flowable.timer(1, MINUTES) // Just to block the main thread for a while
.blockingSubscribe();
}
private static List<String> objectHolder(final String el1,
final String el2,
final String el3,
final String el4,
final String el5,
final String el6) {
return List.of(el1, el2, el3, el4, el5, el6);
}
static Single<String> single1() {
return Single.just("s1");
}
static Single<String> single2() {
return Single.just("s2");
}
static Single<String> single3(String value) {
return single("s3", value);
}
static Single<String> single4(String value) {
return single("s4", value);
}
static Single<String> single5(String value) {
return single("s5", value);
}
static Single<String> single6(String value) {
return single("s6", value);
}
static Single<String> single(String value1, String value2) {
return Single.just(value1).map(l -> l + "_" + value2);
}
This outputs:
[s1]
[s1, s2, s3_s1, s4_s2, s5_s1, s6_s2]
I have a method that try use WebClient to return a Mono
#GetMapping("getMatch")
public Mono<Object> getMatch(#RequestParam Long matchId) {
return WebClient.create(OpenDotaConstant.BASE_URL).get()
.uri("/matches/{matchId}", matchId)
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(Object.class);
}
It can return result that I expected.
Then I try to create another method to support List as params
#GetMapping("getMatches")
public Flux<Object> getMatches(#RequestParam String matchesId) {
List<Long> matchesList = JSON.parseArray(matchesId, Long.class);
return Flux.fromStream(matchesList.parallelStream().map(this::getMatch));
}
But this time return a weird result.
[
{
"scanAvailable": true
},
{
"scanAvailable": true
}
]
I'm new to reactive-programming, What is the correct way to combine Stream and Mono,and then convert to the Flux?
Probably, what you need is the following:
#GetMapping("getMatches")
public Flux<Object> getMatches(#RequestParam String matchesId) {
List<Long> matchesList = JSON.parseArray(matchesId, Long.class);
return Flux.fromStream(matchesList.stream())
.flatMap(this::getMatch);
}
Instead of:
#GetMapping("getMatches")
public Flux<Object> getMatches(#RequestParam String matchesId) {
List<Long> matchesList = JSON.parseArray(matchesId, Long.class);
return Flux.fromStream(matchesList.parallelStream().map(this::getMatch));
}
Notes:
Basically, you expect getMatches endpoint to return Flux<Object>. However, as it is written - it actually returns Flux<Mono<Object>>, therefore you see the strange output. To get Flux<Object>, I suggest, first, create Flux<Long> that are match ids, and then flatMap the result of calling getMatch (that returns Mono<Object>), this finally gives Flux<Object>.
Also, there is no need to use parallelStream(). Because you're already using reactor, everything will be performed concurrently on reactor scheduler.
I looking for a good way to code this:
MyCloudDTO saveInCloud(MyCloudDTO dto) {
//sync http call :-(
return cloudService.save(dto);
}
MyData saveInDb(MyData data) {
//database call
return repository.save(data);
}
MyDTO save(MyDTO dto) {
MyCloudDTO cloudDTO = mapper.map(dto);
dto.setCloudId(saveInCloud(dto).getId());
MyData data = mapper2.map(dto);
dto.setDbId(saveInDb(data).getId());
return dto;
}
With Spring WebFlux and Monos.
The main problem here is that I want to keep the base dto during all the process because the mappers map only the properies they need and leave the the rest.
I found a way to do that but I'm not sure that's the correct/best way to do:
Mono<MyCloudDTO> saveInCloud(MyCloudDTO dto) {
return reactiveCloudService.save(dto);
}
Mono<MyData> saveInDb(MyData data) {
return reactiveRepository.save(data);
}
Mono<MyDTO> save(MyDTO dto) {
Mono<MyDTO> baseMono = Mono.just(dto);
Mono<MyCloudDTO> saveCloudMono = Mono.just(dto)
.map(mapper::map)
.flatMap(this::saveInCloud);
Mono<MyDTO> mergeAfterCloudSave = Mono.zip(
baseMono,
saveCloudMono,
(base, cloudDto) -> base.idCloud(cloudDto.getId())); //the setter return this
Mono<MyData> saveDbMono = mergeAfterCloudSave
.map(mapper2::map)
.flatMap(this::saveInCloud);
Mono<MyDTO> mergeAfterDbSave = Mono.zip(
mergeAfterCloudSave,
saveDbMono,
(base, dbData) -> base.idDb(data.getId())); //the setter return this
return mergeAfterDbSave;
}
Thank you very much!
Saveriu