My system is being overrun by the errors in message queue processing - orocommerce

Question

my system is currently being overrun by background processing errors. The prod.log file is 300 MB in size daily and CPU is constantly 100% every sec due to an error in queue and it keeps retrying non stop.
How do I solve the error? (find the process or message)
How can I quickly stop the process from retrying?
The error in question is generating the following log:
[2020-05-14 07:02:07] consumer.ERROR: Consuming interrupted, reason: An error occurred while running job, id: 1128304 [] []
[2020-05-14 07:02:33] app.ERROR: Transaction aborted wit error: Error in one or more bulk request actions: update: /oro/oro_product_1/2 caused [oro_product_1][2]: document missing [index: oro] update: /oro/oro_product_1/5 caused [oro_product_1][5]: document missing [index: oro] update: /oro/oro_product_1/7 caused [oro_product_1][7]: document missing [index: oro] update: /oro/oro_product_1/11 caused [oro_product_1][11]: document missing [index: oro] update: /oro/oro_product_1/12 caused [oro_product_1][12]: document missing [index: oro] update: /oro/oro_product_1/14 caused [oro_product_1][14]: document missing [index: oro] update: /oro/oro_product_1/26 caused [oro_product_1][26]: document missing [index: oro] update: /oro/oro_product_1/29 caused [oro_product_1][29]: document missing [index: oro] update: /oro/oro_product_1/33 caused [oro_product_1][33]: document missing [index: oro] update: /oro/oro_product_1/36 caused [oro_product_1][36]: document missing [index: oro] update: /oro/oro_product_1/43 caused [oro_product_1][43]: document missing [index: oro] update: /oro/oro_product_1/48 caused [oro_product_1][48]: document missing [index: oro] update: /oro/oro_product_1/51 caused [oro_product_1][51]: document missing [index: oro] update: /oro/oro_product_1/57 caused [oro_product_1][57]: document missing [index: oro] update: /oro/oro_product_1/60 caused [oro_product_1][60]: document missing [index: oro] update: /oro/oro_product_1/4477 caused [oro_product_1][4477]: document missing [index: oro] . [] {"processor":"Oro\\Bundle\\VisibilityBundle\\Async\\Visibility\\CustomerProcessor","message_id":"oro.5e6965d4ebbfa8.58577096","message_body":"{\"id\":105,\"entity_class_name\":\"Oro\\\\Bundle\\\\CustomerBundle\\\\Entity\\\\Customer\"}","message_properties":{"oro.message_queue.client.topic_name":"oro_visibility.visibility.change_customer","oro.message_queue.client.processor_name":"oro_visibility.async.visibility.customer_processor","oro.message_queue.client.queue_name":"oro.default","oro.security.token":"organizationId=1;userId=54;userClass=Oro\\Bundle\\UserBundle\\Entity\\User;roles=ROLE_USER,ROLE_ADMINISTRATOR,ROLE_MANAGER,ROLE_ONLINE_SALES_REP,ROLE_MARKETING_MANAGER,ROLE_LEADS_DEVELOPMENT_REP,ROLE_CATALOG_MANAGER,ROLE_ACCOUNT_MANAGER,ROLE_SALES_ASSISTANT","oro-redeliver-count":392783},"message_headers":{"content_type":null,"message_id":"oro.5e6965d4ebbfa8.58577096","timestamp":0},"message_priority":0,"elapsed_time":"182 ms","memory_usage":"128.18 MB"}
This error keeps repeating continuously in sequence (without any other errors) for about 50 times for a span of about 8 secs. Then immediately starting again 10 secs later and repeating the entire process again.
I am at wits end as there is not much I can see in the log. Thanks for any help!

Answer 1

How do I solve the error? (find the process or message)
All the information you need is in an error log.
used processor is Oro\\Bundle\\VisibilityBundle\\Async\\Visibility\\CustomerProcessor
message id is 5e6965d4ebbfa8
How can I quickly stop the process from retrying?
If you are using the community edition of OroCommerce, you can find the message in oro_message_queue table by id and delete it manually.
The error could happen because you have corrupted records in a database or a bug in code. It's hard to say without debugging.