How to resolve Aerospike error, Operation not allowed at this time? - aerospike

Question

I am using Aerospike 4.8v and persisting my data on disk and I am making parallel write requests to aerospike, if i make 10 parallel requests then it works fine but when I make 100 parallel requests it gives error i.e. Operation not allowed at this time, with code 22. I think that aerospike must be capable of handling hundreds or thousands of requests in parallel but I don't know whats wrong so if anyone can guide that would be helpful.
Error log:
error : { AerospikeError: Operation not allowed at this time.
at Function.fromASError (/data/codebase/lib/node_modules/aerospike/lib/error.js:113:21)
at QueryCommand.convertError (/data/codebase/lib/node_modules/aerospike/lib/commands/command.js:91:27)
at QueryCommand.convertResponse (/data/codebase/lib/node_modules/aerospike/lib/commands/command.js:101:24)
at asCallback (/data/codebase/lib/node_modules/aerospike/lib/commands/command.js:163:24)
name: 'AerospikeError',
code: 22,
command:
QueryCommand {
client:
Client {
domain: null,
_events: {},
_eventsCount: 0,
_maxListeners: undefined,
config: [Object],
as_client: AerospikeClient {},
connected: true,
captureStackTraces: false },
args: [ 'antiSpamming', 'userTargetingMatrix', [Object], undefined ],
captureStackTraces: false,
key: undefined,
ensureConnected: true,
stream:
RecordStream {
aborted: false,
client: [Object],
_events: [Object],
_eventsCount: 3 } },
func: 'as_query_parse_records_async',
file: 'src/main/aerospike/aerospike_query.c',
line: 246,
inDoubt: false }
Warning logs:
Jan 28 06:03:25 ip-1-0-4-78 asd[32437]: Jan 28 2020 06:03:25 GMT: WARNING (scan): (scan_manager.c:103) at scan threads limit - can't start new scan
Jan 28 06:03:25 ip-1-0-4-78 asd[32437]: Jan 28 2020 06:03:25 GMT: WARNING (scan): (scan.c:676) basic scan job 5614303283813349397 failed to start (22)

Answer 1

You are most likely exceeding the default limit related to queries. First, see the definition for error code 22:
For scans with server 4.7 or later, no threads available
(scan-threads-limit reached).
Based on your other question, you're doing scans rather than a query with a secondary index. You'll need to increase that limit, as suggested in the 'Additional Information'. However, you have a pretty weak system in terms of CPU, so you should adjust that value and benchmark, comparing performance before and after with the same workload. In a real production system you'd have multiple nodes, probably want more than two CPU cores, and similarly tune the scan threads as needed.

Answer 2

This happened to me because a job was performing a backup while I was trying to scan the partitions.
Recommendation, do not run an extra job while a backup is in process.