AEROSPIKE_ERR_RECORD_NOT_FOUND just after INSERT - aerospike

Question

I have a strange problem writing data to the aerospike cluster
aql> insert into storebig.Chunks (PK,Data) values ('5cb138284d431abd6a053a56625ec088bfb88912', '1234567890')
OK, 1 record affected.
aql> select * from storebig.Chunks where PK = '5cb138284d431abd6a053a56625ec088bfb88912'
Error: (2) AEROSPIKE_ERR_RECORD_NOT_FOUND
aql> insert into storebig.Chunks (PK,Data) values ('5cb138284d431abd6a053a56625ec088bfb88912', '1234567890')
Error: (1) AEROSPIKE_ERR_SERVER
Same story with the golang client library (of course)
It is very possible cluster is not healfy - some strange messages appears in the server(s) log:
May 06 2015 12:17:49 GMT: WARNING (drv_ssd): (drv_ssd.c::1236) read: read wrong key: expecting de6f0bc93bfdf560 got 8ad3dd7fce1ac7ec
May 06 2015 12:17:49 GMT: WARNING (drv_ssd): (drv_ssd.c::1236) read: read wrong key: expecting de6f0bc93bfdf560 got 8ad3dd7fce1ac7ec
May 06 2015 12:17:50 GMT: WARNING (drv_ssd): (drv_ssd.c::1230) read: bad block magic offset 29843600384
May 06 2015 12:17:50 GMT: WARNING (drv_ssd): (drv_ssd.c::1230) read: bad block magic offset 29843600384
My question is: what can I do to investigate the situation, debug and recover? Where to look and what to try?
Thank you.
With best regards,
Daniel Podolsky
UPDATE
config template (actual config generated from this template on docker container start)
service {
user root
group root
paxos-single-replica-limit 1
pidfile /var/run/aerospike/asd.pid
service-threads 4
transaction-queues 4
transaction-threads-per-queue 4
proto-fd-max 15000
}
logging {
file /storage/logs/aerospike.log {
context any info
}
console {
context any info
}
}
network {
service {
address <%=os.getenv("NODE_EXT_ADDR")%>
port 3000
}
fabric {
address <%=os.getenv("NODE_INT_ADDR")%>
port 3001
}
heartbeat {
mode multicast
address 239.1.99.2
port 9918
interface-address <%=os.getenv("NODE_INT_ADDR")%> interval 150
timeout 10
}
info {
address <%=os.getenv("NODE_INT_ADDR")%>
port 3003
}
}
namespace storebig {
replication-factor 3
memory-size <%=os.getenv("MEM_USE_BIG")%>K
default-ttl 0
high-water-disk-pct 98
high-water-memory-pct 98
stop-writes-pct 95
storage-engine device {
file /storage/data/big.dat
filesize 3T
data-in-memory false
}
}
namespace storefast {
replication-factor 3
memory-size <%=os.getenv("MEM_USE_FAST")%>K
default-ttl 0
high-water-disk-pct 98
high-water-memory-pct 98
stop-writes-pct 95
storage-engine device {
file /storage/data/fast.dat
filesize <%=os.getenv("MEM_USE_FAST")%>K
data-in-memory true
}
}
namespace storetest {
replication-factor 3
memory-size <%=os.getenv("MEM_USE_FAST")%>K
default-ttl 0
high-water-disk-pct 98
high-water-memory-pct 98
stop-writes-pct 95
storage-engine device {
file /storage/data/test.dat
filesize 3T
data-in-memory false
}
}

Answer 1

After reading over your configuration I believe I have found your problem. Individual devices and files in Aerospike can be no larger than 2TiB and yours are configured to 3TiB. Regrettably there currently isn't a check in config parser for this limit and I am unable to find reference in our docs--both of these issues are being taken care of.
You can instead use multiple files to store your data for each namespace (each file limited to 2TB). As discussed elsewhere you will likely see better performance by using multiple files or devices for a given namespace.

Answer 2

reading the Aerospike manual, there is no limitation for device size. only for file size (2TB maximum)
Manual:
Recipe for an SSD Storage Engine
The minimal configuration for an SSD namespace requires setting storage-engine to device and adding a device parameter for each SSD to be used by this namespace. In addition, memory-size may need to be changed from the default of 4GB to a size appropriate for the expected primary index size. For assistance in sizing the primary index, please refer to the Sizing Guide. For performance, we recommend reducing the write-block-size from the default of 1MB to 128 Kb on SSD backed namespaces.
Recipe for a HDD Storage Engine with Data in Memory
The minimal configuration for an HDD with Data-in-Memory namespace involves setting storage-engine to device, setting data-in-memory to true, and finally providing a list of file parameters to indicate where data will be persisted. Also filesize needs to be large enough to support the size of the data on disk (with a maximum allowed value of 2 TiB). Lastly, memory-size may need adjusted from the default of 4GB to a size appropriate to handle the expected primary index size and the expected size of the data in memory. For assistance sizing filesize or memory-size please refer to our Sizing Guide.