Redis Lua script problems only when many calls with different params - redis

Question

I have script that works fine when I simply call it by hands, or when I call it using my API.
The same script works fine when I create 5 threads and each of them calls this script 10000 times with the same parameters.
But when I run performance testing of my API with tester-written script, I see wrong information in Redis and this information simply can't appear in Redis if script works correcty, so I'm assuming there is some problem in it.
Script:
-- ARGV[1] createdAt
-- ARGV[2] storyId
-- ARGV[3] storyType (image or video)
-- ARGV[4] storyFilename (original filename)
-- ARGV[5] storyThumbnail (thumbnail filename)
-- ARGV[6] storyTags
-- ARGV[7] authorId
-- ARGV[8] authorUsername
-- ARGV[9] authorName
-- ARGV[10] authorAvatarThumb
-- ARGV[11] PHP_INT_MAX
-- ARGV[12] timestamp = followers will see stories younger then this
-- ARGV[13..N] followerId
-- Init variables
local createdAt = ARGV[1]
local storyId = ARGV[2]
local storyType = ARGV[3]
local storyFilename = ARGV[4]
local storyThumbnail = ARGV[5]
local storyTags = cjson.decode(ARGV[6])
local authorId = tonumber(ARGV[7])
local authorUsername = ARGV[8]
local authorName = ARGV[9]
local authorAvatarThumb = ARGV[10]
local maxScore = ARGV[11]
local storyAuthorData = {
id = authorId,
username = authorUsername,
name = authorName,
avatarThumb = authorAvatarThumb,
storyCount = redis.call('ZCARD', 'user:' .. authorId .. ':storyIds') + 1,
lastStory = {
id = storyId,
thumbnail = storyThumbnail,
isViewed = false
}
}
local storyData = {
id = storyId,
thumbnail = storyThumbnail,
tags = storyTags,
viewerCount = 0,
isViewed = false
}
if storyType == 'image' then
storyAuthorData.lastStory.image = storyFilename
storyData.image = storyFilename
else
storyAuthorData.lastStory.video = storyFilename
storyData.video = storyFilename
end
local packedStoryData = cmsgpack.pack(storyData);
local packedAuthorData = cmsgpack.pack(storyAuthorData);
local authorStoryCountForFollower = redis.call('ZCOUNT', 'user:' .. authorId .. ':storyIds', '(' .. ARGV[12], '+inf')
storyAuthorData.storyCount = authorStoryCountForFollower
local packedAuthorDataForFollower = cmsgpack.pack(storyAuthorData);
redis.call('ZADD', 'storyIds', createdAt, storyId)
-- Iterate through followerIds
local followerId, followerAuthorsKey, followerAuthors, followerAuthor
for i = 13, #ARGV do
followerId = ARGV[i]
redis.call('SADD', 'story:' .. storyId .. ':projections', followerId)
redis.call('ZADD', 'forUser:' .. followerId .. ':storiesByUser:' .. authorId, createdAt, packedStoryData)
-- Iterate through follower' story authors and update current author
followerAuthorsKey = 'forUser:' .. followerId .. ':storyAuthors'
followerAuthors = redis.call('ZRANGE', followerAuthorsKey, 0, -1)
for j = 1, #followerAuthors do
followerAuthor = followerAuthors[j]
if cmsgpack.unpack(followerAuthor).id == authorId then
redis.call('ZREM', followerAuthorsKey, followerAuthor)
break
end
end
redis.call('ZADD', followerAuthorsKey, createdAt, packedAuthorDataForFollower)
end
-- Process author as follower of himself
redis.call('ZADD', 'user:' .. authorId .. ':storyIds', createdAt, storyId)
redis.call('ZADD', 'forUser:' .. authorId .. ':storiesByUser:' .. authorId, createdAt, packedStoryData)
followerAuthorsKey = 'forUser:' .. authorId .. ':storyAuthors'
followerAuthor = redis.call('ZREVRANGE', followerAuthorsKey, 0, 0, 'WITHSCORES')
if followerAuthor[2] == maxScore then
redis.call('ZREM', followerAuthorsKey, followerAuthor[1])
end
redis.call('ZADD', followerAuthorsKey, maxScore, packedAuthorData)
As you can see in this part:
-- Iterate through follower' story authors and update current author
followerAuthorsKey = 'forUser:' .. followerId .. ':storyAuthors'
followerAuthors = redis.call('ZRANGE', followerAuthorsKey, 0, -1)
for j = 1, #followerAuthors do
followerAuthor = followerAuthors[j]
if cmsgpack.unpack(followerAuthor).id == authorId then
redis.call('ZREM', followerAuthorsKey, followerAuthor)
break
end
end
redis.call('ZADD', followerAuthorsKey, createdAt, packedAuthorDataForFollower)
there is no possibility that for any key forUser:*:storyAuthors the same author would appear twice.
But in Redis I can see:
zrange forUser:5870:storyAuthors 0 -1 WITHSCORES
1) "\x86\xa8username\xaee8dccb3501913e\xa4name\xactaylor evans\xabavatarThumb\xd9+d083e43ff8dbec76904f09abb8be7dab_thumb.jpeg\xa2id\xcd\x14S\xaastoryCount\x0f\xa9lastStory\x84\xa2id\xd9 00001453abe86ec44260d64111619a76\xa9thumbnail\xd9%59810b11537769.89801964_thumbnail.jpg\xa8isViewed\xc3\xa5video\xbb59810b11537769.89801964.mp4"
2) "1501629201.733"
3) "\x86\xa8username\xaee8dccb3501913e\xa4name\xactaylor evans\xabavatarThumb\xd9+d083e43ff8dbec76904f09abb8be7dab_thumb.jpeg\xa2id\xcd\x14S\xa9lastStory\x84\xa2id\xd9 00001453a77458044360d641bc47c73c\xa5image\xbb59810c1122b659.53121588.jpg\xa9thumbnail\xd9%59810c1122b659.53121588_thumbnail.jpg\xa8isViewed\xc2\xaastoryCount\x1c"
4) "1501629457.3821001"
5) "\x86\xa8username\xaee8dccb3501913e\xa4name\xactaylor evans\xabavatarThumb\xd9+d083e43ff8dbec76904f09abb8be7dab_thumb.jpeg\xa2id\xcd\x14S\xa9lastStory\x84\xa2id\xd9 000014539c8c70044360d6417e07a91d\xa8isViewed\xc2\xa9thumbnail\xd9%59810c11637f43.19889868_thumbnail.jpg\xa5video\xbb59810c11637f43.19889868.mp4\xaastoryCount\x1d"
6) "1501629457.7586"
7) "\x86\xa8username\xaea2d4fb6a25ec2a\xa4name\xabjean brewer\xabavatarThumb\xd9+0cd3514371aa7f5ca53b1859b950f92e_thumb.jpeg\xa2id\xcd\x16\xee\xa9lastStory\x83\xa2id\xd9 000016ee38daa80f4360d641f61949da\xa8isViewed\xc3\xa5video\xbb59810c3e432b80.55025328.mp4\xaastoryCount\x04"
8) "1501629502.6382999"
9) "\x86\xa8username\xaea2d4fb6a25ec2a\xa4name\xabjean brewer\xabavatarThumb\xd9+0cd3514371aa7f5ca53b1859b950f92e_thumb.jpeg\xa2id\xcd\x16\xee\xa9lastStory\x83\xa2id\xd9 000016ee5e109c1f4360d6412826f135\xa8isViewed\xc3\xa5video\xbb59810c7e0b7e92.12898014.mp4\xaastoryCount\x1c"
10) "1501629566.4384999"
11) "\x86\xa8username\xae87163046b74033\xa4name\xaadianne kim\xabavatarThumb\xd9+5d59c5cfefd028e9042bff975f92a94e_thumb.jpeg\xa2id\xcd\x04\xf4\xa9lastStory\x83\xa2id\xd9 000004f4c590065f4360d641f56be683\xa8isViewed\xc3\xa5video\xbb59810d7ba433b2.90198935.mp4\xaastoryCount\x1a"
12) "1501629820.1026001"
13) "\x86\xa8username\xae87163046b74033\xa4name\xaadianne kim\xabavatarThumb\xd9+5d59c5cfefd028e9042bff975f92a94e_thumb.jpeg\xa2id\xcd\x04\xf4\xa9lastStory\x83\xa2id\xd9 000004f488d6bc714360d6418e77ad61\xa8isViewed\xc3\xa5video\xbb59810dc68ad4c6.78917372.mp4\xaastoryCount:"
14) "1501629894.9505999"
15) "\x86\xa8username\xae87163046b74033\xa4name\xaadianne kim\xabavatarThumb\xd9+5d59c5cfefd028e9042bff975f92a94e_thumb.jpeg\xa2id\xcd\x04\xf4\xa9lastStory\x83\xa2id\xd9 000004f402baf07e4360d641f1430d08\xa8isViewed\xc3\xa5video\xbb59810dfb69d323.13796483.mp4\xaastoryCountB"
16) "1501629947.7614"
17) "\x86\xa8username\xaea2d4fb6a25ec2a\xa4name\xabjean brewer\xabavatarThumb\xd9+0cd3514371aa7f5ca53b1859b950f92e_thumb.jpeg\xa2id\xcd\x16\xee\xaastoryCount\x1d\xa9lastStory\x84\xa2id\xd9 000016ee4815e74c4360d641c7cb3dfc\xa9thumbnail\xd9%59810d333d9445.06755396_thumbnail.jpg\xa8isViewed\xc3\xa5video\xbb59810d333d9445.06755396.mp4"
18) "4503599627370496"
There are also another problems showing that the data is not consistent.
I've increased RAM size from 2.8Gb to 16Gb and it did not help.
For KEYS params of lua script - number of params depends on how many followers does a user have. Users we use for testing have thousands of followers, so KEYS array can contain up to several thousands of elements.