I'm using Redis (2.4.2) and with the INFO command I can read stats about my Redis server.
There are many stats, including some about how much memory is used. And one is "used_memory_peak" that seems to hold the maximum amount of memory Redis has ever taken.
I've deleted a bunch of key, and I'd like to reset this stat since it affects the scale of my Munin graphs.
There is a CONFIG RESETSTAT command, but it doesn't seem to affect this particular stat.
Any idea how I could do this, without having to export/delete/import my dataset ?
EDIT :
According to #antirez himself (issue 369 on GitHub), this is an intended behavior, but it this feature could be improved to be more useful in a future release.
The implementation of CONFIG RESETSTAT is quite simple:
} else if (!strcasecmp(c->argv[1]->ptr,"resetstat")) {
if (c->argc != 2) goto badarity;
server.stat_keyspace_hits = 0;
server.stat_keyspace_misses = 0;
server.stat_numcommands = 0;
server.stat_numconnections = 0;
server.stat_expiredkeys = 0;
addReply(c,shared.ok);
So it does not initialize the server.stat_peak_memory field used to store the maximum amount of memory ever used by Redis. I don't know if it is a bug or a feature.
Here is a hack to reset the value without having to stop Redis. The idea is to use gdb in batch mode to just change the value of the variable (which is part of a static structure). Normally Redis is compiled with debugging symbols.
# Here we have plenty of things in this instance
> ./redis-cli info | grep peak
used_memory_peak:1363052184
used_memory_peak_human:1.27G
# Let's do some cleaning: everything is wiped out
# don't do this in production !!!
> ./redis-cli flushdb
OK
# Again the same values, while some memory has been freed
> ./redis-cli info | grep peak
used_memory_peak:1363052184
used_memory_peak_human:1.27G
# Here is the magic command: reset the parameter with gdb (output and warnings to be ignored)
> gdb -batch -n -ex 'set variable server.stat_peak_memory = 0' ./redis-server `pidof redis-server`
Missing separate debuginfo for /lib64/libm.so.6
Missing separate debuginfo for /lib64/libdl.so.2
Missing separate debuginfo for /lib64/libpthread.so.0
[Thread debugging using libthread_db enabled]
[New Thread 0x41001940 (LWP 22837)]
[New Thread 0x40800940 (LWP 22836)]
Missing separate debuginfo for /lib64/libc.so.6
Missing separate debuginfo for /lib64/ld-linux-x86-64.so.2
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff51ff000
0x00002af0b5eef218 in epoll_wait () from /lib64/libc.so.6
# And now, result is different: great !
> ./redis-cli info | grep peak
used_memory_peak:718768
used_memory_peak_human:701.92K
This is a hack: think twice before applying this trick on a production instance.
Simple trick to clear peal memory::
Step 1:
/home/logproc/redis/bin/redis-cli BGREWRITEAOF
wait till it finish rewriting aof file.
Step 2:
restart redis db
Done. Thats It.
Related
Are there any specific problems with running Microsoft's BCP utility (on CentOS 7, https://learn.microsoft.com/en-us/sql/linux/sql-server-linux-migrate-bcp?view=sql-server-2017) on multiple threads? Googling could not find much, but am looking at a problem that seems to be related to just that.
Copying a set of large TSV files from HDFS to a remote MSSQL Server with some code of the form
bcpexport() {
filename=$1
TO_SERVER_ODBCDSN=$2
DB=$3
TABLE=$4
USER=$5
PASSWORD=$6
RECOMMEDED_IMPORT_MODE=$7
DELIMITER=$8
echo -e "\nRemoving header from TSV file $filename"
echo -e "Current head:\n"
echo $(head -n 1 $filename)
echo "$(tail -n +2 $filename)" > $filename
echo "First line of file is now..."
echo $(head -n 1 $filename)
# temp. workaround safeguard for NFS latency
#sleep 5 #FIXME: appears to sometimes cause script to hang, workaround implemented below, throws error if timeout reached
timeout 30 sleep 5
echo -e "\nReplacing null literal values with empty chars"
NULL_WITH_TAB="null\t" # WARN: assumes the first field is prime-key so never null
TAB="\t"
sed -i -e "s/$NULL_WITH_TAB/$TAB/g" $filename
echo -e "Lines containing null (expect zero): $(grep -c "\tnull\t" $filename)"
# temp. workaround safeguard for NFS latency
#sleep 5 #FIXME: appears to sometimes cause script to hang, workaround implemented below
timeout 30 sleep 5
/opt/mssql-tools/bin/bcp "$TABLE" in "$filename" \
$TO_SERVER_ODBCDSN \
-U $USER -P $PASSWORD \
-d $DB \
$RECOMMEDED_IMPORT_MODE \
-t "\t" \
-e ${filename}.bcperror.log
}
export -f bcpexport
parallel -q -j 7 bcpexport {} "$TO_SERVER_ODBCDSN" $DB $TABLE $USER $PASSWORD $RECOMMEDED_IMPORT_MODE $DELIMITER \
::: $DATAFILES/$TARGET_GLOB
where $DATAFILES/$TARGET_GLOB constructs a glob that lists a set of files in a directory.
When running this code for a set of TSV files, finding that sometimes some (but not all) of the parallel BCP threads fail, ie. some files successfully copy to MSSQL Server
Starting copy...
5397376 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total : 154902 Average : (34843.8 rows per sec.)
while others output error message
Starting copy...
BCP copy in failed
Usually, see this pattern: a few successful BCP copy-in operations in the first few threads returned, then a bunch of failing threads return their output until run out of files (GNU Parallel returns output only when whole thread done to appear same as if sequential).
Notice in the code there is the -e option to produce an error file for each BCP copy-in operation (see https://learn.microsoft.com/en-us/sql/tools/bcp-utility?view=sql-server-2017#e). When examining the files after observing these failing behaviors, all are blank, no error messages.
Only have seen this with the number of threads >= 10 (and only for certain sets of data (assuming has something to do with total number of files are files sizes, and yet...)), no errors seen so far when using ~7 threads, which further makes me suspect this has something to do with multi-threading.
Monitoring system resources (via free -mh) shows that generally ~13GB or RAM is always available.
May be helpful to note that the data bcp is trying to copy-in may be ~500000-1000000 records long with an upper limit of ~100 columns per record.
Does anyone have any idea what could be going on here? Note, am pretty new to using BCP as well as GNU Parallel and multi-threading.
No, no issues specific to the BCP program being run in multiple threads. You seem to be on the track of what I would say your issue is, system resources. Have you monitored system resources while increasing the number of threads? If anything, there is likely an issue with BCP executing properly when memory/cpu/network resources are low. Regarding the "-e" option, it is meant to output data errors. login errors, bad table names... many other errros are not reported in the file created with the -e option. When you get output using the "-e" option, you'll see info like "value truncated" and such... will give you line numbers and sample data that was at issue.
TLDR: Adding more threads to run concurrently to have bcp copy-in files of data seems to have the affect of overwhelming the endpoint MSSQL Server with write instructions, causing the bcp threads to fail (maybe timeing out?). When the number of threads becomes too many seems to depend on the size of the files getting copy-in'ed by bcp (ie. both the number of records in the file as well as the width of each record (ie. number of columns)).
Long version (more reasons for my theory):
1.
When running a larger number of bcp threads and looking at the processes started on the machine (https://clustershell.readthedocs.io/en/latest/tools/clush.html)
ps -aux | grep bcp
seeing a bunch of sleeping processes (notice the S, see https://askubuntu.com/a/360253/760862) as shown below (added newlines for readability)
me 135296 14.5 0.0 77596 6940 ? S 00:32 0:01
/opt/mssql-tools/bin/bcp TABLENAME in /path/to/tsv/1_16_0.tsv -D -S MyMSSQLServer -U myusername -P -d myDB -c -t \t -e /path/to/logfile
These threads appear to sleep for very long time. Further debugging into why these threads are sleeping suggests that they may in fact be doing their intended job (which would further imply that the problem may be coming from BCP itself (see https://stackoverflow.com/a/52748660/8236733)). From https://unix.stackexchange.com/a/47259/260742 and https://unix.stackexchange.com/a/36200/260742)
A process in S state is usually in a blocking system call, such as reading or writing to a file or the network, or waiting for another called program to finish.
(eg. writing to the MSSQL Server endpoint destination given to bcp in the ODBCDSN)
Your process will be in S state when it is doing reads and possibly writes that are blocking. Can also happen while waiting on semaphores or other synchronization primitives... This is all normal and expected, and not usually a problem... you don't want it to waste CPU while it's waiting for user input.
2. When running different sets of files of varying record-amount-per-file (eg. ranges of 500000 - 1000000 rows/file) and record-width-per-file (~10 - 100 columns/row), found that in cases with either very large data width or amounts, running a fixed set of bcp threads would fail.
Eg. for a set of ~33 TSVs with ~500000 rows each, each row being ~100 columns wide, a set of 30 threads would write the first few OK, but then all the rest would start returning failure messages. Incorporating a bit from #jamie's answer, the fact the the failure messages returned are "BCP copy in failed" errors does not necessarily mean it has do do with the content of the data in question. Having no actual content being written into the -e errorlog files from my process, #jamie's post says this
Regarding the "-e" option, it is meant to output data errors. login errors, bad table names... many other errros are not reported in the file created with the -e option. When you get output using the "-e" option, you'll see info like "value truncated" and such... will give you line numbers and sample data that was at issue.
Meanwhile, a set of ~33 TSVs with ~500000 rows each, each row being ~100 wide, and still using 30 bcp threads would complete quickly and without error (also would be faster when reducing the number of threads or file set). The only difference here being the overall size of the data being bcp copy-in'ed to the MSSQL Server.
All this while
free -mh
still showed that the machine running the threads still had ~15GB of free RAM remaining in each case (which is again why I suspect that the problem has to do with the remote MSSQL Server endpoint rather than with the code or local machine itself).
3. When running some of the tests from (2), found that manually killing the parallel process (via CTL+C) and then trying to remotely truncate the testing table being written to with /opt/mssql-tools/bin/sqlcmd -Q "truncate table mytable" on the local machine would take a very long time (as opposed to manually logging into the MSSQL Server and executing a truncate mytable in the DB). Again this makes me think that this has something to do with the MSSQL Server having too many connections and just being overwhelmed.
** Anyone with any MSSQL Mgmt Studio experience reading this (I have basically none), if you see anything here that makes you think that my theory is incorrect please let me know your thoughts.
For my task I need to load a bulk of data into Redis as soon as possible. It looks like this article is right about my case: https://redis.io/topics/mass-insert
The article starts from giving an example of using multiple inline SET commands with redis-cli. Then they proceed to generating Redis protocol and again use it with redis-cli. They don't explain the reasons or benefits of using Redis protocol.
Using of Redis protocol is a bit harder and it generates a bit more traffic. I wonder, what are the reasons to use Redis protocol rather than simple one-line commands? Probably despite the fact the data is larger, it is easier (and faster) for Redis to parse it?
Good point.
Only a small percentage of clients support non-blocking I/O, and not
all the clients are able to parse the replies in an efficient way in
order to maximize throughput. For all this reasons the preferred way
to mass import data into Redis is to generate a text file containing
the Redis protocol, in raw format, in order to call the commands
needed to insert the required data.
What I understood is that you emulate a client when you use Redis protocol directly, which would benefit from the highlighted points.
Based on the docs you provided, I tried these scripts:
test.rb
def gen_redis_proto(*cmd)
proto = ""
proto << "*"+cmd.length.to_s+"\r\n"
cmd.each{|arg|
proto << "$"+arg.to_s.bytesize.to_s+"\r\n"
proto << arg.to_s+"\r\n"
}
proto
end
(0...100000).each{|n|
STDOUT.write(gen_redis_proto("SET","Key#{n}","Value#{n}"))
}
test_no_protocol.rb
(0...100000).each{|n|
STDOUT.write("SET Key#{n} Value#{n}\r\n")
}
ruby test.rb > 100k_prot.txt
ruby test_no_protocol.rb > 100k_no_prot.txt
time cat 100k.txt | redis-cli --pipe
time cat 100k_no_prot.txt | redis-cli --pipe
I've got these results:
teixeira: ~/stackoverflow $ time cat 100k.txt | redis-cli --pipe
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 100000
real 0m0.168s
user 0m0.025s
sys 0m0.015s
(5 arquivo(s), 6,6Mb)
teixeira: ~/stackoverflow $ time cat 100k_no_prot.txt | redis-cli --pipe
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 100000
real 0m0.433s
user 0m0.026s
sys 0m0.012s
I'm running Redis on windows and have noticed that the size of redis-server.exe decreases over time. When I open Redis, it reads from a dump file and loads all of the hashed key values into memory to about 1.4 GB. However, over time, the amount of memory that redis-server.exe takes up decreases. I have seen it go down to less than 100 MB.
The only reason that I could see this happening is that the keys are expiring and leaving memory, however I have set Redis up so that they never expire. I have also made sure that I have given enough memory.
Some of my settings include:
maxmemory 2gb
maxmemory-policy noeviction
hash-max-zipmap-entries 512
hash-max-zipmap-value 64
activerehashing no
If it's of interest, when I first loaded the keys into Redis, I did it through Python like so:
r.hset(key, field, value)
Any help would be appreciated. I want the keys to be there forever.
This is my output from the INFO command right after I first run it:
redis 127.0.0.1:6379> INFO
redis_version:2.4.6
redis_git_sha1:26cdd13a
redis_git_dirty:0
arch_bits:64
multiplexing_api:winsock2
gcc_version:4.6.1
process_id:9092
uptime_in_seconds:69
uptime_in_days:0
lru_clock:248011
used_cpu_sys:3.34
used_cpu_user:10.06
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
connected_clients:1
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:1129560232
used_memory_human:1.05G
used_memory_rss:1129560232
used_memory_peak:1129560144
used_memory_peak_human:1.05G
mem_fragmentation_ratio:1.00
mem_allocator:libc
loading:0
aof_enabled:0
changes_since_last_save:0
bgsave_in_progress:0
last_save_time:1386600366
bgrewriteaof_in_progress:0
total_connections_received:1
total_commands_processed:0
expired_keys:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
vm_enabled:0
role:master
db0:keys=4007989,expires=0
After I run it when I noticed the memory has decreased in Windows Task Manager, there are not many differences:
uptime_in_seconds:4412 (from 69)
lru_clock:248445 (from 248011)
used_cpu_sys:4.59 (from 3.34)
used_cpu_user:10.25 (from 10.06)
used_memory:1129561240 (from 1129560232)
used_memory_human:1.05G (same!)
used_memory_rss:1129561240 (from 1129560232)
used_memory_peak:1129568960 (from 1129560144)
used_memory_peak_human:1.05G (same!)
mem_fragmentation_ratio:1.00 (same!)
last_save_time:1386600366 (same!)
total_connections_received:4 (from 1)
total_commands_processed:10 (from 0)
expired_keys:0 (same!)
evicted_keys:0 (same!)
keyspace_hits:0 (same!)
keyspace_misses:2 (from 0)
The lookups are taking longer when the memory size is lower. What is going on here?
What version of Redis are you using ?
Do you have a cron of some sort that removes key ? (do a grep on the del command on your codebase just to be sure)
Redis usually runs a single process to manage the in-memory data. However, when the data is persisted to the RDB file, a second process starts to save all the data. During that process, you can see your memory use increase up to double the size of your data set.
I am familiar with how it is done in linux, but I don't know the details about the windows port, so maybe the differences in size you are seeing are because of this second process that is launched periodically? You can easily try if this is the case by issuing a BGSAVE command in redis. This will start the synchronization of data to RDB on the background, so you can see if the memory usage pattern is the one you observed.
If that is the case, then you already know what is going on :)
good luck
I am exploring redis to do pub/sub. I wanted to write a script that uses redis-cli to subscribe to a channel and dump whatever is published to a file. What I notice however is that redis-cli subscripe channel > output does not quite work.
This is because there is no automatic flush of stdout when redis-cli displays the messages associated to the subscription. So the last messages before stopping redis-cli do not appear in the output file.
There is no option you can use to enforce a systematic flush, redis-cli.c needs to be patched. In Redis source code, edit src/redis-cli.c, and find the following piece of code. Add the missing fflush line.
if (config.pubsub_mode) {
if (config.output != OUTPUT_RAW)
printf("Reading messages... (press Ctrl-C to quit)\n");
while (1) {
if (cliReadReply(output_raw) != REDIS_OK) exit(1);
// The following line must be added
fflush(stdout);
}
}
Once redis-cli has been compiled again, it should work as expected.
I need to increase memory in weblogic. I am new in this and I dont know how. I need to set -Xss=4096k . How I can I do it ?
Xss is Thread Stack Size,, it is not the memory size
you can change the memory size by changing the parameters Xmx
the most important parameters are :
-Xms1536m -Xmx1536m -XX:MaxPermSize=512m
Xmx - is the max size of the heap.
Xms - is the initial size of the heap.( give it the same as Xmx )
XX:MaxPermSize - is is used to hold reflective of the VM itself such as class objects and method objects ( it's independent from the heap size,, give it the 1/3 to 1/4 of the Xms size depend in your classes size)
.........
Any Way:
you can change XSS from config.xml
in this path : DOMAIN_NAME/config/config.xml
but you have to shutdown the admin server when you change something in config.xml
, then edit the start properties, or add it under <server> if it's not there:
<server-start>
<arguments>-Xms1536m -Xmx1536m -XX:MaxPermSize=512m -Xss4096k </arguments>
</server-start>
........
[[OR]]
you can change it from the admin console which is easier
access the admin console then go to Environment >> Servers
choose the server you want to change it
form Configuration >> Server Start
you will see box called Arguments:
Add -Xss4096k
Options for the JVM must be set on startup so you need to modify the startup script for WebLogic.
See here:
http://docs.oracle.com/cd/E13222_01/wls/docs100/server_start/overview.html#JavaOptions
Go to setDomainEnv.
Search for the below comment.
#REM IF USER_MEM_ARGS the environment variable is set, use it to override ALL MEM_ARGS values
Paste the below line (3072 for 3GB).
set USER_MEM_ARGS=-Xms128m -Xmx3072m %MEM_DEV_ARGS% %MEM_MAX_PERM_SIZE%
Another way that is simpler is by using the setUserOverrides.sh script or creating one if none exists. See the example below
----setUserOverrides.sh script-----
#!/bin/bash
echo "Setting for UserOverrides.sh"
# global settings (for all managed servers)
export JAVA_OPTIONS="$JAVA_OPTIONS"
# customer settings for each Server
if [ "${SERVER_NAME}" = "AdminServer" ]
then
echo "Customizing ${SERVER_NAME}"
export JAVA_OPTIONS="$JAVA_OPTIONS -server -Xms2g -Xmx2g - Dweblogic.security.SSL.minimumProtocolVersion=TLSv1.1"
fi
if [ "${SERVER_NAME}" = "soa_server1" ]
then
echo "Customizing ${SERVER_NAME}"
export JAVA_OPTIONS="$JAVA_OPTIONS -client -Xms4g -Xmx4g - Dweblogic.security.SSL.minimumProtocolVersion=TLSv1.2"
fi
echo "End setting from UserOverrides.sh"