I want to set a password to connect to a Redis server.
The appropriate way to do that is using the requirepass directive in the configuration file.
http://redis.io/commands/auth
However, after setting the value, I get this upon restarting Redis:
Stopping redis-server: redis-server.
Starting redis-server: Segmentation fault (core dumped)
failed
Why is that?
The password length is limited to 512 characters.
In redis.h:
#define REDIS_AUTHPASS_MAX_LEN 512
In config.c:
} else if (!strcasecmp(argv[0],"requirepass") && argc == 2) {
if (strlen(argv[1]) > REDIS_AUTHPASS_MAX_LEN) {
err = "Password is longer than REDIS_AUTHPASS_MAX_LEN";
goto loaderr;
}
server.requirepass = zstrdup(argv[1]);
}
Now, the parsing mechanism of the configuration file is quite basic. All the lines are split using the sdssplitargs function of the sds (string management) library. This function interprets specific sequence of characters such as:
single and double quotes
\x hex digits
special characters such as \n, \r, \t, \b, \a
Here the problem is your password contains a single double quote character. The parsing fails because there is no matching double quote at the end of the string. In that case, the sdssplitargs function returns a NULL pointer. The core dump occurs because this pointer is not properly checked in the config.c code:
/* Split into arguments */
argv = sdssplitargs(lines[i],&argc);
sdstolower(argv[0]);
This is a bug that should be filed IMO.
A simple workaround would be to replace the double quote character or any other interpreted characters by an hexadecimal sequence (ie. \x22 for the double quote).
Although not documented, it seems there are limitations to the password value, particularly with the characters included, not the length.
I tried with 160 characters (just digits) and it works fine.
This
9hhNiP8MSHZjQjJAWE6PmvSpgVbifQKCNXckkA4XMCPKW6j9YA9kcKiFT6mE
too. But this
#hEpj6kNkAeYC3}#:M(:$Y,GYFxNebdH<]8dC~NLf)dv!84Z=Tua>>"A(=A<
does not.
So, Redis does not support some or all of the "special characters".
Just nailed this one with:
php: urlencode('crazy&char's^pa$$wor|]');
-or-
js: encodeURIComponent('crazy&char's^pa$$wor|]');
Then it can be used anywhere sent to the redis server via (usually) tcp
Related
I cant think of an OS (Linux, Windows, Unix) where this would cause an issue but maybe someone here can tell me if this approach is undesirable.
I would like to use a base64 encoded string as a filename. Something like gH9JZDP3+UEXeZz3+ng7Lw==. Is this likely to cause issues anywhere?
Edit: I will likely keep this to a max of 24 characters
Edit: It looks like I have a character that will cause issues. My function that generated my string is providing stings like: J2db3/pULejEdNiB+wZRow==
You will notice that this has a / which is going to cause issues.
According to this site the / is a valid base64 character so I will not be able to use a base64 encoded string for a filename.
No. You can not use a base64 encoded string for a filename. This is because the / character is valid for base64 strings which will cause issues with file systems.
https://base64.guru/learn/base64-characters
Alternatives:
You could use base64 and then replace unwanted characters but a better option would be to hex encode your original string using a function like bin2hex().
The official RFC 4648 states:
An alternative alphabet has been suggested that would use "~" as the 63rd character. Since the "~" character has special meaning in some file system environments, the encoding described in this section is recommended instead. The remaining unreserved URI character is ".", but some file system environments do not permit multiple "." in a filename, thus making the "." character unattractive as well.
I also found on the serverfault stackexchange I found this:
There is no such thing as a "Unix" filesystem. Nor a "Windows" filesystem come to that. Do you mean NTFS, FAT16, FAT32, ext2, ext3, ext4, etc. Each have their own limitations on valid characters in names.
Also, your question title and question refer to two totally different concepts? Do you want to know about the subset of legal characters, or do you want to know what wildcard characters can be used in both systems?
http://en.wikipedia.org/wiki/Ext3 states "all bytes except NULL and '/'" are allowed in filenames.
http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx describes the generic case for valid filenames "regardless of the filesystem". In particular, the following characters are reserved < > : " / \ | ? *
Windows also places restrictions on not using device names for files: CON, PRN, AUX, NUL, COM1, COM2, COM3, etc.
Most commands in Windows and Unix based operating systems accept * as a wildcard. Windows accepts % as a single char wildcards, whereas shells for Unix systems use ? as single char wildcard.
And this other one:
Base64 only contains A–Z, a–z, 0–9, +, / and =. So the list of characters not to be used is: all possible characters minus the ones mentioned above.
For special purposes . and _ are possible, too.
Which means that instead of the standard / base64 character, you should use _ or .; both on UNIX and Windows.
Many programming languages allow you to replace all / with _ or ., as it's only a single character and can be accomplished with a simple loop.
In Windows, you should be fine as long if you conform to the naming conventions of Windows:
https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file#naming-conventions.
As far a I know, any base64 encoded string does not contain any of the reserves characters.
The thing that is probably going to be a problem is the lengte of the file name.
I have a text file containing unwanted null characters (ASCII NUL, \0). When I try to view it in vi I see ^# symbols, interleaved in normal text. How can I:
Identify which lines in the file contain null characters? I have tried grepping for \0 and \x0, but this did not work.
Remove the null characters? Running strings on the file cleaned it up, but I'm just wondering if this is the best way?
I’d use tr:
tr < file-with-nulls -d '\000' > file-without-nulls
If you are wondering if input redirection in the middle of the command arguments works, it does. Most shells will recognize and deal with I/O redirection (<, >, …) anywhere in the command line, actually.
Use the following sed command for removing the null characters in a file.
sed -i 's/\x0//g' null.txt
this solution edits the file in place, important if the file is still being used. passing -i'ext' creates a backup of the original file with 'ext' suffix added.
A large number of unwanted NUL characters, say one every other byte, indicates that the file is encoded in UTF-16 and that you should use iconv to convert it to UTF-8.
I discovered the following, which prints out which lines, if any, have null characters:
perl -ne '/\000/ and print;' file-with-nulls
Also, an octal dump can tell you if there are nulls:
od file-with-nulls | grep ' 000'
If the lines in the file end with \r\n\000 then what works is to delete the \n\000 then replace the \r with \n.
tr -d '\n\000' <infile | tr '\r' '\n' >outfile
Here is example how to remove NULL characters using ex (in-place):
ex -s +"%s/\%x00//g" -cwq nulls.txt
and for multiple files:
ex -s +'bufdo!%s/\%x00//g' -cxa *.txt
For recursivity, you may use globbing option **/*.txt (if it is supported by your shell).
Useful for scripting since sed and its -i parameter is a non-standard BSD extension.
See also: How to check if the file is a binary file and read all the files which are not?
I used:
recode UTF-16..UTF-8 <filename>
to get rid of zeroes in file.
I faced the same error with:
import codecs as cd
f=cd.open(filePath,'r','ISO-8859-1')
I solved the problem by changing the encoding to utf-16
f=cd.open(filePath,'r','utf-16')
Remove trailing null character at the end of a PDF file using PHP, . This is independent of OS
This script uses PHP to remove a trailing NULL value at the end of a binary file, solving a crashing issue that was triggered by the NULL value. You can edit this script to remove all NULL characters, but seeing it done once will help you understand how this works.
Backstory
We were receiving PDF's from a 3rd party that we needed to upload to our system using a PDF library. In the files being sent to us, there was a null value that was sometimes being appended to the PDF file. When our system processed these files, files that had the trailing NULL value caused the system to crash.
Originally we were using sed but sed behaves differently on Macs and Linux machines. We needed a platform independent method to extract the trailing null value. Php was the best option. Also, it was a PHP application so it made sense :)
This script performs the following operation:
Take the binary file, convert it to HEX (binary files don't like exploding by new lines or carriage returns), explode the string using carriage return as the delimiter, pop the last member of the array if the value is null, implode the array using carriage return, process the file.
//In this case we are getting the file as a string from another application.
// We use this line to get a sample bad file.
$fd = file_get_contents($filename);
//We trim leading and tailing whitespace and convert the string into hex
$bin2hex = trim(bin2hex($fd));
//We create an array using carriage return as the delminiter
$bin2hex_ex = explode('0d0a', $bin2hex);
//look at the last element. if the last element is equal to 00 we pop it off
$end = end($bin2hex_ex);
if($end === '00') {
array_pop($bin2hex_ex);
}
//we implode the array using carriage return as the glue
$bin2hex = implode('0d0a', $bin2hex_ex);
//the new string no longer has the null character at the EOF
$fd = hex2bin($bin2hex);
In TF2 scripting, there is a simple command, echo. This simply logs a message to the console. It can be used with or without double quotes. However, I want to be able to log something to the console involving double quotes--say, the string He said, "nope.". In conventional programming, one would escape the quotes, as so:
echo "He said, \"nope.\""
However, in the TF2 console, this writes:
He said, \ nope.\
Is there a way to use quotes in echo and related commands? (E.g. say)
It's not possible to output double quotes using echo or say, only single quotes. (source) (In fact whenever you type double quotes into regular chat they're automatically changed into single quotes. I don't know why this limitation exists, I'd have to dig around.)
I know this is a very old thread, but in case anyone else is trying to find a fix for this and found this post, let me provide an example of how to use single quotes. I had an alias for switching crosshairs, as such:
alias "default" "cl_crosshair_blue 255; cl_crosshair_red 255; cl_crosshair_green 0; cl_crosshair_file "" "
This is to change the color of my crosshair, but more importantly, reset my crosshair to the "None" in options, letting the crosshair change depending on weapon, as I like most of the default crosshairs- but it would fail at the quotation mark immediately after "file", simply changing the colors and nothing more.
Using single quotes fixes this. Using ALT+0145 and ALT+0146 gives you the starting and ending single quotes (respectively) that you need. Replacing those two double quotes after "file" with the single quotes made the command work as intended.
alias "default" "cl_crosshair_blue 255; cl_crosshair_red 255; cl_crosshair_green 0; cl_crosshair_file ‘’ "
So anytime you absolutely need to use a set quotation marks twice in a command, just use single quotes.
I am aware of the naming conventions for Redis keys (this is a great link here Naming Convention and Valid Characters for a Redis Key ) but what of the values? Will I have an issue if my values include characters such as &^*$#+{ ?
From http://redis.io/topics/data-types:
Redis Strings are binary safe, this means that a Redis string can contain any kind of data, for instance a JPEG image or a serialized Ruby object.
A String value can be at max 512 Megabytes in length.
So those chars you've specified will be fine, as will any other data.
#Ruan is not exactly covering the whole story. I have looked close at that section of the Redis docs and it doesn't cover special characters.
For example, you will need to escape double quotes " with a preceding backslash \" in your key.
Also if you do have special characters in your key i.e, spaces, single or double quotes, you will need to wrap your whole key in double quotes.
The following keys are valid and you can use them to start understanding how special characters are handled.
The following allows spaces in your key.
set "users:100:John Doe" 1234
The following allows special characters by escaping them.
set "metadata:2:moniker\"#\"" 1234
For example:
I get string by command, there is a blank space \x00
127.0.0.1:6379> get "87102213_87102208"
"173275,3915125,10,\x00"
but in code in print log, the blank space is missed. Did you know why?
log_error("reply->str:%s,reply->len:%d",reply->str,reply->len);
reply->str:173275,3915125,10,,reply->len:19
Well \x00 is not a blank space, this is a nul char, which in C also happens to be the string terminator character. Most C API using strings as parameter will consider the string stops at the first \x00 character. This probably includes this log_error function.
However, Redis is binary safe, and all characters are meaningful including the nul char. When this value has been inserted, probably the size was wrong, so the nul char terminating the string was stored as well.