Convert string tensor to lower case - tensorflow

Question

Is there any way to convert a string tensor to lower case, without evaluating in the session ? Some sort of tf.string_to_lower op ?
More specifically, I am reading data from tfrecords files, so my data is made of tensors. I then want to use tf.contrib.lookup.index_table_from_* to lookup indices for words in the data, and I need this to be case-insensitive. Lowering the data before writing it to tfrecords is not an option, as it needs to be kept in original format. One option would be to store both original and lowered, but I'd like to avoid this if possible.

Answer 1

Here's an implementation with tensorflow ops:
def lowercase(s):
ucons = tf.constant_initializer([chr(i) for i in range(65, 91)])
lcons = tf.constant_initializer([chr(i) for i in range(97, 123)])
upchars = tf.constant(ucons, dtype=tf.string)
lchars = tf.constant(lcons, dtype=tf.string)
upcharslut = tf.contrib.lookup.index_table_from_tensor(mapping=upchars, num_oov_buckets=1, default_value=-1)
splitchars = tf.string_split(tf.reshape(s, [-1]), delimiter="").values
upcharinds = upcharslut.lookup(splitchars)
return tf.reduce_join(tf.map_fn(lambda x: tf.cond(x[0] > 25, lambda: x[1], lambda: lchars[x[0]]), (upcharinds, splitchars), dtype=tf.string))
if __name__ == "__main__":
s = "komoDO DragoN "
sess = tf.Session()
x = lowercase(s)
sess.run(tf.global_variables_initializer())
sess.run(tf.tables_initializer())
print(sess.run([x]))
returns [b'komodo dragon ']

Answer 2

You can use tf.py_func to use a python function that manipulates your string and it's executed withing the graph.
You can do something like:
# I suppose your string tensor is tensorA
lower = tf.py_func(lambda x: x.lower(), [tensorA], tf.string, stateful=False)
# Starting from TF 2.0 `tf.py_func` is deprecated so correct code will be
lower = tf.py_function(lambda x: x.numpy().lower(), [tensorA], tf.string)

Answer 3

Unfortunately, tf.py_func doesn't work in all cases as serving or TFT. The following snippet is a simple in-graph TF solution.
import tensorflow as tf
def to_lower_case(text):
chars = tf.strings.unicode_decode(text, input_encoding='UTF-8')
capital_mask = tf.logical_and(tf.greater_equal(chars, 65), tf.less(chars, 91))
chars = chars + tf.cast(capital_mask, tf.int32) * 32
return tf.strings.unicode_encode(chars, output_encoding='UTF-8')
with tf.Session() as sess:
print(sess.run(to_lower_case('Test')))

Answer 4

In Tensorflow 1.14, a lower op has been added. A short code snippet (in eager execution mode) looks like the following:
astring = tf.constant('A String', dtype=tf.string)
tf.strings.lower(astring)
<tf.Tensor: id=79, shape=(), dtype=string, numpy=b'a string'>

Answer 5

If the characters your are using are limited to ASCII characters, I have a working solution for that (in graph). The idea is:
Create a lookup table with keys whose values are in [32, 127), while values the same, except those in [65, 91) replaced with [97, 123). Method: tf.contrib.lookup.HashTable.
Split the string into characters. Method: tf.string_split
Using lookup to map upper case characters to lower case characters. Method: case_table.lookup (if the HashTable was called case_table).
Join the characters back into the string. Method: tf.reduce_join.
A concrete example can be found here: https://github.com/bshao001/ChatLearner/blob/master/chatbot/tokenizeddata.py
This approach should be able to be expanded to other character sets. Notice that if you were trying to convert only those characters that need to be changed (such as 26 English uppercase characters), that would be harder (not sure doable or not) as you will have to use tf.cond method and check if the character is in the key set or not, and would be less efficient too.