Bert tokenizer wont work with tensor format (tensorflow) - tensorflow

Question

This may be a silly question but im new using tf. I have the following code but the tokenizer wont use the strings inside the tensor.
import tensorflow as tf
docs = tf.data.Dataset.from_tensor_slices([['hagamos que esto funcione.'], ["por fin funciona!"]])
from transformers import AutoTokenizer, DataCollatorWithPadding
import numpy as np
checkpoint = "dccuchile/bert-base-spanish-wwm-uncased"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
def tokenize(review):
return tokenizer(review)
tokens = docs.map(tokenize)
I get the folowing output:
ValueError: in user code:
File "<ipython-input-54-3272cedfdcab>", line 13, in tokenize *
return tokenizer(review)
File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py", line 2429, in __call__ *
raise ValueError(
ValueError: text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).
while my expected output is something like this:
tokenizer('esto al fin funciona!')
{'input_ids': [4, 1202, 1074, 1346, 4971, 1109, 5], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}
Any idea how to make it work?

Answer 1

As mentioned in the error you have to pass the inputs to the tokenzier as a string, list(str) or list(list(str)).
Please check the working code below.
import tensorflow as tf
docs = ['hagamos que esto funcione.', "por fin funciona!"]
from transformers import AutoTokenizer, DataCollatorWithPadding
checkpoint = "dccuchile/bert-base-spanish-wwm-uncased"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
def tokenize(review):
return tokenizer(review)
tokens = tokenizer(docs)
The output of the above code is:
{'input_ids': [[4, 8700, 1041, 1202, 13460, 1008, 5], [4, 1076, 1346, 4971, 1109, 5]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1]]}