How to avoid punctuationduring tokenization using Stanford NLP - tokenize

Question

I am using standford core NLP. I have tried the following example. this example can tokenize the words from the text. However it also extract punctuation such as comma, full stop etc. I was wondering how to set the properties that allow not to extract punctuation or alternatively is there any other way to do the same. Here is the code example. I know its easy using Python but not sure how to do it in Java. Please suggest.
props = new Properties();
props.setProperty("annotators", "tokenize, ssplit");
pipeline = new StanfordCoreNLP(props);
String text = "this is simple text written in English,Spanish etc."
// create an empty Annotation just with the given text
Annotation document = new Annotation(text);
pipeline.annotate(document);
List<CoreMap> sentences = document.get(SentencesAnnotation.class);
for(CoreMap sentence: sentences) {
for (CoreLabel token: sentence.get(TokensAnnotation.class)) {
// this is the text of the token
String word = token.get(TextAnnotation.class);
}
}

Answer 1

We don't have any tokenizer option for skipping these, but it shouldn't be difficult. Punctuation strings are a closed class.
You could match tokens which are punctuation using a regular expression. (Use \p{Punct}; see e.g. Punctuation Regex in Java ). Then just drop tokens whose text content matches such a regex.