How to reduce a string to ASCII 7 characters for indexing purposes? - indexing

Question

I am working on an application which must index certain sentences. Currently using Java and PostgreSQL. The sentences may be in several languages like French and Spanish using accents and other non-ASCII symbols.
For each word I want to create an index-able equivalent so that a user can perform a search insensitive to accents (transliteration). For example, when the user searches "nacion" it must find it even if the original word stored by the application was "Nación".
What could be the best strategy for this? I am not necessarily restricted only to PostgreSQL, nor the internal indexed value needs to have any similarity with the original word. Ideally, it should be a generic solution for converting any Unicode string into an ASCII string insensitive to case and accents.
So far I am using a custom function shown below which naively just replaces some letters with ASCII equivalents before storing the indexed value and does the same on query strings.
public String toIndexableASCII (String sStrIn) {
if (sStrIn==null) return null;
int iLen = sStrIn.length();
if (iLen==0) return sStrIn;
StringBuilder sStrBuff = new StringBuilder(iLen);
String sStr = sStrIn.toUpperCase();
for (int c=0; c<iLen; c++) {
switch (sStr.charAt(c)) {
case 'Á':
case 'À':
case 'Ä':
case 'Â':
case 'Å':
case 'Ã':
sStrBuff.append('A');
break;
case 'É':
case 'È':
case 'Ë':
case 'Ê':
sStrBuff.append('E');
break;
case 'Í':
case 'Ì':
case 'Ï':
case 'Î':
sStrBuff.append('I');
break;
case 'Ó':
case 'Ò':
case 'Ö':
case 'Ô':
case 'Ø':
sStrBuff.append('O');
break;
case 'Ú':
case 'Ù':
case 'Ü':
case 'Û':
sStrBuff.append('U');
break;
case 'Æ':
sStrBuff.append('E');
break;
case 'Ñ':
sStrBuff.append('N');
break;
case 'Ç':
sStrBuff.append('C');
break;
case 'ß':
sStrBuff.append('B');
break;
case (char)255:
sStrBuff.append('_');
break;
default:
sStrBuff.append(sStr.charAt(c));
}
}
return sStrBuff.toString();
}

Answer 1

String s = "Nación";
String x = Normalizer.normalize(s, Normalizer.Form.NFD);
StringBuilder sb=new StringBuilder(s.length());
for (char c : x.toCharArray()) {
if (Character.getType(c) != Character.NON_SPACING_MARK) {
sb.append(c);
}
}
System.out.println(s); // Nación
System.out.println(sb.toString()); // Nacion
How this works:
It splits up international characters to NFD decomposition (ó becomes o◌́), then strips the combining diacritical marks.
Character.NON_SPACING_MARK contains combining diacritical marks (Unicode calls it Bidi Class NSM [Non-Spacing Mark]).

Answer 2

The one obvious improvement for your current code: use a Map<Character, Character> that you prefill with your mappings.
And then simply check if that Map has a mapping; of so; use that; otherwise use the original character.
And as Androbin explains, there are special maps that do not rely on objects, but work with primitive types, like this trove. So, depending on your solution and requirements; you could look into that.