How fix double encoding in PostgreSQL? - sql

Question

I have a table in PostgreSQL with words, but some words have invalid UTF-8 chars like 0xe7e36f and 0xefbfbd.
How I can identify all chars inside words that are invalid and replace they with some symbol like ??
EDIT: My database is in UTF-8, but I think there are double encoding from various other encodings. I think this because when I tried to convert to one type as LATIN1, I get an error saying that some char don't exists in that encoding, when I change to LATIN2 I get
the same error, but with another character.
So, what is possible to do to solve this?

Answer 1

Usage
It's a solution for my specific case, but maybe with some modifications can help another people.
Usage
SELECT fix_wrong_encoding('LATIN1');
Function
-- Convert words with wrong encoding
CREATE OR REPLACE FUNCTION fix_wrong_encoding(encoding_name VARCHAR)
RETURNS VOID
AS $$
DECLARE
r RECORD;
counter INTEGER;
token_id INTEGER;
BEGIN
counter = 0;
FOR r IN SELECT t.id, t.text FROM token t
LOOP
BEGIN
RAISE NOTICE 'Converting %', r.text;
r.text := convert_from(convert_to(r.text,encoding_name),'UTF8');
RAISE NOTICE 'Converted to %', r.text;
RAISE NOTICE 'Checking existence.';
SELECT id INTO token_id FROM token WHERE text = r.text;
IF (token_id IS NOT NULL) THEN
BEGIN
RAISE NOTICE 'Token already exists. Updating ids in textblockhastoken';
IF(token_id = r.id) THEN
RAISE NOTICE 'Token is the same.';
CONTINUE;
END IF;
UPDATE textblockhastoken SET tokenid = token_id
WHERE tokenid = r.id;
RAISE NOTICE 'Removing current token.';
DELETE FROM token WHERE id = r.id;
END;
ELSE
BEGIN
RAISE NOTICE 'Token don''t exists. Updating text in token';
UPDATE token SET text = r.text WHERE id = r.id;
END;
END IF;
EXCEPTION WHEN untranslatable_character THEN
--do nothing
WHEN character_not_in_repertoire THEN
--do nothing
END;
counter = counter + 1;
RAISE NOTICE '% token converted', counter;
END LOOP;
END
$$
LANGUAGE plpgsql;