iTextSharp - Incorrect text position - pdf

Question

When extracting the position of the words in this example:
http://www.dertour.de/static/agb/2015/sommer/DER_Deutschland_So15.pdf
with iTextSharp 5.5.8
I'm getting 'incorrect' coordinates for some words. For example on line 17 of the first paragraph: 'gehen oder im Widerspruch zur Reiseaus-'
the x-values of the left,top position of the words are 118, 217, 296, 350, 524, 587. Only the first value seems correct (118,208,277,320,487,540). The x-value of the right-bottom point of the space-character between 'gehen' and 'oder' is 208, which seems correct and also seems to be the correct x-pos for the word 'oder'. Maybe it has something to do with the fillmode of the paragraph, but I'm not sure which actions I should perform to get the right coordinates.
I'm using LocationTextExtractionStrategy and calculate the word-positions to a 300 dpi coordinate system.
public override void RenderText(TextRenderInfo renderInfo)
{
// for the provided example
// uUnit = 1
// originX = 33.862
// originY = 33.555
// dpi = 300
// above values where calculated with code:
// PdfNumber userUnit = pageDict.GetAsNumber(PdfName.USERUNIT);
// if (userUnit != null)
// {
// uUnit = userUnit.FloatValue;
// }
// Rectangle dim = reader.GetPageSize(i);
// float originX = dim.Left;
// float originY = dim.Bottom;
// calculate coordinates:
renderInfo.GetText();
LineSegment segment = renderInfo.GetBaseline();
List<TextRenderInfo> charInfo = renderInfo.GetCharacterRenderInfos().ToList();
foreach (TextRenderInfo item in charInfo)
{
LineSegment char_segment = item.GetBaseline();
int char_left = (int)Math.Round((char_segment.GetStartPoint()[0] - originX) * dpi * uUnit / 72.0f);
int char_top = (int)Math.Round((item.GetAscentLine().GetEndPoint()[1] - originY) * dpi * uUnit / 72.0f);
int char_right = (int)Math.Round((char_segment.GetEndPoint()[0] - originX) * dpi * uUnit / 72.0f);
int char_bottom = (int)Math.Round((item.GetDescentLine().GetStartPoint()[1] - originY) * dpi * uUnit / 72.0f);
}
}

Answer 1

This indeed is a bug in iText & iTextSharp:
The lines with the extremely inaccurate x coordinates are those for which a large wordspacing value is set, e.g. your line:
0.2861 Tw T*
[<0047004500480045004E0000>-286<004F0044004500520000>-286<0049004D0000>-231<003700490044004500520053005000520055004300480000>-286<005A005500520000>-286<00320045004900530045004100550053000D>]TJ
(That 0.2861 argument for Tw is large.)
According to the ToUnicode map of the font in question the 0000 at the end of each word maps to the space character. Thus, iText here adds the word spacing value when calculating the x coordinates because according to the PDF specification ISO 32000-1:
Word spacing works the same way as character spacing but shall apply only to the ASCII SPACE character
(First sentence of section 9.3.3 Word Spacing)
Unfortunately it does not take into account
Word spacing shall be applied to every occurrence of the single-byte character code 32 in a string when using
a simple font or a composite font that defines code 32 as a single-byte code. It shall not apply to occurrences of
the byte value 32 in multiple-byte codes.
(Last sentence of section 9.3.3 Word Spacing)
At the 0000 above, therefore, word spacing must not be applied even though it is mapped to the space character because
the font encoding in question is purely multi-byte and
even in case of single-byte encoded space characters the word spacing is applied only at the single-byte code 32, not at a code which merely maps to the space character with ASCII code 32.
Usually this is not a problem during text extraction, usually PDF generators which encode space characters using multi-byte encodings are aware that word spacing does not apply for them and, therefore, don't change the word spacing from its default 0 value, so the iText bug here does no harm. Usage of word spacing instructions usually indicates that fonts are used which do map the single-byte code 32 to the space character.
Your PDF, on the other hand, seems to not have been created with that fact on the mind, it looks like first the word spacing has been set (0.2861 Tw), and after recognizing that it made no difference, explicit gaps have been added (-286 in the TJ instruction). (Or that was part of the development history of the PDF generator in question.)
Please be aware that positive values in the TJ argument mean a shift to the left, so negative values (as claimed for the -286 above) indeed widen or add gaps:
array TJ Show one or more text strings, allowing individual glyph positioning. Each element of array shall be either a string or a number. If the element is a string, this operator shall show the string. If it is a number, the operator shall adjust the text position by that amount; that is, it shall translate the text matrix, Tm . The number shall be expressed in thousandths of a unit of text space (see 9.4.4, "Text Space Details"). This amount shall be subtracted from the current horizontal or vertical coordinate, depending on the writing mode. In the default coordinate system, a positive adjustment has the effect of moving the next glyph painted either to the left or down by the given amount. Figure 46 shows an example of the effect of passing offsets to TJ.
(Table 109 – Text-showing operators in ISO 32000-1)