iText LocationTextExtractionStrategy/HorizontalTextExtractionStrategy splits text into single characters - pdf

Question

I used a extended version of LocationTextExtractionStrategy to extract connected texts of a pdf and their positions/sizes. I did this by using the locationalResult. This worked well until I tested a pdf containing texts with a different font (ttf). Suddenly these texts are splitted into single characters or small fragments.
For example "Detail" is not any more one object within the locationalResult list but splitted into six items (D, e, t, a, i, l)
I tried using the HorizontalTextExtractionStrategy by making the getLocationalResult method public:
public List<TextChunk> GetLocationalResult()
{
return (List<TextChunk>)locationalResultField.GetValue(this);
}
and using the PdfReaderContentParser to extract the texts:
reader = new PdfReader("some_pdf");
PdfReaderContentParser parser = new PdfReaderContentParser(reader);
var strategy = parser.ProcessContent(i, HorizontalTextExtractionStrategy());
foreach (HorizontalTextExtractionStrategy.HorizontalTextChunk chunk in strategy.GetLocationalResult())
{
// Do something with the chunk
}
but this also returns the same result.
Is there any other way to extract connected texts from a pdf?

Answer 1

I used a extended version of LocationTextExtractionStrategy to extract connected texts of a pdf and their positions/sizes. I did this by using the locationalResult. This worked well until I tested a pdf containing texts with a different font (ttf). Suddenly these texts are splitted into single characters or small fragments.
This problem is due to wrong expectations concerning the contents of the LocationTextExtractionStrategy.locationalResult private list member variable.
This list of TextChunk instances contains the pieces of text as they were forwarded to the strategy from the parsing framework (or probably as they were preprocessed by some filter class), and the framework forwards each single string it encounters in a content stream separately.
Thus, if a seemingly connected word in the content stream actually is drawn using multiple strings, you get multiple TextChunk instances for it.
There actually is some "intelligence" in the method getResultantText joining these chunks properly, adding a space where necessary and so on.
In case of your document, "DETAIL " usually is drawn like this:
[<0027> -0.2<00280037> 0.2<0024002c> 0.2<002f> -0.2<0003>] TJ
As you see there are slight text insertion point moves between 'D' and 'E', 'T' and 'A', 'I' and 'L', and 'L' and ' '. (Such mini moves usually represent kerning.) Thus, you'll get individual TextChunk instances for 'D', 'ET', 'AI', and 'L '.
Admittedly, the LocationTextExtractionStrategy.locationalResult member is not very well documented; but as it is a private member, this IMHO is forgivable.
That this worked well for many documents is due to many PDF creators not applying kerning and simply drawing connected text using single string objects.
The HorizontalTextExtractionStrategy is derived from the LocationTextExtractionStrategy and mainly differs from it in the way it arranges the TextChunk instances to a single string. Thus, you'll see the same fragmentation here.
Is there any other way to extract connected texts from a pdf?
If you want "connected texts" as in "atomic string objects in the content stream", you already have them.
If you want "connected texts" as in "visually connected texts, no matter where the constituent letters are drawn in the content stream", you have to glue those TextChunk instances together like the LocationTextExtractionStrategy and HorizontalTextExtractionStrategy do in getResultantText in combination with the comparison methods in their respective TextChunkLocationDefaultImp and HorizontalTextChunkLocation implementations.

Answer 2

After debugging deep into the iTextSharp library I figured out that my texts are drawn with the TJ operator as mkl also mentioned.
[<0027> -0.2<00280037> 0.2<0024002c> 0.2<002f> -0.2<0003>] TJ
iText processes these texts not as a single PdfString but as an array of PdfObjects which ends up in calling renderListener.RenderText(renderInfo) for each PdfString item in it (see ShowTextArray class and DisplayPdfString method). In the RenderText method however the information about the relation of the pdf strings within the array got lost and every item is added to locationalResult as an independent object.
As my goal is to extract the "argument of a single text drawing instruction" I extended the PdfContentStreamProcessor class about a new method ProcessTexts which returns a list of these atomic strings. My workaround is not very pretty as I had to copy paste some private fields and methods from the original source but it works for me.
class PdfContentStreamProcessorEx : PdfContentStreamProcessor
{
private IDictionary<int, CMapAwareDocumentFont> cachedFonts = new Dictionary<int, CMapAwareDocumentFont>();
private ResourceDictionary resources = new ResourceDictionary();
private CMapAwareDocumentFont font = null;
public PdfContentStreamProcessorEx(IRenderListener renderListener) : base(renderListener)
{
}
public List<string> ProcessTexts(byte[] contentBytes, PdfDictionary resources)
{
this.resources.Push(resources);
var texts = new List<string>();
PRTokeniser tokeniser = new PRTokeniser(new RandomAccessFileOrArray(new RandomAccessSourceFactory().CreateSource(contentBytes)));
PdfContentParser ps = new PdfContentParser(tokeniser);
List<PdfObject> operands = new List<PdfObject>();
while (ps.Parse(operands).Count > 0)
{
PdfLiteral oper = (PdfLiteral)operands[operands.Count - 1];
if ("Tj".Equals(oper.ToString()))
{
texts.Add(getText((PdfString)operands[0]));
}
else if ("TJ".Equals(oper.ToString()))
{
string text = string.Empty;
foreach (PdfObject entryObj in (PdfArray)operands[0])
{
if (entryObj is PdfString)
{
text += getText((PdfString)entryObj);
}
}
texts.Add(text);
}
else if ("Tf".Equals(oper.ToString()))
{
PdfName fontResourceName = (PdfName)operands[0];
float size = ((PdfNumber)operands[1]).FloatValue;
PdfDictionary fontsDictionary = resources.GetAsDict(PdfName.FONT);
CMapAwareDocumentFont _font;
PdfObject fontObject = fontsDictionary.Get(fontResourceName);
if (fontObject is PdfDictionary)
_font = GetFont((PdfDictionary)fontObject);
else
_font = GetFont((PRIndirectReference)fontObject);
font = _font;
}
}
this.resources.Pop();
return texts;
}
string getText(PdfString #in)
{
byte[] bytes = #in.GetBytes();
return font.Decode(bytes, 0, bytes.Length);
}
private CMapAwareDocumentFont GetFont(PRIndirectReference ind)
{
CMapAwareDocumentFont font;
cachedFonts.TryGetValue(ind.Number, out font);
if (font == null)
{
font = new CMapAwareDocumentFont(ind);
cachedFonts[ind.Number] = font;
}
return font;
}
private CMapAwareDocumentFont GetFont(PdfDictionary fontResource)
{
return new CMapAwareDocumentFont(fontResource);
}
private class ResourceDictionary : PdfDictionary
{
private IList<PdfDictionary> resourcesStack = new List<PdfDictionary>();
virtual public void Push(PdfDictionary resources)
{
resourcesStack.Add(resources);
}
virtual public void Pop()
{
resourcesStack.RemoveAt(resourcesStack.Count - 1);
}
public override PdfObject GetDirectObject(PdfName key)
{
for (int i = resourcesStack.Count - 1; i >= 0; i--)
{
PdfDictionary subResource = resourcesStack[i];
if (subResource != null)
{
PdfObject obj = subResource.GetDirectObject(key);
if (obj != null) return obj;
}
}
return base.GetDirectObject(key); // shouldn't be necessary, but just in case we've done something crazy
}
}
}