EDITED at 17:54 26.02.2020
PDFs in the following link look sane but give garbled text when copying because they lack cmap. I don't understand why rendering is performed properly and want to know which information is used to determine characters to render.
https://github.com/angea/PDF101/tree/master/handcoded/textextract
Please note that I'm asking for mechanism but not for solution.
ACCORDING TO https://tex.stackexchange.com/questions/526157/what-is-identity-h-encoding-should-it-be-avoided-and-if-so-how
CMAP is a mapping between CID and GID. It can render when CMAP and file is available.
TOUNICODE CMAP is a mapping between CID and GID. It makes you can extract information from PDF.
There are two fonts on page 1, and both are missing any encoding information in the PDF metadata. In particular there is no ToUnicode map.
Therefore PDF readers have to rely on the font itself, and possibly the character codes used in the content stream.
In the screenshot below, the left side is the font data in the PDF, and on the right is the content stream of the first page. As you can see, the first character code is 0x2e, which maps to the glyph "T" but in Unicode U+002e is "period". The next character code is 0x08, which is a control character. This is why, if you select text from the PDF, the first character would be "." and the second would be garbage.
Why PDF without cmap can render characters?
Because the fonts internal CMap maps the character codes in the PDF page content stream to the correct glyph in the font, so you see glyphs that make sense. However, both the PDF and the font itself, are missing any sensible Unicode mappings, hence you get garbage when you copy+paste the text.
Related
A refrence to pdf informs that a pdf dictionary to defined a font resource needs to contain a property /Widhts giving this information:
(Required except for the standard 14 fonts; indirect reference
preferred) An array of ( LastChar − FirstChar + 1) widths, each
element being the glyph width for the character code that equals
FirstChar plus the array index. For character codes outside the range
FirstChar to LastChar , the value of MissingWidth from the
FontDescriptor entry for this font is used. The glyph widths are
measured in units in which 1000 units corresponds to 1 unit in text
space. These widths must be consistent with the actual widths given in
the font program. (See implementation note 61 in Appendix H.)
emphasis added.
What good is it to provide the widths again is they are obviously included in the font program?
Plainly: Can somebody confirm or reject wether the information one is supposed to provide here, the glyph width is blantantly redundant information, considering it is even mentioned to be contained in the font-program?
Or do some font programs inlcude glyphs without specifying their widths?
Is it because there are font programs that do not include the widths, or is this merely an execercise in patience, indented to complicate the generation of PDF files, hoping people then stick to Adobe software?
Are the /Widths entries required to test if a referenced font (being not embedded), is "correct" (i.e. the pdf viewer is supposed to check if the font-program wanted by the pdf, might be the one found on the platform, comparing the /Widths)?
The Widths array is documented as being present so that application programs can determine the metrics of glyphs without being required to decode a font. This might be of use (for example) when drawing a selection box around text, or highlighting text in some manner.
See pages 393 and 394 of the PDF 1.7 specification:
The width information for each glyph is stored both in the font
dictionary and in the font program itself. (The two sets of widths
must be identical; storing this information in the font dictionary,
although redundant, enables a consumer application to determine glyph positioning without having to look inside
the font program.)
I should also mention that there are many PDF producers which regard abusing the Widths array as a convenient way to alter the spacing of a font. Where the Widths of the Font array do not match the metrics of the glyphs in the font program, Acrobat uses the Widths array values (which is the implementation note in Appendix H referred to by the text you quoted). I also seem to recall that the latest version of the specification lifts the exception for the base 14 fonts, all fonts are supposed to have a /Widths array now.
We've got numerouus examples of PDF files where the metrics array do not match the Widths in the font program.
Note that the Preflight checker in Acrobat Pro, when checking for PDF/A compatibility, will throw an error if the Widths and metrics differ.
So while it is technically true that the /Widths array is redundant, because the same information can be retrieved from the font, it is convenient for some applications to have the informaiton in a more readily accessible form and if (as a PDF consumer) you hope to match the rendering from Acrobat, you need to use it.
I was trying to convert a pdf document into text file. everything works until i open the output file to see its unreadable the characters are in some Chinese font
" 琀攀猀琀 "
this is my command line
gswin64c.exe -ps2ascii -sDEVICE=txtwrite -sOutputFile=outputtext.txt test.pdf
im i doing something wrong?
You haven't posted the file, so its not possible to be absolutely certain, however....
Almost certainly the text in your PDF file is not encoded using an ASCII encoding scheme (possibly contains sunset fonts), and does not contain a ToUnicode CMap for the font in question. Additionally the glyph names are not standard names (or its a TrueType font, which don't have named glyphs).
Without any of the above information txtwrite doesn't have any clue what the character codes represent, and so simply emits them verbatim.
Given that you are seeing Chinese glyphs, I would suspect that the original font is a CIDFont, probably a TrueType font, subset and has no ToUnicode CMap.
In this case, the only way to get the text out will be to use OCR.
I have a pdf which contains many fonts and What is the best way to check whether it contains font that belongs to Arial font family ?
Is this possible in any language?
I couldn't find any library or language that could do this.
So, I tried by converting pdf to image using ImageMagick and segmented all alphabets present in the image(pdf).And then I tried to compare all segmented alphabets to segmented images of arial font family alphabets which worked fine.
I created all datasets using MS Word.But arial font family looks different in different editors.By 'looks different', I mean the segmented image of same alphabet has different pixel values in different editors.And also alphabet of 10pt size has different dimensions in different editors.So, this method doesn't work.
Any Suggestions on how to do this? May be using svg file or ps file
I also came to learn that, In pdf's alphabets are rendered using Bezier curves, where each bezier curve is drawn using some control points and nodes.
Are these control points, same for all alphabets that belong to one font family? If Yes, how to extract control points of alphabets in pdf as these can be used to detect font family.
There can be three types of text in your document:
text that isn't real text, but part of a raster image,
vector text drawn by PDF syntax without using a real font,
vector text using a real font.
The answer to your question depends on the type of text you're confronted with:
There is no way to extract font information if the text is not real text, but part of a raster image. You need an OCR tool to convert the pixels into characters, but you won't get any info about the font family. You could try to compare pixels, but you've already tried that and you've discovered that this isn't trivial (one might consider your current solution as a bad workaround / bad design).
You describe text that is drawn on a page using Bézier curves. Although, it's possible to draw text like this, you won't find many PDFs that are drawn like this. The reason is obvious: every time you'd need a specific glyph, let's say A, you would need to add the syntax to draw that glyph on the page, leading to plenty of redundant PDF syntax.
PDFs usually work with fonts. A font is stored in a PDF file using a font dictionary. The syntax that makes up a page refers to that font using a name that can be chosen by the PDF producer, but that corresponds with an entry in the page resources that contains a reference to the font dictionary. Each font has an encoding mapping characters to glyphs. In the page content we use characters, based on these characters, glyphs will be selected in the font.
You are asking about the font family. This information is stored in the font dictionary. Take a look at my answer to the question What are the ways of checking if piece of text in PDF documernt is bold using iTextSharp and you'll get an idea of what such a font dictionary looks like.
Do you see the /BaseFont entry in the font dictionary? It has values such as JOJJAH+TT116t00. In this case, the name of the font is "TT116t00", but what is "JOJJAH"? That is explained in my answer to the question What are the extra characters in the font name of my PDF?
Not all fonts are embedded. Sometimes the name of the font is sufficient for the viewer to know what the glyphs look like. For instance: there are 14 Standard Type 1 fonts that every viewer should be able to render.
Arial isn't one of those fonts, so if you want to be sure that Arial is rendered correctly, that font needs to be embedded. The font dictionary will refer to a Font Descriptor where you'll find the syntax to draw glyphs using linear paths, Bézier curves, etc. Suppose that you need the character A, then the font descriptor will contain some syntax that knows how to draw that character. The font dictionary will also have a map that maps the character A to the glyph A. Now when you need that glyph in your content, you can just use the character A and that will refer to the syntax that draws the glyph A. That syntax is stored inside the PDF only once.
Suppose that a PDF has the full Arial font embedded, then the value of /BaseFont would be Arial. However, if we'd embed the full Arial font, the PDF would be bloated. There are way too many characters in Arial; we don't need them all. That's why we'll only embed one or more subsets. When you see 6 characters followed by a + sign in the /BaseFont entry, you have discovered a font subset.
Getting the /BaseFont entry of a font dictionary can be done using different libraries. On the official iText site, we have different Q&As that explain how to Inspect a PDF. There's also an example that lists the fonts used in a PDF. Maybe that can be helpful.
NOTE: as explained in the help section, more specifically on the page What topics can I ask about here?, you will find rule #4: Questions asking us to recommend or find a book, tool, software library, tutorial or other off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam.
I have given you the general information about where to find font information inside a PDF, but it's not allowed for you to ask questions to recommend the best tool to do this. Sorry for that.
I am attempting to gain a better understanding of how a PDF stores text. Generally speaking, when a PDF is created from an application like MS Word (or in my case SQL Server Reporting Services), how is text stored by the PDF? I would hope that the resulting document isn't OCR'ed in this particular scenario the way it would be if the original PDF document had been created from an image.
To get a bit more detailed, I am trying to understand how text extractors for PDFs work. My initial understanding of PDF was that it stored (PostScript) instructions on how to draw the "image" of the document to a page or a printer, and that there was no actual text contained within the document itself. Subsequently, I was thinking that a text extractor might reverse-engineer such instructions to generate the text that the PDF would otherwise generate. I am not confident of this, though.
PDF contains several different types of objects; not only vectorial or raster drawing instructions. Text in in particular is represented by text elements. These include a string of characters that should be drawn at certain positions using a specific font.
Text extraction from PDFs can be a complicated affair because the file format is oriented for page layout. A text element may be an entire paragraph, or a single character. Even a single word may consist of several text elements if different typefaces are mixed. Also, the characters are not necessarily encoded in a standard encoding such as Unicode. They may be encoded in a way specific to a particular font.
If you are lucky enough to deal with Tagged PDF files such as PDF/A or PDF/UA, text extraction can be a lot easier because text spans are identified as such, and a mapping to Unicode characters is defined.
Wikipedia doesn't have the complete specification but does serve as an introduction: http://en.wikipedia.org/wiki/Portable_Document_Format#Text
I am extracting text from a PDF file using PdfBox. When the PDF does not contain any embedded fonts everything works fine. The problem occurs when there are some TrueType embedded fonts. I discovered that in same cases the embedded fonts replace the shape of default characters with some other shapes. For example a char code for 'ï' is used to encode 'ł'. I am aware that I cannot get the real shape of the character without any mapping or OCR. I would like to know which characters might be redefined by the embedded characters. My question is how can I know which characters in the PDF stream are defined by the embedded fonts?