PDF m l operators - pdf

I am using a PDF parser to extract lines from a pdf document. It fails on a specific doc generated pdf. The smallest pdf that it fails for has a 1 cell 1 row table, but the stream shows a 2 cell 1 row table. I have these questions:-
Why does the stream show 2 cells instead of just 1?
What are those re operators for, as there are no rectangles?
Who generates these instructions, is it MS Word? Or the PDF Printing application (Cute PDF Writer)?
Here is the pdf :-
Here is the relevant stream:-
stream
q 0.12 0 0 0.12 0 0 cm
/R7 gs
q
647 5996 m
700 5996 l
700 5885 l
647 5885 l
h
W n
0 0 0 rg
q
8.33333 0 0 8.33333 0 0 cm BT
/R8 11.04 Tf
0.998087 0 0 1 77.64 709.2 Tm
()Tj
ET
Q
Q
q
700 5996 m
746 5996 l
746 5885 l
700 5885 l
h
W n
0 0 0 rg
q
8.33333 0 0 8.33333 0 0 cm BT
/R8 11.04 Tf
0.998087 0 0 1 84 709.2 Tm
()Tj
ET
Q
Q
0 0 0 rg
600 5996 4 4 re
f
600 5996 4 4 re
f
604 5996 3892 4 re
f
4496 5996 4 4 re
f
4496 5996 4 4 re
f
600 5884 4 112 re
f
600 5880 4 4 re
f
600 5880 4 4 re
f
604 5880 3892 4 re
f
4496 5884 4 112 re
f
4496 5880 4 4 re
f
4496 5880 4 4 re
f
q
8.33333 0 0 8.33333 0 0 cm BT
/R8 11.04 Tf
0.998087 0 0 1 72 695.28 Tm
()Tj
ET
Q
Q
endstream
and here is the image drawn using the m and l instructions above :-

Why does the stream show 2 cells instead of just 1?
The stream does not show any cells at all. Only tagged PDFs may have a certain awareness of tables and table cells but your PDF does not look tagged.
What you (considering your question title) appear to mean are the sequences
647 5996 m
700 5996 l
700 5885 l
647 5885 l
h
W n
and
700 5996 m
746 5996 l
746 5885 l
700 5885 l
h
W n
But all they do is intersecting the current clip path with a rectangle. Thus, following drawing operations are restricted to the respective rectangle. Such restriction can be found in PDFs in many situations, table cells are only one of them, and such clip path changes are not even necessary for table cells...
Furthermore, considering the preceding transformation matrix change
0.12 0 0 0.12 0 0 cm
the rectangles above are fairly small, each probably large enough for a single character.
What are those re operators for, as there are no rectangles?
Well, they are rectangles.
Very small in height and/or width, but rectangles nonetheless.
And they are filled rectangles, cf. the f operator.
To make a long story short, the "lines" around the area we perceive as a table cell, are actually filled rectangles:
604 5996 3892 4 re
600 5884 4 112 re
604 5880 3892 4 re
4496 5884 4 112 re
Furthermore the corners of the cell are drawn as tiny squares (and each corner twice):
600 5996 4 4 re
600 5996 4 4 re
4496 5996 4 4 re
4496 5996 4 4 re
600 5880 4 4 re
600 5880 4 4 re
4496 5880 4 4 re
4496 5880 4 4 re
Thus, these re instructions give you the border edges and corners of what we perceive as table cell.
Who generates these instructions, is it MS Word? Or the PDF Printing application (Cute PDF Writer)?
The concrete instructions you see are PDF instructions. Thus, your printing application creates them.
Of course, though, your printing application creates them because that is how it interprets the MS Word output...

Cute PDF Writer apparently (from a quick glance on their web page) uses the Windows printing system. In general, in cases like this, you print from MS Word, and MS Word will try to use Windows methods to draw the lines and other items, which the printer driver (Cute PDF Writer in this case) will then translate to PDF commands. An intermediate stage with first rendering to PostScript and then translating to PDF is also possible.
So, that would mean that MS Word is responsible for the fact that two cells are drawn.
I only see one rectangle in the image of the PDF that you posted, so I'm not sure what is happening here. Also, I can't explain the other re commands. The rectangles in the second image look like they might be a frame around a two-on-one printed page, but the coordinates look strange, so it could also be something else.

Related

Edit binary data in PDF with SED / BBE (change colors in a PDF)

I want to change some background colors in a batch of PDF's
I found out that the color information is stored in the first stream - endstream block
in a format like such: 1 1 1 sc which in this example represents white #FFFFFF
here an example after I decode the binary stream with
qpdf --qdf --object-streams=disable IN.pdf OUT.pdf
stream
q Q q /Cs1 cs 0.9686275 0.9725490 0.9764706 sc 0 12777 m 600 12777 l 600 0
l 0 0 l h f 0 12777 m 600 12777 l 600 0 l 0 0 l h f ➡️1 1 1 sc⬅️ 0 12575 m 600
12575 l 600 12308 l 0 12308 l h f 0.1254902 0.2666667 0.3921569 sc 0 872 m
600 872 l 600 462 l 0 462 l h f 0 462 m 600 462 l 600 0 l 0 0 l h f ➡️1 1 1
sc⬅️ 0 12297 m 600 12297 l 600 5122 l 0 5122 l h f 0.7411765 0.8980392 0.9725490
sc 23 7249 m 577 7249 l 577 6007 l 23 6007 l h f 1 0.9215686 0.9333333 sc
23 5848 m 577 5848 l 577 5533 l 23 5533 l h f 0.9686275 0.9725490 0.9764706
sc 23 5510 m 577 5510 l 577 5156 l 23 5156 l h f ➡️1 1 1 sc⬅️ 0 5110 m 600 5110
...
endstream
If I open the PDF in TextEdit and manually replace 1 1 1 sc with 0 1 0 sc my white background immediately changes to green after saving the PDF file.
How can I do this in an automated way with a Text Tool?
sed 's/1 1 1 sc/0 1 0 sc/' IN.pdf > OUT.pdf
gives me the error: sed: RE error: illegal byte sequence
bbe -e 's/0 1 1 sc/0 1 0 sc/' IN.pdf > OUT.pdf
no errors, OUT.pdf is written but no colors have changed
echo 'hello 1 1 1 sc world' | bbe -e 's/1 1 1 sc/0 1 0 sc/'
seems to work fine...
In the above stream (the first stream block) in the 1-page PDF file I need to replace only the second and third find. The second one has a line break?
It is not completely clear what you are doing.
You mention commands:
qpdf --qdf --object-streams=disable IN.pdf OUT.pdf
sed 's/1 1 1 sc/0 1 0 sc/' IN.pdf > OUT.pdf
bbe -e 's/0 1 1 sc/0 1 0 sc/' IN.pdf > OUT.pdf
It is not obvious if IN.pdf in the sed or bbe commands is the same IN.pdf file as the qpdf command.
If all three commands are using the same file as input, then that can explain why bbe fails.
Another possibility is that the bbe command shown is the command you are actually using and not a typo. It does not actually look for the string 1 1 1 sc.
sed is not designed to work with binary data.
Although the GNU implementation has a non-standard -z option to help read binary files, it still works on a form of "lines". Perl can be used as an improved sed here.
To change only the first three instances of the string 1 1 1 sc in the file, you could try:
qpdf --qdf --object-streams=disable IN.pdf - |\
perl -0777 -pe 'for $i (1..3) { s/1 1 1 sc/0 1 0 sc/ }' |\
qpdf - OUT.pdf
In this Perl command:
-0777 - treat entire input as single record
-pe - run command on each record, then print (like sed)
for $i (1..3) { ... } - run three times
s/.../.../ - similar to sed's s/// command
I think I will tackle this task with PikePDF, a Python library which seems to be able to work with content streams: https://pikepdf.readthedocs.io/en/latest/topics/content_streams.html
I was just able to Pretty Print the content streams by using:
#!/usr/bin/env python
from pikepdf import Pdf
import pikepdf
with pikepdf.open('IN.pdf') as pdf:
page = pdf.pages[0]
instructions = pikepdf.parse_content_stream(page)
data = pikepdf.unparse_content_stream(instructions)
print(data.decode('ascii'))
Now working my way to actual Edit the content stream ..........
Here the stream fragment from my question, pretty printed:
q
Q
q
/Cs1 cs
0.9686275 0.9725490 0.9764706 sc
0 12777 m
600 12777 l
600 0 l
0 0 l
h
f
0 12777 m
600 12777 l
600 0 l
0 0 l
h
f
➡️1 1 1 sc⬅️
0 12575 m
600 12575 l
600 12308 l
0 12308 l
h
f
0.1254902 0.2666667 0.3921569 sc
0 872 m
600 872 l
600 462 l
0 462 l
h
f
0 462 m
600 462 l
600 0 l
0 0 l
h
f
➡️1 1 1 sc⬅️
0 12297 m
600 12297 l
600 5122 l
0 5122 l
h
f
0.7411765 0.8980392 0.9725490 sc
23 7249 m
577 7249 l
577 6007 l
23 6007 l
h
f
1 0.9215686 0.9333333 sc
23 5848 m
577 5848 l
577 5533 l
23 5533 l
h
f
0.9686275 0.9725490 0.9764706 sc
23 5510 m
577 5510 l
577 5156 l
23 5156 l
h
f
➡️1 1 1 sc⬅️
0 5110 m
600 5110
Some more info about the color value:
Just divide the RGB values by 255
for example:
DeepSkyBlue = #00bfff = RGB(0, 191, 255)
0/255 = 0
191/255 = 0.7490196
255/255 = 1
0 0.7490196 1 sc

Why is this vertical text positioning working?

The PDF content below renders with the correct vertical positions, but how?
1 0 0 -1 0 792 cm
q
.75 0 0 .75 72 192.75 cm
BT
/F4 14.666667 Tf
1 0 0 -1 0 .80265617 Tm
0 -13.2773438 Td <0030> Tj
12.2087708 0 Td <0024> Tj
8.6870575 0 Td <003C> Tj
9.7756042 0 Td <0032> Tj
11.4001007 0 Td <0035> Tj
ET
Q
q
.75 0 0 .75 72 222.75 cm
BT
/F4 14.666667 Tf
1 0 0 -1 4.0719757 .80265617 Tm
0 -13.2773438 Td <002C> Tj
4.0719757 0 Td <0003> Tj
4.0719757 0 Td <0057> Tj
4.0719757 0 Td <004B> Tj
8.1511078 0 Td <004C> Tj
3.2561493 0 Td <0051> Tj
8.1511078 0 Td <004E> Tj
ET
Q
Renders correctly:
MAJOR
I think
However I can't understand how the y positions are calculated to do this (x is fine). The Text Rendering Matrix (TRM) is given by Text Matrix (TM) multiplied by Current Transformation Matrix (CTM) PDF1.7 Reference section 9.4.4. CTM is the identity matrix multiplied by each "cm" operation.
So for the first snippet,
CTM = [1 0 0 -1 0 792] x [0.75 0 0 0.75 72 192.75] = [0.75 0 0 -0.75 72 786.75]
TRM is TM x CTM:
TRM = [1 0 0 -1 0 0.8026] x [0.75 0 0 -0.75 72 786.75] = [0.75 0 0 0.75 72 786.1]
So, ignoring small details, the text will be rendered around y = 786 (actually 776 I reckon)
For the second snippet,
CTM = [1 0 0 -1 0 792] x [0.75 0 0 0.75 72 222.75] = [0.75 0 0 -0.75 72 816.75]
TRM = [1 0 0 -1 4.072 0.802] x [0.75 0 0 -0.75 72 816.75] = [0.75 0 0 0.75 75.05 816.4]
Again, ignoring small details, the text will be rendered around y = 816 (actually 806 I reckon)
But the y origin is the bottom of the page, and 816 is greater than 786. So how come the second snippet of text renders correctly below the first? I'm clearly missing something in the calculations, but I can't see what. Any ideas?
The error in your calculations is that you apply the cm matrix by multiplication from the right side. You instead have to apply it from the left side.
I.e. for the first snippet you have
CTM = [0.75 0 0 0.75 72 192.75] × [1 0 0 -1 0 792] = [0.75 0 0 -0.75 72 599.25]
and for the second snippet
CTM = [0.75 0 0 0.75 72 222.75] × [1 0 0 -1 0 792] = [0.75 0 0 -0.75 72 569.25]
With these current transformation matrices the rendered result is to be expected.
If you wonder how you should have known that you need to multiply from the left side...
This result is true in general for PDF: when a sequence of transformations is carried out, the matrix representing the combined transformation (M′) is calculated by premultiplying the matrix representing the additional transformation (MT) with the one representing all previously existing transformations (M):
𝑀′ = 𝑀𝑇 × 𝑀
(ISO 32000-2 section 8.3.4 "Transformation matrices")
Without going deep into matrices (not my forte, there is a slight error in my initial maths so images new corrected) you are working downwards from top left based on an inverted start point of 0 792 cm (Top Left corner)
The start of that snippet is above MAJOR 72 192.75 cm
Without outher transformations the text would be "UpsideDown" with M facing towards the bottom then the second 1 0 0 -1 mirrors it back upright and 0.8 "raises" it towards bottom so baseline is 193.5 ish from topleft at which point you "add" 0 -13.2773438 Td so the baseline is now about 205 from top left
Likewise, the origin for the second row is 72 222.75 cm down from above datum.
In both cases you placed their mirrored baseline even lower at 0 -13.2773438 Td thus both lines will be lower than shown above. In part due to the matrix inversions.
so here the second baseline is now at about 72 234 cm down from top left as subject to similar maths is roughly 222.75+.802+13.277 down but scale can also have effect.
Generally its best to use real time viewer of alterations (however this is not the best way just an example that by playing with rounded values I can see the effects).

Detect PDF form field radio button (radiobutton) shape / style

I need to programmatically categorize which shape a pdf form field radiobutton has. Therefore I created a test pdf using *crobat. I added a radiobutton group where each widget is using a different style.
One way could be to check the CA key of the appearance characteristics dictionary (MK) which is mapped to the ZapfDingbats font:
/MK<</BC[0.0]>> //CIRCLE (normally l)
/MK<</BC[0.0]/CA(4)>> //CHECK
/MK<</BC[0.0]/CA(8)>> //CROSS
/MK<</BC[0.0]/CA(u)>> //DIAMOND
/MK<</BC[0.0]/CA(n)>> //SQUARE
/MK<</BC[0.0]/CA(H)>> //STAR
However in the example PDF for the circle the CA key does not exist (it should have been /CA(l)). To implicitly assume a round shape does not seem correct.
Another idea would be to look at the appearance dictionary. For the example given in the pdf spec it seems possible:
stream
q
0 0 1 rg
BT
/ZaDb 12 Tf
0 0 Td
(l) Tj
ET
Q
endstream
However the normal appearance generated by *crobat looks like that:
stream
q
1 0 0 1 9 9 cm
8.5 0 m
8.5 4.6946 4.6946 8.5 0 8.5 c
-4.6946 8.5 -8.5 4.6946 -8.5 0 c
-8.5 -4.6946 -4.6946 -8.5 0 -8.5 c
4.6946 -8.5 8.5 -4.6946 8.5 0 c
s
Q
0.501953 G
q
0.7071 0.7071 -0.7071 0.7071 9 9 cm
7.5 0 m
7.5 4.1423 4.1423 7.5 0 7.5 c
-4.1423 7.5 -7.5 4.1423 -7.5 0 c
S
Q
0.75293 G
q
0.7071 0.7071 -0.7071 0.7071 9 9 cm
-7.5 0 m
-7.5 -4.1423 -4.1423 -7.5 0 -7.5 c
4.1423 -7.5 7.5 -4.1423 7.5 0 c
S
Q
q
1 0 0 1 9 9 cm
3.5 0 m
3.5 1.9331 1.9331 3.5 0 3.5 c
-1.9331 3.5 -3.5 1.9331 -3.5 0 c
-3.5 -1.9331 -1.9331 -3.5 0 -3.5 c
1.9331 -3.5 3.5 -1.9331 3.5 0 c
f
Q
endstream
My question: Is there a way to detect that a widget annotation has a round shape / circular style? I know that any arbitrary shape can be defined as an appearance however for the use case at hand the differentiation of those 6 styles is more than enough.
If the answer somehow depends on the pdf lib (due to certain functionality): currently openPDF is used and other libs like pdfbox or iText are in use, too.
First you can check if the /CA entry is 'l'. If /CA does not exist you can check if the appearance stream contains 'c', 'v', or 'y' operators (curve operators). If they are present you can assume a circular style.
This is a empiric approach but it might work for you situation.

My program reads PDF and try to find coordinate of each glyph in user space

it goes like this
q
0.1199951 0 0 0.1199951 0 0 cm
1 g
824 4101 267 389 re
f
Q
q
0.1199951 0 0 0.1199951 0 0 cm
1 g
824 4853 267 25 re
f
Q
q
0.1199951 0 0 0.1199951 0 0 cm
1 g
824 5241 267 25 re
f
Q
q
0.1199951 0 0 0.1199951 0 0 cm
1 g
1090 578 3081 1988 re
f
Q
q
0.1199951 0 0 0.1199951 0 0 cm
603 586 m
603 1800 l
649 1800 l
649 586 l
h
W n
8.3336724 0 0 8.3336724 0 0 cm
BT
/T1_0 5.04 Tf
0 1.0002 -1 0 76.8 70.32 Tm
(J)Tj
I want to ask what should be coordinate for J ?
My cropbox is 0 0 612 792 , Rotate value is 90.
So according to me
Th=1 default,
Tfs=5.04, from {/T1_0 5.04 Tf}
Trise=0 default,
teststatematrix
5.04 1 0
0 5.04 0
0 0 1
Tm
0 1.0002 0
-1 0 0
76.8 70.32 1
TRM = textstatematrix X Tm
-1 5.041 0
-5.040 0 0
76.800 70.320 1
So
[x,y,1] = [76.8, 70.32, 1] X TRM = [-354.413 457.469 1]
So x coordinate in user space is coming to be a negative number. Can you please Explain What mistake i am doing?
The matrix Trm calculated by the OP as
-1 5.041 0
-5.040 0 0
76.800 70.320 1
is the text rendering matrix described as follows in the PDF specification:
Conceptually, the entire transformation from text space to device space may be represented by a text rendering matrix, Trm:
(section 9.4.2, ISO 32000-1:2008)
The OP's mistake is not in calculating this matrix but in using it: This matrix contains the entire transformation from text space to device space,
Tj and other text-showing operators shall position the origin of the first glyph to be painted at the origin of text space.
(section 9.2.4 ISO 32000-1:2008)
and
The glyph origin is the point (0, 0) in the glyph coordinate system
(ibidem)
To determine, therefore, where the OP's
(J)Tj
puts the origin of the glyph J, one has to apply that matrix to the origin (0, 0), not to (76.8, 70.32) as the OP did.
Thus,
[x,y,1] = [0, 0, 1] X Trm = [76.8, 70.32, 1]
i.e. the coordinates of J are (76.8, 70.32) in device space. As the OP assumed the initial transformation matrix to have been the identity matrix, this device space essentially is the default user space.
Unfortunately the OP did not explain the coordinates in which coordinate system he is looking for. Thus, these coordinates probably are not the coordinates he was looking for.

Pdf setting the font color to the text

I am trying to add some text to a pdf file manually.I was able to add new text with a specific font. But i am not able to set the font color. So how can i do it manually?
(I just want to change these manually as i already have the code where i write these byte to make the pdf file)
Also how can i use graphic states specified in the pdf standard to manipulate the text so that feature changes does not affect the color changes etc.How exactly i can use the graphic state?
Source pdf file click here
Modified pdf file clcik here
The PDF color operators are listed in Table 74 of the PDF specification ISO 32000-1:2008.
In your case your added content stream is
104 0 obj
<</Length 105 0 R>>stream
/Helv 8 Tf
BT
1 0 0 1 15.67 150 Tm
(l)Tj
ET
/Helv 8 Tf
BT
1 0 0 1 17.88 190 Tm
(abcdefghijklmnopqr)Tj
ET
endstream
endobj
If e.g. you want the writing to be filled with red in a RGB color space, you add an 1 0 0 rg:
104 0 obj
<</Length 105 0 R>>stream
BT
1 0 0 1 15.67 150 Tm
/Helv 8 Tf
1 0 0 rg
[...]
EDIT
If you are afraid that that change may affect later text, remember to use the Graphics State Stack operators q and Q (cf. section 8.4.2 of the PDF specification). E.g.
q
0 1 -1 0 595.22 0 cm
q
BT
1 0 0 1 36 540 Tm
/Xi0 12 Tf
0.75 g
(Hello people!)Tj
0 g
ET
Q
Q
(Copied from How to add text object to existing pdf)