Does DM scripting have an equivalent to c++ pointers? - dm-script

As the title asks. I am interested in knowing if the DM scripting language is capable of handling references and pointers, like you would find in c/c++ as *a and &a. I'm guessing the capability exists at some level, owing to the way inbuilt functions like fitgaussian() work. If this capability is open to the user, it would make it easier to code up some classes and functions I want to create.

Yes, it does and it actually is quite similar to C++.
(But the simplification of the script language obscures it a bit.)
I'm trying to shed some light on it:
All objects in DM (Image, TagGroup, Component, ROI, ImageDisplay, etc. ) are automatically and always passed by reference, not value.
You can see this in the following example code:
void ModifyImage( image imgTemp )
{
imgTemp = irow
}
Image img := RealImage( "Test", 4, 100, 100 )
img = icol
img.ShowImage()
OKDialog("Act!")
ModifyImage( img )
img.ShowImage()
The actual image img of the main script changes. The routine ModifyImage received and used a reference to the image object.
Hint: This is a very typical source for bugs when dealing with TagGroups. When one f.e. gets a tag from an image and then modifies it in a method, it changes the tag of the image!
In order to have methods not modify the provided parameter objects passed in by reference, one has to explicitly create a clone of the object first. For this reason, all the object-types in the scripting languages provide ...clone() commands.
void ModifyImage( image imgTemp )
{
imgTemp = irow
}
Image img := RealImage( "Test", 4, 100, 100 )
img = icol
img.ShowImage()
OKDialog("Act!")
ModifyImage( img.ImageClone() )
Now, the img variable of the main script is not changed. (Because we create a copy in memory, and then pass the reference to the copy.)
However, primitive types (string, number) are by default passed by value, not reference.
Again, this can be seen by a little example.
void ModifyString( string tempStr )
{
tempStr = "Changed"
}
string str = "Original"
Result("\n" + str )
OKDialog("Act!")
ModifyString( str )
Result(" --> " + str )
Here the variable str of the main script is not changed by ModifyString because the values was passed into tempStr, and not a reference. The same is true for number type variables.
If one wants to pass a primitive type by reference, this can be denoted with a & in the methods signature.
The example from above now modified:
void ModifyString( string &tempStr )
{
tempStr = "Changed"
}
string str = "Original"
Result("\n" + str )
OKDialog("Act!")
ModifyString( str )
Result(" --> " + str )
The & can also be used for non-primitive objects in DM, if the passed in pointer is allowed to be changed:
Here is an example for this. The method does not modify the provided image, but instead creates a new one and passes back the reference into the pointer.
void ModifyImage( image &imgTemp )
{
imgTemp := RealImage( "New Test" , 4, 200, 200 )
imgTemp = iradius
}
Image img := RealImage( "Test", 4, 100, 100 )
img = icol
img.ShowImage()
OKDialog("Act!")
ModifyImage( img )
img.ShowImage()
Finally: The * operator to denote pointers is never used in DM-scripting.

Related

How to add annotation information in each frame of a stack image?

I know how to add the arrows in a single image displayed front, but now I need to add the arrow annotations on each frame of a image stack to indicate the contrast change position and show them using the GSM "slice player". How to do it?
There is no difference between a 2D image and a 3D stack in DigitalMicrograph. Both are just dimensional data. As such "slices" in a Stack do neither have their individual tags nor annotations - there is just a single imageTagGroup and a single imageDisplays.
So to achieve what you want you need a different approach. You need to move your annotation whenever the display updates to shows a different slice.
In order to do this, you need to add a display-listener to your image display and act on the slice_property_changed event.
A basic example script for this:
Class CStackAnno
{
ImageDisplay disp
Component arrow
Number ListenerID
// This method is called whenever the imagedisplay fires the slice update event
void OnSlicePropChanged( object self, Number disp_flags, ImageDisplay disp, Number flags1, Number flags2, object slice_id_beg, object slice_id_end )
{
image img := disp.ImageDisplayGetImage()
if ( 3 != img.ImageGetNumDimensions() ) return
if ( !arrow.ComponentIsValid() ) return
number sx = img.imageGetDimensionsize(0)
number sy = img.imageGetDimensionsize(1)
number sz = img.imageGetDimensionsize(2)
number start, end
disp.ImageDisplayGetDisplayedLayers( start, end )
number kLineEndPoint = 2
arrow.ComponentSetControlPoint( kLineEndPoint, sx/sz * start, sx/sz * start, 0 )
}
Object Launch( object self, image Img )
{
if ( !img.ImageIsValid() ) Throw( "Invalid input image." )
if ( 3 != img.ImageGetNumDimensions() ) Throw( "This script only supports 3D images." )
disp = img.ImageGetImageDisplay(0)
// Register DisplayListener to catch when it updates
ListenerID = disp.ImageDisplayAddEventListener( self, "slice_property_changed:OnSlicePropChanged" )
// Add the annotation
arrow = NewArrowAnnotation( img.ImageGetDimensionSize(1)/5, img.ImageGetDimensionSize(0)*4/5, 0, 0 )
arrow.ComponentSetForegroundColor( 0, 0.5 , 1 )
arrow.ComponentSetBackgroundColor( 0, 0.8 , 1 )
arrow.ComponentSetDrawingMode( 1 )
disp.ComponentAddChildAtEnd( arrow )
return self
}
}
//Main call
image fImg
GetFrontImage(fImg)
Alloc(CStackAnno).Launch(fImg)

How to apply virtual apperture with 4D-STEM dataset in EFFICIENT way?

I would like to apply arbitrarily defined bit mask as virtual aperture and apply it to 4D-STEM data set in an EFFICIENT way.
I did it using the SliceN function and apply the mask pixel-by-pixel, which is very slow for large datasets. How to optimize it to so to run faster?
Image 4DSTEM := GetFrontImage() // dimention [ScanX, ScanY, Dx, Dy]
Image mask: = iradius // just an arbitrary mask (aperture)
Image out // dimention [ScanX, ScanY]
for (number i=0; i<ScanX; i++)
{ for (number j=0; j<ScanY; j++)
{
Diff2D = 4DSTEM.SliceN(4,2,i,j,0,0,2,Dx,1,3,Dy,1)
out.setpixel(i,j, sum(diff2D*mask))
}
}
out.showimage()
for an [100,100,512,512] dataset, that took few minutes to finish. When I have to repeat the operation several times, that is way to slow compare to matrix operation. but I dont know how to make it in an efficient way.
Thanks!
you're hitting the limitations of scripting languages here. Using sliceN is already pretty much the optimum you can get to, unfortunately. Everything else in speed optimization requires parallelized, compiled code. (i.e. you could code C++ code and use the SDK to compile your own plugin.)
However, there is a bit of room for improvement over your example.
First of all, your example above doesn't run :c) But that is quickly fixed.
Point #1:
Try to avoid number type casting. DM script only knows number but internally there is a difference between the proper number types (integer, floating point, signed/unsigned, byte-size). The script languages uses real-4-byte as the default unless told differently explicitly. And some methods will return real-4-byte by default. For this reason, the processing will be fastest, if both data and mask use real-4-byte data as well.
In my testing, the time-difference between running with uint16 data plus uint8 mask and *real4 data plus real4 mask) was significant! Nearly 30% time difference.
Point #2:
Don't copy you sliced image! Use := not = for your Dif2D.
The SliceN command returns an expression directly addressing the required memory. You can use it directly in any other expression (like I do below) or you can assign an image variable to it using := to give it a name.
The speed increase is not huge, but it's one copy-operation less per loop iteration.
Point #3:
You additional knowledge: Now for arbitrary masks there is not much you can do, but most often masks are zero-valued over large stretches and it is possible to define a smaller ROI containing all non-zero points. If this is the case, you can limit your math operations to that region.
i.e. instead of multiplying the whole DP with the same sized mask, just use a smaller mask and use the according sub-section of the DP.
This can actually make a big difference, but it will depend on your mask.
Of course you need to "find" this ROI first. In my script below I'm having a helper method to do that, utilizing the comparatively fast max() command and image rotation as trick for speed-up.
Point #4:
...would be to get rid of the double-for loop and replace it with image-expressions. Unfortunately, DigitalMicrograph does currently (GMS 3.3) not support this for 4D or 5D data.
The script below executed on a [53 x 52 x 512 x 512] STEM DI (of real-4 byte data) gave me the following timings:
Original: 12.80910 sec
Test 1 : 10.77700 sec
Test 2 : 1.83017 sec
// Helper class for timing
class CTimer{
number s
string n
~CTimer(object self){result("\n"+n+": "+ (GetHighResTickCount()-s)/GetHighResTicksPerSecond()+" sec");}
object Start(object self, string n_) { n=n_; s=GetHighResTickCount(); return self;}
}
// Helper method to find best non-zero containing ROI
void GetNonZeroArea( image src, number &t, number &l, number &b, number &r )
{
image work = !!src // Make a binary image which is 0 only where src==0
number d
max(work,d,t) // get "first" non-zero pixel coordinate, this is y = dist from TOP
rotateRight(work) // rotate image right
max(work,d,l) // get "first" non-zero pixel coordinate, this is y = dist from LEFT
rotateRight(work) // rotate image right
max(work,d,b) // get "first" non-zero pixel coordinate, this is y = dist from BOTTOM
b = work.ImageGetDimensionSize(1) - b // Opposite side!
rotateRight(work) // rotate image right
max(work,d,r) // get "first" non-zero pixel coordinate
r = work.ImageGetDimensionSize(1) - r // Opposite side!
}
// The original proposed script (plus fixes to make it actually run)
image Original(image STEM4D, image mask)
{
Number ScanX = STEM4D.ImageGetDimensionSize(0)
Number ScanY = STEM4D.ImageGetDimensionSize(1)
Number Dx = STEM4D.ImageGetDimensionSize(2)
Number Dy = STEM4D.ImageGetDimensionSize(3)
Image out := RealImage("Test1",4,ScanX,ScanY)
for (number i=0; i<ScanX; i++)
{ for (number j=0; j<ScanY; j++)
{
image Diff2D = STEM4D.SliceN(4,2,i,j,0,0,2,Dx,1,3,Dy,1)
out.setpixel(i,j, sum(Diff2D*mask))
}
}
return out
}
// Remove copying the slice, just reference it
image Test1(image STEM4D, image mask)
{
Number ScanX = STEM4D.ImageGetDimensionSize(0)
Number ScanY = STEM4D.ImageGetDimensionSize(1)
Number Dx = STEM4D.ImageGetDimensionSize(2)
Number Dy = STEM4D.ImageGetDimensionSize(3)
Image out := RealImage("Test1",4,ScanX,ScanY)
for (number i=0; i<ScanX; i++)
{ for (number j=0; j<ScanY; j++)
{
image Diff2D := STEM4D.SliceN(4,2,i,j,0,0,2,Dx,1,3,Dy,1)
out.setpixel(i,j, sum(Diff2D*mask))
}
}
return out
}
// Limit mask size to what is needed!
image Test2(image STEM4D, image mask )
{
Number ScanX = STEM4D.ImageGetDimensionSize(0)
Number ScanY = STEM4D.ImageGetDimensionSize(1)
Number Dx = STEM4D.ImageGetDimensionSize(2)
Number Dy = STEM4D.ImageGetDimensionSize(3)
Image out := RealImage("Test1",4,ScanX,ScanY)
Number t,l,b,r
GetNonZeroArea(mask,t,l,b,r)
Number w = r - l
Number h = b - t
image subMask := mask.slice2(l,t,0, 0,w,1, 1,h,1 )
for (number i=0; i<ScanX; i++)
for (number j=0; j<ScanY; j++)
out.setpixel(i,j, sum(STEM4D.SliceN(4,2,i,j,l,t,2,w,1,3,h,1)*subMask))
return out
}
Image src := GetFrontImage() // dimention [ScanX, ScanY, Dx, Dy]
Number ScanX = src.ImageGetDimensionSize(0)
Number ScanY = src.ImageGetDimensionSize(1)
Number Dx = src.ImageGetDimensionSize(2)
Number Dy = src.ImageGetDimensionSize(3)
Number r = 50 // mask radius
Image maskImg := RealImage("Mask",4,Dx,Dy)
maskImg = iradius < r ? 1 : 0 // just an aperture mask
image resultImg
{
object timer = Alloc(CTimer).Start("Original")
resultImg := Original(src,maskImg)
}
resultImg.SetName("Oringal")
resultImg.ShowImage()
{
object timer = Alloc(CTimer).Start("Test 1")
Test1(src,maskImg).ShowImage()
}
resultImg.SetName("Test 1")
resultImg.ShowImage()
{
object timer = Alloc(CTimer).Start("Test 2")
Test2(src,maskImg).ShowImage()
}
resultImg.SetName("Test 2")
resultImg.ShowImage()
Compiled code comparison:
Now, it should be added that the above script still is rather slow. Because it is iterating and using script language. The fully compiled c++ code of DigitalMicrograph is much faster. So if you have the licensed packages giving you the SI menu, then you want to use the SI/Map/Signal command. This is near-instantaneous for the example STEM DI I've mentioned above. My other answer shows how one could utilize this functionality by script.
As mentioned in my other answer, a real speed-win comes when compiled, parallelized code is used. DigitalMicrograph does this, after all, in the available SI "signal" map functionality. This feature is not available in the free version, but if you have Spectrum-Imaging acquisition, you most likely have the appropriated license as well.
The answer below utilizes this functionality by accessing the UI with the command ChooseMenuItem() and applying a few more tricks. The script is a bit lengthy, but its parts also show some other nice tricks worthwhile knowing:
TestSignalIntegrationInSI is the main script demoing how things can work.
CreatePickerByScript shows how one can create picker-spectra on SIs. This is used to open a 'Picker Diffraction Pattern' image from the STEM DI.
AddTestMasksToDP_ROIs programmatically adds ROIs to the diffraction pattern to be used as mask
AddTestMasksToDP_Threshold programmatically adds an intensity-threshold mask to be used as mask.
AddTestMasksToDP_DPMasks programmatically adds the various types of diffraction-masks to be used as mask
GetIntegratedSignalViaSIMenu is the central step of the script. With a picker-DP and required 'masks' on it front-most, the menu command is called to perform the signal-extraction (as fast as possible.) Then the displayed result-image is returned.
GetNewestImage is just a utility method showing how on can access the latest memory-created image.
Here is the script:
image GetNewestImage()
{
// New images get the next higher imageID.
// This can be used to identify the "latest" created image.
if ( 0 == CountImages() ) Throw( "No image in memory!" )
// We create a temp. image to get the uppder limit
number lastID = RealImage("Dummy",4,1).ImageGetID()
// Then we search for the next lower existing one
image lastImg
for( number ID = lastID - 1; ID>0; ID-- )
{
lastImg := FindImageByID(ID)
if ( lastImg.ImageIsValid() ) break
}
return lastImg
}
image CreatePickerByScript( image SI, number t, number l, number b, number r )
{
if ( SI.ImageGetNumDimensions()<3 ) Throw( "Sorry, LineScans are not supprorted here." )
// Adding a non-volatile ROI of specific RoiNAME acts as if using
// the picker-tool. The ID string must be unique!
ROI pickerROI = NewROI()
pickerROI.RoiSetVolatile( 0 )
string uniqueID = GetDate(0)+"#"+GetTime(1)+";"+round(random()*1000)
pickerROI.RoiSetName( "SICursor(##"+uniqueID+"##)" )
SI.ImageGetImageDisplay(0).ImageDisplayAddROI( pickerROI )
// This creates the picker image.
// So the child is now the "newest" image in memory
image child := GetNewestImage()
return child
}
void AddTestMasksToDP_ROIs( image DP )
{
// Add ROIs to the DP which are your masks (any numebr and type of ROI works)
imageDisplay DPdisp = DP.ImageGetImageDisplay(0)
number dpX = DP.ImageGetDimensionSize(0)
number dpY = DP.ImageGetDimensionSize(1)
// Only simple RECT ROIs are supported
ROI maskRoi1 = NewROI()
maskRoi1.ROISetRectangle( dpY*0.1, dpX*0.1, dpY*0.8, dpX*0.3 )
DPdisp.ImageDisplayAddROI(maskRoi1)
// Arbitrary multi-vertex (use for ovals etc.)
ROI maskRoi2 = NewROI()
maskRoi2.ROISetRectangle( dpY*0.7, dpX*0.1, dpY*0.9, dpX*0.9 )
DPdisp.ImageDisplayAddROI(maskRoi2)
}
void AddTestMasksToDP_Threshold( image DP )
{
// Add intensity treshhold mask (highest 95% intensity range)
imageDisplay DPdisp = DP.ImageGetImageDisplay(0)
DPdisp.RasterImageDisplaySetThresholdOn( 1 )
number low = max(DP) * 0.05
number high = max(DP)
DPdisp.RasterImageDisplaySetThresholdLimits( low, high )
}
void AddTestMasksToDP_DPMasks( image DP )
{
// Add Diffraction masks to the DP
imageDisplay DPdisp = DP.ImageGetImageDisplay(0)
// Spot masks (always symmetric pair)
Component spotMask = NewComponent(8,0,0,0,0) // 8 = Spotmask
spotMask.ComponentSetControlPoint(4, 0, 0,0) // 4 = TopLeft of one spot [Size only]
spotMask.ComponentSetControlPoint(7,10,10,0) // 7 = BottomRight of one spot [Size only]
spotMask.ComponentSetControlPoint(8,150,0,0) // 8 = Spot position [center]
DPdisp.ComponentAddChildAtEnd(spotMask)
// Bandpass mask (Only circles are correctly supported)
Component bandpassMask = NewComponent(15,0,0,0,0) // 15 = Bandpass (ring)
number r1 = 100
number r2 = 120
bandpassMask.ComponentSetControlPoint(7,r1,r1,0) // 7 = BottomRight of one ring [Size only]
bandpassMask.ComponentSetControlPoint(14,r2,r2,0) // 14 = BottomRight of one ring [Size only]
DPdisp.ComponentAddChildAtEnd(bandpassMask)
// Wege mask (symmetric)
Component wedgeMask = NewComponent(19,0,0,0,0) // 19 = wedgemask (ringsegment)
wedgeMask.ComponentSetControlPoint(9,10,20,0) // 9 = One wedge vector
wedgeMask.ComponentSetControlPoint(10,-20,40,0) // 10 = Other wedge vector
DPdisp.ComponentAddChildAtEnd(wedgeMask)
// Array mask (symmetric)
Component arrayMask = NewComponent(9,0,0,0,0) // 9 = arrayMask (ringsegment)
arrayMask.ComponentSetControlPoint(9,-70,-60,0) // 9 = One array vector
arrayMask.ComponentSetControlPoint(10,99,-99,0) // 10 = Other array vector
arrayMask.ComponentSetControlPoint(4, 0, 0,0) // 4 = TopLeft of one spot [Size only]
arrayMask.ComponentSetControlPoint(7,20,20,0) // 7 = BottomRight of one spot [Size only]
DPdisp.ComponentAddChildAtEnd(arrayMask)
}
image GetIntegratedSignalViaSIMenu( image pickerChild )
{
// Call the Menu to do the work
// The picker-spectrum or DP needs to be front-most
pickerChild.SelectImage()
ChooseMenuItem("SI","Map","Signal")
// The created signal map is NOT the newest image
// (some internal iamges are created for the mask)
// but it is the front-most displayed one.
image signalMap := GetFrontImage()
return signalMap
}
image GetMaskFromSignalMap( image signalMap, number DPx, number DPy )
{
// The actual mask is stored in the tags
string tagPath = "Processing:[0]:Parameters:Mask"
tagGroup tg = signalMap.ImageGetTagGroup()
if ( !tg.TagGroupDoesTagExist(tagPath) )
Throw( "Sorry, no mask tag found." )
image mask := RealImage("Mask",4,DPx, DPy )
if ( !tg.TagGroupGetTagAsArray(tagPath,mask) )
Throw( "Sorry, could not retrieve mask. Maybe wrong size?" )
return mask
}
void TestSignalIntegrationInSI()
{
image STEMDI := GetFrontImage()
image DP := STEMDI.CreatePickerByScript(0,0,1,1)
if ( TwoButtonDialog( "Add ROIs as mask?", "Yes", "No" ) )
AddTestMasksToDP_ROIs( DP )
else if ( TwoButtonDialog( "Add intensity treshold as mask?", "Yes", "No" ) )
AddTestMasksToDP_Threshold( DP )
else if ( TwoButtonDialog( "Add diffraction masks as mask?", "Yes", "No" ) )
AddTestMasksToDP_DPMasks( DP )
image signalMap := GetIntegratedSignalViaSIMenu( DP )
number dpX = DP.ImageGetDimensionSize(0)
number dpY = DP.ImageGetDimensionSize(1)
// We may want to close the DP again. No longer needed
//DP.DeleteImage()
// Verification: Get Mask image form SignalMap
image usedMask := GetMaskFromSignalMap( signalMap, dpX, dpY )
usedMask.SetName( "This mask was used." )
usedMask.ShowImage()
}
TestSignalIntegrationInSI()
The solution below utilizes the intrinsic expression loops by performing in-place multiplication and then projection.
Disappointingly, it turns out the solution is actually a bit slower then the for-loop with the SliceN command.
For the same test-data of size [53 x 52 x 512 x 512] the resulting timing is:
Data copy: 1.28073 sec
Inplace multiply: 30.1978 sec
Project 1/2: 1.1208 sec
Project 2/2: 0.0019557 sec
InPlace multiplication with projections (total): 32.9045 sec
InPlace multiplication with projections (total): 34.9853 sec
// Helper class for timing
class CTimer{
number s
string n
~CTimer(object self){result("\n"+n+": "+ (GetHighResTickCount()-s)/GetHighResTicksPerSecond()+" sec");}
object Start(object self, string n_) { n=n_; s=GetHighResTickCount(); return self;}
}
image MaskMultipliedSum( image STEM4D, image MASK2D, number copyFirst )
{
// Boring feasability checks...
if ( 4 != STEM4D.ImageGetNumDimensions() )
Throw( "Input data is not 4D." )
if ( 2 != MASK2D.ImageGetNumDimensions() )
Throw( "Input mask is not 2D." )
Number ScanX = STEM4D.ImageGetDimensionSize(0)
Number ScanY = STEM4D.ImageGetDimensionSize(1)
Number Dx = STEM4D.ImageGetDimensionSize(2)
Number Dy = STEM4D.ImageGetDimensionSize(3)
if ( Dx != MASK2D.ImageGetDimensionSize(0) )
Throw ("X dimension of mask does not match input data." )
if ( Dy != MASK2D.ImageGetDimensionSize(1) )
Throw ("Y dimension of mask does not match input data." )
// Do the maths!
image workCopy4D
if ( copyFirst )
{
object timer = Alloc(CTimer).Start("Data copy")
workCopy4D = STEM4D
}
else
workCopy4D := STEM4D
{
object timer = Alloc(CTimer).Start("Inplace multiply")
workCopy4D *= MASK2D[idimindex(2),idimindex(3)]
}
// Now we want to "sum up" over Dx and Dy
image p1,p2
{
object timer = Alloc(CTimer).Start("Project 1/2")
p1 := project( workCopy4D, 3 )
}
{
object timer = Alloc(CTimer).Start("Project 2/2")
p2 := project( p1, 2 )
}
return p2
}
image stack4D, mask2D
If ( GetTwoLabeledImagesWithPrompt("Please select 4D data and 2D mask", "Select input", "4D data", stack4D, "2D mask", mask2D ) )
{
number doCopy = TwoButtonDialog("Create workcopy?","Yes (takes time)","No (overwrites input data!)")
object timer = Alloc(CTimer).Start("InPlace multiplication with projections (total)")
MaskMultipliedSum(stack4D,mask2D,doCopy).ShowImage()
}

Detect Bold, Italic and Strike Through text using PDFBox with VB.NET

Is there a way to preserve the text formatting when extracting a PDF with PDFBox?
I have a program that parses a PDF document for information. When a new version of the PDF is released the authors use bold or italic text to indicate new information and Strike through or underlined to indicated omitted text. Using the base Stripper class in PDFbox returns all the text but the formatting is removed so I have no way of telling if the text is new or omitted. I'm currently using the project example code below:
Dim doc As PDDocument = Nothing
Try
doc = PDDocument.load(RFPFilePath)
Dim stripper As New PDFTextStripper()
stripper.setAddMoreFormatting(True)
stripper.setSortByPosition(True)
rtxt_DocumentViewer.Text = stripper.getText(doc)
Finally
If doc IsNot Nothing Then
doc.close()
End If
End Try
I have my parsing code working well if I simply copy and paste the PDF text into a richtextbox which preservers the formatting. I was thinking of doing this programatically by opening the PDF, select all, Copy, close the document then paste it in my richtextbox but that seems clunky.
As the OP mentioned in a comment that a Java example would do and I've yet only used PDFBox with Java, this answer features a Java example. Furthermore, this example has been developed and tested with PDFBox version 1.8.11 only.
A customized text stripper
As already mentioned in a comment,
The bold and italic effects in the OP's sample document are generated by using a different font (containing bold or italic versions of the letters) to draw the text. The underline and strike-through effects in the sample document are generated by drawing a rectangle under / through the text line which has the width of the text line and a very small height. To extract these information, therefore, one has to extend the PDFTextStripper to somehow react to font changes and rectangles nearby text.
This is an example class extending the PDFTextStripper just like that:
public class PDFStyledTextStripper extends PDFTextStripper
{
public PDFStyledTextStripper() throws IOException
{
super();
registerOperatorProcessor("re", new AppendRectangleToPath());
}
#Override
protected void writeString(String text, List<TextPosition> textPositions) throws IOException
{
for (TextPosition textPosition : textPositions)
{
Set<String> style = determineStyle(textPosition);
if (!style.equals(currentStyle))
{
output.write(style.toString());
currentStyle = style;
}
output.write(textPosition.getCharacter());
}
}
Set<String> determineStyle(TextPosition textPosition)
{
Set<String> result = new HashSet<>();
if (textPosition.getFont().getBaseFont().toLowerCase().contains("bold"))
result.add("Bold");
if (textPosition.getFont().getBaseFont().toLowerCase().contains("italic"))
result.add("Italic");
if (rectangles.stream().anyMatch(r -> r.underlines(textPosition)))
result.add("Underline");
if (rectangles.stream().anyMatch(r -> r.strikesThrough(textPosition)))
result.add("StrikeThrough");
return result;
}
class AppendRectangleToPath extends OperatorProcessor
{
public void process(PDFOperator operator, List<COSBase> arguments)
{
COSNumber x = (COSNumber) arguments.get(0);
COSNumber y = (COSNumber) arguments.get(1);
COSNumber w = (COSNumber) arguments.get(2);
COSNumber h = (COSNumber) arguments.get(3);
double x1 = x.doubleValue();
double y1 = y.doubleValue();
// create a pair of coordinates for the transformation
double x2 = w.doubleValue() + x1;
double y2 = h.doubleValue() + y1;
Point2D p0 = transformedPoint(x1, y1);
Point2D p1 = transformedPoint(x2, y1);
Point2D p2 = transformedPoint(x2, y2);
Point2D p3 = transformedPoint(x1, y2);
rectangles.add(new TransformedRectangle(p0, p1, p2, p3));
}
Point2D.Double transformedPoint(double x, double y)
{
double[] position = {x,y};
getGraphicsState().getCurrentTransformationMatrix().createAffineTransform().transform(
position, 0, position, 0, 1);
return new Point2D.Double(position[0],position[1]);
}
}
static class TransformedRectangle
{
public TransformedRectangle(Point2D p0, Point2D p1, Point2D p2, Point2D p3)
{
this.p0 = p0;
this.p1 = p1;
this.p2 = p2;
this.p3 = p3;
}
boolean strikesThrough(TextPosition textPosition)
{
Matrix matrix = textPosition.getTextPos();
// TODO: This is a very simplistic implementation only working for horizontal text without page rotation
// and horizontal rectangular strikeThroughs with p0 at the left bottom and p2 at the right top
// Check if rectangle horizontally matches (at least) the text
if (p0.getX() > matrix.getXPosition() || p2.getX() < matrix.getXPosition() + textPosition.getWidth() - textPosition.getFontSizeInPt() / 10.0)
return false;
// Check whether rectangle vertically is at the right height to underline
double vertDiff = p0.getY() - matrix.getYPosition();
if (vertDiff < 0 || vertDiff > textPosition.getFont().getFontDescriptor().getAscent() * textPosition.getFontSizeInPt() / 1000.0)
return false;
// Check whether rectangle is small enough to be a line
return Math.abs(p2.getY() - p0.getY()) < 2;
}
boolean underlines(TextPosition textPosition)
{
Matrix matrix = textPosition.getTextPos();
// TODO: This is a very simplistic implementation only working for horizontal text without page rotation
// and horizontal rectangular underlines with p0 at the left bottom and p2 at the right top
// Check if rectangle horizontally matches (at least) the text
if (p0.getX() > matrix.getXPosition() || p2.getX() < matrix.getXPosition() + textPosition.getWidth() - textPosition.getFontSizeInPt() / 10.0)
return false;
// Check whether rectangle vertically is at the right height to underline
double vertDiff = p0.getY() - matrix.getYPosition();
if (vertDiff > 0 || vertDiff < textPosition.getFont().getFontDescriptor().getDescent() * textPosition.getFontSizeInPt() / 500.0)
return false;
// Check whether rectangle is small enough to be a line
return Math.abs(p2.getY() - p0.getY()) < 2;
}
final Point2D p0, p1, p2, p3;
}
final List<TransformedRectangle> rectangles = new ArrayList<>();
Set<String> currentStyle = Collections.singleton("Undefined");
}
(PDFStyledTextStripper.java)
In addition to what the PDFTextStripper does, this class also
collects rectangles from the content (defined using the re instruction) using an instance of the AppendRectangleToPath operator processor inner class,
checks text for the style variants from the sample document in determineStyle, and
whenever the style changes, adds the new style to the result in writeString.
Beware: This merely is a proof of concept! In particular
the implementations of the tests in TransformedRectangle.underlines(TextPosition) and TransformedRectangle#strikesThrough(TextPosition) are very simplistic and only work for horizontal text without page rotation and horizontal rectangular strikeThroughs and underlines with p0 at the left bottom and p2 at the right top;
all rectangles are collected, not checking whether they actually are filled with a visible color;
the tests for "bold" and "italic" merely inspect the name of the used font which may not suffice in general.
A test output
Using the PDFStyledTextStripper like this
String extractStyled(PDDocument document) throws IOException
{
PDFTextStripper stripper = new PDFStyledTextStripper();
stripper.setSortByPosition(true);
return stripper.getText(document);
}
(from ExtractText.java, called from the test method testExtractStyledFromExampleDocument)
one gets the result
[]This is an example of plain text
[Bold]This is an example of bold text
[]
[Underline]This is an example of underlined text[]
[Italic]This is an example of italic text
[]
[StrikeThrough]This is an example of strike through text[]
[Italic, Bold]This is an example of bold, italic text
for the OP's sample document
PS The code of the PDFStyledTextStripper meanwhile has been slightly changed to also work for a sample document shared in a github issue, in particular the code of its inner class TransformedRectangle, cf. here.

Generate a tree of structs with testing/quick, respecting invariants

I have a tree of structs which I'd like to test using testing/quick, but constraining it to within my invariants.
This example code works:
var rnd = rand.New(rand.NewSource(time.Now().UnixNano()))
type X struct {
HasChildren bool
Children []*X
}
func TestSomething(t *testing.T) {
x, _ := quick.Value(reflect.TypeOf(X{}), rnd)
_ = x
// test some stuff here
}
But we hold HasChildren = true whenever len(Children) > 0 as an invariant, so it'd be better to ensure that whatever quick.Value() generates respects that (rather than finding "bugs" that don't actually exist).
I figured I could define a Generate function which uses quick.Value() to populate all the variable members:
func (X) Generate(rand *rand.Rand, size int) reflect.Value {
x := X{}
throwaway, _ := quick.Value(reflect.TypeOf([]*X{}), rand)
x.Children = throwaway.Interface().([]*X)
if len(x.Children) > 0 {
x.HasChildren = true
} else {
x.HasChildren = false
}
return reflect.ValueOf(x)
}
But this is panicking:
panic: value method main.X.Generate called using nil *X pointer [recovered]
And when I change Children from []*X to []X, it dies with a stack overflow.
The documentation is very thin on examples, and I'm finding almost nothing in web searches either.
How can this be done?
Looking at the testing/quick source code it seems that you can't create recursive custom generators and at the same time reuse the quick library facilities to generate the array part of the struct, because the size parameter, that is designed to limit the number of recursive calls, cannot be passed back into quick.Value(...)
https://golang.org/src/testing/quick/quick.go (see around line 50)
in your case this lead to an infinite tree that quickly "explodes" with 1..50 leafs at each level (that's the reason for the stack overflow).
If the function quick.sizedValue() had been public we could have used it to accomplish your task, but unfortunately this is not the case.
BTW since HasChildren is an invariant, can't you simply make it a struct method?
type X struct {
Children []*X
}
func (me *X) HasChildren() bool {
return len(me.Children) > 0
}
func main() {
.... generate X ....
if x.HasChildren() {
.....
}
}

COM: Create a VT_ARRAY with VT_BSTR values

I'm a COM newbie and I think what I have is correct, but the runtime doesn't like it. Any help is much appreciated.
I need to invoke a COM function that takes in a single dimensional array of BSTRs. Specifically, the documentation says the parameter must be:
Function: AddFiles ( [in] VARIANT * filePaths )
filePaths The single-dimensioned array of full paths to each file or
folder. filePaths can be of type
VT_ARRAY|VT_VARIANT, where each entry
is a VT_BSTR, or VT_ARRAY|VT_BSTR.
I have a vector<wstring> myPaths of paths which I want to pass into the function that takes the parameter above. Here's the code I wrote. Calling AddFiles on myComObject results in an AV (myComObject is not null, and I can invoke other methods on it):
...
VARIANT filePaths;
VariantInit( &filePaths );
filePaths.vt = VT_ARRAY|VT_VARIANT;
filePaths.parray = SafeArrayCreateVector( VT_BSTR, 0, (unsigned int) myPaths.size() );
long i = 0;
for( vector<wstring>::iterator it = myPaths.begin();
it != myPaths.end();
it++, i++ )
{
BSTR myPath= SysAllocString(it->c_str());
SafeArrayPutElement( filePaths.parray, &i, myPath);
}
myComObject->AddFiles( &filePaths );
...
The COM object isn't my code and I can't debug into it, but I suspect I'm not creating that array properly - based on the requirement of the AddFiles function and the code I have, anyone have ideas about what I might be doing wrong?
If myComObject->AddFiles can only deal with VT_ARRAY|VT_VARIANT, the following should work too.
VARIANT myPath;
VariantInit(&myPath);
myPath.vt = VT_BSTR;
myPath.bstrVal = SysAllocString(it->c_str());
SafeArrayPutElement(filePaths.parray, &i, &myPath);
Don't you want:
filePaths.vt = VT_ARRAY|VT_BSTR;
Since you're creating a SafeArray of BSTRs?