How to get an image from a long string data in dm-script - dm-script

I would like to get an image data from a string data array.
The below script runs well but speed is low.
(The actual length of the string data is much longer than in the example below.)
I guess pixel addressing in the for loop would take a time.
image str2img(string str)
{
image img:=RealImage("",4,10,1)
string tempstr=str
for(number i=0;i<10;i++)
{
if(find(tempstr,",")!=-1)
{
img[i,0]=tempstr.left(find(tempstr,",")).val()
tempstr=tempstr.right(tempstr.len()-find(tempstr,",")-1)
result(tempstr+"\n")
}else
{
img[i,0]=tempstr.val()
}
}
return img
}
string input="1,2,3,4,5,6,7,8,9,10"
image output=str2img(input)
output.showimage()
Then I wrote the following script to use stream.
However I got the error massage 'Non-numeric text encountered'.
image str2img(string str)
{
TagGroup Tg=NewTagGroup()
Tg.TagGroupSetTagAsString("data",str)
object fstream=NewStreamFromBuffer(0)
TagGroupWriteTagDataToStream(Tg,"data",fstream,0)
fstream.StreamSetPos(0,0)
number bLinesAreRows=1
number bSizeByCount=1
number dtype=2 //2 for real4 (float)
object imgSizeObj = Alloc( "ImageData_ImageDataSize" )
image img := ImageImportTextData( "Imag Name " , fstream , dtype , imgSizeObj , bLinesAreRows , bSizeByCount )
return img
}
string input="1,2,3,4,5,6,7,8,9,10"
image output=str2img(input)
output.showimage()
Is the "ImageImportTextData()" function valid only for reading a saved file?
Or are there any efficient way to obtain an image from a long string data?

Very good question and I like they way you were going about it.
No, ImageImportTextData() works for any stream as you will see in the example below.
However, the command requires text-lines to be finalized by line-breaks if you want it to count, and there seems to be an issue with String-tags streaming. I never use this, as there are dedicated commands to stream text.
So, your fixed script looks like:
image str2img(string str)
{
object fstream=NewStreamFromBuffer(0)
fStream.StreamWriteAsText(0,str) // Write text to stream directly
fstream.StreamSetPos(0,0)
number bLinesAreRows=1
number bSizeByCount=1
number dtype=2 //2 for real4 (float)
object imgSizeObj = Alloc( "ImageData_ImageDataSize" )
image img := ImageImportTextData( "Imag Name " , fstream , dtype , imgSizeObj , bLinesAreRows , bSizeByCount )
return img
}
string input="1,2,3,4,5,6,7,8,9,10\n" // Note final line-break if you want to count.
image output=str2img(input)
output.showimage()

Related

How can I use the StreamWriteAsText() to write data of the Number type?

My ultimate goal is to write a file of image data and the time it was taken, for multiple times. This could be used to produce time vs intensity plots.
To do this, I am trying to write a 1D image to a file stream repeatedly in time using the ImageWriteImageDataToStream() function. I go about this by attaching a Listener object to the camera view I am reading out and this listener executes a function that writes the image to a file stream using ImageWriteImageDataToStream() every time the data changes (messagemap = "data_changed:MyFunctiontoExecute") .
My question is, is there a way to also write a time stamp to this same file stream?
All I can find is StreamWriteAsText(), which takes a String data type. Can I convert time which is a Number type to a String type?
Does anyone have a better way to do this?
My solution at the moment is to create a separate file at the same time and record the timing using WriteFile(), so not using a file stream.
//MyFunctiontoExecute, where Img is the 1D image at the current time
My_file_stream.StreamSetPos(2,0)
ImageWriteImageDataToStream(Img, My_file_stream, 0)
//Write the time to the same file
Number tmp_time = GetHighResTickCount() - start_time
My_file_stream.StreamSetPos(2,0)
My_file_stream.StreamWriteAsText(0,tmp_time) //does not work
//instead using a different file
WriteFile(My_extrafileID,tmp_time+"/n")
I think your concept of streaming is wrong. When you stream to a file, at the end of the toStream() commands, the stream-position is already at the end. So you don't set the position.
Your script essentially tells the computer to set the stream back to that starting position and then to write the text - overwriting the data.
You only need the 'StreamSetPos()' command when you want to jump over some sections during reading (useful when defining import-scripts for specific file formats, for example. Or to extract only specific sub-sets from a file.).
If all you want to do is "stream-out some raw-data", you do exactly that: Just call the commands after each other:
void WriteDataPlusDateToStream( object fStream, image img, string dateStr )
{
number endian = 0
number encoding = 0
img.ImageWriteImageDataToStream(fStream,endian)
fStream.StreamWriteAsText(encoding,dateStr)
}
Similarly, you just "stream-in" by just following the same sequence:
void ReadDataPlusDateFromStream( object fStream, image img, string &dateStr )
{
number endian = 0
number encoding = 0
img.ImageReadImageDataFromStream(fStream,endian)
fStream.StreamReadTextLine(encoding,dateStr)
}
Two things are important here:
in ImageReadImageDataFromStream it is the size and data-type of the image img which defines how many bytes are read from the stream and how they are interpreted. Therefore img must have been pre-created and of fitting size and file-type.
in StreamReadTextLine the stream will continue to read in as text until it encounters the end-of-line character (\n) or the end of the stream. Therefore make sure to write this end-of-line character when streaming-out. Alternatively, you can make sure that the strings are always of a specific size and then use StreamReadAsText with the appropriate length specified.
Using the two methods above, you can use the following test-script as a starting point:
void WriteDataPlusDateToStream( object fStream, image img, string dateStr )
{
number endian = 0
number encoding = 0
img.ImageWriteImageDataToStream(fStream,endian)
fStream.StreamWriteAsText(encoding,dateStr)
}
void ReadDataPlusDateFromStream( object fStream, image img, string &dateStr )
{
number endian = 0
number encoding = 0
img.ImageReadImageDataFromStream(fStream,endian)
fStream.StreamReadTextLine(encoding,dateStr)
}
void writeTest(string path)
{
Result("\n Writing to :" + path )
image testImg := RealImage("Test",4,100)
string dateStr;
number loop = 5;
number doAutoClose = 1
object fStream = NewStreamFromFileReference( CreateFileForWriting(path), doAutoClose )
for( number i=0; i<loop; i++ )
{
testImg = icol * random()
dateStr = GetDate(1)+"#"+GetTime(1)+"|"+Format(GetHighResTickCount(),"%.f") + "\n"
fStream.WriteDataPlusDateToStream(testImg,dateStr)
sleep(0.33)
}
}
void readTest(string path)
{
Result("\n Reading form :" + path )
image testImg := RealImage("Test",4,100)
string dateStr;
number doAutoClose = 1
object fStream = NewStreamFromFileReference( OpenFileForReading(path), doAutoClose )
while ( fStream.StreamGetPos() < fStream.StreamGetSize() )
{
fStream.ReadDataPlusDateFromStream(testImg,dateStr)
result("\n time:"+dateStr)
testImg.ImageClone().ShowImage()
}
}
string path = "C:/test.dat"
ClearResults()
writeTest(path)
readTest(path)
Note, that when streaming "binary data" like this, it is you who defines the file-format. You must make sure that the writing and reading code matches up.

How can I log something in USQL UDO?

I have custom extractor, and I'm trying to log some messages from it.
I've tried obvious things like Console.WriteLine, but cannot find where output is. However, I found some system logs in adl://<my_DLS>.azuredatalakestore.net/system/jobservice/jobs/Usql/.../<my_job_id>/.
How can I log something? Is it possible to specify log file somewhere on Data Lake Store or Blob Storage Account?
A recent release of U-SQL has added diagnostic logging for UDOs. See the release notes here.
// Enable the diagnostics preview feature
SET ##FeaturePreviews = "DIAGNOSTICS:ON";
// Extract as one column
#input =
EXTRACT col string
FROM "/input/input42.txt"
USING new Utilities.MyExtractor();
#output =
SELECT *
FROM #input;
// Output the file
OUTPUT #output
TO "/output/output.txt"
USING Outputters.Tsv(quoting : false);
This was my diagnostic line from the UDO:
Microsoft.Analytics.Diagnostics.DiagnosticStream.WriteLine(System.String.Format("Concatenations done: {0}", i));
This is the whole UDO:
using System.Collections.Generic;
using System.IO;
using System.Text;
using Microsoft.Analytics.Interfaces;
namespace Utilities
{
[SqlUserDefinedExtractor(AtomicFileProcessing = true)]
public class MyExtractor : IExtractor
{
//Contains the row
private readonly Encoding _encoding;
private readonly byte[] _row_delim;
private readonly char _col_delim;
public MyExtractor()
{
_encoding = Encoding.UTF8;
_row_delim = _encoding.GetBytes("\n\n");
_col_delim = '|';
}
public override IEnumerable<IRow> Extract(IUnstructuredReader input, IUpdatableRow output)
{
string s = string.Empty;
string x = string.Empty;
int i = 0;
foreach (var current in input.Split(_row_delim))
{
using (System.IO.StreamReader streamReader = new StreamReader(current, this._encoding))
{
while ((s = streamReader.ReadLine()) != null)
{
//Strip any line feeds
//s = s.Replace("/n", "");
// Concatenate the lines
x += s;
i += 1;
}
Microsoft.Analytics.Diagnostics.DiagnosticStream.WriteLine(System.String.Format("Concatenations done: {0}", i));
//Create the output
output.Set<string>(0, x);
yield return output.AsReadOnly();
// Reset
x = string.Empty;
}
}
}
}
}
And these were my results found in the following directory:
/system/jobservice/jobs/Usql/2017/10/20.../diagnosticstreams
good question. I have been asking myself the same thing. This is theoretical, but I think it would work (I'll updated if I find differently).
One very hacky way is that you could insert rows into a table with your log messages as a string column. Then you can select those out and filter based on some log_producer_id column. You also get the benefit of logging if part of the script works, but later parts do not assuming the failure does not roll back. Table can be dumped at end as well to file.
For the error cases, you can use the Job Manager in ADLA to open the job graph and then view the job output. The errors often have detailed information for data-related errors (e.g. row number in file with error and a octal/hex/ascii dump of the row with issue marked with ###).
Hope this helps,
J
ps. This isn't a comment or an answer really, since I don't have working code. Please provide feedback if the above ideas are wrong.

How to modify footnote placeholder in docx4j

I have a docx file which contains a footnote. I have a placeholder in the footnote text that needs to be replaced. While extracting the nodes and modifying the textvalue that placeholder went unpassed. For some reason I think it is not picking up the text provided in the footnote.
Can u please guide me as to how u get to replace a placeholder in the footnote.
Approach 1
faster if you haven't yet caused unmarshalling to occur:
FootnotesPart fp = wordMLPackage.getMainDocumentPart().getFootnotesPart();
fp.variableReplace(mappings);
Approach 2
FootnotesPart fp = wordMLPackage.getMainDocumentPart().getFootnotesPart();
// unmarshallFromTemplate requires string input
String xml = XmlUtils.marshaltoString(fp.getJaxbElement(), true);
// Do it...
Object obj = XmlUtils.unmarshallFromTemplate(xml, mappings);
// Inject result into docx
fp.setJaxbElement((CTFootnotes) obj);
Since #JasonPlutext's answer did not work for my case I am posting what worked for me
FootnotesPart fp = template.getMainDocumentPart().getFootnotesPart();
List<Object> texts = fp.getJAXBNodesViaXPath("//w:t", true);
for(Object obj : texts) {
Text text = (Text) ((JAXBElement) obj).getValue();
String textValue = text.getValue();
// do variable replacement
text.setValue(textValue);
}
But still I face the issue when exporting this as pdf using Docx4J.toPDF(..);
The output does not pick up the footnote reference.

How to get the last input ID in a textfile?

Can someone help me in my problem?
Because I'm having a hard time of on how to get the last input ID in a text file. My back end is a text file.
Thanks.
this is the sample content of the text file of my program.
ID|CODE1|CODE2|EXPLAIN|CODE3|DATE|PRICE1|PRICE2|PRICE3|
02|JKDHG|hkjd|Hfdkhgfdkjgh|264|56.46.54|654 654.87|878 643.51|567 468.46|
03|DEJSL|hdsk|Djfglkdfjhdlf|616|46.54.56|654 654.65|465 465.46|546 546.54|
01|JANE|jane|Jane|251|56.46.54|534 654.65|654 642.54|543 468.74|
how would I get the last input id so that the id of the input line wouldn't back to number 1?
Make a function that read file and return list of lines(string) like this:
public static List<string> ReadTextFileReturnListOfLines(string strPath)
{
List<string> MyLineList = new List<string>();
try
{
// Create an instance of StreamReader to read from a file.
StreamReader sr = new StreamReader(strPath);
string line = null;
// Read and display the lines from the file until the end
// of the file is reached.
do
{
line = sr.ReadLine();
if (line != null)
{
MyLineList.Add(line);
}
} while (!(line == null));
sr.Close();
return MyLineList;
}
catch (Exception E)
{
throw E;
}
}
I am not sure if
ID|CODE1|CODE2|EXPLAIN|CODE3|DATE|PRICE1|PRICE2|PRICE3|
is part of the file but you have to adjust the index of the element you want to get
, then get the element in the list.
MyStringList(1).split("|")(0);
If you're looking for the last (highest) number in the ID field, you could do it with a single line in LINQ:
Dim MaxID = (From line in File.ReadAllLines("file.txt")
Skip 1
Select line.Split("|")(0)).Max()
What this code does is gets an array via File.ReadAllLines, skips the first line (which appears to be a header), splits each line on the delimiter (|), takes the first element from that split (which is ID) and selects the maximum value.
In the case of your sample input, the result is "03".

How do I save RichTextBox content into SQL varbinary (byte array) column?

I want to save content of a RichTextBox to varbinary (= byte array) in XamlPackage format.
I need technicial advise on how to it.
I actually need to know how to convert between FlowDocument to byte array.
Is it even recommended to store it as varbinary, or this is a bad idea?
Update
Code snippet:
///Load
byte[] document = GetDocumentFromDataBase();
RickTextBox tb = new RickTextBox();
TextRange tr = new TextRange(tb.Document.ContentStart, tb.Document.ContentEnd)
tr.Load(--------------------------) //Load from the byte array.
///Save
int maxAllowed = 1024;
byte[] document;
RichTextBox tb = new RichTextBox();
//User entered text and designs in the rich text
TextRange tr = new TextRange(tb.Document.ContentStart, tb.Document.ContentEnd)
tr.Save(--------------------------) //Save to byte array
if (document.Length > maxAllowed)
{
MessageBox.Show((document.Length - maxAllowed) + " Exceeding limit.");
return;
}
SaveToDataBase();
TextRange
I can't find my full example right now, but you can use XamlReader and XamlWriter to get the document into and out of a string. From there, you can use UnicodeEncoding, AsciiEncoding or whatever encoder you want to get it into and out of bytes.
My shorter example for setting the document from a string...
docReader is my flow document reader
private void SetDetails(string detailsString)
{
if (docReader == null)
return;
if (String.IsNullOrEmpty(detailsString))
{
this.docReader.Document = null;
return;
}
using (
StringReader stringReader = new StringReader(detailsString))
{
using (System.Xml.XmlReader reader = System.Xml.XmlReader.Create(stringReader))
{
this.docReader.Document = XamlReader.Load(reader) as FlowDocument;
}
}
}