I've found the following code in my boss's project:
Dim strBuilder As New System.Text.StringBuilder("", 1000000)
Before I call him out on it, I'd like to confirm whether this line actually sets a megabyte (or two megabytes in Unicode?) of memory aside for that one stringbuilder?
That initializes a Char() of length 1000000.
So the actual size needed in memory is 2000000 Bytes = ~2 MB since a char is unicode and needs 2 bytes.
Edit: Just in case your boss doesn't believe, this is reflected with ILSpy:
// System.Text.StringBuilder
[SecuritySafeCritical]
public unsafe StringBuilder(string value, int startIndex, int length, int capacity)
{
if (capacity < 0)
{
throw new ArgumentOutOfRangeException("capacity", Environment.GetResourceString("ArgumentOutOfRange_MustBePositive", new object[]
{
"capacity"
}));
}
if (length < 0)
{
throw new ArgumentOutOfRangeException("length", Environment.GetResourceString("ArgumentOutOfRange_MustBeNonNegNum", new object[]
{
"length"
}));
}
if (startIndex < 0)
{
throw new ArgumentOutOfRangeException("startIndex", Environment.GetResourceString("ArgumentOutOfRange_StartIndex"));
}
if (value == null)
{
value = string.Empty;
}
if (startIndex > value.Length - length)
{
throw new ArgumentOutOfRangeException("length", Environment.GetResourceString("ArgumentOutOfRange_IndexLength"));
}
this.m_MaxCapacity = 2147483647;
if (capacity == 0)
{
capacity = 16;
}
if (capacity < length)
{
capacity = length;
}
this.m_ChunkChars = new char[capacity];
this.m_ChunkLength = length;
fixed (char* ptr = value)
{
StringBuilder.ThreadSafeCopy(ptr + (IntPtr)startIndex, this.m_ChunkChars, 0, length);
}
}
You could try calling GC.GetTotalMemory() before and after that allocation, and see if it increases. Note: this is not a good, scientific way to do this, but may prove your point.
Related
I am trying to work with payloads in Lucene 6 but I am having troubles. The idea is to index payloads and use them in a CustomScoreQuery to check if the payload of a query term matches the payload for the document term.
Here is my payload filter:
#Override
public final boolean incrementToken() throws IOException {
if (!this.input.incrementToken()) {
return false;
}
// get the current token
final char[] token = Arrays.copyOfRange(this.termAtt.buffer(), 0, this.termAtt.length());
String stoken = String.valueOf(token);
String[] parts = stoken.split(Constants.PAYLOAD_DELIMITER);
if (parts.length > 1 && parts.length == 2){
termAtt.setLength(parts[0].length());
// the rest is the payload
BytesRef br = new BytesRef(parts[1]);
System.out.println(br);
payloadAtt.setPayload(br);
}else if (parts.length > 1){
// skip
}else{
// no payload here
payloadAtt.setPayload(null);
}
return true;
}
It seems to be adding the payload, however when I try to access the payload in CustomScoreQuery it just keeps returning null.
public float determineBoost(int doc) throws IOException{
float boost = 1f;
LeafReader reader = this.context.reader();
System.out.println("Has payloads:" + reader.getFieldInfos().hasPayloads());
// loop through each location of the term and boost if location matches the payload
if (reader != null){
PostingsEnum posting = reader.postings(new Term(this.field, term.getTerm()), PostingsEnum.POSITIONS);
System.out.println("Term: " + term.getTerm());
if (posting != null){
// move to the document currently looking at
posting.advance(doc);
int count = 0;
while (count < posting.freq()){
BytesRef load = posting.getPayload();
System.out.println(posting);
System.out.println(posting.getClass());
System.out.println(posting.attributes());
System.out.println("Load: " + load);
// if the location matches in the term location than boos the term by the boost factor
try {
if(load != null && term.containLocation(new Payload(load))){
boost = boost * this.boost;
}
} catch (PayloadException e) {
// do not care too much, the payload is unrecognized
// this is not going to change the boost factor
}
posting.nextPosition();
count += 1;
}
}
}
return boost;
}
For my two tests it keeps stating the load is null. Any suggestions or help?
On certain images, when I call:
PdfImageObject pimg = new PdfImageObject(stream);
Image bmp = pimg.GetDrawingImage();
The Image that is returned is twisted. I've seen this before and it usually has to do with byte alignment but I'm not sure how to get around this.
The /DecodeParms for this object are /EndOfLine true /K 0 /Columns 3300.
I have tried using the GetStreamBytesRaw() with BitMiracle.LibTiff and with it I can get the data formatted properly although the image is rotated. I'd prefer for GetDrawingImage() to decode the data properly if possible, assuming that is the problem.
I could provide the PDF via email if requested.
Thanks,
Darren
For anyone else that runs across this scenario here is my solution. The key to this puzzle was understanding that /K 0 is G3, /K -1 (or anything less than 0) is G4 /K 1 (or anything greater than 0) is G3-2D.
The twisting happens when you try to make G3 compressed data fit into a G4 image which it appears that is what iTextSharp may be doing. I know it definitely does not work with how I have iTextSharp implemented in my project. I confess that I cannot decipher all the decoding stuff that iTextSharp is doing so it could be something I'm missing too.
EndOfLine didn't have any part in this puzzle but I still think putting line feeds in binary data is a strange practice.
99% of this code came from BitMiracle.LibTiff.Net - Thank you.
int nK = 0;// Default to 0 like the PDF Spec
PdfObject oDecodeParms = stream.Get(PdfName.DECODEPARMS);
if (oDecodeParms is PdfDictionary)
{
PdfObject oK0 = ((PdfDictionary)oDecodeParms).Get(PdfName.K);
if (oK0 != null)
nK = ((PdfNumber)oK0).IntValue;
}
using (MemoryStream ms = new MemoryStream())
{
using (Tiff tiff = Tiff.ClientOpen("custom", "w", ms, new TiffStream()))
{
tiff.SetField(TiffTag.IMAGEWIDTH, width);
tiff.SetField(TiffTag.IMAGELENGTH, height);
if (nK == 0 || nK > 0) // 0 = Group 3, > 0 = Group 3 2D
tiff.SetField(TiffTag.COMPRESSION, Compression.CCITTFAX3);
else if (nK < 0) // < 0 = Group 4
tiff.SetField(TiffTag.COMPRESSION, Compression.CCITTFAX4);
tiff.SetField(TiffTag.BITSPERSAMPLE, bpc);
tiff.SetField(TiffTag.SAMPLESPERPIXEL, 1);
tiff.WriteRawStrip(0, rawBytes, rawBytes.Length); //saving the tiff file using the raw bytes retrieved from the PDF.
tiff.Close();
}
TiffStreamForBytes byteStream = new TiffStreamForBytes(ms.ToArray());
using (Tiff input = Tiff.ClientOpen("bytes", "r", null, byteStream))
{
int stride = input.ScanlineSize();
Bitmap result = new Bitmap(width, height, pixelFormat);
ColorPalette palette = result.Palette;
palette.Entries[0] = System.Drawing.Color.White;
palette.Entries[1] = System.Drawing.Color.Black;
result.Palette = palette;
for (int i = 0; i < height; i++)
{
Rectangle imgRect = new Rectangle(0, i, width, 1);
BitmapData imgData = result.LockBits(imgRect, ImageLockMode.WriteOnly, pixelFormat);
byte[] buffer = new byte[stride];
input.ReadScanline(buffer, i);
System.Runtime.InteropServices.Marshal.Copy(buffer, 0, imgData.Scan0, buffer.Length);
result.UnlockBits(imgData);
}
}
}
/// <summary>
/// Custom read-only stream for byte buffer that can be used
/// with Tiff.ClientOpen method.
/// </summary>
public class TiffStreamForBytes : TiffStream
{
private byte[] m_bytes;
private int m_position;
public TiffStreamForBytes(byte[] bytes)
{
m_bytes = bytes;
m_position = 0;
}
public override int Read(object clientData, byte[] buffer, int offset, int count)
{
if ((m_position + count) > m_bytes.Length)
return -1;
Buffer.BlockCopy(m_bytes, m_position, buffer, offset, count);
m_position += count;
return count;
}
public override void Write(object clientData, byte[] buffer, int offset, int count)
{
throw new InvalidOperationException("This stream is read-only");
}
public override long Seek(object clientData, long offset, SeekOrigin origin)
{
switch (origin)
{
case SeekOrigin.Begin:
if (offset > m_bytes.Length)
return -1;
m_position = (int)offset;
return m_position;
case SeekOrigin.Current:
if ((offset + m_position) > m_bytes.Length)
return -1;
m_position += (int)offset;
return m_position;
case SeekOrigin.End:
if ((m_bytes.Length - offset) < 0)
return -1;
m_position = (int)(m_bytes.Length - offset);
return m_position;
}
return -1;
}
public override void Close(object clientData)
{
// nothing to do
return;
}
public override long Size(object clientData)
{
return m_bytes.Length;
}
}
I'm very new to Arduino and C programming.
I'm making a GPS speedo and I'm trying to read in some serial, store a value from a substring and echo it back via serial.
At the moment I'm having problems storing the substring.
I've gotten to the point where I'm able to get some data between < and >.
But the data doesn't come in like that. It's a NMEA data stream and the data I want is between ,N, and ,K,.
So I've been trying to replace ,N, with < and ,K, with > .
Just can't get it to work. I get error: request for member 'replace' in 'c', which is of non-class type 'char'
Here's my code so far....
int indata = 0;
int scrubdata = 0;
char inString[32];
int stringPos = 0;
boolean startRead = false; // is reading?
void setup() {
Serial.begin(4800);
}
void loop() {
String pageValue = readPage();
Serial.print(pageValue);
}
String readPage(){
//read the page, and capture & return everything between '<' and '>'
stringPos = 0;
memset( &inString, 0, 32 ); //clear inString memory
while(true){
if (Serial.available() > 0) {
char c = Serial.read();
c.replace(",N,", "<");
c.replace(",K,", ">");
if (c == '<' ) { //'<' is our begining character
startRead = true; //Ready to start reading the part
}
else if(startRead){
if(c != '>'){ //'>' is our ending character
inString[stringPos] = c;
stringPos ++;
}
else{
//got what we need here! We can disconnect now
startRead = false;
return inString;
}
}
}
}
}
By Default:
Serial.read() returns an int if you must process the data this way, try casting it to char with:
char c = (char) Serial.read();
Another way to do this:
Would be to seek your beginning string (discarding un-needed data) using Serial.find() then reading data until you met your end character ",K," with Serial.readBytesUntil()
Something like this would work quite well:
char inData[64]; //adjust for your data size
Serial.setTimeout(2000); //Defaults to 1000 msecs set if necessary
Serial.find(",N,"); //Start of Data
int bRead = Serial.readBytesUntil(",K,", inData, 64); //Read until end of data
inData[bRead] = 0x00; //Zero terminate if using this as a string
return inData;
I have been asked to provide a WCF service that allows a blob (potentially 1GB) to be downloaded in chunks as an offset byte[] for consumption by a Silverlight application. Essentially, the operation will have a parameter for number of bytes to offset and the max number of bytes to return, nothing complex I think.
The code I have so far is:
[OperationContract]
public byte[] Download(String url, int blobOffset, int bufferSize)
{
var blob = new CloudBlob(url);
using(var blobStream = blob.OpenRead())
{
var buffer = new byte[bufferSize];
blobStream.Seek(blobOffset, SeekOrigin.Begin);
int numBytesRead = blobStream.Read(buffer, 0, bufferSize);
if (numBytesRead != bufferSize)
{
var trimmedBuffer = new byte[numBytesRead];
Array.Copy(buffer, trimmedBuffer, numBytesRead);
return trimmedBuffer;
}
return buffer;
}
}
I have tested this (albeit with relatively small files < 2MB) and it does work, but my questions are:
Can someone suggest improvements to the code?
Is there a better approach given the requirement?
using (BlobStream blobStream = blob.OpenRead())
{
bool getSuccess = false;
int getTries = 0;
rawBytes = new byte[blobStream.Length];
blobStream.Seek(0, SeekOrigin.Begin);
int blockSize = 4194304; //Start at 4 mb per batch
int index = 0;
int documentSize = rawBytes.Length;
while (getTries <= 10 && !getSuccess)
{
try
{
int batchSize = blockSize;
while (index < documentSize)
{
if ((index + batchSize) > documentSize)
batchSize = documentSize - index;
blobStream.Read(rawBytes, index, batchSize);
index += batchSize;
}
getSuccess = true;
}
catch (Exception e)
{
if (getTries > 9)
throw e;
else
blockSize = blockSize / 2; // Reduce by half for each attempt
}
finally
{ getTries++; }
}
}
You could return the blob as a stream instead of a byte array. There is a code sample in a related question here: Returning Azure BLOB from WCF service as a Stream - Do we need to close it?
Note there are some restrictions on which bindings you can use when you return a stream.
When I issue cmd17 with address (0x00000000) to my card from PIC-18F4520 on SPI bus, I get a correct return R1 token from the command issue. Then, after a few loops checking, I get a 0xFE marker returned from my issuing SPI_Put_Char(0xFF). Data should then start so I read 512 bytes into my IO_Buffer array. As I scan the returns, I got many 0x00 bytes. Oddly, and repeatedly, at about pos 448 in sector 0, some data comes over - a few bytes here and there - then the the final 32 bytes (I can only view 32 at a time on my LCD screen) are all zeroes followed by the 0x55AA marker expected at the end of the boot sector.
The odd thing, is that using disk investigator reveals the SD card has the proper sector zero information - MSDOS message, EB jump code, all sorts of stuff. My read command gives all that back as zeroes. I just don't get what's happening.
Other information: I boot with the cmd0, cmd8, cmd58 and OCR reads fine. Then acmd41 (looping cmd55 followed by APP_SEND_OP_COND). All seem to respond and give expected marker. Finally, I even use SEND_CID to get the card information. that returns MID=3 OID=SD and a verion SD017 followed by other information - seems all to be correct.
I have tried adding pull up and pull down resistors on DOUT from card but doesn't affect any results.
I am desperate for ideas to try to get this card to read correctly. I have (BTW) tried two other cards. They give different specific results, but qualitatively the same - initialization, OCR, and CID read all work okay. Data read gives mostly zeroes followed by some reproducible but sparse bytes, and a 0xAA55 marker!?!
My SanDisk 1GB SD card is running at 3.296 volts which seems stable during card reading.
Here's some code:
bit MMC_Command(unsigned char cmd, unsigned short AdrH, unsigned short AdrL, unsigned char *response)
{
unsigned char response_length;
unsigned char MMC_Counter_Byte = 255;
unsigned char current_response;
switch (cmd)
{
case MMC_SEND_IF_COND:
case MMC_READ_OCR:
response_length = 5;
break;
case MMC_SEND_STATUS:
response_length = 2;
break;
default:
response_length = 1;
};
DEV_xSELECT = DEV_MMC;
SPI_Put_Char(cmd);
SPI_Put_Char(AdrH >> 8);
SPI_Put_Char(AdrH & 0x00FFU);
SPI_Put_Char(AdrL >> 8);
SPI_Put_Char(AdrL & 0x00FFU);
SPI_Put_Char(0x95U); //CRC = 0x95 to get to SPI, then value not important, so always use this for convenience
do
{
response[0] = SPI_Put_Char(0xFF);
} while ((response[0] & 0x80) && --MMC_Counter_Byte);
if (!MMC_Counter_Byte)
{
//SPI_Put_Char(0xFF); //some say is necessary
DEV_xSELECT = DEV_NONE;
return FALSE;
};
for (current_response = 1; current_response < response_length; current_response++)
{
response[current_response] = SPI_Put_Char(0xFF);
};
SPI_Put_Char(0xFF); //some say is necessary
DEV_xSELECT = DEV_NONE;
return TRUE;
};
unsigned char MMC_Init_SD(void)
{
unsigned long MMC_Counter_Word;
unsigned char response[5];
DEV_xSELECT = DEV_MMC;
for (MMC_Counter_Word = 0; MMC_Counter_Word < 20; MMC_Counter_Word++)
{
SPI_Put_Char(0xFFU);
};
DEV_xSELECT = DEV_NONE;
for (MMC_Counter_Word = 0; MMC_Counter_Word < 10; MMC_Counter_Word++)
{
SPI_Put_Char(0xFFU);
};
MMC_Counter_Word = 255;
do
{
MMC_Command(MMC_GO_IDLE_STATE, 0x0000, 0x0000, response); //cmd0
} while (--MMC_Counter_Word && (response[0] != 0x01));
if (!MMC_Counter_Word) //if counter timed out, error
{
return FALSE;
};
MMC_Command(MMC_SEND_IF_COND, 0x0000, 0x01AA, response); //cmd8
if (response[0] != 0x05)
{
return FALSE; //other card type
};
MMC_Command(MMC_READ_OCR, 0x0000, 0x0000, response); //cmd58
MMC_Counter_Word = 0xFFFFU;
do
{
if (MMC_Command(MMC_APP_CMD, 0x0000, 0x0000, response)) //cmd55
{
MMC_Command(MMC_APP_SEND_OP_COND, 0x4001, 0x0000, response); //acmd41
SPI_Put_Char(0xFF);
}
else
{
return FALSE;
};
} while (--MMC_Counter_Word && ((response[0] & 1) == 1));
if (!MMC_Counter_Word)
{
return FALSE;
};
if (MMC_Command(MMC_SEND_CID, 0x0000, 0x0000, response)) //cmd10
{
DEV_xSELECT = DEV_MMC;
MMC_Counter_Word = 255;
while (--MMC_Counter_Word && (SPI_Put_Char(0xFF) != 0xFE));
if (!MMC_Counter_Word)
{
DEV_xSELECT = DEV_NONE;
return FALSE;
};
//code for reading 16 byte OCR goes here
SPI_Put_Char(0xFFU);
SPI_Put_Char(0xFFU); //cycle through 16-bit CRC
SPI_Put_Char(0xFFU); //1GB Sandisk SD seems to require another dummy
DEV_xSELECT = DEV_NONE;
Delay_Sec(2);
LCD_CLS();
}
else
{
return FALSE;
};
return TRUE;
};
bit MMC_Fill_IO_Buffer(unsigned long sector)
{
unsigned short MMC_Fill_Index_Byte;
unsigned char MMC_Counter_Byte = 255;
unsigned char response[1];
if (MMC_Command(MMC_READ_SINGLE_BLOCK, 0x0000, 0x0000, response)) //cmd10
{
DEV_xSELECT = DEV_MMC;
MMC_Counter_Byte = 255;
while (--MMC_Counter_Byte && (SPI_Put_Char(0xFF) != 0xFE));
if (!MMC_Counter_Byte)
{
DEV_xSELECT = DEV_NONE;
return FALSE;
};
}
else
{
return FALSE;
};
for (MMC_Fill_Index_Byte = 0; MMC_Fill_Index_Byte < 512 ; MMC_Fill_Index_Byte++)
{
IO_Buffer[MMC_Fill_Index_Byte] = SPI_Put_Char(0xFF);
};
SPI_Put_Char(0xFFU);
SPI_Put_Char(0xFFU); //cycle through 16-bit CRC
SPI_Put_Char(0xFFU); //1GB Sandisk SD seems to require another dummy
DEV_xSELECT = DEV_NONE;
//following is IO_Buffer displaying code.
//LCD_CLS();
//for (MMC_Counter_Byte = 0; MMC_Counter_Byte < 42; MMC_Counter_Byte++)
//{
// LCD_Draw_Byte_Hex(IO_Buffer[MMC_Counter_Byte + 448]);
//};
//while (1);
return TRUE;
};
Thanks ahead of time!
Your sector 0 looks like a vaild partition table. If you read from the drive letter using disk investigator you may ended up reading the sector 0 of the partition and not from the sd-card itself. This program does not seem to be able to read from a physical device, so you can not use it to read the partition table.
Finally found the solution to this!
It turns out you were reading the MBR, which is located at the address 0 on the SD card. To find the location of the boot sector, one needs to read the appropriate entry in the MBR. The entries start at the address 0x01be and are 16 bytes each. The point of interest in the entry lies at the offset 0x08, is 4 bytes long and is called an LBA. [Wikipedia] To get the address of the boot sector location, one would multiply the LBA by the size of a sector (512 bytes). [Microchip forum]
For an example, see my other answer.