Guess a buffer size and recall an api function or call it everytime twotimes to get exact buffer size - api

I want to retrieve the version information of msi-package(s)
What's the better way?
First: Guessing a buffer that is large enough and recall if it doesn't fit (ERROR_MORE_DATA)
1 func call vs. 3 func calls and buffer can be bigger then needed
Second: Call the api function to get the buffer size and then recall it to get the string with a (perfect) matching buffer size
2 func calls every time with a perfect buffer size
It's about (1 or 3) function call(s) vs. 2 function calls every time.
Is there any best practice for this "problem"?
I hope to get a generalized answer (assume calling function is really time consuming and/or buffer size can be very different (10 bytes to 200 megabyte) for further code writing. :-)
pseudo code:
First:
StringBuffer = 10 // (byte) guessing returned string will fit in 10 bytes
result = MsiGetProductInfoW(
product,
INSTALLPROPERTY_VERSIONSTRING,
VersionString,
StringBuffer
); //maybe it fits in 10
if result = ERROR_MORE_DATA then //doesnt fit in 10 so recall to get the correct buffer size
begin
MsiGetProductInfoW(
product,
INSTALLPROPERTY_VERSIONSTRING,
nil,
StringBuffer
);
Inc(StringBuffer); // cause null-terminated string
// recall it with matching
MsiGetProductInfoW(
product,
INSTALLPROPERTY_VERSIONSTRING,
VersionString,
StringBuffer
);
end;
Second:
StringBuffer = 0;
// get buffer size
MsiGetProductInfoW(
product,
INSTALLPROPERTY_VERSIONSTRING,
nil,
StringBuffer
);
Inc(StringBuffer); // cause null-terminated string
// use it with the correct buffersize
MsiGetProductInfoW(
product,
INSTALLPROPERTY_VERSIONSTRING,
VersionString,
StringBuffer
);
Thank you!

In your First option, you can skip the second call, because even on the failing first call, the needed size should be stored in StringBuffer.
This makes the choice (1 or 2) vs. (always 2). That should be clear enough. Further, it shouldn't be hard to come up with a reasonable-sized buffer, that will pass 90+% of the time.

Related

Why is my custom block going twice into general_work() function in GNU Radio?

I am creating a custom block "Combine" that gets 20 bytes of data from the first input. The value of first input specifies the number of bytes to be read from the second input, which are read and wrote to the output file.
Whenever I execute the flowgraph, the printing shows that the code goes twice into the general work function. It reads the correct data in the first time and the second time, it just reads bogus values and writes this incorrect data to the output sink.
I am using the following signatures for the input:
Combine_impl::Combine_impl()
: gr::block("Combine",
gr::io_signature::make(2, 2, sizeof(unsigned char)),
gr::io_signature::make(1, 1, sizeof(unsigned char)))
{}
I think my problem is with the forecast function and the usage of consume each function. I have tried doing this in forecast but it still goes twice into the general_work function and writes incorrect data to the output file.
ninput_items_required[0] = 20;
ninput_items_required[1] = 7; //because the first input has a value of 7 and will read 7 bytes of data from the second input
Can someone please help me this to determine what exactly is going wrong over here? Also, how is the consume_each() function supposed to be utilized over here?
I modified the forecast function to take the exact number of items that were utilized from each input:
void Combine_impl::forecast (int noutput_items, gr_vector_int &ninput_items_required)
{
ninput_items_required[0] = 20;
ninput_items_required[1] = 7;
}
Instead of using consume_each() which specifies the number of bytes consumed in all of the inputs, I used the consume() function which specifies this number separately for each input:
consume(0, 20); //20 bytes of input at index 0 were consumed
consume(1, 7); //7 bytes of input at index 1 were consumed
And instead of returning noutput_items from the general_work function, I return the following. It exactly the specifies the number of bytes that are to returned, whose value is different than noutput_items.
return (20 + 7);

Can I append data to the buffer while repeating SSL_write()?

The following description is about int SSL_write(SSL *ssl, const void *buf, int num):
When an SSL_write() operation has to be repeated because of
SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, it must be repeated with
the same arguments.
What does the word "same" mean exactly? Is it right if I remain the same buf pointer while appending data to this buffer?
same means ssl points to the same SSL object, num is the same value as before, buf points to the same buffer, and the first num bytes of that buffer have not changed. You can add data to the buffer beyond the first num bytes without impact.

Can I measure a TableBatchOperation's size?

The .Net SDK documentation for TableBatchOperation says that
A batch operation may contain up to 100 individual table operations, with the requirement that each operation entity must have same partition key. A batch with a retrieve operation cannot contain any other operations. Note that the total payload of a batch operation is limited to 4MB.
It's easy to ensure that I don't add more than 100 individual table operations to the batch: in the worst case, I can check the Count property. But is there any way to check the payload size other than manually serialising the operations (at which point I've lost most of the benefit of using the SDK)?
As you add entities you can track the size of the names plus data. Assuming you're using a newer library where the default is Json, the additional characters added should be relatively small (compared to the data if you're close to 4MB) and estimable. This isn't a perfect route, but it would get you close.
Serializing as you go especially if you're actually getting close to the 100 entity limit or the 4MB limit frequently is going to lose you a lot of perf, aside from any convenience lost. Rather than trying to track as you go either by estimating size or serializing, you might be best off sending the batch request as-is and if you get a 413 indicating request body too large, catch the error, divide the batch in 2, and continue.
I followed Emily Gerner's suggestion using optimistic inserts and error handling, but using StorageException.RequestInformation.EgressBytes to estimate the number of operations which fit in the limit. Unless the size of the operations varies wildly, this should be more efficient. There is a case to be made for not raising len every time, but here's an implementation which goes back to being optimistic each time.
int off = 0;
while (off < ops.Count)
{
// Batch size.
int len = Math.Min(100, ops.Count - off);
while (true)
{
var batch = new TableBatchOperation();
for (int i = 0; i < len; i++) batch.Add(ops[off + i]);
try
{
_Tbl.ExecuteBatch(batch);
break;
}
catch (Microsoft.WindowsAzure.Storage.StorageException se)
{
var we = se.InnerException as WebException;
var resp = we != null ? (we.Response as HttpWebResponse) : null;
if (resp != null && resp.StatusCode == HttpStatusCode.RequestEntityTooLarge)
{
// Assume roughly equal sizes, and base updated length on the size of the previous request.
// We assume that no individual operation is too big!
len = len * 4000000 / (int)se.RequestInformation.EgressBytes;
}
else throw;
}
}
off += len;
}

Dealing with Int64 value with Booksleeve

I have a question about Marc Gravell's Booksleeve library.
I tried to understand how booksleeve deal the Int64 value (i have billion long value in Redis actually)
I used reflection to undestand the Set long value overrides.
// BookSleeve.RedisMessage
protected static void WriteUnified(Stream stream, long value)
{
if (value >= 0L && value <= 99L)
{
int i = (int)value;
if (i <= 9)
{
stream.Write(RedisMessage.oneByteIntegerPrefix, 0, RedisMessage.oneByteIntegerPrefix.Length);
stream.WriteByte((byte)(48 + i));
}
else
{
stream.Write(RedisMessage.twoByteIntegerPrefix, 0, RedisMessage.twoByteIntegerPrefix.Length);
stream.WriteByte((byte)(48 + i / 10));
stream.WriteByte((byte)(48 + i % 10));
}
}
else
{
byte[] bytes = Encoding.ASCII.GetBytes(value.ToString());
stream.WriteByte(36);
RedisMessage.WriteRaw(stream, (long)bytes.Length);
stream.Write(bytes, 0, bytes.Length);
}
stream.Write(RedisMessage.Crlf, 0, 2);
}
I don't understand why, with more than two digits int64, the long is encoding in ascii?
Why don't use byte[] ? I know than i can use byte[] overrides to do this, but i just want to understand this implementation to optimize mine. There may be a relationship with the Redis storage.
By advance thank you Marc :)
P.S : i'm still very enthusiastic about your next major version, than i can use long value key instead of string.
It writes it in ASCII because that is what the redis protocol demands.
If you look carefully, it is always encoded as ASCII - but for the most common cases (0-9, 10-99) I've special-cased it, as these are very simple results:
x => $1\r\nX\r\n
xy => $2\r\nXY\r\n
where x and y are the first two digits of a number in the range 0-99, and X and Y are those digits (as numbers) offset by 48 ('0') - so decimal 17 becomes the byte sequence (in hex):
24-32-0D-0A-31-37-0D-0A
Of course, that can also be achieved simply via the writing each digit sequentially and offsetting the digit value by 48 ('0'), and handling the negative sign - I guess the answer there is simply "because I coded it the simple but obviously correct way". Consider the value -123 - which is encoded as $4\r\n-123\r\n (hey, don't look at me - I didn't design the protocol). It is slightly awkward because it needs to calculate the buffer length first, then write that buffer length, then write the value - remembering to write in the order 100s, 10s, 1s (which is much harder than writing the other way around).
Perfectly willing to revisit it - simply: it works.
Of course, it becomes trivial if you have a scratch buffer available - you just write it in the simple order, then reverse the portion of the scratch buffer. I'll check to see if one is available (and if not, it wouldn't be unreasonable to add one).
I should also clarify: there is also the integer type, which would encode -123 as :-123\r\n - however, from memory there are a lot of places this simply does not work.

Hacky Sql Compact Workaround

So, I'm trying to use ADO.NET to stream a file data stored in an image column in a SQL Compact database.
To do this, I wrote a DataReaderStream class that takes a data reader, opened for sequential access, and represents it as a stream, redirecting calls to Read(...) on the stream to IDataReader.GetBytes(...).
One "weird" aspect of IDataReader.GetBytes(...), when compared to the Stream class, is that GetBytes requires the client to increment an offset and pass that in each time it's called. It does this even though access is sequential, and it's not possible to read "backwards" in the data reader stream.
The SqlCeDataReader implementation of IDataReader enforces this by incrementing an internal counter that identifies the total number of bytes it has returned. If you pass in a number either less than or greater than that number, the method will throw an InvalidOperationException.
The problem with this, however, is that there is a bug in the SqlCeDataReader implementation that causes it to set the internal counter to the wrong value. This results in subsequent calls to Read on my stream throwing exceptions when they shouldn't be.
I found some infomation about the bug on this MSDN thread.
I was able to come up with a disgusting, horribly hacky workaround, that basically uses reflection to update the field in the class to the correct value.
The code looks like this:
public override int Read(byte[] buffer, int offset, int count)
{
m_length = m_length ?? m_dr.GetBytes(0, 0, null, offset, count);
if (m_fieldOffSet < m_length)
{
var bytesRead = m_dr.GetBytes(0, m_fieldOffSet, buffer, offset, count);
m_fieldOffSet += bytesRead;
if (m_dr is SqlCeDataReader)
{
//BEGIN HACK
//This is a horrible HACK.
m_field = m_field ?? typeof (SqlCeDataReader).GetField("sequentialUnitsRead", BindingFlags.NonPublic | BindingFlags.Instance);
var length = (long)(m_field.GetValue(m_dr));
if (length != m_fieldOffSet)
{
m_field.SetValue(m_dr, m_fieldOffSet);
}
//END HACK
}
return (int) bytesRead;
}
else
{
return 0;
}
}
For obvious reasons, I would prefer to not use this.
However, I do not want to buffer the entire contents of the blob in memory either.
Does any one know of a way I can get streaming data out of a SQL Compact database without having to resort to such horrible code?
I contacted Microsoft (through the SQL Compact Blog) and they confirmed the bug, and suggested I use OLEDB as a workaround. So, I'll try that and see if that works for me.
Actually, I decided to fix the problem by just not storing blobs in the database to begin with.
This eliminates the problem (I can stream data from a file), and also fixes some issues I might have run into with Sql Compact's 4 GB size limit.