I have a very long array that doesn't seem to load, how would I tell if it is too long? According to adobe, the maximum size of a variable is 12 bytes however I have previously been able to load arrays that are much larger than that. On another note, can you do something like this shrink the size of an array?
var LEFT = "com.games.Note.TYPE_LEFT";
var RIGHT = "com.games.Note.TYPE_RIGHT";
var songData = new Array(RIGHT,LEFT,RIGHT,LEFT,RIGHT,LEFT,RIGHT,LEFT)
Related
I am trying to get a bitmap From byte array
val bitmap_tmp =
Bitmap.createBitmap(height, width, Bitmap.Config.ARGB_8888)
val buffer = ByteBuffer.wrap(decryptedText)
bitmap_tmp.copyPixelsFromBuffer(buffer)
callback.bitmap(bitmap_tmp)
I am facing a error in the below line :
bitmap_tmp.copyPixelsFromBuffer(buffer)
The Error Reads As:
java.lang.RuntimeException: Buffer not large enough for pixels
I have tried Different Solutions found on stack Like Add the line before error but still it crashes:
buffer.rewind()
However the Weird part is the same code at a different place for the same image [Same image with same dimensions] get perfectly functioned and I get the bitmap but here it crashes.
How do I solve this?
Thanks in Adv
The error message makes it sound like the buffer you're copying from isn't large enough, like it needs to contain at least as many bytes as necessary to overwrite every pixel in your bitmap (which has a set size and pixel config).
The documentation for the method doesn't make it clear, but here's the source for the Bitmap class, and in that method:
if (bufferBytes < bitmapBytes) {
throw new RuntimeException("Buffer not large enough for pixels");
}
So yeah, you can't partially overwrite the bitmap, you need enough data to fill it. And if you check the source, that depends on the buffer's current position and limit (it's not just its capacity, it's how much data is remaining to be read).
If it works elsewhere, I'm guessing decryptedText is different there, or maybe you're creating your Bitmap with a different Bitmap.Config (like ARGB_8888 requires 4 bytes per pixel)
I create an image in Vulkan and I get an alignment requirement in the memory requirements of 131072. This seems like an enormous alignment and I'm not sure why anything bigger than 128 or 256 may be needed. It's so big that my memory allocation algorithm can't even handle it, and will never be able to practically handle it given that each allocation of this strict an alignment will waste too much space. What's the deal behind this? Here is how I create the image:
VkImageCreateInfo create_info{};
create_info.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO;
create_info.imageType = VK_IMAGE_TYPE_2D;
create_info.pNext = nullptr;
create_info.sharingMode = VK_SHARING_MODE_EXCLUSIVE;
create_info.samples = VkSampleCountFlagBits::VK_SAMPLE_COUNT_1_BIT;
create_info.queueFamilyIndexCount = 0;
image_create_info.extent.width = 1716;
image_create_info.extent.height = 1731;
image_create_info.extent.depth = 1;
image_create_info.usage = VkImageUsageFlagBits::VK_IMAGE_USAGE_SAMPLED_BIT;
image_create_info.tiling = VkImageTiling::VK_IMAGE_TILING_OPTIMAL;
image_create_info.initialLayout = VkImageLayout::VK_IMAGE_LAYOUT_UNDEFINED;
image_create_info.flags = 0;
image_create_info.mipLevels = 1;
image_create_info.format = VK_FORMAT_R8G8B8A8_UINT;
image_create_info.arrayLayers = 1;
VkImage vk_image;
VkResult result = vkCreateImage((VkDevice)VK::logicalDevice, &image_create_info, nullptr, &vk_image);
VkMemoryRequirements requirements;
vkGetImageMemoryRequirements(VK::logicalDevice, vk_image, &requirements);
Another interesting thing about the requirements returned by the function is that the memory size requirement for format VK_FORMAT_R8G8B8A8_UINT is about 12 mb, which makes sense, but with a format of VK_FORMAT_R8G8B8_UINT (so without the alpha channel), it gives a size requirement of only 3 mb, about a quarter of the size. Have I run into some sort of bug?
I know the dimensions of the image I created aren't power of two, but surely this shouldn't lead to such strange behaviour, should it?
It's so big that my memory allocation algorithm can't even handle it and will never be able to practically handle it given that each allocation of this strict an alignment will waste too much space.
Then fix that.
Implementations are allowed to require all kinds of alignments, especially for optimally-tiled images. 128KiB alignment is hardly unreasonable for images. So your sub-allocator needs to be able to account for this.
As for "waste too much space," perhaps you should take another look at those numbers. The example texture must take up at least 11'881'584 bytes. 128KiB is slightly more than 1% of that storage. That's not a lot of waste.
Using Kotlin, how would I transform a Sequence of detail objects into a Sequence of summary objects, where each summary object represents a fixed number of detailed objects? It's kind of like map, except it's not one-to-one and it's kind of like reduce except it's not to a single accumulator?
For example:
var hourlyInfo: Sequence<HourlyData> = ...
var dailyInfo: Sequence<DailySummary> = hourlyInfo.somethingMagical()
What about the same problem with a stream instead of a sequence?
If there's a known fixed number of items, you use chunked and then map. When the sequence is iterated, actual lists of each chunk of items will be allocated.
var dailyInfo: Sequence<DailySummary> = hourlyInfo
.chunked(12)
.map(::DailySummary) // for example if it has a constructor taking a List of 12 HourlyData
If the List of HourlyData isn't ordered, or there isn't some fixed size, then I suppose you have to group by some property in HourlyData and then map it. This does not preserve Sequence status.
var dailyInfo: Sequence<DailySummary> = hourlyInfo
.groupBy(HourlyData::hourOfDay)
.map { (hour, listOfHourlyData) -> DailySummary(hour, listOfHourlyData) }
I have a data file in CSV (Comma-Separated-Value) format that has about 50 million lines in it.
Each line is read into a string, parsed, and then used to fill in the fields of an object of type FOO. The object then gets added to a List(of FOO) that ultimately has 50 million items.
That all works, and fits in memory (at least on an x64 machine), but its SLOW. It takes like 5 minutes every time load and parse the file into the list. I would like to make it faster. How can I make it faster?
The important parts of the code are shown below.
Public Sub LoadCsvFile(ByVal FilePath As String)
Dim s As IO.StreamReader = My.Computer.FileSystem.OpenTextFileReader(FilePath)
'Find header line
Dim L As String
While Not s.EndOfStream
L = s.ReadLine()
If L = "" Then Continue While 'discard blank line
Exit While
End While
'Parse data lines
While Not s.EndOfStream
L = s.ReadLine()
If L = "" Then Continue While 'discard blank line
Dim T As FOO = FOO.FromCSV(L)
Add(T)
End While
s.Close()
End Sub
Public Class FOO
Public time As Date
Public ID As UInt64
Public A As Double
Public B As Double
Public C As Double
Public Shared Function FromCSV(ByVal X As String) As FOO
Dim T As New FOO
Dim tokens As String() = X.Split(",")
If Not DateTime.TryParse(tokens(0), T.time) Then
Throw New Exception("Could not convert CSV to FOO: Invalid ISO 8601 timestamp")
End If
If Not UInt64.TryParse(tokens(1), T.ID) Then
Throw New Exception("Could not convert CSV to FOO: Invalid ID")
End If
If Not Double.TryParse(tokens(2), T.A) Then
Throw New Exception("Could not convert CSV to FOO: Invalid Format for A")
End If
If Not Double.TryParse(tokens(3), T.B) Then
Throw New Exception("Could not convert CSV to FOO: Invalid Format for B")
End If
If Not Double.TryParse(tokens(4), T.C) Then
Throw New Exception("Could not convert CSV to FOO: Invalid Format for C")
End If
Return T
End Function
End Class
I did some benchmarking and here are the results.
The complete algorithm above took 314 seconds to load the whole file and put the objects into the list.
With the body of FromCSV() reduced to just returning a new object of type FOO with default field values, the whole process took 84 seconds. Therefore it appears that processing the line of text into the object fields is taking 230 seconds (73% of the total time).
Doing everything but parsing the ISO 8601 date string takes 175 seconds. Therefore it appears that processing the date string takes 139 seconds, which is 60% of the text processing time, just for that one field.
Just reading the lines in the file without any processing or object creating takes 41 seconds.
Using StreamReader.ReadBlock to read the whole file in chunks of about 1KB takes 24s, but its a minor improvement in the grand scheme of things and probably not worth the added complexity. In order to use TryParse I would now need to manually create the temporary strings rather than using String.Split().
At this point the only path I see is to just display status to the user every few seconds so they don't wonder if the program is frozen or something.
UPDATE
I created two new functions. One can save the dataset from memory into a binary file using System.IO.BinaryWriter. The other function can load that binary file back into memory using System.IO.BinaryReader. The binary versions were considerably faster than CSV versions, and the binary files take up much less space.
Here are the benchmark results (same dataset for all tests):
LOAD CSV: 340s
SAVE CSV: 312s
SAVE BIN: 29s
LOAD BIN: 41s
CSV FILE SIZE: 3.86GB
BIN FILE SIZE: 1.63GB
I have a lot of experience with CSV, and the bad news is that you aren't going to be able to make this a whole lot faster. CSV libraries aren't going to be of much assistance here. The difficult problem with CSV, that libraries attempt to handle, is dealing with fields that have embedded commas, or newlines, which require quoting and escaping. Your dataset doesn't have this issue, since none of the columns are strings.
As you have discovered, the bulk of the time is spent in the parse methods. Andrew Morton had a good suggestion, using TryParseExact for DateTime values can be a quite a bit faster than TryParse. My own CSV library, Sylvan.Data.Csv (which is the fastest available for .NET), uses an optimization where it parses primitive values directly out of the stream read buffer without converting to string first (only when running on .NET core), that can also speed things up a bit. However, I wouldn't expect it to be possible to cut the processing time in half while sticking with CSV.
Here is an example of using my library, Sylvan.Data.Csv to process the CSV in C#.
static List<Foo> Read(string file)
{
// estimate of the average row length based on Andrew Morton's 4GB/50m
const int AverageRowLength = 80;
var textReader = File.OpenText(file);
// specifying the DateFormat will cause TryParseExact to be used.
var csvOpts = new CsvDataReaderOptions { DateFormat = "yyyy-MM-ddTHH:mm:ss" };
var csvReader = CsvDataReader.Create(textReader, csvOpts);
// estimate number of rows to avoid growing the list.
var estimatedRows = (int)(textReader.BaseStream.Length / AverageRowLength);
var data = new List<Foo>(estimatedRows);
while (csvReader.Read())
{
if (csvReader.RowFieldCount < 5) continue;
var item = new Foo()
{
time = csvReader.GetDateTime(0),
ID = csvReader.GetInt64(1),
A = csvReader.GetDouble(2),
B = csvReader.GetDouble(3),
C = csvReader.GetDouble(4)
};
data.Add(item);
}
return data;
}
I'd expect this to be somewhat faster than your current implementation, so long as you are running on .NET core. Running on .NET framework the difference, if any, wouldn't be a significant. However, I don't expect this to be acceptably fast for your users, it will still likely take tens of seconds, or minutes to read the whole file.
Given that, my advice would be to abandon CSV altogether, which means you can abandon parsing which is what is slowing things down. Instead, read and write the data in binary form. Your data records have a nice property, in that they are fixed width: each record contains 5 fields that are 8 bytes (64bit) wide, so each record requires exactly 40 bytes in binary form. 50m x 40 = 2GB. So, assuming Andrew Morton's estimate of 4GB for the CSV is correct, moving to binary will halve the storage needs. Immediately, that means there is half as much disk IO needed to read the same data. But beyond that, you won't need to parse anything, the binary representation of the value will essentially be copied directly to memory.
Here are some examples of how to do this in C# (don't know VB very well, sorry).
static List<Foo> Read(string file)
{
var stream = File.OpenRead(file);
// the exact number of records can be determined by looking at the length of the file.
var recordCount = stream.Length / 40;
var data = new List<Foo>(recordCount);
var br = new BinaryReader(stream);
for (int i = 0; i < recordCount; i++)
{
var ticks = br.ReadInt64();
var id = br.ReadInt64();
var a = br.ReadDouble();
var b = br.ReadDouble();
var c = br.ReadDouble();
var f = new Foo()
{
time = new DateTime(ticks),
ID = id,
A = a,
B = b,
C = c,
};
data.Add(f);
}
return data;
}
static void Write(List<Foo> data, string file)
{
var stream = File.Create(file);
var bw = new BinaryWriter(stream);
foreach(var item in data)
{
bw.Write(item.time.Ticks);
bw.Write(item.ID);
bw.Write(item.A);
bw.Write(item.B);
bw.Write(item.C);
}
}
This should almost certainly be an order of magnitude faster than a CSV-based solution. The question then becomes: is there some reason that you must use CSV? If the source of the data is out of your control, and you must use CSV, I would then ask: will the data file change every time, or will it only be appended to with new data? If it is appended to, I would investigate a solution where each time the app starts convert only the new section of appended CSV data and add it to a binary file that you will then load everything from. Then you only have to pay the cost of processing the new CSV data each time, and will load everything quickly from the binary form.
This could be made even faster by creating fixed layout struct (Foo), allocating an array of them, and using span-based trickery to read the array data directly from the FileStream. This can be done because all of your data elements are "blittable". This would be the absolute fastest way to load this data into your program. Start with the BinaryReader/Writer and if you find that still isn't fast enough, then investigate this.
If you find this solution to work, I'd love to hear the results.
I want to retrieve the version information of msi-package(s)
What's the better way?
First: Guessing a buffer that is large enough and recall if it doesn't fit (ERROR_MORE_DATA)
1 func call vs. 3 func calls and buffer can be bigger then needed
Second: Call the api function to get the buffer size and then recall it to get the string with a (perfect) matching buffer size
2 func calls every time with a perfect buffer size
It's about (1 or 3) function call(s) vs. 2 function calls every time.
Is there any best practice for this "problem"?
I hope to get a generalized answer (assume calling function is really time consuming and/or buffer size can be very different (10 bytes to 200 megabyte) for further code writing. :-)
pseudo code:
First:
StringBuffer = 10 // (byte) guessing returned string will fit in 10 bytes
result = MsiGetProductInfoW(
product,
INSTALLPROPERTY_VERSIONSTRING,
VersionString,
StringBuffer
); //maybe it fits in 10
if result = ERROR_MORE_DATA then //doesnt fit in 10 so recall to get the correct buffer size
begin
MsiGetProductInfoW(
product,
INSTALLPROPERTY_VERSIONSTRING,
nil,
StringBuffer
);
Inc(StringBuffer); // cause null-terminated string
// recall it with matching
MsiGetProductInfoW(
product,
INSTALLPROPERTY_VERSIONSTRING,
VersionString,
StringBuffer
);
end;
Second:
StringBuffer = 0;
// get buffer size
MsiGetProductInfoW(
product,
INSTALLPROPERTY_VERSIONSTRING,
nil,
StringBuffer
);
Inc(StringBuffer); // cause null-terminated string
// use it with the correct buffersize
MsiGetProductInfoW(
product,
INSTALLPROPERTY_VERSIONSTRING,
VersionString,
StringBuffer
);
Thank you!
In your First option, you can skip the second call, because even on the failing first call, the needed size should be stored in StringBuffer.
This makes the choice (1 or 2) vs. (always 2). That should be clear enough. Further, it shouldn't be hard to come up with a reasonable-sized buffer, that will pass 90+% of the time.