How to dump the NTFS $Bitmap file - ntfs

For a project, I want to get the list of all free/used clusters on an NTFS partition.
For this i have to dump the $Bitmap file and parse its contents.
There are few API's and examples on the web, but however they don't seem to work. Is there a simple way/ code sample to just copy the $Bitmap file somewhere.
Is using FSCTL_GET_VOLUME_BITMAP the only way? Ideally I would like to do it in C#.

NFI.EXE which is (used to be) part of the "oem support tools" can enumerate all NTFS partition items. It might also be capable to dump the content of $Bitmap.

You definitely want to go the easy route and use the IOCTL rather than trying to read $Bitmap directly. Of course, you don't have to do it yourself if somebody has done it for you. It turns out that an MSDN blogger has already written a nice little wrapper for you:
http://blogs.msdn.com/jeffrey_wall/archive/2004/09/13/229137.aspx
The whole class is over 300 lines of code, so I won't post it all, but here's the function that gets the volume bitmap:
/// <summary>
/// Get cluster usage for a device
/// </summary>
/// <param name="DeviceName">use "c:"</param>
/// <returns>a bitarray for each cluster</returns>
static public BitArray GetVolumeMap(string DeviceName)
{
IntPtr pAlloc = IntPtr.Zero;
IntPtr hDevice = IntPtr.Zero;
try
{
hDevice = OpenVolume(DeviceName);
Int64 i64 = 0;
GCHandle handle = GCHandle.Alloc(i64, GCHandleType.Pinned);
IntPtr p = handle.AddrOfPinnedObject();
// alloc off more than enough for my machine
// 64 megs == 67108864 bytes == 536870912 bits == cluster count
// NTFS 4k clusters == 2147483648 k of storage == 2097152 megs == 2048 gig disk storage
uint q = 1024 * 1024 * 64; // 1024 bytes == 1k * 1024 == 1 meg * 64 == 64 megs
uint size = 0;
pAlloc = Marshal.AllocHGlobal((int)q);
IntPtr pDest = pAlloc;
bool fResult = DeviceIoControl(
hDevice,
FSConstants.FSCTL_GET_VOLUME_BITMAP,
p,
(uint)Marshal.SizeOf(i64),
pDest,
q,
ref size,
IntPtr.Zero);
if (!fResult)
{
throw new Exception(Marshal.GetLastWin32Error().ToString());
}
handle.Free();
/*
object returned was...
typedef struct
{
LARGE_INTEGER StartingLcn;
LARGE_INTEGER BitmapSize;
BYTE Buffer[1];
} VOLUME_BITMAP_BUFFER, *PVOLUME_BITMAP_BUFFER;
*/
Int64 StartingLcn = (Int64)Marshal.PtrToStructure(pDest, typeof(Int64));
Debug.Assert(StartingLcn == 0);
pDest = (IntPtr)((Int64)pDest + 8);
Int64 BitmapSize = (Int64)Marshal.PtrToStructure(pDest, typeof(Int64));
Int32 byteSize = (int)(BitmapSize / 8);
byteSize++; // round up - even with no remainder
IntPtr BitmapBegin = (IntPtr)((Int64)pDest + 8);
byte[] byteArr = new byte[byteSize];
Marshal.Copy(BitmapBegin, byteArr, 0, (Int32)byteSize);
BitArray retVal = new BitArray(byteArr);
retVal.Length = (int)BitmapSize; // truncate to exact cluster count
return retVal;
}
finally
{
CloseHandle(hDevice);
hDevice = IntPtr.Zero;
Marshal.FreeHGlobal(pAlloc);
pAlloc = IntPtr.Zero;
}
}

Related

STM32 Crash on Flash Sector Erase

I'm trying to write 4 uint32's of data into the flash memory of my STM32F767ZI so I've looked at some examples and in the reference manual but still I cannot do it. My goal is to write 4 uint32's into the flash and read them back and compare with the original data, and light different leds depending on the success of the comparison.
My code is as follows:
void flash_write(uint32_t offset, uint32_t *data, uint32_t size) {
FLASH_EraseInitTypeDef EraseInitStruct = {0};
uint32_t SectorError = 0;
HAL_FLASH_Unlock();
EraseInitStruct.TypeErase = FLASH_TYPEERASE_SECTORS;
EraseInitStruct.VoltageRange = FLASH_VOLTAGE_RANGE_3;
EraseInitStruct.Sector = FLASH_SECTOR_11;
EraseInitStruct.NbSectors = 1;
//EraseInitStruct.Banks = FLASH_BANK_1; // or FLASH_BANK_2 or FLASH_BANK_BOTH
st = HAL_FLASHEx_Erase(&EraseInitStruct, &SectorError);
if (st == HAL_OK) {
for (int i = 0; i < size; i += 4) {
st = HAL_FLASH_Program(FLASH_TYPEPROGRAM_WORD, FLASH_USER_START_ADDR + offset + i, *(data + i)); //This is what's giving me trouble
if (st != HAL_OK) {
// handle the error
break;
}
}
}else {
// handle the error
}
HAL_FLASH_Lock();
}
void flash_read(uint32_t offset, uint32_t *data, uint32_t size) {
for (int i = 0; i < size; i += 4) {
*(data + i) = *(__IO uint32_t*)(FLASH_USER_START_ADDR + offset + i);
}
}
int main(void) {
uint32_t data[] = {'a', 'b', 'c', 'd'};
uint32_t read_data[] = {0, 0, 0, 0};
HAL_Init();
SystemClock_Config();
MX_GPIO_Init();
flash_write(0, data, sizeof(data));
flash_read(0, read_data, sizeof(read_data));
if (compareArrays(data,read_data,4))
{
HAL_GPIO_WritePin(GPIOB, GPIO_PIN_7,SET);
}
else
{
HAL_GPIO_WritePin(GPIOB, GPIO_PIN_14,SET);
}
return 0;
}
The problem is that before writing data I must erase a sector, and when I do it with the HAL_FLASHEx_Erase(&EraseInitStruct, &SectorError), function, the program always crashes, and sometimes even corrupts my codespace forcing me to update firmware.
I've selected the sector farthest from the code space but still it crashes when i try to erase it.
I've read in the reference manual that
Any attempt to read the Flash memory while it is being written or erased, causes the bus to
stall. Read operations are processed correctly once the program operation has completed.
This means that code or data fetches cannot be performed while a write/erase operation is
ongoing.
which I believe means the code should ideally be run from RAM while we operate on the flash, but I've seen other people online not have this issue so I'm wondering if that's the only problem I have. With that in mind I wanted to confirm if this is my only issue, or if I'm doing something wrong?
In your loop, you are adding multiples of 4 to i, but then you are adding i to data. When you add to a pointer it is automatically multiplied by the size of the pointed type, so you are adding multiples of 16 bytes and reading past the end of your input buffer.
Also, make sure you initialize all members of EraseInitStruct. Uncomment that line and set the correct value!

Byte[r,c,c] to Halcon HObject/HImage to byte[]

I am using Zivid.NET, Halcon.NET and ML.NET together. Zivid provides me with a 3D byte array (row, column, channel), Halcon uses HImages/HObjects, ML.NET functionality expects a 1D byte array (same as File.ReadAllBytes())
So far I used a workaround where:
I save()'d Zivid's imageRGBA as a PNG,
which I read with Halcon's read_image() that gives me a HObject.
After some graphical work I saved the HObject again as a PNG using write_image().
Using File.ReadAllBytes() to read that PNG I get the byte[] that my ML.NET functionalities expect.
But this is far from ideal with larger amounts of data.
What I need is:
a way to convert byte[r,c,c] images to HObject/HImage.
a way to convert HObject/HImage images to byte[].
Halcon's read_image() and write_image() don't seem to have any options for this and I haven't found anything helpful so far.
To create an HImage object from byte, you need a pointer to an array and then it's simple:
public HImage(string type, int width, int height, IntPtr pixelPointer)
To get a pointer and acess the data from HImage, the following function is needed:
IntPtr HImage.GetImagePointer1(out string type, out int width, out int height)
To convert from Zivid.NET's byte[r,c,c] to HImage one can:
var byteArr = imgRGBA.ToByteArray();
byte[,] redByteArray = new byte[1200, 1920];
byte[,] greenByteArray = new byte[1200, 1920];
byte[,] blueByteArray = new byte[1200, 1920];
for (int row = 0; row < 1200; row++)
for (int col = 0; col < 1920; col++)
{
redByteArray[row, col] = byteArr[row, col, 0];
greenByteArray[row, col] = byteArr[row, col, 1];
blueByteArray[row, col] = byteArr[row, col, 2];
}
GCHandle pinnedArray_red = GCHandle.Alloc(redByteArray, GCHandleType.Pinned);
IntPtr pointer_red = pinnedArray_red.AddrOfPinnedObject();
GCHandle pinnedArray_green = GCHandle.Alloc(greenByteArray, GCHandleType.Pinned);
IntPtr pointer_green = pinnedArray_green.AddrOfPinnedObject();
GCHandle pinnedArray_blue = GCHandle.Alloc(blueByteArray, GCHandleType.Pinned);
IntPtr pointer_blue = pinnedArray_blue.AddrOfPinnedObject();
GenImage3(out HObject imgHImage, "byte", 1920, 1200, pointer_red, pointer_green, pointer_blue);
pinnedArray_red.Free();
pinnedArray_green.Free();
pinnedArray_blue.Free();
Still working on the second part..
A better / more efficient method is very welcome.

redis bitset -- how to upload an existing bitset array

I have huge data of bitset, stored in db. I want to upload the same to redis bitset, so I can perform bit operations on it. Is there a way to upload this data from either redis-cli or javascript code? I am using bitset.js npm module to load the bitset in my program from db.
One obvious way is to iterate my bitset array within my javascript code and keep calling redis.setbit(...) multiple times. Is there a way to upload all of them at once? If so how?
A bitset in Redis is actually just a string, so you can assign to it directly all at once. The bits in the string are the bits of the bitfield, set in left-to-right order. I.e. setting bit number 0 to 1 yields the binary number 10000000, or a single byte with the value 128. This looks like "\x80" when Redis prints it, which you can see for yourself by running setbit foo 0 1 and then get foo in Redis.
So to construct the right string to send to Redis, we just need to read the bits out of your BitSet and construct a buffer, one byte at a time, with the appropriate bits set.
Below is code that uses bitset.js and the redis npm module to transfer a BitSet in JavaScript into a Redis key. Note that this code assumes that the bitfield fits comfortably in memory.
let redis = require('redis'),
BitSet = require('./bitset');
let client = redis.createClient();
// create some data
let bs = new BitSet;
bs.set(0, 1);
bs.set(31, 1);
// calculate how many bytes we'll need
var numBytes = Math.ceil(bs.msb()/8);
// construct a buffer with that much space
var buffer = new Buffer(numBytes);
// for each byte
for (var i = 0; i < numBytes; i++) {
var byte = 0;
// iterate over each bit
for (var j = 0; j < 8; j++) {
// slide previous bits to the left
byte <<= 1;
// and set the rightmost bit
byte |= bs.get(i*8+j);
}
// put this byte in the buffer
buffer[i] = byte;
}
// now we have a complete buffer to use as our value in Redis
client.set('bitset', buffer, function (err, result) {
client.getbit('bitset', 31, function (err, result) {
console.log('Bit 31 = ' + result);
client.del('bitset', function () {
client.quit();
});
});
});

How do I copy writeablebitmap to clipboard from an elevated trust silverlight application using P/Invoke?

I've got a silverlight 5 application that is running elevated trust in-browser. This allows us to do things that wouldn't ordinarily be possible in silverlight like having more access to the clipboard via P/Invoke.
What I need to be able to do is copy controls to the clipboard so they could be pasted into Word or Outlook. I can convert the controls to an image via WriteableBitmap but copying the data to the clipboard is something I'm having issues with.
Calling code:
WriteableBitmap bmp = new WriteableBitmap(elements[0], new ScaleTransform() { ScaleX = 1.0, ScaleY = 1.0 });
int[] p = bmp.Pixels;
int len = p.Length * 4;
byte[] result = new byte[len];
Buffer.BlockCopy(p, 0, result, 0, len);
CopyToClipboardViaPInvoke(result, ClipboardFormat.CF_BITMAP);
Copy function:
private void CopyToClipboardViaPInvoke(byte[] data, ClipboardFormat format)
{
IntPtr p = IntPtr.Zero;
if (Native.OpenClipboard(p))
{
GCHandle pinnedArray = GCHandle.Alloc(data, GCHandleType.Pinned);
IntPtr pointer = pinnedArray.AddrOfPinnedObject();
try
{
Native.EmptyClipboard();
IntPtr result = Native.SetClipboardData(format, pointer);
}
finally
{
Native.CloseClipboard();
pinnedArray.Free();
}
}
}
The result says it's successful, but paste does nothing. IsClipboardFormatAvailable also states that the format is available on the clipboard. I've also tried various ClipboardFormat inputs and other methods of converting the control to an image without any luck.
Update 1
Thanks to suggestions from user629926 I've got what I think is a little closer but I'm still missing something.
Native.EmptyClipboard();
IntPtr bmp = IntPtr.Zero;
GCHandle pinnedArray = GCHandle.Alloc(bytes, GCHandleType.Pinned);
IntPtr bmpPointer = pinnedArray.AddrOfPinnedObject();
Native.StartupInput sin = new Native.StartupInput() { GdiplusVersion = 1 };
Native.StartupOutput sout = new Native.StartupOutput();
IntPtr gdip = IntPtr.Zero;
int startup = Native.GdiplusStartup(out gdip, ref sin, out sout);
int created = Native.GdipCreateBitmapFromScan0(width, height, width * 4, 0x0026200A, bmpPointer, out bmp);
IntPtr result = Native.SetClipboardData(format, bmp);
Native.DeleteObject(bmp);
Native.GdiplusShutdown(ref gdip);
Use GdipCreateBitmapFromScan0 to get HBitmap from you array and pass it to clipboard and the DeleteObject to free it.
[DllImport("gdiplus.dll")]
static extern int GdipCreateBitmapFromScan0(int width, int height, int stride, int format, IntPtr scan0, out IntPtr bitmap);
[DllImport("gdi32.dll")]
public static extern bool DeleteObject(IntPtr hObject);

What is the fastest way to compare two byte arrays?

I am trying to compare two long bytearrays in VB.NET and have run into a snag. Comparing two 50 megabyte files takes almost two minutes, so I'm clearly doing something wrong. I'm on an x64 machine with tons of memory so there are no issues there. Here is the code that I'm using at the moment and would like to change.
_Bytes and item.Bytes are the two different arrays to compare and are already the same length.
For Each B In item.Bytes
If B <> _Bytes(I) Then
Mismatch = True
Exit For
End If
I += 1
Next
I need to be able to compare as fast as possible files that are potentially hundreds of megabytes and even possibly a gigabyte or two. Any suggests or algorithms that would be able to do this faster?
Item.bytes is an object taken from the database/filesystem that is returned to compare, because its byte length matches the item that the user wants to add. By comparing the two arrays I can then determine if the user has added something new to the DB and if not then I can just map them to the other file and not waste hard disk drive space.
[Update]
I converted the arrays to local variables of Byte() and then did the same comparison, same code and it ran in like one second (I have to benchmark it still and compare it to others), but if you do the same thing with local variables and use a generic array it becomes massively slower. I’m not sure why, but it raises a lot more questions for me about the use of arrays.
What is the _Bytes(I) call doing? It's not loading the file each time, is it? Even with buffering, that would be bad news!
There will be plenty of ways to micro-optimise this in terms of looking at longs at a time, potentially using unsafe code etc - but I'd just concentrate on getting reasonable performance first. Clearly there's something very odd going on.
I suggest you extract the comparison code into a separate function which takes two byte arrays. That way you know you won't be doing anything odd. I'd also use a simple For loop rather than For Each in this case - it'll be simpler. Oh, and check whether the lengths are correct first :)
EDIT: Here's the code (untested, but simple enough) that I'd use. It's in C# for the minute - I'll convert it in a sec:
public static bool Equals(byte[] first, byte[] second)
{
if (first == second)
{
return true;
}
if (first == null || second == null)
{
return false;
}
if (first.Length != second.Length)
{
return false;
}
for (int i=0; i < first.Length; i++)
{
if (first[i] != second[i])
{
return false;
}
}
return true;
}
EDIT: And here's the VB:
Public Shared Function ArraysEqual(ByVal first As Byte(), _
ByVal second As Byte()) As Boolean
If (first Is second) Then
Return True
End If
If (first Is Nothing OrElse second Is Nothing) Then
Return False
End If
If (first.Length <> second.Length) Then
Return False
End If
For i as Integer = 0 To first.Length - 1
If (first(i) <> second(i)) Then
Return False
End If
Next i
Return True
End Function
The fastest way to compare two byte arrays of equal size is to use interop. Run the following code on a console application:
using System;
using System.Runtime.InteropServices;
using System.Security;
namespace CompareByteArray
{
class Program
{
static void Main(string[] args)
{
const int SIZE = 100000;
const int TEST_COUNT = 100;
byte[] arrayA = new byte[SIZE];
byte[] arrayB = new byte[SIZE];
for (int i = 0; i < SIZE; i++)
{
arrayA[i] = 0x22;
arrayB[i] = 0x22;
}
{
DateTime before = DateTime.Now;
for (int i = 0; i < TEST_COUNT; i++)
{
int result = MemCmp_Safe(arrayA, arrayB, (UIntPtr)SIZE);
if (result != 0) throw new Exception();
}
DateTime after = DateTime.Now;
Console.WriteLine("MemCmp_Safe: {0}", after - before);
}
{
DateTime before = DateTime.Now;
for (int i = 0; i < TEST_COUNT; i++)
{
int result = MemCmp_Unsafe(arrayA, arrayB, (UIntPtr)SIZE);
if (result != 0) throw new Exception();
}
DateTime after = DateTime.Now;
Console.WriteLine("MemCmp_Unsafe: {0}", after - before);
}
{
DateTime before = DateTime.Now;
for (int i = 0; i < TEST_COUNT; i++)
{
int result = MemCmp_Pure(arrayA, arrayB, SIZE);
if (result != 0) throw new Exception();
}
DateTime after = DateTime.Now;
Console.WriteLine("MemCmp_Pure: {0}", after - before);
}
return;
}
[DllImport("msvcrt.dll", CallingConvention = CallingConvention.Cdecl, EntryPoint="memcmp", ExactSpelling=true)]
[SuppressUnmanagedCodeSecurity]
static extern int memcmp_1(byte[] b1, byte[] b2, UIntPtr count);
[DllImport("msvcrt.dll", CallingConvention = CallingConvention.Cdecl, EntryPoint = "memcmp", ExactSpelling = true)]
[SuppressUnmanagedCodeSecurity]
static extern unsafe int memcmp_2(byte* b1, byte* b2, UIntPtr count);
public static int MemCmp_Safe(byte[] a, byte[] b, UIntPtr count)
{
return memcmp_1(a, b, count);
}
public unsafe static int MemCmp_Unsafe(byte[] a, byte[] b, UIntPtr count)
{
fixed(byte* p_a = a)
{
fixed (byte* p_b = b)
{
return memcmp_2(p_a, p_b, count);
}
}
}
public static int MemCmp_Pure(byte[] a, byte[] b, int count)
{
int result = 0;
for (int i = 0; i < count && result == 0; i += 1)
{
result = a[0] - b[0];
}
return result;
}
}
}
If you don't need to know the byte, use 64-bit ints that gives you 8 at once. Actually, you can figure out the wrong byte, once you've isolated it to a set of 8.
Use BinaryReader:
saveTime = binReader.ReadInt32()
Or for arrays of ints:
Dim count As Integer = binReader.Read(testArray, 0, 3)
Better approach... If you are just trying to see if the two are different then save some time by not having to go through the entire byte array and generate a hash of each byte array as strings and compare the strings. MD5 should work fine and is pretty efficient.
I see two things that might help:
First, rather than always accessing the second array as item.Bytes, use a local variable to point directly at the array. That is, before starting the loop, do something like this:
array2 = item.Bytes
That will save the overhead of dereferencing from the object each time you want a byte. That could be expensive in Visual Basic, especially if there's a Getter method on that property.
Also, use a "definite loop" instead of "for each". You already know the length of the arrays, so just code the loop using that value. This will avoid the overhead of treating the array as a collection. The loop would look something like this:
For i = 1 to max Step 1
If (array1(i) <> array2(i))
Exit For
EndIf
Next
Not strictly related to the comparison algorithm:
Are you sure your bottleneck is not related to the memory available and the time used to load the byte arrays? Loading two 2 GB byte arrays just to compare them could bring most machines to their knees. If the program design allows, try using streams to read smaller chunks instead.