We are working on sending UDP packets to PSI SI. We are developing PSI SI generator.
But we are stuck on CheckSum CRC32 - we are not able to find the check sum. I tried on few code from the internet . It comes up with some checksum but that checksum doesnt match with wireshark check sum .
We have wireshark dump of the PSISI packets working with correct checksum .
Can anyoone help me in calculating the checksum for the PSI SI ?
Regards,
vipul
I am just developing a DVB-S Head station and manipulating the SI-data I ran into the same problem. The solution is to read the ISO/IEC 13818-1 exactly and use the right algorithm.
ISO/IEC 13818-1 describes that the beginning of a section of a PSI-Table is indicated by a pointer field in the same Transport Stream packet payload. This means, that there always is a pointer field in front of the section data and this pointer field must not be put in the checksum calculation. The first byte of the pointer field is the length of the field data. In most cases there is no field data and you find a simple zero in front of your section data that starts with the table id of the section. Don´t take this zero into the checksum calculation.
MPEG´s CRC 32 is a cyclic unreflected redundancy check that starts with 0xffffffff and takes the highest bits first. The magic value is 0x04C11DB7 that can be easily derived by the polynom specified in ISO/IEC 13818-1 Annex B with assigning each bit to the polynomial exponent.
Putting all together you have this simple code to calculate the checksum:
uint calcCrc32(byte[] sectionData, int sectionDataLength)
{
uint crc32 = 0xffffffff;
for (int i = 1 + sectionData[0]; i < sectionDataLength; i++)
{
byte b = sectionData[i];
for (int bit = 0; bit < 8; bit++)
{
if ((crc32 >= 0x80000000) != (b >= 0x80))
crc32 = (crc32 << 1) ^ 0x04C11DB7;
else
crc32 = (crc32 << 1);
b <<= 1;
}
}
return crc32;
}
Related
I want to write for example the number 32 to the 16-24 bits of a register. This register is 100 bits long and the rest or some of the register contains "reserved bits" that shouldn't be write to (According to the datasheet.) or lets say it contains other values I don't want to change(Previous settings.).
If it was only a few bits I could set each one of them with the R &= ~(1 << x) or R |= 1 << x for each bit. But if it was a number, It'd be a huge pain to turn 32 to binary and do it one by one. I see some of the examples basically do something like R = 0x20 << 16. but I'm confused. wouldn't that ruin every other bit and set the reserved bits to 0 messing with the MCU Operation?
I want to write for example the number 32 to the 16-24 bits of a register. This register is 100 bits long and the rest or some of the register contains "reserved bits" that shouldn't be write to (According to the datasheet.) or lets say it contains other values I don't want to change(Previous settings.).
You want to perform a Read-Modify-Write. In this case, you are interested in setting bits 16-24 to a specific value. Assuming those values are zero, you can do that like this:
my_register |= (32 << 16);
This is a Bitwise-OR operation and that is important to note because it keeps whatever the value of the bits were.
Assuming those values are non-zero, you will want to clear those bits first, then write the new value. You can do that like this:
my_register &= ~(0xFF << 16); //Clear bits 16-24
my_register |= (0x20 << 16); //Set bits 16-24 to 32
The above uses Bitwise AND, Bitwise OR, and Bitwise inversion. Again, these operations maintain the values of other bits.
I see some of the examples basically do something like R = 0x20 << 16.
but I'm confused. wouldn't that ruin every other bit and set the
reserved bits to 0 messing with the MCU Operation?
That's not necessarily true. Those bits are likely write protected, or the default value for those bits might be 0 so writing 0 to them has no effect. It just depends on the MCU itself.
Here a function for understanding the principe:
unsigned SetSomeBits(unsigned Var, unsigned StartBitNumber, unsigned NumberOfBits, unsigned Value2Set)
{
unsigned Mask = (1<<NumberOfBits)-1; //With NumberOfBits=3 Mask becomes 0b000111
Mask <<= StartBitNumber;
//Mask contains now 0 at do-not-touch bit positions
//Mask contains now 1 at to-be-changed bit positions
Var &= ~Mask; //Zero out the to-be-changed bits
return Var | (Value2Set<<StartBitNumber); //Set the requested bits
}
...and here as a macro:
#define SET_SOME_BITS(Var, StartBitNumber, NumberOfBits, Value2Set) ((Var) & ~(((1<<(NumberOfBits))-1)<<(StartBitNumber)) | (Value2Set)<<(StartBitNumber))
Both versions fail if Value2Set doesn't fit into NumberOfBits.
I have confusion in this particular line-->
result = (double) hi * (1 << 30) * 4 + lo;
of the following code:
void access_counter(unsigned *hi, unsigned *lo)
// Set *hi and *lo to the high and low order bits of the cycle
// counter.
{
asm("rdtscp; movl %%edx,%0; movl %%eax,%1" // Read cycle counter
: "=r" (*hi), "=r" (*lo) // and move results to
: /* No input */ // the two outputs
: "%edx", "%eax");
}
double get_counter()
// Return the number of cycles since the last call to start_counter.
{
unsigned ncyc_hi, ncyc_lo;
unsigned hi, lo, borrow;
double result;
/* Get cycle counter */
access_counter(&ncyc_hi, &ncyc_lo);
lo = ncyc_lo - cyc_lo;
borrow = lo > ncyc_lo;
hi = ncyc_hi - cyc_hi - borrow;
result = (double) hi * (1 << 30) * 4 + lo;
if (result < 0) {
fprintf(stderr, "Error: counter returns neg value: %.0f\n", result);
}
return result;
}
The thing I cannot understand is that why is hi being multiplied with 2^30 and then 4? and then low added to it? Someone please explain what is happening in this line of code. I do know that what hi and low contain.
The short answer:
That line turns a 64bit integer that is stored as 2 32bit values into a floating point number.
Why doesn't the code just use a 64bit integer? Well, gcc has supported 64bit numbers for a long time, but presumably this code predates that. In that case, the only way to support numbers that big is to put them into a floating point number.
The long answer:
First, you need to understand how rdtscp works. When this assembler instruction is invoked, it does 2 things:
1) Sets ecx to IA32_TSC_AUX MSR. In my experience, this generally just means ecx gets set to zero.
2) Sets edx:eax to the current value of the processor’s time-stamp counter. This means that the lower 64bits of the counter go into eax, and the upper 32bits are in edx.
With that in mind, let's look at the code. When called from get_counter, access_counter is going to put edx in 'ncyc_hi' and eax in 'ncyc_lo.' Then get_counter is going to do:
lo = ncyc_lo - cyc_lo;
borrow = lo > ncyc_lo;
hi = ncyc_hi - cyc_hi - borrow;
What does this do?
Since the time is stored in 2 different 32bit numbers, if we want to find out how much time has elapsed, we need to do a bit of work to find the difference between the old time and the new. When it is done, the result is stored (again, using 2 32bit numbers) in hi / lo.
Which finally brings us to your question.
result = (double) hi * (1 << 30) * 4 + lo;
If we could use 64bit integers, converting 2 32bit values to a single 64bit value would look like this:
unsigned long long result = hi; // put hi into the 64bit number.
result <<= 32; // shift the 32 bits to the upper part of the number
results |= low; // add in the lower 32bits.
If you aren't used to bit shifting, maybe looking at it like this will help. If lo = 1 and high = 2, then expressed as hex numbers:
result = hi; 0x0000000000000002
result <<= 32; 0x0000000200000000
result |= low; 0x0000000200000001
But if we assume the compiler doesn't support 64bit integers, that won't work. While floating point numbers can hold values that big, they don't support shifting. So we need to figure out a way to shift 'hi' left by 32bits, without using left shift.
Ok then, shifting left by 1 is really the same as multiplying by 2. Shifting left by 2 is the same as multiplying by 4. Shifting left by [omitted...] Shifting left by 32 is the same as multiplying by 4,294,967,296.
By an amazing coincidence, 4,294,967,296 == (1 << 30) * 4.
So why write it in that complicated fashion? Well, 4,294,967,296 is a pretty big number. In fact, it's too big to fit in an 32bit integer. Which means if we put it in our source code, a compiler that doesn't support 64bit integers may have trouble figuring out how to process it. Written like this, the compiler can generate whatever floating point instructions it might need to work on that really big number.
Why the current code is wrong:
It looks like variations of this code have been wandering around the internet for a long time. Originally (I assume) access_counter was written using rdtsc instead of rdtscp. I'm not going to try to describe the difference between the two (google them), other than to point out that rdtsc does not set ecx, and rdtscp does. Whoever changed rdtsc to rdtscp apparently didn't know that, and failed to adjust the inline assembler stuff to reflect it. While your code might work fine despite this, it might do something weird instead. To fix it, you could do:
asm("rdtscp; movl %%edx,%0; movl %%eax,%1" // Read cycle counter
: "=r" (*hi), "=r" (*lo) // and move results to
: /* No input */ // the two outputs
: "%edx", "%eax", "%ecx");
While this will work, it isn't optimal. Registers are a valuable and scarce resource on i386. This tiny fragment uses 5 of them. With a slight modification:
asm("rdtscp" // Read cycle counter
: "=d" (*hi), "=a" (*lo)
: /* No input */
: "%ecx");
Now we have 2 fewer assembly statements, and we only use 3 registers.
But even that isn't the best we can do. In the (presumably long) time since this code was written, gcc has added both support for 64bit integers and a function to read the tsc, so you don't need to use asm at all:
unsigned int a;
unsigned long long result;
result = __builtin_ia32_rdtscp(&a);
'a' is the (useless?) value that was being returned in ecx. The function call requires it, but we can just ignore the returned value.
So, instead of doing something like this (which I assume your existing code does):
unsigned cyc_hi, cyc_lo;
access_counter(&cyc_hi, &cyc_lo);
// do something
double elapsed_time = get_counter(); // Find the difference between cyc_hi, cyc_lo and the current time
We can do:
unsigned int a;
unsigned long long before, after;
before = __builtin_ia32_rdtscp(&a);
// do something
after = __builtin_ia32_rdtscp(&a);
unsigned long long elapsed_time = after - before;
This is shorter, doesn't use hard-to-understand assembler, is easier to read, maintain and produces the best possible code.
But it does require a relatively recent version of gcc.
I need to read 8-byte integers from a stream. I could not find any documentation how to read 8-byte integers in DM. It would be something similar to a long long integer.
Is there a trick how to stream 8-byte integers from file in GMS 2.x ?
We can use the "Stream" object to read/import data of various kinds. Please refer to the DM Help > Scripting > File Input and Output:
Other examples can also be found at DM-Script-Database :
Read-Ser (http://donation.tugraz.at/dm/source_codes/127)
JEMS_.ems file reader (http://donation.tugraz.at/dm/source_codes/108)
Hope this helps.
I used the following (stupid) method to do so:
number readint32(object s){
number stream_byte_order=2
number result=0
TagGroup tg = NewTagGroup();
tg.TagGroupSetTagAsLong( "SInt32_0", 0 )
TagGroupReadTagDataFromStream( tg, "SInt32_0", s, stream_byte_order );
tg.TagGroupGetTagAsLong( "SInt32_0", result)
return result
}
number readint64(object s){
//new for reading 8-byte integer in TIA ver >3.7
//DM automatic convert result to float when the second 4-byte >1
number result = readint32(s)+ (readint32(s)*4294967296)
// 4294967296 equals to 0xFFFFFFFF in hex form
return result
}
It works with reading ser <2GB, but does not for larger file. I still did not figure it out...
#09-04-2016
Now i got a solution to the data offset problem in ser:
Here is the solution:
Void b_readint64(object s, number &lo, number &hi){
//new for reading 8-byte (64bit) integer in TIA ver >3.7
//read the low and high section individually and later work
//together with StreamSetPos32singed, StreamSetPos64 funcsions
lo = b_readint32(s)
hi = b_readint32(s)
}
Void StreamSetPos32Signed(object s, number base, number lo){
if (lo>0) StreamSetPos(s, base, lo)
else StreamSetPos(s, base, 4294967296+lo)
}
Void StreamSetPos64(object s, number base, number lo, number hi){
if (hi!=0){
StreamSetPos(s, base, 0)
for (number i=0; i<hi; i++) StreamSetPos(s, 1, 4294967296)
StreamSetPos32Signed(s, 1, lo)
} else StreamSetPos32signed(s, base, lo)
}
BTW, I just uploaded this upgraded script to
http://portal.tugraz.at/portal/page/portal/felmi/DM-Script/DM-Script-Database
There is nothing like an 8-byte integer in DigitalMicrograph. You can use the streaming to read in two successive 4-byte sections as integers (See answer above) and then display them as binary using binary() or hexadecimal using hex(), but you will have to do the maths yourself for the "meaning" of the 8-byte integer (storing it as real-number). You can use the binary operators & | ^ for bitwise numeric, when needed.
I want to know, if there an easy way to get the total length in bytes of the NSStream object. So, for example, in C# I can get a Stream.Length property, and that'll be the answer. In objective-c, so far, I haven't found anything like that. The simplest solution, I could imagine would be "read bytes to buffer and count their number":
long totalLength = 0;
while((result = [sInput read:buffer maxLength:BUFFER_SIZE]) != 0) {
if(result > 0) {
totalLength += result;
}
As stated in docs return value for read method is:
A positive number indicates the number of bytes read;
0 indicates that the end of the buffer was reached;
A negative number means that the operation failed.
finally the size values would contain the total length.
Is this a correct way to solve the issue, or is there a simpler way? Btw, is my code correct? (I'm not confident in my obj-c skills yet)
This question may already have been asked but nothing on SO actually gave me the answer I need.
I am trying to reverse engineer someone else's vb.NET code and I am stuck with what a Xor is doing here. Here is 1 line of the body of a soap request that gets parsed (some values have been obscured so the checksum may not work in this case):
<HD>CHANGEDTHIS01,W-A,0,7753.2018E,1122.6674N, 0.00,1,CID_V_01*3B</HD>
and this is the snippet of vb code that checks it
LastStar = strValues(CheckLoop).IndexOf("*")
StrLen = strValues(CheckLoop).Length
TransCheckSum = Val("&h" + strValues(CheckLoop).Substring(LastStar + 1, (StrLen - (LastStar + 1))))
CheckSum = 0
For CheckString = 0 To LastStar - 1
CheckSum = CheckSum Xor Asc(strValues(CheckLoop)(CheckString))
Next '
If CheckSum <> TransCheckSum Then
'error with the checksum
...
OK, I get it up to the For loop. I just need an explanation of what the Xor is doing and how that is used for the checksum.
Thanks.
PS: As a bonus, if anyone can provide a c# translation I would be most grateful.
Using Xor is a simple algorithm to calculate a checksum. The idea is the same as when calculating a parity bit, but there is eight bits calculated across the bytes. More advanced algorithms like CRC and MD5 are often used to calculate checksums for more demanding applications.
The C# code would look like this:
string value = strValues[checkLoop];
int lastStar = value.IndexOf("*");
int transCheckSum = Convert.ToByte(value.Substring(lastStar + 1, 2), 16);
int checkSum = 0;
for (int checkString = 4; checkString < lastStar; checkString++) {
checkSum ^= (int)value[checkString];
}
if (checkSum != transCheckSum) {
// error with the checksum
}
I made some adjustments to the code to accomodate the transformation to C#, and some things that makes sense. I declared the variables used, and used camel case rather than Pascal case for local variables. I use a local variable for the string, instead of getting it from the collection each time.
The VB Val method stops parsing when it finds a character that it doesn't recognise, so to use the framework methods I assumed that the length of the checksum is two characters, so that it can parse the string "3B" rather than "3B</HD>".
The loop starts at the fourth character, to skip the first "<HD>", which should logically not be part of the data that the checksum should be calculated for.
In C# you don't need the Asc function to get the character code, you can just cast the char to an int.
The code is basically getting the character values and doing a Xor in order to check the integrity, you have a very nice explanation of the operation in this page, in the Parity Check section : http://www.cs.umd.edu/class/sum2003/cmsc311/Notes/BitOp/xor.html