I'm trying to parse some whatsapp data serialized but during the process the is some C2 and C3 added.
For example the string :
F0-9F-8F-B3-E2-80-8D-F0-9F-8C-88
Is transformed in :
C3-B0-C2-9F-C2-8F-C2-B3-C3-A2-C2-80-C2-8D-C3-B0-C2-9F-C2-8C-C2-88
We can see also that the F0 -> B0 and the E2 -> A2.
These tranformation only happend with the C3 bytes
Moreover the C2 and C3 bytes are not counted on the size.
I really don't know what these bytes are representing, does anyone has a clue ?
The orginal message part look like this :
12-0D-0A-0B-C3-B0-C2-9F-C2-8F-C2-B3-C3-A2-C2-80-C2-8D-C3-B0-C2-9F-C2-8C-C2-88
Edit
The data are comming from whatsapp-web web-socket, they are decrypted and then parsed with protobuf. The main issue is that the datas comming from whatsapp-web are not proper protobuf datas (I was trying to parse them with this website https://protogen.marcgravell.com/decode). But if the C2 are remove and in general the upper modifications are applied the message deserialize correctly. One more thing that I noticed is that it only apply for long messages and messages with smiley.
This is a whole message (in hex and different from the upper one):
0A-40-0A-1A-30-30-30-30-30-30-30-30-30-30-30-40-73-2E-77-68-61-74-73-61-70-70-2E-6E-65-74-10-00-1A-20-20-21-22-23-24-25-26-27-28-29-30-31-32-33-34-35-36-37-38-39-40-41-42-43-44-45-46-47-48-49-50-51-12-0D-0A-0B-C3-B0-C2-9F-C2-8F-C2-B3-C3-A2-C2-80-C2-8D-C3-B0-C2-9F-C2-8C-C2-88-18-C3-A3-C2-A0-C3-9B-C3-B6-05-20-00
Related
For a C based embedded hardware module, the configuration structure is made up of several fields in a particular layout, e.g. take this 8-byte struct for example:
Offset
Datatype
Field
0
UInt8
fieldA
1
UInt16
some_value
3
UInt32
another_value
7
UInt8
aByte
This 8-byte config can be read and written via NFC communication. So there is an Android app which reads this data (as a sequence of 8 bytes) and can write it back, so that the firmware on the embedded hardware module (written in C) can "understand" it.
The task is now to decode the 8-byte sequence of e.g. 12 ab cd 04 fe ff 56 77 (little-endian!) into the decoded values:
Field
Bytes to be decoded
Decoded, human-readable number
fieldA
12
0x12
some_value
ab cd
0xCDAB
another_value
04 fe ff 56
0x56FFFE04
aByte
77
0x77
Note that this is a Kotlin or Java question, no C question ;)
Now my question is about finding a Kotlin way to decode such a binary struct into the respective values (as shown above), so that the values can be presented to the app user. And, encode the values (after the user would have edited some values) back into the binary structure of 8 bytes.
Note that endianness is also an issue. In general, the target system is a little-endianed ARM and the Android app also runs on a little-endian system, so there might be no issue. However, this is by coincidence, and I would like to make this explicit.
What could be a Kotlin way of decoding/encoding numbers into such bytes, using explicit endian conversion of necessary?
If it was for Python, the struct library with its pack and unpack function are PERFECT for such tasks. But how to do this in Kotlin? I would love to see exactly such functions ...
The best I can think of is to just wrap the byte array in a ByteBuffer and read it one by one.
Suppose you have:
data class SomeStructure(
val fieldA: UByte,
val someValue: UShort,
val anotherValue: UInt,
val aByte: UByte,
)
You can do:
val byteArray: ByteArray = ....
val buffer = ByteBuffer.wrap(byteArray).order(ByteOrder.LITTLE_ENDIAN)
val someStruct = SomeStructure(
buffer.get().toUByte(),
buffer.getShort().toUShort(),
buffer.getInt().toUInt(),
buffer.get().toUByte(),
)
println(someStruct)
Note that the get and getXXX methods of ByteBuffer advances the "reading" position, hence mutating the ByteBuffer, so if you want to re-read the buffer again for whatever reason after creating the SomeStructure, you should flip it, or just create a new byte buffer.
You could also make this a secondary constructor of SomeStructure.
constructor(buffer: ByteBuffer): this(
buffer.get().toUByte(),
buffer.getShort().toUShort(),
buffer.getInt().toUInt(),
buffer.get().toUByte(),
)
This way, you could even support reading data classes with references to SomeStructures from byte buffers:
data class SomeOtherStructure(
val struct1: SomeStructure,
val struct2: SomeStructure,
) {
constructor(buffer: ByteBuffer): this(
SomeStructure(buffer), // recall that this advances the current position of the buffer
SomeStructure(buffer)
)
}
I want to store the data received from API and store it as a value in Redis.The response would be in different languages
eg:
I want to store the response ದಿನಾಂಕ .But whenever i try to access the value throug key the response would be something like "\xe0\xb2\xa6\xe0\xb2\xbf\xe0\xb2\xa8\xe0\xb2\xbe\xe0\xb2\x82\xe0\xb2\x95" .
Please help if possible :)
That's simply the array of bytes (hexadecimal representation) of your string (UTF-8 representation).
For example, input e0 b2 a6 e0 b2 bf e0 b2 a8 e0 b2 be e0 b2 82 e0 b2 95 in https://onlineutf8tools.com/convert-bytes-to-utf8.
All you should do is use a function (depending on your programming language or Redis client) to convert an array of bytes into its UTF-8 string representation.
See Python-redis keys() returns list of bytes objects instead of strings.
Simple question:
Lets say i have got 2 cells (A1 and A2). on A1 i have a value of 16000000 and on cell A2 a value of 230000.
both values are in bytes. what i need is to be able to convert them to MB and kB respectively with VBA code.
i can start dividing A1/1024/1024 and A2/1024 but what i would like to know is how do i make the vba code diferentiate between the 2 values and then convert them to their respective category.
i could convert both of them to MB by dividing by 1024 but that would leave me with cell A2 with a value of 0.23MB.
i am not asking for code to be written for me, just for an explanation on how to do this if possible.
If you can handle using 10³ (aka 1,000) as a Kb instead of 2¹⁰ (aka 1,024) and 10⁶ (aka 1,000,000) as a Mb instead of 2²⁰ (aka 1,048,576) then a Custom Number Format Code will preserve the raw underlying value while displaying the unit of size.
range("B2:B99").NumberFormat = "[Color10][>999999]0,, \M\b;[Color13][>999]0, K\b;[Color3]0 \B;"
I have also added blue for Mb, green for Kb and red for B.
I am trying to decipher a trace of USB I/O traffic produced by usbmon and am having some issues getting my head around the endianness. For the sake of example, here are two lines from the trace I am working with:
ffff8800650e7000 433121059 S Ci:2:000:0 s 80 06 0100 0000 0040 64 <
ffff8800650e7000 433121661 C Ci:2:000:0 0 18 = 12010002 00000040 da0b8781 00010102 0301
I initially had no suspicion whatsoever of anything other than big-endianness in the trace, but then I saw da0b8781 in the second line, which corresponds to the identity of the USB device I am tracing which has a vendor ID of 0x0bda and product ID of 0x8187 (note the reversal of byte-order in the trace).
So at this point I thought that maybe within a given field of a usbmon trace, the bytes were always in reverse byte order and should be interpreted as such. But to the contrary, let's examine a small part near the end of the first trace line, ... 0040 64
0040 is a hex field representing the maximum accepted response size. 64 is a decimal field that should represent exactly the same thing. 0x0040 = 64 decimal, without switching the byte order to 0x4000, which would then != 64 decimal. So it's at this point I started to get a bit uncertain about the byte-order of the different parts of the usbmon trace.
Next I thought, maybe it's just the data portions of the usbmon trace that are in reverse byte order. So I thought perhaps I should really be reading
...12010002 00000040 da0b8781 00010102 0301
as
1030 20101000 1878b0ad 04000000 20001021...
Nope, that doesn't seem to be right either. The USB Specification states that the vendor Id (0x0bda in my case) should be at byte offset 8 for this particular string. If we leave the above string in its original order, then the vendor Id does start at byte offset 8 (12010002 00000040 consumes the first 8 bytes), but if we reverse it as I have above, then it starts at byte offset 6 (1030 20101000 only consumes the first 6 bytes).
So my best guess now is that usbmon displays everything big-endian, accept that it switches to reverse byte order within each 4-byte word, but for data only. Can anyone offer some clarification on whether this is correct, or whether there may be something else I'm missing?
May be a bit late for you but I've tried usbmon (and found it OK)
you may want to take a look at evtest
http://www.freedesktop.org/wiki/Evtest
I am doing a sms project using VB .net to connect to my gsm modem via serialport. All the connection and everything was working great and I can send and receive sms.
Then here come a problem, I need to send long sms more than 160 chars. I read and did some research and I know that I need to use the pdu and udh. I managed to understand slightly of the pdu packet but is still quite confuse on that part.
Anyone can briefly explain the exact way to do the concatented sms (long sms)?
Another problem is how can I use vb to encode the data into pdu gsm 7bit data? Is there any built in functions or do I need to write them myself?
This may help out with the concatenated SMS and GSM encoding, but I've never used a GSM modem so hopefully somebody else can fill in the gaps.
To convert your .NET string to a GSM encoded string, I've written an open-source library available on Github that might help you. However it doesn't do the 7 bit packing you might need.
For the long message support you need to add a User Data Header at the start of each message.
First split the message in to multiple parts each 153 GSM (7bit) bytes long. Don't forget that extended characters such as { take two bytes, see the GSM character set for a full list.
Then add the UDH bytes to the beginning that allow concatenation. I've put an example below for a two part message.
First SMS: 05 00 03 A6 02 01 .. bytes that make up the first part text..
Second SMS: 05 00 03 A6 02 02 .. bytes that make up the second part text..
Bytes Description
05 (UDHL) Length of the UDH. So the following five bytes are UDH
00 (IEI) This identifier says this is a concatenated message
03 (IEDL) It says that next 3 bytes are the data for this IE.
A6 Reference number of this concatenated message.
02 There are two parts to this concatenated message.
01or02 This indicates whether the SMS is the first or second part.
UHDL = User Data Header Length
IEI = Information Element Identifier (00 is a concatenated message)
IEDL = Information Element Data Length (A concatenated message header length is always 3)
For the reference number, the easiest way is just to assign an random number in the range 0-255, each part of the message must have the same reference number.