I'm attempting to decode a manchester-encoded packet using GNU Radio Companion. I've been following this example where the author decodes packets from a Somfy window blinds remote. From what I've read in that article and this mailing list, the Viterbi Combo block is the way to perform manchester decoding.
The Viterbi Combo block requires the block size (frame size) beforehand. This isn't a problem in the case of the first article because the frame size is fixed and known beforehand. In my case, however, the frame size is variable and conveyed in the first octet of the header. i.e.
[preamble][sync][header][data][crc]
There are several blocks for extracting length information out of a stream, but these assume that the manchester decoding has already been done. Is there a way to do manchester decoding without knowing the block size? Is this a case where I need to make my own custom block?
Related
Last time mkl helped me a lot, hopefully he (or someone else) can help me with these questions too. Unfortunately I couldn't get access to the ISO norm (ISO 32000-1 or 32000-2).
Are these bytes used for padding? I have tried several files, and they all have padding characters. This is quite remarkable, as I would expect that this substantial amount of low entropy bytes should significantly lower the average entropy of the PDF file. However, this does not seem to be the case, as the average entropy of a PDF file is almost eight bits)
Furthermore, this (meta)data should be part of an object stream, and therefore compressed, but this is not the case (Is there a specific reason for this)..?) (magenta = high entropy/random, how darker the color, how lower the entropy, In generated this image with http://binvis.io/#/)
These are the entropy values of a .doc file (**not **docx), that I converted to a PDF with version 1.4, as this version should not contain object streams etc. However, the entropy values of this file are still quite high. I would think that the entropy of a PDF with version <1.5 would have a lower entropy value on average, as it does not use object streams, but the results are similar to a PDF with version 1.5
I hope somebody can help me with these questions. Thank you.
Added part:
The trailer dictionary has a variable length, and with PDF 1.5 (or higher) it can be part of the central directory stream, not only the length but also the position/offset of the trailer dictionary can vary (although is it.. because it seems that even if the trailer dictionary is part of the central directory stream, it is always at the end of the file?, at least... in all the PDFs I tested). The only thing I don't really understand is that for some reason the researchers of this study assumed that the trailer has a fixed size and a fixed position (the last 164 bytes of a file).
They also mention in Figure 8 that a PDF file encrypted by EasyCrypt, has some structure in both the header and the trailer (which is why it has a lower entropy value compared to a PDF file encrypted with ransomware).
However, when I encrypt a file with EasyCrypt (I tried three different symmetric encryption algorithms: AES 128 bit, AES 256 bit and RC2) and encrypt several PDF files (with different versions), I get a fully encrypted file, without any structure/metadata that is not encrypted (neither in the header nor in the trailer). However, when I encrypt a file with Adobe Acrobat Pro, the structure of the PDF file is preserved. This makes sense, since the PDF extension has its own standardised format for encrypting files, but I don't really understand why they mention that EasyCrypt conforms to this standardised format.
PDF Header encrypted with EasyCrypt:
PDF Header encrypted with Adobe Acrobat Pro:
Unfortunately I couldn't get access to the ISO norm (ISO 32000-1 or 32000-2).
https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf
Are these bytes used for padding?
Those bytes are part of a metadata stream. The format of the metadata is XMP. According to the XMP spec:
Padding It is recommended that applications allocate 2 KB to 4 KB of padding to the packet. This allows the XMP to be edited in place, and expanded if necessary, without overwriting existing application data. The padding must be XML-compatible whitespace; the recommended practice is to use the space character (U+0020) in the appropriate encoding, with a newline about every 100 characters.
So yes, these bytes are used for padding.
Furthermore, this (meta)data should be part of an object stream, and therefore compressed, but this is not the case (Is there a specific reason for this)..?)
Indeed, there is. The pdf document-wide metadata streams are intended to be readable by applications, too, that don't know the PDF format but do know the XMP format. Thus, these streams should not be compressed or encrypted.
...
I don't see a question in that item.
Added part
the position/offset of the trailer dictionary can vary (although is it.. because it seems that even if the trailer dictionary is part of the central directory stream, it is always at the end of the file?, at least... in all the PDFs I tested)
Well, as the stream in question contains cross reference information for the objects in the PDF, it usually is only finished pretty late in the process of creating the PDF an, therefore, added pretty late to the PDF file. Thus, an end-ish position of it usually is to be expected.
The only thing I don't really understand is that for some reason the researchers of this study assumed that the trailer has a fixed size and a fixed position (the last 164 bytes of a file).
As already discussed, assuming a fixed position or length of the trailer in general is wrong.
If you wonder why they assumed such a fixed size nonetheless, you should ask them.
If I were to guess why they did, I'd assume that their set of 200 PDFs simply was not generic. In the paper they don't mention how they selected those PDFs, so maybe they used a batch they had at their hands without checking how special or how generic it was. If those files were generated by the same PDF creator, chances indeed are that the trailers have a constant (or near constant) length.
If this assumption is correct, i.e. if they worked with a not-generic set of test files only, then their results, in particular their entropy values and confidence intervals and the concluded quality of the approach, are questionable.
They also mention in Figure 8 that a PDF file encrypted by EasyCrypt, has some structure in both the header and the trailer (which is why it has a lower entropy value compared to a PDF file encrypted with ransomware).
However, when I encrypt a file with EasyCrypt (I tried three different symmetric encryption algorithms: AES 128 bit, AES 256 bit and RC2) and encrypt several PDF files (with different versions), I get a fully encrypted file, without any structure/metadata that is not encrypted (neither in the header nor in the trailer).
In the paper they show a hex dump of their file encrypted by EasyCrypt:
Here there is some metadata (albeit not PDF specific) that should show less entropy.
As your EasyCrypt encryption results differ, there appear to be different modes of using EasyCrypt, some of which add this header and some don't. Or maybe EasyCrypt used to add such headers but doesn't anymore.
Either way, this again indicates that the research behind the paper is not generic enough, taking just the output of one encryption tool in one mode (or in one version) as representative example for data encrypted by non-ransomware.
Thus, the results of the article are of very questionable quality.
the PDF extension has its own standardised format for encrypting files, but I don't really understand why they mention that EasyCrypt conforms to this standardised format.
If I haven't missed anything, they merely mention that A constant regularity exists in the header portion of the normally encrypted files, they don't say that this constant regularity does conform to this standardised format.
I'm using a HackRF One device and its corresponding osmocom Sink block inside of gnuradio-companion. Because the input to this block is Complex (i.e. a pair of Floats), I could conceivably send it an enormously large value. At some point the osmocom Sink will hit a maximum value and stop driving the attached HackRF to output stronger signals.
I'm trying to figure out what that maximum value is.
I've looked through the documentation at a number of different sites, for both the HackRF One and the osmocom source and can't find an answer. I tried looking through the source code itself, but couldn't see any clear indication there, although I may have missed something there.
http://sdr.osmocom.org/trac/wiki/GrOsmoSDR
https://github.com/osmocom/gr-osmosdr
I also thought of deriving the value empirically, but didn't trust my equipment to get a precise measure of when the block started to hit the rails.
Any ideas?
Thanks
Friedman
I'm using a HackRF One device and its corresponding osmocom Sink block inside of gnuradio-companion. Because the input to this block is Complex (i.e. a pair of Floats), I could conceivably send it an enormously large value.
No, the complexes z must meet
because the osmocom sink/the underlying drivers and devices map that -1 – +1 range to the range of the I and Q DAC values.
You're right, though, it's hard to measure empirically, because typically, the output amplifiers go into nonlinearity close to the maximum DAC outputs, and on top of that, everything is frequency-dependent, so e.g. 0.5+j0.5 at 400 MHz doesn't necessarily produce the same electrical field strength as 0.5+j0.5 at 1GHz.
That's true for all non-calibrated SDR devices (which, aside from your typical multi-10k-Dollar Signal Generator, is everything, unless you calibrate for all frequencies of interest yourself).
I'm doing a study on HEVC and very new to video compression. The first thing i want to do is see the effect of packet loss on the decoded video. I want to modify the decoder so that i can introduce packet loss on the hevc bitstream. I'll be using error patterns generated by NS2. What part of the decoder should i focus on? How can i insert the error patterns to the hevc bitstream? What specific variables determine the frame and slice number? I'm using HM 16.6. Thanks
I once developed a Python tool that hacks into the bitstream and flip bits around. What I did is that I read the bitstream file generated by the encoder linearly, and randomized the bit flipping process. Because I know the structure of the NAL units, from the standard specs, I could tell where my corrupted bit is. The best part to start manipulating is the NAL unit headers. The video, sequence, slice headers. You can tell where they are based on the packetization process parameters. It has been a long time, so I forgot the details. Headers do not tell everything about the bitstream especially the frame, slice number. It could be lice-based or tile-based or I forgot what else. But that you can tell from the headers. Now the decoder will know the frame, slice number as it decodes and follow the encoder's guidelines. So what you will need to know will be in the headers. Check the latest working draft and study the header information.
I am posting this as an answer rather than a comment since I dont have reputations to comment.
In HEVC a slice is a group of consecutive CTU (Coding Tree Unit)s within a frame. Furthermore, HEVC introduces splitting of slices into so-called slice segments, mainly for low-delay appliances . When a slice is split into multiple slice segments, only the first slice segment carries the header information required to decode the whole slice. The remaining dependant slice segments refer to the initial slice segment during the decoding. Each slice segment (or slice if no segmentation is used) maps to a single data unit called a ‘NAL (Network Abstraction Layer) unit’. A NAL unit is the video packet payload in the physical channel, and therefore a packet loss during the transmission corresponds to a NAL unit loss and consequently a slice loss.
If you study the HEVC standard you will encounter another type of data unit called an Access unit. An access unit is a collection of NAL units(hence a collection of slices). During the transmission delimiters are placed between NAL units and also between Access units in order to separately identify each type. In a HEVC coded stream an access unit delimiter is defined as x00 00 00 01 whereas a NAL unit delimiter is defined as x00 00 01. So basically if you want to introduce a packet loss in a given frame, first you need to identify the correct frame by counting the access unit delimiters in the bitstream. Later delete all the bits between the desired NAL unit delimiters.
I've been looking for an answer everywhere and I was only able to find some bits and pieces. What I want to do is to load multiple mp3 files (kind of temporarily merge them) and then cut them into pieces using silence detection.
My understanding is that I can use Mp3FileReader for this but the questions are:
1. How do I read say 20 seconds of audio from an mp3 file? Do I need to read 20 times reader.WaveFormat.AverageBytesPerSecond? Or maybe keep on reading frames until the sum of Mp3Frame.SampleCount / Mp3Frame.SampleRate exceeds 20 seconds?
2. How do I actually detect the silence? I would look at an appropriate number of the consecutive samples to check if they are all below some threshold. But how do I access the samples regardless of them being 8 or 16bit, mono or stereo etc.? Can I directly decode an MP3 frame?
3. After I have detected silence at say sample 10465, how do I map it back to the mp3 frame index to perform the cutting without re-encoding?
Here's the approach I'd recommend (which does involve re-encoding)
Use AudioFileReader to get your MP3 as floating point samples directly in the Read method
Find an open source noise gate algorithm, port it to C#, and use that to detect silence (i.e. when noise gate is closed, you have silence. You'll want to tweak threshold and attack/release times)
Create a derived ISampleProvider that uses the noise gate, and in its Read method, does not return samples that are in silence
Either: Pass the output into WaveFileWriter to create a WAV File and and encode the WAV file to MP3
Or: use NAudio.Lame to encode directly without a WAV step. You'll probably need to go from SampleProvider back down to 16 bit WAV provider first
BEFORE READING BELOW: Mark's answer is far easier to implement, and you'll almost certainly be happy with the results. This answer is for those who are willing to spend an inordinate amount of time on it.
So with that said, cutting an MP3 file based on silence without re-encoding or full decoding is actually possible... Basically, you can look at each frame's side info and each granule's gain & huffman data to "estimate" the silence.
Find the silence
Copy all the frames from before the silence to a new file
now it gets tricky...
Pull the audio data from the frames after the silence, keeping track of which frame header goes with what audio data.
Start writing the second new file, but as you write out the frames, update the main_data_begin field so the bit reservoir is in sync with where the audio data really is.
MP3 is a compressed audio format. You can't just cut bits out and expect the remainder to still be a valid MP3 file. In fact, since it's a DCT-based transform, the bits are in the frequency domain instead of the time domain. There simply are no bits for sample 10465. There's a frame which contains sample 10465, and there's a set of bits describing all frequencies in that frame.
Plain cutting the audio at sample 10465 and continuing with some random other sample probably causes a discontinuity, which means the number of frequencies present in the resulting frame skyrockets. So that definitely means a full recode. The better way is to smooth the transition, but that's not a trivial operation. And the result is of course slightly different than the input, so it still means a recode.
I don't understand why you'd want to read 20 seconds of audio anyway. Where's that number coming from? You usually want to read everything.
Sound is a wave; it's entirely expected that it crosses zero. So being close to zero isn't special. For a 20 Hz wave (threshold of hearing), zero crossings happen 40 times per second, but each time you'll have multiple samples near zero. So you basically need multiple samples that are all close to zero, but on both sides. 5 6 7 isn't much for 16 bits sounds, but it might very well be part of a wave that will have a maximum at 10000. You really should check for at least 0.05 seconds to catch those 20 Hz sounds.
Since you detected silence in a 50 millisecond interval, you have a "position" that's approximately several hundred samples wide. With any bit of luck, there's a frame boundary in there. Cut there. Else it's time for reencoding.
I'm currently using AES (256) with CBC mode to encrypt data. I store the initialization vector with the encrypted data. Right now I'm just adding the IV to the beggining of the encrypted data, then on decrypt, reading it in as a hard coded length of bytes.
If the initialization vector length changes in the future, this method will break.
So my questions is:
Will longer AES key sizes in the future = longer IVs? Or, in other words, will the block size of AES change in the future?
If so, what would be the best way of dealing with this? Using the first byte as an indicator of how long the IV is, then reading in that many bytes?
Rijndael does support larger block sizes, but AES is currently fixed at a 128 bit block. It seems relatively unlikely that the larger Rijndael block sizes will be standardized by NIST, since this would effectively be a completely new algorithm, one that hasn't been implemented by anyone. If NIST feels the need for block cipher with a larger size, it's likely the would simply run a new contest.
However what I would recommend is that, rather than the IV length, you include near the start of your message some kind of algorithm identifier (a single byte is all you'll need), which will allow you not just the flexibility to handle larger IVs, but also extend your format in other ways in the future, for instance a new algorithm. Eg 0 == AES-256/CBC, 1 == AES-256/GCM, 2=AES-2.0/CBC, 3=AES-256/CBC with special extra header somewhere, etc, etc.
PS - don't forget to also use a message authentication code, since otherwise you expose yourself to a variety of easy message modification attacks.
The purpose of the initialization vector is to randomize the first block, so that the same data encrypted twice with the same key will not produce the same output.
From an information-theoretic point of view, there are "only" 2^128 distinct IVs for AES, because those are all the possible random values you might XOR with your first block of actual data. So there is never any reason to have an IV larger than the cipher's block size.
Larger block sizes could justify larger IVs. Larger key sizes do not.
A larger block size would mean a different algorithm by definition. So however you tag your data to indicate what algorithm you are using, that is how you will tell what block size (and therefore IV size) to use.
As an alternative solution you could switch to AES-CTR mode. Counter mode requires a Nonce, but the Nonce does not have to be tied to the AES block size. If the AES block size were increased (unlikely, as Jack says) then you could retain the same size Nonce.