I'm trying to find an alternative to QR codes (I'd also be willing to accept an entirely novel solution and implement it myself) that meets certain specifications.
First, the codes will often end up on thin pipes, and so need to be readable around a cylinder. The advantage to this is that the effect on the image from wrapping it around a cylinder is easy to express geometrically, and the codes will never be placed on a very irregular shape.
Second, read accuracy must be very high, as any read mistake would be extremely costly. If this means larger codes with more redundancy for better error correction, so be it.
Third, ability to be read by the average smartphone camera from a few inches out.
Fourth, storage space of around half a kilobyte per code.
Do you know of such a code?
The Data Matrix Rectangular Extension (DMRE) improves upon the standard set of rectangular Data Matrix symbol sizes in an algorithmically compatible manner, thus increasing the range of suitable applications with no real downsides.
Reliable cylindrical marking is a primary use case.
Regardless of symbology you will be unable to approach sufficient data density to achieve 0.5KB of binary data in a single compact, narrow symbol scanned using a standard camera phone. However, most 2D symbologies (DMRE included) support a feature called Structured Append that allows chaining of multiple symbols that can be scanned in any order to produce a single read when all components are accounted for.
If the data to be encoded is known to be highly structured (e.g. mostly numeric or alphanumeric) then the internal encoding process of Data Matrix will be more optimised than for general binary data. For example, the largest DMRE symbol (26×64) will provide up to 236 numeric characters, ~175 alphanumeric characters and only 116 bytes.
If the default error recovery rate is insufficient then including a checksum in the data may be appropriate.
DMRE has just been voted to be accepted as an ISO/IEC project and will likely become an international standard enjoying broad hardware and software support in due course.
Another option may be to investigate PDF417 which has a broader range of symbols sizes, however the data density is somewhat less than Data Matrix.
DMRE references: AIM specification and explanatory notes.
Related
I am reading a book on advanced iOS programming which says "It's generally much faster to work on planar formats than interleaved formats. If at all possible, get your data into planar format as soon as you can , and leave it in the at format through the entire transformation process".
I got to wondering why that is. The only reason I could come up with was that as you are iterating over your pixels, that you can use ++ operator instead of things like w*y+x to calculate your offsets.
I am familiar with planar vs interleaved as well as RGBA vs YUV. I've written a lot of c code to do things like image format conversion, rotations, flips, resizing, etc... The conversions were typically either to or from i420.
Is there something I'm overlooking?
Sweeping generalizations like the one you quote are often of limited practical applicability (i.e. almost, but not quite, entirely generalized from one or two data points ;-)
You say you have written a lot of C code for image processing. Good, take it to the next step: learn how to use a profiler, write microbenchmarks to unit-test the performance of your algorithms, especially when you are exploring the solution space.
I'm looking for an efficient way to represent a chess position.
my "efficiency" criteria are these:
requires as little memory as possible
given a position representation and a legal move (represented in the usual way, e.g. as in pgn format) it is easy to compute the resulting poisition representation. that is, the function new_position = compute_position(old_position, move) can be (or is already) implemented in a very efficient way (from a run-time perspective).
it is easy to compare two positions to see if they are identical
The reason I try to deviate from standard representations is that my requirements are slightly different. Specifically, I am not trying to develop a chess engine, so move generation and related issues are not required. I only need to follow some existing games, and represent certain positions, and store them in a database.
If there is a software package that that already gives this functionality, that would be great. if you have ideas on how to do it, I'd be happy to develop it :)
Thanks...
I'll offer some possible solutions that use "as little memory as possible" as per your specification. I know that you can make some of these more compact, but doing so makes it much more difficult to work with.
64 bits: I don't know what kind of information you need to be able to retrieve from the representation, but a technique that fulfills all your given requirements is Zobrist hashing. It requires very little memory (you can use a 64 bit key and then store 232 positions before a collision is expected to occur). It is very easy and efficient to update incrementally (bitwise XOR operations) and of course easy to compare. However, if you need to display the position or retrieve any information about pieces, this leaves you out of luck.
328 bits: you can use a piece list, with each piece (4 bits) located at a square (6 bits). Since there are 32 pieces, and you need to keep track of castling rights (4 bits), en passant file (3 bits), and side to move (1 bit).
468 bits: bitboard representation. I know you're not developing a chess engine but this is still a very compact representation. You need 7 bitboards (all white, all black, all queens, all rooks, all bishops, all knights, all pawns), keep track of kings separately (12 bits), and all the other miscellaneous information. Yes this would be a bit of work to implement, but this allows you to do many things not possible in other representations, so this is great if you need to analyze positions.
There's a free PGN viewer here: http://chesstempo.com/pgn-viewer.html
Regarding the space requirement, you could compress FEN using a compression algorithm of your choice, without losing the ability to check for equality. Depending how the moves are encoded (g1f3 versus Nf3) you may require a legal move generator for requirement 2).
Can somebody help me, to find a library, or a detailed description of algorithm, that could embed a Digital watermark(invisible watermark, just a kind of steganography) to a jpeg/png file. But the quality of algorithm, should be great. It should be possible to extract this mark after rotation and expansion(if possible) of image.
Mark is just a key 32bytes.
I found a good site, but the algorithm are made for the NetPBM format, that is dead...
I know that there is a LSB method, but it is not stable to the expansion. Are there something better?
Changing metadata, is not suitable, because it is visible changes.
This maybe won't really be an answer, as I don't think it would be easy to give a magical, precise answer on this question.Watermarking is complex, and the best way to do it is by yourself : this will make things more hard for an attacker trying to reverse engineer your code. One could even read your question here, guess what library you used, and attack your system more easily.
Making Steganography resist to expansion in JPEG images is also very hard, because the JPEG compression is reapplied after the expansion. There are in fact a bunch of JPEG steganography algorithms. Which one you should use, depends on what exactly do you require :
Data confidentiality ?
Message presence confidentiality ?
Message coherence after JPEG changes ?
Resistance to "Known Cover" attacks (when attackers try to find the message, based on the steganographic system) ?
Resistance to "Known Message" attacks (when attackers try to find the steganographic system used, based on the message) ?
From what I know, usually, algorithm that resist to JPEG changes (picture recompression) are often really easier to attack, whereas algorithms that run the "encode" stage during the JPEG compression (after the DCT (lossy) transform, and before the Huffmann (non-lossy) transform) are more prone to resist.
Also, one key factor about steganography is scale : if you have only 32bytes of data to encode in a, say, 256*256px image, don't use an algo that can encode 512bytes of data in the same size. Either use a scalable algorithm, either use an algorithm at its efficient scale.
Also, the best way to do good steganography is to know its limitations,and to know how steganalyzers work. Try these tools, so you can understand what attackers will do to your picture.^
Now, I cannot tell you what steganographic system will be the best for you, but I can give you some indications :
jSteg - Quite old, I don't think it will resist to JPEG changes
OutGuess - Quite old too, but one of the best algorithms
F5 (and F3/F4) - More recent, good algorithm, scientifical research behind.
stegHide
I think all of these are LSB based : the encoding is done during the JPEG compression, after the DCT and Quantization. The only non LSB-based steganography system I heard of was mentionned in this research paper, however, I did not read it to the end yet, so I cannot tell if this will meet your needs.
However, I'm not sure there exists a real steganography algorithm resisting to JPEG compression, to JPEG resize and rotation, resisting to visual and statisticals attacks. Or I'm not aware of it.
Sorry for the lack of precise answer, I tried to give you what I know on the subject, as it's always better to be more informed. Sorry also for the lack of proper English, I'm French, nobody's perfect :)
Pistache is right in what he told you regarding the watermarking implementation algorithms. I will try to help you by showing one algorithm for the given requirements.
Before explaining you the algorithms first I guess that the distinction between the JPG and PNG formats should be done.
JPEG is a lossy format, i.e, the images are susceptible to compression that could remove the watermark. When you open an image for manipulation purposes and you save it, upon the writing procedure, a compression is made by using DCT filtering that removes some important components of the image.
On the other hand, PNG format is lossless, and that means that images are not susceptible to such kind of compression when stored after manipulation.
As a matter of fact, JPEG is used as a watermarking scheme attack due to its compressing characteristic that could remove the watermark if an attacker performed the compression.
Now that you know the difference between both formats, I can tell you a suitable algorithm resistant to the attacks that you mentioned.
Regarding methods to embed a watermark message for PNG files you can use the histogram embedding method. The histogram embedding method changes values on the histogram by changing the values of the neighbor bins. For example imagine that you have a PNG image in grayscale.
Therefore, you'll have only one channel for embedding and that means that you have one histogram with 256 bins. By selecting the neighbor bins x and x+1, you change the values of x and x+1 by moving the pixels with the bright x to x+1 or the other way around, so that (x/(x+1))>T for embedding a '1' or ((x+1)/x)>T for embedding a '0'.
You can repeat the same procedure for the whole histogram length and therefore you can embed in the best case up to 128bits. However this payload is less than what you asked. Therefore I suggest you to split the image into parts, for example blocks, and if you split one image into 4 components you'd be able to embed in the best case up to
512 bits which means 64 bytes.
This method although is very, susceptible to filtering and compression if applied straight in the space domain. Therefore, I suggest you to compute before the DWT of the image and use its low-frequency sub-band. This will provide you better transparency and robustness increased for the warping, resizing etc attacks and compression or filtering as well.
There are other approaches such as LPM (Log Polar Maps) but they are very complex to implement and I think for your case this approach would be fine.
I can suggest you two papers, the first is:
Watermarking digital image and video data. A state-of-the-art overview
This paper will give you some basic notions of watermarking and explain more in detail the LSB algorithm. And the second paper is:
Real-Time Compressed- Domain Video Watermarking Resistance to Geometric Distortions
This paper will explain the algorithm that I just explained now.
Cheers,
I do not know if you are considering approaches different to steganography. Instead of storing data hidden in the pixel data you could create a new data block in the JPEG file and store encripted data.
Take a look at the JPEG file structure on Wikipedia
You can create an application specific data block, using the marker 0xFF 0xEn. Doing so, any change in the image pixels do not change the information stored in the image. Moreover, many image editing software respect custom data blocks and will keep them even after image manipulation.
what is the basic idea behind steganography?ie ,how do you get the hidden information?
suppose if it is an image and some text is within that image...how do you get that text?..
Every stenography algorithm is different in that respect. Every algorithm hides the information differently and thus getting the information back is different.
A simple example goes like this - Each pixel of the image is composed of 3 bytes, one for red, green and blue. Most people can't detect a difference of one bit in the color in an image so one option is to use the least significant bit of each color channel for your data. This way you can store 3 bits of information in every pixel with very little effect on the general quality of the image.
To get the information back you'll need to read the first bit of every color channel of every pixel and gather all the bits together.
This is just a very simple and almost trivial way to do stenography. Real stenography algorithms are somewhat more involved. Like in cryptography, there is no way to generically "unhide" all stenography. you need to know which algorithm you're trying to decode.
The very basic idea is that images contain tons of redundant information that your eye cannot see. For instance if you changed the last bit on each pixel there would be no visible change as almost all of the information about the color is the other bits. So you can encode messages using the last bit (the most basic algorithm). The histogram however will be changed and a large message will easily be detectable. As far a decoding the message itself, well, the message itself is probably using public key encryption so you will never know what the actual payload was.
Steganography unlike cryptography is considered broken if Eve (who is eavesdropping and practising steganalysis) knows that there is a message at all. The assumptions are based on that Alice and Bob are being watched and any communication is sign that they are up to something (aka prisoners, restrictive governments, all governments in the future hehe ;-))
And of course the algorithms become much more complex that just flipping the last bits, but encoding data that will not affect the structure of image (and become vulnerable to statistical attacks.) :
I read this book last summer and I thought is was an excellent introduction (it has a lot a psuedocode of the algorithms used)
http://www.amazon.com/Steganography-Digital-Media-Principles-Applications/dp/0521190193
Steganography, coming from greek Steganos(i'm greek :P) is the art of hiding messages. While cryptography is about scrambling a message, steganography is about a person not being able to locate the message.
There are many tools that do this procedure for you. Writing a tool like that can be a complicated procedure i think, though i haven't tried to do that. You would need to create a sophisticated approach of correctly using unused or seemingly not important image pixels or data, in order to add your own message, file etc. For some more information, please take a look at : http://www.symantec.com/connect/articles/steganography-revealed
I need a FAST decompression routine optimized for restricted resource environment like embedded systems on binary (hex data) that has following characteristics:
Data is 8bit (byte) oriented (data bus is 8 bits wide).
Byte values do NOT range uniformly from 0 - 0xFF, but have a poisson distribution (bell curve) in each DataSet.
Dataset is fixed in advanced (to be burnt into Flash) and each set is rarely > 1 - 2MB
Compression can take as much as time required, but decompression of a byte should take 23uS in the worst case scenario with minimal memory footprint as it will be done on a restricted resource environment like an embedded system (3Mhz - 12Mhz core, 2k byte RAM).
What would be a good decompression routine?
The basic Run-length encoding seems too wasteful - I can immediately see that adding a header setion to the compressed data to put to use unused byte values to represent oft repeated patterns would give phenomenal performance!
With me who only invested a few minutes, surely there must already exist much better algorithms from people who love this stuff?
I would like to have some "ready to go" examples to try out on a PC so that I can compare the performance vis-a-vis a basic RLE.
The two solutions I use when performance is the only concern:
LZO Has a GPL License.
liblzf Has a BSD License.
miniLZO.tar.gz This is LZO, just repacked in to a 'minified' version that is better suited to embedded development.
Both are extremely fast when decompressing. I've found that LZO will create slightly smaller compressed data than liblzf in most cases. You'll need to do your own benchmarks for speeds, but I consider them to be "essentially equal". Both are light-years faster than zlib, though neither compresses as well (as you would expect).
LZO, in particular miniLZO, and liblzf are both excellent for embedded targets.
If you have a preset distribution of values that means the propability of each value is fixed over all datasets, you can create a huffman encoding with fixed codes (the code tree has not to be embedded into the data).
Depending on the data, I'd try huffman with fixed codes or lz77 (see links of Brian).
Well, the main two algorithms that come to mind are Huffman and LZ.
The first basically just creates a dictionary. If you restrict the dictionary's size sufficiently, it should be pretty fast...but don't expect very good compression.
The latter works by adding back-references to repeating portions of output file. This probably would take very little memory to run, except that you would need to either use file i/o to read the back-references or store a chunk of the recently read data in RAM.
I suspect LZ is your best option, if the repeated sections tend to be close to one another. Huffman works by having a dictionary of often repeated elements, as you mentioned.
Since this seems to be audio, I'd look at either differential PCM or ADPCM, or something similar, which will reduce it to 4 bits/sample without much loss in quality.
With the most basic differential PCM implementation, you just store a 4 bit signed difference between the current sample and an accumulator, and add that difference to the accumulator and move to the next sample. If the difference it outside of [-8,7], you have to clamp the value and it may take several samples for the accumulator to catch up. Decoding is very fast using almost no memory, just adding each value to the accumulator and outputting the accumulator as the next sample.
A small improvement over basic DPCM to help the accumulator catch up faster when the signal gets louder and higher pitch is to use a lookup table to decode the 4 bit values to a larger non-linear range, where they're still 1 apart near zero, but increase at larger increments toward the limits. And/or you could reserve one of the values to toggle a multiplier. Deciding when to use it up to the encoder. With these improvements, you can either achieve better quality or get away with 3 bits per sample instead of 4.
If your device has a non-linear μ-law or A-law ADC, you can get quality comparable to 11-12 bit with 8 bit samples. Or you can probably do it yourself in your decoder. http://en.wikipedia.org/wiki/M-law_algorithm
There might be inexpensive chips out there that already do all this for you, depending on what you're making. I haven't looked into any.
You should try different compression algorithms with either a compression software tool with command line switches or a compression library where you can try out different algorithms.
Use typical data for your application.
Then you know which algorithm is best-fitting for your needs.
I have used zlib in embedded systems for a bootloader that decompresses the application image to RAM on start-up. The licence is nicely permissive, no GPL nonsense. It does make a single malloc call, but in my case I simply replaced this with a stub that returned a pointer to a static block, and a corresponding free() stub. I did this by monitoring its memory allocation usage to get the size right. If your system can support dynamic memory allocation, then it is much simpler.
http://www.zlib.net/