I'm looking for an efficient way to represent a chess position.
my "efficiency" criteria are these:
requires as little memory as possible
given a position representation and a legal move (represented in the usual way, e.g. as in pgn format) it is easy to compute the resulting poisition representation. that is, the function new_position = compute_position(old_position, move) can be (or is already) implemented in a very efficient way (from a run-time perspective).
it is easy to compare two positions to see if they are identical
The reason I try to deviate from standard representations is that my requirements are slightly different. Specifically, I am not trying to develop a chess engine, so move generation and related issues are not required. I only need to follow some existing games, and represent certain positions, and store them in a database.
If there is a software package that that already gives this functionality, that would be great. if you have ideas on how to do it, I'd be happy to develop it :)
Thanks...
I'll offer some possible solutions that use "as little memory as possible" as per your specification. I know that you can make some of these more compact, but doing so makes it much more difficult to work with.
64 bits: I don't know what kind of information you need to be able to retrieve from the representation, but a technique that fulfills all your given requirements is Zobrist hashing. It requires very little memory (you can use a 64 bit key and then store 232 positions before a collision is expected to occur). It is very easy and efficient to update incrementally (bitwise XOR operations) and of course easy to compare. However, if you need to display the position or retrieve any information about pieces, this leaves you out of luck.
328 bits: you can use a piece list, with each piece (4 bits) located at a square (6 bits). Since there are 32 pieces, and you need to keep track of castling rights (4 bits), en passant file (3 bits), and side to move (1 bit).
468 bits: bitboard representation. I know you're not developing a chess engine but this is still a very compact representation. You need 7 bitboards (all white, all black, all queens, all rooks, all bishops, all knights, all pawns), keep track of kings separately (12 bits), and all the other miscellaneous information. Yes this would be a bit of work to implement, but this allows you to do many things not possible in other representations, so this is great if you need to analyze positions.
There's a free PGN viewer here: http://chesstempo.com/pgn-viewer.html
Regarding the space requirement, you could compress FEN using a compression algorithm of your choice, without losing the ability to check for equality. Depending how the moves are encoded (g1f3 versus Nf3) you may require a legal move generator for requirement 2).
Related
I would like to use AI/TensorFlow/Machine Learning of some description in order to recognise patterns in a set of data.
Most of the samples of machine learning seem to be based on decision making, whether a value falls above or a below a line, then the decision is good or bad. But what I seem to have is a set of data, that may or may not have any relationship, may or may not have a pattern, and a single entity by itself is neither good nor bad, but the whole set of data together needs to be used to work out the type of data.
The data in question is a set of readings over a period of time from an automotive CANBUS reader - so hexadecimal values containing a Command ID, and 1 or more values, usually in the format: FFF#FF:FF:FF:FF:FF:FF:FF:FF
eg:
35C#F4:C6:4E:7C:31:2B:60:28
One canbus command, may contain one or more sensor readers - so for example F4 above might represent the steering position, C6 might indicate the pitch of the vehicle, 4E might indicate the roll.
Each sensor may take up one or more octets, 7C:31 might indicate the speed of the vehicle, and "B" of "2B" might indicate whether the engine is running or not.
I can detect the data with a human eye, and can see that the relevent item might be linear, random, static (ie a limited set of values) or it might be a bell curve.
Im new to statistical analysis and machine learning so do not know the terminology. In the first instance Im looking for the terminology and references to appropriate material that will help me achieve my goal.
My goal is given a sample of data from a CANBUS reader, to scan all the values, using each and every possible combination of numbers (eg octets 1, 1+2, 1+2+3... 2, 2+3, 2+3+4... 3, 3+4 etc) to detect patterns within the data and work out whether those patterns are linear, curve, static, or random.
I want to basically read as many CAN-BUS readings from as many cars as I can, throw it at a program to do some analysis and learning and hopefully provide me with possibilities to investigate further so I can monitor various systems on different cars.
It seems like a relatively simple premise, but extremely hard for me to define.
I want to experiment with massive parallel chess computing. From what I saw and understood in wikis and source code of some engines is that in most (all?) implementations of the min-max(negamax, alpha-beta, ...)-algorithm there is one internal position that gets updated for every new branch and then gets undone after receiving the evaluation of that branch.
What are the benefits of that technic compared to just generating a new position-object and passing that to the next branch?
This is what I have done in my previous engines and I believe this method is superior for the purpose of parallelism.
The answer to your question depends heavily on a few factors, such as how you are storing the chess board, and what your make/unmake move function looks like. I'm sure there are ways of storing the board that would be better suited towards your method, and indeed historically some top tiered chess engine (in particular crafty) used to use that method, but you are correct in saying that modern engines no longer do it that way. Here is a link to a discussion about this very point:
http://www.talkchess.com/forum/viewtopic.php?t=50805
If you want to understand why this is, then you must understand how today's engines represent the board. The standard implementation revolves around bitboards, and that requires 12- 64 bit integers per position, in addition to a redundant mailbox (array in non computer chess jargon) used in conjunction. Copying all that is usually more expensive than a good makeMove/unMakeMove function.
I also want to point out that a hybrid approach is also common. Usually make and unmake is used on the board itself, where as other information like en passant squares and castle rights are copied and changed like you suggested.
Interestingly, for board representations of < ~300 bytes, it IS cheaper to copy the board on each move (on modern x86), as opposed to making and unmaking the move.
As you suggest the immutable properties of copying the board on each move are from a programming perspective, including parallelising, very attractive.
My board rep is 208 bytes. C++ compiled with g++ 7.4.0.
Empricism shows that board copy is 20% faster than move make/unmake. I presume the code is using 32/64 byte-wide AVX instructions for the copy. In theory you can do 32/64 byte copy per cycle.
Just for you: https://github.com/rolandpj1968/oom-skaak/tree/init-impl-legal-move-gen-move-do-undo-2
I'm trying to find an alternative to QR codes (I'd also be willing to accept an entirely novel solution and implement it myself) that meets certain specifications.
First, the codes will often end up on thin pipes, and so need to be readable around a cylinder. The advantage to this is that the effect on the image from wrapping it around a cylinder is easy to express geometrically, and the codes will never be placed on a very irregular shape.
Second, read accuracy must be very high, as any read mistake would be extremely costly. If this means larger codes with more redundancy for better error correction, so be it.
Third, ability to be read by the average smartphone camera from a few inches out.
Fourth, storage space of around half a kilobyte per code.
Do you know of such a code?
The Data Matrix Rectangular Extension (DMRE) improves upon the standard set of rectangular Data Matrix symbol sizes in an algorithmically compatible manner, thus increasing the range of suitable applications with no real downsides.
Reliable cylindrical marking is a primary use case.
Regardless of symbology you will be unable to approach sufficient data density to achieve 0.5KB of binary data in a single compact, narrow symbol scanned using a standard camera phone. However, most 2D symbologies (DMRE included) support a feature called Structured Append that allows chaining of multiple symbols that can be scanned in any order to produce a single read when all components are accounted for.
If the data to be encoded is known to be highly structured (e.g. mostly numeric or alphanumeric) then the internal encoding process of Data Matrix will be more optimised than for general binary data. For example, the largest DMRE symbol (26×64) will provide up to 236 numeric characters, ~175 alphanumeric characters and only 116 bytes.
If the default error recovery rate is insufficient then including a checksum in the data may be appropriate.
DMRE has just been voted to be accepted as an ISO/IEC project and will likely become an international standard enjoying broad hardware and software support in due course.
Another option may be to investigate PDF417 which has a broader range of symbols sizes, however the data density is somewhat less than Data Matrix.
DMRE references: AIM specification and explanatory notes.
what is the basic idea behind steganography?ie ,how do you get the hidden information?
suppose if it is an image and some text is within that image...how do you get that text?..
Every stenography algorithm is different in that respect. Every algorithm hides the information differently and thus getting the information back is different.
A simple example goes like this - Each pixel of the image is composed of 3 bytes, one for red, green and blue. Most people can't detect a difference of one bit in the color in an image so one option is to use the least significant bit of each color channel for your data. This way you can store 3 bits of information in every pixel with very little effect on the general quality of the image.
To get the information back you'll need to read the first bit of every color channel of every pixel and gather all the bits together.
This is just a very simple and almost trivial way to do stenography. Real stenography algorithms are somewhat more involved. Like in cryptography, there is no way to generically "unhide" all stenography. you need to know which algorithm you're trying to decode.
The very basic idea is that images contain tons of redundant information that your eye cannot see. For instance if you changed the last bit on each pixel there would be no visible change as almost all of the information about the color is the other bits. So you can encode messages using the last bit (the most basic algorithm). The histogram however will be changed and a large message will easily be detectable. As far a decoding the message itself, well, the message itself is probably using public key encryption so you will never know what the actual payload was.
Steganography unlike cryptography is considered broken if Eve (who is eavesdropping and practising steganalysis) knows that there is a message at all. The assumptions are based on that Alice and Bob are being watched and any communication is sign that they are up to something (aka prisoners, restrictive governments, all governments in the future hehe ;-))
And of course the algorithms become much more complex that just flipping the last bits, but encoding data that will not affect the structure of image (and become vulnerable to statistical attacks.) :
I read this book last summer and I thought is was an excellent introduction (it has a lot a psuedocode of the algorithms used)
http://www.amazon.com/Steganography-Digital-Media-Principles-Applications/dp/0521190193
Steganography, coming from greek Steganos(i'm greek :P) is the art of hiding messages. While cryptography is about scrambling a message, steganography is about a person not being able to locate the message.
There are many tools that do this procedure for you. Writing a tool like that can be a complicated procedure i think, though i haven't tried to do that. You would need to create a sophisticated approach of correctly using unused or seemingly not important image pixels or data, in order to add your own message, file etc. For some more information, please take a look at : http://www.symantec.com/connect/articles/steganography-revealed
I need a FAST decompression routine optimized for restricted resource environment like embedded systems on binary (hex data) that has following characteristics:
Data is 8bit (byte) oriented (data bus is 8 bits wide).
Byte values do NOT range uniformly from 0 - 0xFF, but have a poisson distribution (bell curve) in each DataSet.
Dataset is fixed in advanced (to be burnt into Flash) and each set is rarely > 1 - 2MB
Compression can take as much as time required, but decompression of a byte should take 23uS in the worst case scenario with minimal memory footprint as it will be done on a restricted resource environment like an embedded system (3Mhz - 12Mhz core, 2k byte RAM).
What would be a good decompression routine?
The basic Run-length encoding seems too wasteful - I can immediately see that adding a header setion to the compressed data to put to use unused byte values to represent oft repeated patterns would give phenomenal performance!
With me who only invested a few minutes, surely there must already exist much better algorithms from people who love this stuff?
I would like to have some "ready to go" examples to try out on a PC so that I can compare the performance vis-a-vis a basic RLE.
The two solutions I use when performance is the only concern:
LZO Has a GPL License.
liblzf Has a BSD License.
miniLZO.tar.gz This is LZO, just repacked in to a 'minified' version that is better suited to embedded development.
Both are extremely fast when decompressing. I've found that LZO will create slightly smaller compressed data than liblzf in most cases. You'll need to do your own benchmarks for speeds, but I consider them to be "essentially equal". Both are light-years faster than zlib, though neither compresses as well (as you would expect).
LZO, in particular miniLZO, and liblzf are both excellent for embedded targets.
If you have a preset distribution of values that means the propability of each value is fixed over all datasets, you can create a huffman encoding with fixed codes (the code tree has not to be embedded into the data).
Depending on the data, I'd try huffman with fixed codes or lz77 (see links of Brian).
Well, the main two algorithms that come to mind are Huffman and LZ.
The first basically just creates a dictionary. If you restrict the dictionary's size sufficiently, it should be pretty fast...but don't expect very good compression.
The latter works by adding back-references to repeating portions of output file. This probably would take very little memory to run, except that you would need to either use file i/o to read the back-references or store a chunk of the recently read data in RAM.
I suspect LZ is your best option, if the repeated sections tend to be close to one another. Huffman works by having a dictionary of often repeated elements, as you mentioned.
Since this seems to be audio, I'd look at either differential PCM or ADPCM, or something similar, which will reduce it to 4 bits/sample without much loss in quality.
With the most basic differential PCM implementation, you just store a 4 bit signed difference between the current sample and an accumulator, and add that difference to the accumulator and move to the next sample. If the difference it outside of [-8,7], you have to clamp the value and it may take several samples for the accumulator to catch up. Decoding is very fast using almost no memory, just adding each value to the accumulator and outputting the accumulator as the next sample.
A small improvement over basic DPCM to help the accumulator catch up faster when the signal gets louder and higher pitch is to use a lookup table to decode the 4 bit values to a larger non-linear range, where they're still 1 apart near zero, but increase at larger increments toward the limits. And/or you could reserve one of the values to toggle a multiplier. Deciding when to use it up to the encoder. With these improvements, you can either achieve better quality or get away with 3 bits per sample instead of 4.
If your device has a non-linear μ-law or A-law ADC, you can get quality comparable to 11-12 bit with 8 bit samples. Or you can probably do it yourself in your decoder. http://en.wikipedia.org/wiki/M-law_algorithm
There might be inexpensive chips out there that already do all this for you, depending on what you're making. I haven't looked into any.
You should try different compression algorithms with either a compression software tool with command line switches or a compression library where you can try out different algorithms.
Use typical data for your application.
Then you know which algorithm is best-fitting for your needs.
I have used zlib in embedded systems for a bootloader that decompresses the application image to RAM on start-up. The licence is nicely permissive, no GPL nonsense. It does make a single malloc call, but in my case I simply replaced this with a stub that returned a pointer to a static block, and a corresponding free() stub. I did this by monitoring its memory allocation usage to get the size right. If your system can support dynamic memory allocation, then it is much simpler.
http://www.zlib.net/