How do I determine a sufficient sample size to test an algorithm that can not be unit tested (~pattern recognition).
I have a relatively simple algorithm that uses vehicle position data, and bridge positional data, to determine whether a vehicle has crossed a bridge or not (true/false). The algorithm is allowed to give false positives but must never give a false negative.
I have tested the algorithm manually 400 times (200 instances where it is known the vehicle crossed, and 200 instances where it is known the vehicle did not cross). It has performed very well with no false negative results.
My concern is that I can not feasibly test the many thousand bridges for every concievable gps approach, and I must rely on a certain sample of tested bridges to be confident in my algorithm. I have read the wikipedia page on sample size and I do not see how it applies to my situation.
Related
I am looking for a machine learning strategy that will allow me to load in test data, along with a test outcome, and optimize for a particular scenario to adjust future testing parameters.
(See example in the edit)
Original example: For example, consider that I have a 3 dimensional space (environmental chamber) that I place a physical test device into. I will then select a number of locations and physical attributes with which to test the device. First I will select to test the device at every location configuration, across multiple temperatures, humidities, and pressures. At each test increment, or combination of variables, I log the value of each feature, e.g. x,y,z positional data, as well as the temperature, humidity, pressure, etc.. after setting these parameters I will initiate a software test on the physical device that is affected by the environmental factors in ways too complex for me to predict. This software test can output three outputs that vary with an unknown (until tested) probability based on the logged physical parameters. Of the three outputs, one is a failure, one is a success, and one is that the test finishes without any meaningful output (we can ignore this case in processing).
Once the test finishes testing every physical parameter I provide it, I would like to then run an algorithm on this test data to select the controllable parameters e.g. x,y,z positions, or temperature to maximize my chance of a successful test while also minimizing my chance at a failure (in some cases we find parameters that have a high chance at failure and a high chance at success, failures are more time expensive, thus we need to avoid them). The algorithm should use these to suggest an alternative set of test parameter ranges to initiate the next test iteration that it believes will get us closer to our goal.
Currently, I can maximize the success case by using an iterative decision tree and ignoring the results that end in failure.
Any ideas are appreciated
Edit:
Another example (this is contrived, lets not get into the details of PRNGS)-- Lets say that I have an embedded device that has a hardware pseudo random number generator (PRNG) that is affected by environmental factors, such as heat and magnetometer data. I have a program on the device that uses this PRNG to give me a random value. Suppose that this PRNG just barely achieves sufficient randomization in the average case, in the best case gives me a good random value, and in the worst case fails to provide a random value. By changing the physical parameters of the environment around the device I can find values which with some probability cause this PRNG to fail to give me a random number, give me an 'ok' random number, and which cause it to succeed in generating a cryptographically secure random number. Lets suppose in cases in which it fails to generate a good enough random number the program enters a long running loop trying to find one before ultimately failing, which we would like to avoid as it is computationally expensive. To test this, we first start off by testing every combination of variables in which we can control (temperature, position, etc..) perhaps by jumping several degrees at a time, to give a rough picture on what the device's response looks like over a large range. I would like then to run an algorithm on this test data, narrow my testing windows and iterate over the newly selected feature parameters to arrive at an optimized solution. In this case, since failures are expensive, the optimized solution would be something that minimizes failures, while simultaneously maximizing successes.
I'm working on project, connected to hardware, especially sensors and actuators. Both sensors and actuators can be digital and analog. Example of all four types:
digital sensor: returns one of several possible states (i.e. "presence"/"absence"/"can't find out")
analog sensor: returns any value from range, but results are round according to scale interval (i.e. temperature sensor returns any value from 0 to 60 Celsius, with one digit precision, for example 18.5 C)
digital actuator: can get one of several possible states (i.e. motor, that opens window, can set window "open"/"half-open"/"closed")
analog actuator: can gat any value from range, and value is also rounded according to scale interval (i.e. fan rotations per second can be from 0 to 10, for example 9.5)
The question is, how I can represent these classes, without violating OOP principles, at least make design clear and logical.
One of my solutions is on the picture:
But I think this solution is "ugly"
Next solution is here:
This solution is also bad, because of redundant attributes and not uniform methods' names.
What do you suggest? Maybe there exists design pattern that solves my problem?
Thanks for help!
Finally I came to solution.
In my concrete case, Sensor and Actuator classes are just abstraction for data about current value and socket port number. So I decide to get rid of difference and make one class for both sensors and actuators. And discrete and continuous devices have it's own classes.
I would solve this by using two different operations to return either analog or digital values. The reason is simply that you will not really mix them. If you read temperatures you will readAnalog() to get the value and for a switch you would call readBinary(). This way you can implement that reading a float from a binary would raise an exception.
Also, I would not mix this in a Device. Actuators and sensors are very different and the only thing they might inherit is the plastic housing (and even that seems strange).
I am encoding the interval [x:y] to binary codes like 10101111, so for population, it is like [[1,0,1,1],[0,1,0,1]].
I defined the fitness function directly using the value of the function (sin(x)^2).
For selection, i am using tournament selection and for crossover, only simple exchange part of the chromosome like this: 1(10)0 and 0(01)1 -> 1(01)0 and 0(10)1.
For mutation, using Bit inversion.
The algorithm kind of works, it can generate the global minimum sometimes, and sometimes local ones. but I don't see the function of crossover in this problem, because the feature of the 'x' is being broken every time (i think), I don't know why, and if it is even right way to code the crossover or maybe the encoding part.
I'm afraid that there isn't a "right way" to crossover.
There are many crossover operator (e.g. Comparison of a Crossover Operator in Binary-coded Genetic Algorithms - STJEPAN PICEK, MARIN GOLUB) that can be used in binary coded genetic algorithm, but:
depending on the properties of a problem one or another crossover operator will have better result.
every crossover operator has its advantages and downfalls, so choosing one ultimately represents the question of someone's requirements and experiments
undergone.
in many situations uniform and two-point crossover are good choices.
Crossover is the major exploratory mechanism of the genetic algorithm, but the driving force behind GA is the cooperation between selection, crossover and mutation (mutation prevents convergence of the population and introduces variation).
Usually a mutation-only approach doesn't have enough exploration strength to reach to the minimum and the success is largely due to distribution of solutions in the initial population.
For continuous function optimization you should also check differential evolution.
Decision problems are not suited for use in evolutionary algorithms since a simple right/wrong fitness measure cannot be optimized/evolved. So, what are some methods/techniques for converting decision problems to optimization problems?
For instance, I'm currently working on a problem where the fitness of an individual depends very heavily on the output it produces. Depending on the ordering of genes, an individual either produces no output or perfect output - no "in between" (and therefore, no hills to climb). One small change in an individual's gene ordering can have a drastic effect on the fitness of an individual, so using an evolutionary algorithm essentially amounts to a random search.
Some literature references would be nice if you know of any.
Application to multiple inputs and examination of percentage of correct answers.
True, a right/wrong fitness measure cannot evolve towards more rightness, but an algorithm can nonetheless apply a mutable function to whatever input it takes to produce a decision which will be right or wrong. So, you keep mutating the algorithm, and for each mutated version of the algorithm you apply it to, say, 100 different inputs, and you check how many of them it got right. Then, you select those algorithms that gave more correct answers than others. Who knows, eventually you might see one which gets them all right.
There are no literature references, I just came up with it.
Well i think you must work on your fitness function.
When you say that some Individuals are more close to a perfect solution can you identify this solutions based on their genetic structure?
If you can do that a program could do that too and so you shouldn't rate the individual based on the output but on its structure.
Assume you have access to an "oracle" implementation whose output you trust to be correct.
The most obvious way to do this seems to be to run a set of known plaintext/hash combinations through the implementation and see if they come out as expected. An arbitrary number of these cases could be constructed by generating random plaintexts (using a static seed to keep it deterministic) and using the oracle to find their hashes.
The major problem I see with this is that it's not guaranteed to hit possible corner cases. Generating more cases will reduce the likelihood of missing corner cases, but how many cases is enough?
There's also the side issue of specifying the lengths of these random plaintexts because MD5 takes an arbitrary-length string as input. For my purposes, I don't care about long inputs (say, anything longer than 16 bytes), so you can use the fact that this is a "special purpose" MD5 implementation in your answer if it makes things simpler or you can just answer for the general case if it's all the same.
If you have an algorithmic error, it's extremely likely that every hash will be wrong. Hashes are unforgiving by nature.
Since the majority of possible errors will be exposed quickly, you really won't need that many tests. The main things to cover are the edge cases:
Length=0 (input is empty)
Length=1
Length=16
Input contains at least one byte with value 0
Repeated patterns of bytes in the input (would this be a meaningful edge case for MD5?)
If those all pass, perhaps along with tests for one or two more representative inputs, you could be pretty confident in your algorithm. There aren't that many edge cases (unless someone more familiar with the algorithm's details can think of some more).