Here's an example of Soundex code in SQL:
SELECT SOUNDEX('Smith'), SOUNDEX('Smythe');
----- -----
S530 S530
How does 'Smith' become S530?
In this example, the first digit is S because that's the first character in the input expression, but how are the remaining three digits are calculated?
Take a look a this article
The first letter of the code corresponds to the first letter of the
name. The remainder of the code consists of three digits derived from
the syllables of the word according to the following code:
1 = B, F, P, V
2 = C, G, J, K, Q, S, X, Z
3 = D, T
4 = L
5 = M,N
6 = R
The double letters with the same Soundex code, A, E, I, O, U, H, W, Y,
and some prefixes are being disregarded...
So for Smith and Smythe the code is created like this:
S S -> S
m m -> 5
i y -> 0
t t -> 3
h h -> 0
e -> -
What is Soundex?
Soundex is:
a phonetic algorithm for indexing names by sound, as pronounced in English; first developed by Robert C. Russell and Margaret King Odell in 1918
How does it Work?
There are several implementations of Soundex, but most implement the following steps:
Retain the first letter of the name and drop all other occurrences of vowels and h,w:
|a, e, i, o, u, y, h, w | → "" |
Replace consonants with numbers as follows (after the first letter):
| b, f, p, v | → 1 |
| c, g, j, k, q, s, x, z | → 2 |
| d, t | → 3 |
| l | → 4 |
| m, n | → 5 |
| r | → 6 |
Replace identical adjacent numbers with a single value (if they were next to each other prior to step 1):
| M33 | → M3 |
Cut or Pad with zeros or cut to produce a 4 digit result:
| M3 | → M300 |
| M34123 | → M341 |
Here's an interactive demo in jsFiddle:
And here's a demo in SQL using SQL Fiddle
In SQL Server, SOUNDEX is often used in conjunction with DIFFERENCE, which is used to score how many of the resulting digits are identical (just like the game mastermind†), with higher numbers matching most closely.
What are the Alternatives?
It's important to understand the limitations and criticisms of soundex and where people have tried to improve it, notably only being rooted in English pronunciation and also discards a lot of data, resulting in more false positives.
Both Metaphone & Double Metaphone still focus on English pronunciations, but add much more granularity to the nuances of speech in Enlgish (ie. PH → F)
Phil Factor wrote a Metaphone Function in SQL with the source on github
Soundex is most commonly used on identifying similar names, and it'll have a really hard time finding any similar nicknames (i.e. Robert → Rob or Bob). Per this question on a Database of common name aliases / nicknames of people, you could incorporate a lookup against similar nicknames as well in your matching process.
Here are a couple free lists of common nicknames:
SOEMPI - name_to_nick.csv | Github
carltonnorthern - names.csv | Github
Further Reading:
Fuzzy matching using T-SQL
SQL Server – Do You Know Soundex Functions?
Related
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 9 years ago.
I just took my midterm but couldn't answer this question.
Can someone please give a couple of examples of the language and construct a grammar for the language or
at least show me how i will go about it?
Also how to write grammar for L:
L = {an bm | n,m = 0,1,2,..., n <= 2m } ?
Thanks in advance.
How to write grammar for formal language?
Before read my this answer you should read first: Tips for creating Context free grammars.
Grammar for {an bm | n,m = 0,1,2,..., n <= 2m }
What is you language L = {an bm | n,m = 0,1,2,..., n <= 2m } description?
Language description:
The language L is consist of set of all strings in which symbols a followed by symbols b, where number of symbol b are more than or equals to half of number of a's.
To understand more clearly:
In pattern an bm, first symbols a come then symbol b. total number of a 's is n and number of b's is m. The inequality equation says about relation between n and m. To understand the equation:
given: n <= 2m
=> n/2 <= m means `m` should be = or > then n/2
=> numberOf(b) >= numberOf(a)/2 ...eq-1
So inequality of n and m says:
numberOf(b) must be more than or equals to half of numberOf(a)
Some example strings in L:
b numberOf(a)=0 and numberOf(b)=1 this satisfy eq-1
bb numberOf(a)=0 and numberOf(b)=2 this satisfy eq-1
So in language string any number of b are possible without a's. (any string of b) because any number is greater then zero (0/2 = 0).
Other examples:
m n
--------------
ab numberOf(a)=1 and numberOf(b)=1 > 1/2
abb numberOf(a)=1 and numberOf(b)=2 > 1/2
abbb numberOf(a)=1 and numberOf(b)=3 > 1/2
aabb numberOf(a)=2 and numberOf(b)=2 > 2/2 = 1
aaabb numberOf(a)=3 and numberOf(b)=2 > 3/2 = 1.5
aaaabb numberOf(a)=4 and numberOf(b)=2 = 4/2 = 2
Points to be note:
all above strings are possible because number of b's are either equal(=) to half of the number of a or more (>).
and interesting point to notice is that total a's can also be more then number of b's, but not too much. Whereas number of b's can be more then number of a's by any number of times.
Two more important case are:
only a as a string not possible.
note: null ^ string is also allowed because in ^ , numberOf(a) = numberOf(b) = 0 that satisfy equation.
At once, it look that writing grammar is tough but really not...
According to language description, we need following kinds of rules:
rule 1: To generate ^ null string.
N --> ^
rule 2: To generate any number of b
B --> bB | b
Rule 3: to generate a's:
(1) Remember you can't generate too many a's without generating b's.
(2) Because b's are more then = to half of a's; you need to generate one b for every alternate a
(3) Only a as a string not possible so for first (odd) alternative you need to add b with an a
(4) Whereas for even alternative you can discard to add b (but not compulsory)
So you overall grammar:
S --> ^ | A | B
B --> bB | b
A --> aCB | aAB | ^
C --> aA | ^
here S is start Variable.
In the above grammar rules you may have confusion in A --> aCB | aAB | ^, so below is my explanation:
A --> aCB | aAB | ^
^_____^
for second alternative a
C --> aA <== to discard `b`
and aAB to keep b
let us we generate some strings in language using this grammar rules, I am writing Left most derivation to avoid explanation.
ab S --> A --> aCB --> aB --> ab
abb S --> A --> aCB --> aB --> abB --> abb
abbb S --> A --> aCB --> aB --> abB --> abB --> abbB --> abbb
aabb S --> A --> aAB --> aaABB --> aaBB --> aabB --> aabb
aaabb S --> A --> aCB --> aaAB --> aaaABB --> aaaBB --> aaabB --> aaabb
aaaabb S --> A --> aCB --> aaAB --> aaaCBB --> aaaaABB --> aaaaBB
--> aaaabB
--> aaaabb
One more for non-member string:
according to language a5 b2 = aaaaabb is not possible. because 2 >= 5/2 = 2.5 ==> 2 >= 2.5 inequality fails. So we can't generate this string using grammar too. I try to show below:
In our grammar to generate extra a's we have to use C variable.
S --> A
--> aCB
--> aaAB
--> aa aCB B
--> aaa aA BB
--> aaaa aCB BB
---
^
here with first `a` I have to put a `b` too
While my answer is done but I think you can change A's rules like:
A --> aCB | A | ^
Give it a Try!!
EDIT:
as #us2012 commented: It would seem to me that then, S -> ^ | ab | aaSb | Sb would be a simpler description. I feel this question would be good for OP and other also.
OP's language:
L = {an bm | n,m = 0,1,2,..., n <= 2m}.
#us2012's Grammar:
S -> ^ | ab | aaSb | Sb
#us2012's question:
Whether this grammar also generates language L?
Answer is Yes!
The inequality in language between number of a's = n and number of b = m is n =< 2m
We can also understand as:
n =< 2m
that is
numberOf(a) = < twice of numberOf(b)
And In grammar, when even we add one or two a's we also add one b . So ultimately number of a can't be more then twice of number of b.
Grammar also have rules to generate. any numbers of b's and null ^ strings.
So the simplified Grammar provided by #us2012 is CORRECT and also generates language L exactly.
Notice: The first solution came from derivation as I written in am linked answer, I started with language description then tried to write some basic rules and progressively I could write complete grammar.
Whereas #us2012's answer came by aptitude, you can gain the aptitude to write grammar by reading others' solutions and writing your own for some - just like how you learn programming.
Can someone please explain the Differential Evolution method? The Wikipedia definition is extremely technical.
A dumbed-down explanation followed by a simple example would be appreciated :)
Here's a simplified description. DE is an optimisation technique which iteratively modifies a population of candidate solutions to make it converge to an optimum of your function.
You first initialise your candidate solutions randomly. Then at each iteration and for each candidate solution x you do the following:
you produce a trial vector: v = a + ( b - c ) / 2, where a, b, c are three distinct candidate solutions picked randomly among your population.
you randomly swap vector components between x and v to produce v'. At least one component from v must be swapped.
you replace x in your population with v' only if it is a better candidate (i.e. it better optimise your function).
(Note that the above algorithm is very simplified; don't code from it, find proper spec. elsewhere instead)
Unfortunately the Wikipedia article lacks illustrations. It is easier to understand with a graphical representation, you'll find some in these slides: http://www-personal.une.edu.au/~jvanderw/DE_1.pdf .
It is similar to genetic algorithm (GA) except that the candidate solutions are not considered as binary strings (chromosome) but (usually) as real vectors. One key aspect of DE is that the mutation step size (see step 1 for the mutation) is dynamic, that is, it adapts to the configuration of your population and will tend to zero when it converges. This makes DE less vulnerable to genetic drift than GA.
Answering my own question...
Overview
The principal difference between Genetic Algorithms and Differential Evolution (DE) is that Genetic Algorithms rely on crossover while evolutionary strategies use mutation as the primary search mechanism.
DE generates new candidates by adding a weighted difference between two population members to a third member (more on this below).
If the resulting candidate is superior to the candidate with which it was compared, it replaces it; otherwise, the original candidate remains unchanged.
Definitions
The population is made up of NP candidates.
Xi = A parent candidate at index i (indexes range from 0 to NP-1) from the current generation. Also known as the target vector.
Each candidate contains D parameters.
Xi(j) = The jth parameter in candidate Xi.
Xa, Xb, Xc = three random parent candidates.
Difference vector = (Xb - Xa)
F = A weight that determines the rate of the population's evolution.
Ideal values: [0.5, 1.0]
CR = The probability of crossover taking place.
Range: [0, 1]
Xc` = A mutant vector obtained through the differential mutation operation. Also known as the donor vector.
Xt = The child of Xi and Xc`. Also known as the trial vector.
Algorithm
For each candidate in the population
for (int i = 0; i<NP; ++i)
Choose three distinct parents at random (they must differ from each other and i)
do
{
a = random.nextInt(NP);
} while (a == i)
do
{
b = random.nextInt(NP);
} while (b == i || b == a);
do
{
c = random.nextInt(NP);
} while (c == i || c == b || c == a);
(Mutation step) Add a weighted difference vector between two population members to a third member
Xc` = Xc + F * (Xb - Xa)
(Crossover step) For every variable in Xi, apply uniform crossover with probability CR to inherit from Xc`; otherwise, inherit from Xi. At least one variable must be inherited from Xc`
int R = random.nextInt(D);
for (int j=0; j < D; ++j)
{
double probability = random.nextDouble();
if (probability < CR || j == R)
Xt[j] = Xc`[j]
else
Xt[j] = Xi[j]
}
(Selection step) If Xt is superior to Xi then Xt replaces Xi in the next generation. Otherwise, Xi is kept unmodified.
Resources
See this for an overview of the terminology
See Optimization Using Differential Evolution by Vasan Arunachalam for an explanation of the Differential Evolution algorithm
See Evolution: A Survey of the State-of-the-Art by Swagatam Das and Ponnuthurai Nagaratnam Suganthan for different variants of the Differential Evolution algorithm
See Differential Evolution Optimization from Scratch with Python for a detailed description of an implementation of a DE algorithm in python.
The working of DE algorithm is very simple.
Consider you need to optimize(minimize,for eg) ∑Xi^2 (sphere model) within a given range, say [-100,100]. We know that the minimum value is 0. Let's see how DE works.
DE is a population-based algorithm. And for each individual in the population, a fixed number of chromosomes will be there (imagine it as a set of human beings and chromosomes or genes in each of them).
Let me explain DE w.r.t above function
We need to fix the population size and the number of chromosomes or genes(named as parameters). For instance, let's consider a population of size 4 and each of the individual has 3 chromosomes(or genes or parameters). Let's call the individuals R1,R2,R3,R4.
Step 1 : Initialize the population
We need to randomly initialise the population within the range [-100,100]
G1 G2 G3 objective fn value
R1 -> |-90 | 2 | 1 | =>8105
R2 -> | 7 | 9 | -50 | =>2630
R3 -> | 4 | 2 | -9.2| =>104.64
R4 -> | 8.5 | 7 | 9 | =>202.25
objective function value is calculated using the given objective function.In this case, it's ∑Xi^2. So for R1, obj fn value will be -90^2+2^2+2^2 = 8105. Similarly it is found for all.
Step 2 : Mutation
Fix a target vector,say for eg R1 and then randomly select three other vectors(individuals)say for eg.R2,R3,R4 and performs mutation. Mutation is done as follows,
MutantVector = R2 + F(R3-R4)
(vectors can be chosen randomly, need not be in any order).F (scaling factor/mutation constant) within range [0,1] is one among the few control parameters DE is having.In simple words , it describes how different the mutated vector becomes. Let's keep F =0.5.
| 7 | 9 | -50 |
+
0.5 *
| 4 | 2 | -9.2|
+
| 8.5 | 7 | 9 |
Now performing Mutation will give the following Mutant Vector
MV = | 13.25 | 13.5 | -50.1 | =>2867.82
Step 3 : Crossover
Now that we have a target vector(R1) and a mutant vector MV formed from R2,R3 & R4 ,we need to do a crossover. Consider R1 and MV as two parents and we need a child from these two parents. Crossover is done to determine how much information is to be taken from both the parents. It is controlled by Crossover rate(CR). Every gene/chromosome of the child is determined as follows,
a random number between 0 & 1 is generated, if it is greater than CR , then inherit a gene from target(R1) else from mutant(MV).
Let's set CR = 0.9. Since we have 3 chromosomes for individuals, we need to generate 3 random numbers between 0 and 1. Say for eg, those numbers are 0.21,0.97,0.8 respectively. First and last are lesser than CR value, so those positions in the child's vector will be filled by values from MV and second position will be filled by gene taken from target(R1).
Target-> |-90 | 2 | 1 | Mutant-> | 13.25 | 13.5 | -50.1 |
random num - 0.21, => `Child -> |13.25| -- | -- |`
random num - 0.97, => `Child -> |13.25| 2 | -- |`
random num - 0.80, => `Child -> |13.25| 2 | -50.1 |`
Trial vector/child vector -> | 13.25 | 2 | -50.1 | =>2689.57
Step 4 : Selection
Now we have child and target. Compare the obj fn of both, see which is smaller(minimization problem). Select that individual out of the two for next generation
R1 -> |-90 | 2 | 1 | =>8105
Trial vector/child vector -> | 13.25 | 2 | -50.1 | =>2689.57
Clearly, the child is better so replace target(R1) with the child. So the new population will become
G1 G2 G3 objective fn value
R1 -> | 13.25 | 2 | -50.1 | =>2689.57
R2 -> | 7 | 9 | -50 | =>2500
R3 -> | 4 | 2 | -9.2 | =>104.64
R4 -> | -8.5 | 7 | 9 | =>202.25
This procedure will be continued either till the number of generations desired has reached or till we get our desired value. Hope this will give you some help.
I have a table which contains the edges from node x to node y in a graph.
n1 | n2
-------
a | a
a | b
a | c
b | b
b | d
b | c
d | e
I would like to create a (materialized) view which denotes the shortest number of nodes/hops a path contains to reach from x to node y:
n1 | n2 | c
-----------
a | a | 0
a | b | 1
a | c | 1
a | d | 2
a | e | 3
b | b | 0
b | d | 1
b | c | 1
b | e | 2
d | e | 1
How should I model my tables and views to facilitate this? I guess I need some kind of recursion, but I believe that is pretty difficult to accomplish in SQL. I would like to avoid that, for example, the clients need to fire 10 queries if the path happens to contain 10 nodes/hops.
This works for me, but it's kinda ugly:
WITH RECURSIVE paths (n1, n2, distance) AS (
SELECT
nodes.n1,
nodes.n2,
1
FROM
nodes
WHERE
nodes.n1 <> nodes.n2
UNION ALL
SELECT
paths.n1,
nodes.n2,
paths.distance + 1
FROM
paths
JOIN nodes
ON
paths.n2 = nodes.n1
WHERE
nodes.n1 <> nodes.n2
)
SELECT
paths.n1,
paths.n2,
min(distance)
FROM
paths
GROUP BY
1, 2
UNION
SELECT
nodes.n1,
nodes.n2,
0
FROM
nodes
WHERE
nodes.n1 = nodes.n2
Also, I am not sure how good it will perform against larger datasets. As suggested by Mark Mann, you may want to use a graph library instead, e.g. pygraph.
EDIT: here's a sample with pygraph
from pygraph.algorithms.minmax import shortest_path
from pygraph.classes.digraph import digraph
g = digraph()
g.add_node('a')
g.add_node('b')
g.add_node('c')
g.add_node('d')
g.add_node('e')
g.add_edge(('a', 'a'))
g.add_edge(('a', 'b'))
g.add_edge(('a', 'c'))
g.add_edge(('b', 'b'))
g.add_edge(('b', 'd'))
g.add_edge(('b', 'c'))
g.add_edge(('d', 'e'))
for source in g.nodes():
tree, distances = shortest_path(g, source)
for target, distance in distances.iteritems():
if distance == 0 and not g.has_edge((source, target)):
continue
print source, target, distance
Excluding the graph building time, this takes 0.3ms while the SQL version takes 0.5ms.
Expanding on Mark's answer, there are some very reasonable approaches to explore a graph in SQL as well. In fact, they'll be faster than the dedicated libraries in perl or python, in that DB indexes will spare you the need to explore the graph.
The most efficient of index (if the graph is not constantly changing) is a nested-tree variation called the GRIPP index. (The linked paper mentions other approaches.)
If your graph is constantly changing, you might want to adapt the nested intervals approach to graphs, in a similar manner that GRIPP extends nested sets, or to simply use floats instead of integers (don't forget to normalize them by casting to numeric and back to float if you do).
Rather than computing these values on the fly, why not create a real table with all interesting pairs along with the shortest path value. Then whenever data is inserted, deleted or updated in your data table, you can recalculate all of the shortest path information. (Perl's Graph module is particularly well-suited to this task, and Perl's DBI interface makes the code straightforward.)
By using an external process, you can also limit the number of recalculations. Using PostgreSQL triggers would cause recalculations to occur on every insert, update and delete, but if you knew you were going to be adding twenty pairs of points, you could wait until your inserts were completed before doing the calculations.
I have this language:
{an bm | m+n is an even number}
What's the proper grammar for this?
S -> aaS | aB | bbC | ε
B -> bbB | b
C -> bbC | ε
you see, it is a regular language. 'S' stands for "we have constructed an even number of a's and more a's may follow, 'B' stands for "we have constructed an uneven number of a's and now an uneven number of b's follows. 'C' stands for "we have constructed an even number of a's and now an even number of b's follows.
ε stands for "", the empty string
I'm studying for a finite automata & grammars test and I'm stuck with this question:
Construct a grammar that generates L:
L = {a^n b^m c^m+n|n>=0, m>=0}
I believe my productions should go along this lines:
S->aA | aB
B->bB | bC
C->cC | c Here's where I have doubts
How can my production for C remember the numbers of m and n? I'm guessing this must rather be a context-free grammar, if so, how should it be?
Seems like it should be like:
A->aAc | aBc | ac | epsilon
B->bBc | bc | epsilon
You need to force C'c to be counted during construction process. In order to show it's context-free, I would consider to use Pump Lemma.
S -> X
X -> aXc | Y
Y -> bYc | e
where e == epsilon and X is unnecessary but
added for clarity
Yes, this does sound like homework, but a hint:
Every time you match an 'a', you must match a 'c'. Same for matching a 'b'.
S->aSc|A
A->bAc|λ
This means when ever you get a at least you have 1 c or if you get a and b you must have 2 c.
i hope it has been helpful
Well guys, this is how I'll do it:
P={S::=X|epsilon,
X::=aXc|M|epsilon,
M::=bMc|epsilon}
My answer:
S -> aAc | aSc
A -> bc | bAc
where S is the start symbol.
S-> aBc/epsilon
B-> bBc/S/epsilon
This takes care of the order of the alphabets as well