Related
I want to code the Metaphone 3 algorithm myself. Is there a description? I know the source code is available for sale but that is not what I am looking for.
Since the author (Lawrence Philips) decided to commercialize the algorithm itself it is more than likely that you will not find description. The good place to ask would be the mailing list: https://lists.sourceforge.net/lists/listinfo/aspell-metaphone
but you can also checkout source code (i.e. the code comments) in order to understand how algorithm works:
http://code.google.com/p/google-refine/source/browse/trunk/main/src/com/google/refine/clustering/binning/Metaphone3.java?r=2029
From Wikipedia, the Metaphone algorithm is
Metaphone is a phonetic algorithm, an algorithm published in 1990 for indexing words by their English pronunciation. It fundamentally improves on the Soundex algorithm by using information about variations and inconsistencies in English spelling and pronunciation to produce a more accurate encoding, which does a better job of matching words and names which sound similar [...]
Metaphone 3 specifically
[...] achieves an accuracy of approximately 99% for English words, non-English words familiar to Americans, and first names and family names commonly found in the United States, having been developed according to modern engineering standards against a test harness of prepared correct encodings.
The overview of the algorithm is:
The Metaphone algorithm operates by first removing non-English letters and characters from the word being processed. Next, all vowels are also discarded unless the word begins with an initial vowel in which case all vowels except the initial one are discarded. Finally all consonents and groups of consonents are mapped to their Metaphone code. The rules for grouping consonants and groups thereof then mapping to metaphone codes are fairly complicated; for a full list of these conversions check out the comments in the source code section.
Now, onto your real question:
If you are interested in the specifics of the Metaphone 3 algorithm, I think you are out of luck (short of buying the source code, understanding it and re-creating it on your own): the whole point of not making the algorithm (of which the source you can buy is an instance) public is that you cannot recreate it without paying the author for their development effort (providing the "precise algorithm" you are looking for is equivalent to providing the actual code itself). Consider the above quotes: the development of the algorithm involved a "test harness of [...] encodings". Unless you happen to have such test harness or are able to create one, you will not be able to replicate the algorithm.
On the other hand, implementations of the first two iterations (Metaphone and Double Metaphone) are freely available (the above Wikipedia link contains a score of links to implementations in various languages for both), which means you have a good starting point in understanding what the algorithm is about exactly, then improve on it as you see fit (e.g. by creating and using an appropriate test harness).
The link by #Bo now refers to (now defucnt) project entire source code.
Hence here is the new link with direct link to Source code for Metaphone 3
https://searchcode.com/codesearch/view/2366000/
by Lawrence Philips
Metaphone 3 is designed to return an approximate phonetic key
(and an alternate * approximate phonetic key when appropriate) that
should be the same for English * words, and most names familiar in
the United States, that are pronounced similarly. * The key value
is not intended to be an exact phonetic, or even phonemic, *
representation of the word. This is because a certain degree of
'fuzziness' has * proven to be useful in compensating for variations
in pronunciation, as well as * misheard pronunciations. For example,
although americans are not usually aware of it, * the letter 's' is
normally pronounced 'z' at the end of words such as "sounds".
The 'approximate' aspect of the encoding is implemented according to the following rules: * * (1) All vowels are
encoded to the same value - 'A'. If the parameter encodeVowels * is
set to false, only initial vowels will be encoded at all. If
encodeVowels is set * to true, 'A' will be encoded at all places in
the word that any vowels are normally * pronounced. 'W' as well as
'Y' are treated as vowels. Although there are differences in * the
pronunciation of 'W' and 'Y' in different circumstances that lead to
their being * classified as vowels under some circumstances and as
consonants in others, for the purposes * of the 'fuzziness' component
of the Soundex and Metaphone family of algorithms they will * be
always be treated here as vowels. * * (2) Voiced and
un-voiced consonant pairs are mapped to the same encoded value. This
means that: * 'D' and 'T' -> 'T' * 'B' and 'P' -> 'P' * 'G' and 'K' -> 'K' * 'Z' and 'S' -> 'S' * 'V' and 'F' -> 'F' * * - In addition to the above voiced/unvoiced rules,
'CH' and 'SH' -> 'X', where 'X' * represents the "-SH-" and "-CH-"
sounds in Metaphone 3 encoding.
I thought it is wrong to have the general community be denied an algorithm (not code)
I am selling source, so the algorithm is not hidden. I am asking $40.00 for a copy of the source code, and asking other people who are charging for their software or services that use Metaphone 3 to pay me a licensing fee, and also asking that the source code not be distributed by other people (except for an exception I made for Google Refine - i can only request that you do not redistribute the copy of Metaphone 3 found there separately from the Refine package.)
Actually Metaphone3 is an algorithm with many very specific rules being a result of some test cases analysis. So it's not only a pure algorithm but it comes with extra domain knowledge. To obtain these knowledge and specific rules the author needed to put in a great effort. That's why this algorithm is not open-source.
There is an alternative anyway which is open-source: Double Metaphone.
See here: https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/language/DoubleMetaphone.html
This is not a commercial post and I have no relationship with the owner but it is worth saying that an implementation of Metaphone3 is available as commercial software from its creator amporphics.com. It looks like his personal store. It is a Java app but I bought the Python version and it works fine.
The Why Metaphone3? page says:
One common solution to spelling variation is the database approach.
Some very impressive work has been done accumulating personal name
variations from all over the world. (Of course, we are always very
pleased when the companies that retail these databases advertise that
they also use some version of Metaphone to improve their flexibility
:-) )
But - there are some problems with this approach:
They only work well until they encounter a spelling variation or a
new word or name that is not already in their database.
Then they don't work at all.
Metaphone 3 is an algorithmic approach that will deliver a phonetic
lookup key for anything you enter into it.
Personal names, that is, first names and family names, are not the
same as company names. In fact, the name of a company or agency may
contain words of any kind, not just names. Database solutions usually
don't cover possible spelling variations, or for that matter
misspellings, for regular 'dictionary' words. Or if they do, not very
thoroughly.
Metaphone 3 was developed to account for all spelling variations
commonly found in English words, first and last names found in the
United States and Europe, and non-English words whose native
pronunciations are familiar to Americans. It doesnt care what kind of
a word you are trying to match.
For what it is worth, we licensed the code since it is affordable and it is easy to use. I can't speak as to performance yet. There are good alternatives on PyPi but I can't find them at the moment.
I have Persons table in SQL Server 2008.
My goal is to find Persons who have almost similar addresses.
The address is described with columns state, town, street, house, apartment, postcode and phone.
Due to some specific differences in some states (not US) and human factor (mistakes in addresses etc.), address is not filled in the same pattern.
Most common mistakes in addresses
Case sensitivity
Someone wrote "apt.", another one "apartment" or "ap." (although addresses aren't written in English)
Spaces, dots, commas
Differences in writing street names, like 'Dr. Jones str." or "Doctor Jones street" or "D. Jon. st." or "Dr Jones st" etc.
The main problem is that data isn't in the same pattern, so it's really difficult to find similar addresses.
Is there any algorithm for this kind of issue?
Thanks in advance.
UPDATE
As I mentioned address is separated into different columns. Should I generate a string concatenating columns or do your steps for each column?
I assume I shouldn't concatenate columns, but if I'll compare columns separately how should I organize it? Should I find similarities for each column an union them or intersect or anything else?
Should I have some statistics collecting or some kind of educating algorithm?
Suggest approaching it thus:
Create word-level n-grams (a trigram/4-gram might do it) from the various entries
Do a many x many comparison for string comparison and cluster them by string distance. Someone suggested Levenshtein; there are better ones for this kind of task, Jaro-Winkler Distance and Smith-Waterman work better. A libraryt such as SimMetrics would make life a lot easier
Once you have clusters of n-grams, you can resolve the whole string using the constituent subgrams i.e. D.Jones St => Davy Jones St. => DJones St.
Should not be too hard, this is an all-too-common problem.
Update: Based on your update above, here are the suggested steps
Catenate your columns into a single string, perhaps create a db "view" . For example,
create view vwAddress
as
select top 10000
state town, street, house, apartment, postcode,
state+ town+ street+ house+ apartment+ postcode as Address
from ...
Write a separate application (say in Java or C#/VB.NET) and Use an algorithm like JaroWinkler to estimate the string distance for the combined address, to create a many x many comparison. and write into a separate table
address1 | address n | similarity
You can use Simmetrics to get the similarity thus:
JaroWinnkler objJw = new JaroWinkler()
double sim = objJw.GetSimilarity (address1, addres n);
You could also trigram it so that an address such as "1 Jones Street, Sometown, SomeCountry" becomes "1 Jones Street", "Jones Street Sometown", and so on....
and compare the trigrams. (or even 4-grams) for higher accuracy.
Finally you can order by similarity to get a cluster of most similar addresses and decide an approprite threshold. Not sure why you are stuck
I would try to do the following:
split up the address in multiple words, get rid of punctuation at the same time
check all the words for patterns that are typically written differently and replace them with a common name (e.g. replace apartment, ap., ... by apt, replace Doctor by Dr., ...)
put all the words back in one string alphabetically sorted
compare all the addresses using a fuzzy string comparison algorithm, e.g. Levenshtein
tweak the parameters of the Levenshtein algorithm (e.g. you want to allow more differences on longer strings)
finally do a manual check of the strings
Of course, the solution to keep your data 'in shape' is to have explicit fields for each of your characteristics in your database. Otherwise, you will end up doing this exercise every few months.
The main problem I see here is to exactly define equality.
Even if someone writes Jon. and another Jone. - you will never be able to say if they are the same. (Jon-Jonethan,Joneson,Jonedoe whatever ;)
I work in a firm where we have to handle exact this problem - I'm afraid I have to tell you this kind of checking the adress lists for navigation systems is done "by hand" most of the time. Abbrevations are sometimes context dependend, and there are other things that make this difficult. Ofc replacing string etc is done with python - but telling you the MEANING of such an abbr. can only done by script in a few cases. ("St." -> Can be "Saint" and "Street". How to decide? impossible...this is human work.).
Another big problem is as you said "Is there a street "DJones" or a person? Or both? Which one is ment here? Is this DJones the same as Dr Jones or the same as Don Jones? Its impossible to decide!
You can do some work with lists as presented by another answer here - but it will give you enough "false positives" or so.
You have a postcode field!!!
So, why don't you just buy a postcode table for your country
and use that to clean up your street/town/region/province information?
I did a project like this in the last centuary. Basicly it was a consolidation of two customer files after a merger, and, involved names and addresses from three different sources.
Firstly as many posters have suggested, convert all the common words and abbreveations and spelling mistakes to a common form "Apt." "Apatment" etc. to "Apt".
Then look through the name and identifiy the first letter of the first name, plus the first surname. (Not that easy consider "Dr. Med. Sir Henry de Baskerville Smythe") but dont worry where there are amiguities just take both! So if you lucky you get HBASKERVILLE and HSMYTHE. Now get rid of all the vowels as thats where most spelling variations occur so now you have HBSKRVLL HSMTH.
You would also get these strings from "H. Baskerville","Sir Henry Baskerville Smith" and unfortunately "Harold Smith" but we are talking fuzzy matching here!
Perform a similar exercise on the street, and apartment and postcode fields. But do not throw away the original data!
You now come to the interesting bit first you compare each of the original strings and give say 50 points for each string that matches exactly. Then go through you "normalised" strings and give say 20 points for each one that matches exactly. Then go through all the strings and give say 5 points for each four character or more substring they have in common. For each pair compared you will end up with some with scores > 150 which you can consider as a certain match, some with scores less than 50 which you can consider not matched and some inbetween which have some probability of matching.
You need some more tweaking to improve this by adding various rules like "subtract 20 points for a surname of 'smith'". You really have to keep running and tweaking until you get happy with the resulting matches, but, once you look at the results you get a pretty good feel which score to consider a "match" and which are the false positives you need to get rid of.
I think the amount of data could affect what approach works best for you.
I had a similar problem when indexing music from compilation albums with various artists. Sometimes the artist came first, sometimes the song name, with various separator styles.
What I did was to count the number of occurrences on other entries with the same value to make an educated guess wether it was the song name or an artist.
Perhaps you can use soundex or similar algorithm to find stuff that are similar.
EDIT: (maybe I should clarify that I assumed that artist names were more likely to be more frequently reoccurring than song names.)
One important thing that you mention in the comments is that you are going to do this interactively.
This allows to parse user input and also at the same time validate guesses on any abbreviations and to correct a lot of mistakes (the way for example phone number entry works some contact management systems - the system does the best effort to parse and correct the country code, area code and the number, but ultimately the user is presented with the guess and has the chance to correct the input)
If you want to do it really good then keeping database/dictionaries of postcodes, towns, streets, abbreviations and their variations can improve data validation and pre-processing.
So, at least you would have fully qualified address. If you can do this for all the input you will have all the data categorized and matches can then be strict on certain field and less strict on others, with matching score calculated according weights you assign.
After you have consistently pre-processed the input then n-grams should be able to find similar addresses.
Have you looked at SQL Server Integration Services for this? The Fuzzy Lookup component allows you to find 'Near matches': http://msdn.microsoft.com/en-us/library/ms137786.aspx
For new input, you could call the package from .Net code, passing the value row to be checked as a set of parameters, you'd probably need to persist the token index for this to be fast enough for user interaction though.
There's an example of address matching here: http://msdn.microsoft.com/en-us/magazine/cc163731.aspx
I'm assuming that response time is not critical and that the problem is finding an existing address in a database, not merging duplicates. I'm also assuming the database contains a large number of addresses (say 3 million), rather than a number that could be cleaned up economically by hand or by Amazon's Mechanical Turk.
Pre-computation - Identify address fragments with high information content.
Identify all the unique words used in each database field and count their occurrences.
Eliminate very common words and abbreviations. (Street, st., appt, apt, etc.)
When presented with an input address,
Identify the most unique word and search (Street LIKE '%Jones%') for existing addresses containing those words.
Use the pre-computed statistics to estimate how many addresses will be in the results set
If the estimated results set is too large, select the second-most unique word and combine it in the search (Street LIKE '%Jones%' AND Town LIKE '%Anytown%')
If the estimated results set is too small, select the second-most unique word and combine it in the search (Street LIKE '%Aardvark%' OR Town LIKE '%Anytown')
if the actual results set is too large/small, repeat the query adding further terms as before.
The idea is to find enough fragments with high information content in the address which can be searched for to give a reasonable number of alternatives, rather than to find the most optimal match. For more tolerance to misspelling, trigrams, tetra-grams or soundex codes could be used instead of words.
Obviously if you have lists of actual states / towns / streets then some data clean-up could take place both in the database and in the search address. (I'm very surprised the Armenian postal service does not make such a list available, but I know that some postal services charge excessive amounts for this information. )
As a practical matter, most systems I see in use try to look up people's accounts by their phone number if possible: obviously whether that is a practical solution depends upon the nature of the data and its accuracy.
(Also consider the lateral-thinking approach: could you find a mail-order mail-list broker company which will clean up your database for you? They might even be willing to pay you for use of the addresses.)
I've found a great article.
Adding some dlls as sql user-defined functions we can use string comparison algorithms using SimMetrics library.
Check it
http://anastasiosyal.com/archive/2009/01/11/18.aspx
the possibilities of such variations are countless and even if such an algorithm exists, it can never be fool-proof. u can't have a spell checker for nouns after all.
what you can do is provide a drop-down list of previously entered field values, so that they can select one, if a particular name already exists.
its better to have separate fields for each value like apartments and so on.
You could throw all addresses at a web service like Google Maps (I don't know whether this one is suitable, though) and see whether they come up with identical GPS coordinates.
One method could be to apply the Levenshtein distance algorithm to the address fields. This will allow you to compare the strings for similarity.
Edit
After looking at the kinds of address differences you are dealing with, this may not be helpful after all.
Another idea is to use learning. For example you could learn, for each abbreviation and its place in the sentence, what the abbreviation means.
3 Jane Dr. -> Dr (in 3rd position (or last)) means Drive
Dr. Jones St -> Dr (in 1st position) means Doctor
You could, for example, use decision trees and have a user train the system. Probably few examples of each use would be enough. You wouldn't classify single-letter abbreviations like D.Jones that could be David Jones, or Dr. Jones as likely. But after a first level of translation you could look up a street index of the town and see if you can expand the D. into a street name.
Again, you would run each address through the decision tree before storing it.
It feels like there should be some commercial products doing this out there.
A possibility is to have a dictionary table in the database that maps all the variants to the 'proper' version of the word:
*Value* | *Meaning*
Apt. | Apartment
Ap. | Apartment
St. | Street
Then you run each word through the dictionary before you compare.
Edit: this alone is too naive to be practical (see comment).
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
I am a programmer and need a practical approach to storing street address structures of the world in a database. So which is the best and common database design for storing street addresses? It should be simple to use, fast to query and dynamic to store all street addresses of the world.
It is possible to represent addresses from lots of different countries in a standard set of fields. The basic idea of a named access route (thoroughfare) which the named or numbered buildings are located on is fairly standard, except in China sometimes. Other near universal concepts include: naming the settlement (city/town/village), which can be generically referred to as a locality; naming the region and assigning an alphanumeric postcode. Note that postcodes, also known as zip codes, are purely numeric only in some countries. You will need lots of fields if you really want to be generic.
The Universal Postal Union (UPU) provides address data for lots of countries in a standard format. Note that the UPU format holds all addresses (down to the available field precision) for a whole country, it is therefore relational. If storing customer addresses, where only a small fraction of all possible addresses will be stored, its better to use a single table (or flat format) containing all fields and one address per row.
A reasonable format for storing addresses would be as follows:
Address Lines 1-4
Locality
Region
Postcode (or zipcode)
Country
Address lines 1-4 can hold components such as:
Building
Sub-Building
Premise number (house number)
Premise Range
Thoroughfare
Sub-Thoroughfare
Double-Dependent Locality
Sub-Locality
Frequently only 3 address lines are used, but this is often insufficient. It is of course possible to require more lines to represent all addresses in the official format, but commas can always be used as line separators, meaning the information can still be captured.
Usually analysis of the data would be performed by locality, region, postcode and country and these elements are fairly easy for users to understand when entering data. This is why these elements should be stored as separate fields. However, don't force users to supply postcode or region, they may not be used locally.
Locality can be unclear, particularly the distinction between map locality and postal-locality. The postal locality is the one deemed by a postal authority which may sometimes be a nearby large town. However, the postcode will usually resolve any problems or discrepancies there, to allow correct delivery even if the official post-locality is not used.
Have a look at Database Answers. Specifically, this covers many cases:
(All variable length character datatype)
AddressId
Line1
Line2
Line3
City
ZipOrPostcode
StateProvinceCounty
CountryId
OtherAddressDetails
Ask yourself what is the main purpose of storing this data? Do you intend to actually send mail to the person at the address? Track demographics, populations? Be able to ask callers for their correct address as part of some basic authentication/verification? All of the above? None of the above?
Depending on your actual need, you will determine either a) it doesn't really matter, and you can go for a free-text approach, or b) structured/specific fields for all countries, or c) country specific architecture.
Sometimes the closest you can get to a street address is the city.
I once had a project to put all the Secondary Schools in India in Google Maps. I wrote a spiffy program using the Google API and thought it would be quite easy.
Then I got the data from the client. Some school addresses were things like "Across from the market, next to the barber" or "Near old bus stand".
It made my task much harder since, unfortunately, the Google API does not support that format.
For international addresses, it is remarkably hard to find a way to format the information if it is broken down into fields. As a for instance, an Italian address uses:
<street address>
<zip> <town> <region>
<country>
Such as
Via Eroi della Repubblica
89861 Tropea VV
Italy
That is rather different from the order for US addresses - on the second line.
See also the SO questions:
How many address fields would you use for a UK database?
Do you break up addresses into street / city / state / zip?
How do you deal with duplicate street suffixes?
Best practices for storing postal addresses in a database (RDBMS)?
Also check out tag 'postal-code'.
Edit: Reverse order of region and town - per UPU
Maybe this is useful:
https://gist.github.com/259744
For a project I collected a table of informations about all countries of the world, including ISO codes, top level domain, phone code, car sign, length and regex of zip.
Country names and comments unfortunately only in German...
Differently of other answers here, I believe it's possible to have an structured address database.
Just out of the hat, I can think of the following structure:
Country
Region (State / Province)
Locality (City / Municipality)
Sub-Locality (County / other sub-division of a locality)
Street
But how to query it fast enough?
One way I always think it can be accomplished is to ask for the ZIP Code (or Postal Code) which varies from country to country, but is solid within the country.
This way you can structure your data around the information provided by the postal offices around the world.
Depends on how free-form you are prepared to go with the fields. One free-form address field will obviously always do, but be of relatively little help narrowing down geography.
The problem you'll have is that there is too much variation in the level of geographical hierarchy across countries. Heck, some countries do not even have 'street addresses' everywhere.
I recommend you don't try to make it too clever.
Len Silverston of Universal Data Model fame recommends a separate hierarchy of GEOGRAPHIC BOUNDARIES and depending on how much free-formed-ness you're willing to accept either simple STREET ADDRESS LINEs or per-country derivatives.
No, absolutely not. If you compare the way US and Japanese addresses work, you'll see that it's not possible.
UPDATE:
On second thought, anything can be done, but there's a trade-off.
One approach is to model the problem with address and address_attribute tables, with a 1:m relationship between them, anything can be modeled. The address_attribute table would have a pk, a name, a value, and an fk that points back to its address parent's pk. It's almost like using a Map with name, value pairs.
The trade-off is having to do a JOIN every time you want an address. You also have to interrogate the names of the address_attributes to figure out what you're dealing with each time.
Another approach would be to do more comprehensive research on how addresses are modeled around the world. In an object-oriented world you might have the western Address class (street1/street2/city/state/zip) and others for Japan, China, as many as needed to tile the address space. Then you'd have a master Address table and child tables to the other types with a 1:1 relationship between them.
How does Amazon or eBay do it? They ship internationally. Do they have locale-specific UI features? I've only used the US locale.
No, there are no standard addressing scheme. It usually varies from country-to-country.
Even the Universal Postal Union said on Adressing the world, an address for everyone that there is none. The best solution for this is to use the 2/3-letter country code standards known as ISO 3166 and treat everything else by country's standards.
However, if you really are desperate to use easily accessible tools for your project, you can try Google Place API.
Your design should strongly depend from your purpose. Some people have posted how to structure data. So if you simply want to send s-mail to someone, it will do. Things begin to complicate if you want to use this data for navigation. Car navigation will require additional structures to contain traffic info (eg one-way roads), while foot navigation will require a lot of additional data. Here is small example: in my city, my neighborhood is near the park. Next to the park is former airfield (in fact, one of the oldest in Europe) turned into aviation museum. Next to aviation museum is a business park. Street number for museum is 39, while business park numbers start with 39A. So it may seem that 39 and 39A are close – but it takes about a mile to walk from one to another (and even longer if going by car) .
This is just a small example taken from my city, I think you can probably find a lot of exceptions (especially in rural or wilder parts of every country).
Say I am storing addresses in a DB table, in this fairly common break down:
address_street_line_1,
address_street_line_2,
address_city,
address_state,
address_zip,
address_country_id
(Note: I have read the questions on splitting down further, street type, house number, etc. and for this application I think it would unnecessarily complicate things.)
To work best with international users, which of these fields should NOT be required?
I'm thinking this:
address_street_line_1 REQUIRED
address_city REQUIRED
address_country_id REQUIRED
Should I require state or zip?
Thanks!
Xavier
You can probably only require one field: country.
But what you should really be doing is making the logic dependent on country. Take a look at Address Formats by Country for a comprehensive list. That isn't just about required fields either. It's also about correct formatting. A US address might be:
8031 Main Street
Springfield OH 12345
USA
whereas in Switzerland:
Bodenstr. 173
8043 Zürich
Schweiz
Note: the street numbers and post codes are in the "reverse" order for Switzerland (compared to what English speaking countries use).
Also, your data types need to be broad enough to cover data used in other countries. Zip/post code should absolutely not be a numeric type. For example, "EC2R 8AH" is a valid UK postcode.
That goes back to this principle: if you don't perform arithmetic on it, it's not a numeric type. It's text.
Also, try not to call it Zip Code to end users. That's a US only term. Pretty much everywhere else its call a Postcode, Post code or Postal Code. Also note that the UK postal codes are alphanumeric and include a space.
Not all countries even use postal codes, for example they were rarely used in New Zealand prior to 2006 or so. I think Ireland doesn't use them at all.
If you're truly international, city-states such as Singapore don't actually need a City field.
In the user interface, you can (and perhaps should) make the postcode required for countries where you already know it's required, since that isn't likely to change. And, if you make the UI dynamic enough, you can call it "Zip code" if the selected country is the United States, "Postal code" for Canada, "Postcode" for the UK, etc.
How about making none required? If the user wants to be contacted they'll enter enough information. Or, enter a single text field and let them enter free form information. They know better than you what fields are required for postal deliveries to make it to their door.
I would say everything except street_line_2 and state- and think of 'zip' as more of a postal codes instead of zip code - as you can tell from the variety of format based on the country of origin, this should have a pretty open format.
Even in the U.S., most of the address is not required. A large fraction of U.S. zip codes are allocated to various businesses and organizations - any mail to one of those zips will be delivered the same regardless of the rest of the address. For instance:
General Electric
Schenectady, NY 12345
Internal Revenue Service
Ogden, UT 84201-0027
The city and state are nice, but the mail will probably get delivered without.
The best way that I have found to solve this problem is by abstracting the logic in your application layer, and not the persistence layer. One of the cleanest/simplest ways I've seen this done is by passing the user's data in a value object (creating a common interface that's easy to validate against) to a validator with the current country code, which makes sure all the required attributes are set properly in the value object for that locale. Assuming it passes validation, pass the value object along to the persistence side of your application for storage.
The key here is the value object - you're creating a common interface that multiple pieces of your application can talk to, validate, and read/write from. You can then also use that same value object when displaying the address: have your persistence layer get the information, put it in the value object, pass it to a factory with the current locale which returns the desired address format, and send that output to the front end.
There are no states in New Zealand, so it should definately be optional. So I think you have the right answer in your question.
If you are not going to do any specific lookup, like searching by postal code or by city, I'd say to all combine the address in a single field. This way you will support the different address from different countries.
You will also support address oddities.
If you fear that the requirements are going to change, you could store the address as a Xml field. Modern database like Sql Server 2005 and 2008 can have an index on a Xml node inside a Xml column as long as you are using a schema.
It all come down to requirements. If the client need to group the data inside a grid by country, then you need a country column.
Making fields required is always a tradeoff. If the person doesn't want to fill in the info then they won't -- they'll put in a period, or garbage to get past the "required field" nanny.
I only require street_address_1 in my apps. Also, for the US and many countries, you can buy the mapping between the postal/zip code and the canonical city/state. It's not expensive. (The mapping between individual street addresses and zip is much more expensive.)
For the US, see http://www.usps.com/ncsc/addressinfo/citystate.htm
If you're including an Ajax web interface, ask for the country first, then the post code. If in the US, then use Ajax to fetch and fill in the city/state for the user from the zip.
Some non-US countries, eg UK, can have 3 lines of street addresses if you're asking people to fill in their "preferred address" Eg:
Mirassou (You can register a building's name with the post office
High Street as an alternative to its street number)
Old Town
City, Bucks postal_code
Larry
Actually, city isn't even required in the US.
Many people have rural addresses on state and county roads. Publication 28 at the postal service web site has details. Different companies end up using the "city" field to store other information. This also applies to military base addresses.
Publication 28 link
I'm working on a company search API using Lucene.
My Lucene company index has got 2 companies:
1.Abigail Adams National Bancorp, Inc.
2.National Bancorp
If the user types in National Bancorp, then only company # 2(ie. National Bancorp) should be returned and not #1.....ie. only exact matches should be returned.
How do I achieve this functionality?
Thanks for reading.
You can use KeywordAnalyzer to index and search on this field. Keyword Analyzer will generate only one token for the entire string.
I googled a lot with no help for the same problem. After scratching my head for a while I found the solution. Search the string within double quotes, that will solve your problem.
National Bancorp will return both #1 and #2 but "National Bancorp" will return only #2.
This is something that may warrant the use of the shingle filter. This filter groups multiple words together. For example, Abigail Adams National Bancorp with a ShingleFilter of 3 tokens would produce (assuming a simple WhitespaceAnalyzer) [Abigail], [Abigail Adams], [Abigail Adams National], [Adams National Bancorp], [Adams National], [Adams], [National], [National Bancorp] and [Bancorp].
If a user the queries for National Bancorp, you will get an exact match on National Bancorp itself, and a lower scored exact match on Abigail Adams National Bancorp (lower scored because this one has much more tokens in the field, thus lowering the idf). I think it makes sense to return both documents on such a query.
You may want to apply the shingle filter at query time as well, depending on the use case.
You may want to reconsider your requirements, depending on whether or not I correctly understood your question. Please bear with me if I did misunderstand you.
Just a little food for thought:
If you only want exact matches returned, then why are you searching in the first place?
Are you sure that the user expects exact matches? I typically search assuming that the search engine will accommodate missing words.
Suppose the user searched for National Bank but National Bank was no longer in your index. Would you still want Abigail Adams National Bancorp, Inc to be excluded from the results simply because it was not an exact match?
In light of this, I would suggest you continue to present all possible matches (exact or not) to the user and let them decide for themselves which is most appropriate for them. I say this simply because you may not be thinking the same way as all of your users. Lucene will take care of making sure the closest matches rank highest in the results, helping them make quicker choices.
I have the same requirements of exact matching. I have used queryBuilder of org.hibernate.search.query.dsl and the query is:
query = queryBuilder.phrase().withSlop(0).onField(field)
.sentence(searchTerm).createQuery();
Its working for me.