I'm trying to determine if there's a database or API available that I could use to get detailed roadway data, meaning the shape and length of roads, intersection points, speed limits, exit numbers for highways, etc.
Obviously Google, MapQuest and others have access to this information and use it to give you driving directions, but I'd like to do something else with the data.
Thanks for any information.
You can use OpenStreetMap database, which contains the mentioned data and much more. You can read more on OpenStreetMap Wiki
Please mind the OpenStreetMap license
Related
I am making a little personal project.
Ideally I would like to be able to make programmatically a google search and have the count of results. (My goal is to compare the results count between a lot (100000+) of different phrases).
Is there a free way to make a web search and compare the popularity of different texts, by using Google Bing or whatever (the source is not really important).
I tried Google but seems that freely I can do only 10 requests per day.
Bing is more permissive (5000 free requests per month).
Is there other tools or way to have a count of number of results for a particular sentence freely ?
Thanks in advance.
There are several things you're going to need if you're seeking to create a simple search engine.
First of all you should read and understand where the field of information retrieval started with G. Salton's paper or at least read the wiki page on the vector space model. It will require you learning at least some undergraduate linear algebra. I suggest Gilbert Strang's MIT video lectures for this.
You can then move to the Brin/Page Pagerank paper which outlays the original concept behind the hyperlink matrix and quickly calculating eigenvectors for ranking or read the wiki page.
You may also be interested in looking at the code for Apache Lucene
To get into contemporary search algorithm techniques you need calculus and regression analysis to learn machine learning and deep learning as the current google search has moved away from Pagerank and utilizes these. This is partially due to how link farming enabled people to artificially engineer search results and the huge amount of meta data that modern browsers and web servers allow to be collected.
EDIT:
For the webcrawler only portion I'd recommend WebSPHINX. I used this in my senior research in college in conjunction with Lucene.
I have come across a problem that I 've never solved before but I find it frequently implemented in various apps so I would like to ask if there is a common way to solve it. I have a set of analytics data each representing some logging action (i.e. info, warn etc). Each of this items has a location and a type (i.e. action). There can be millions of these items per area (depending on the area size or map zoom).
I am looking for the best way to store this set of data in my database. I am very comfortable with SQL Server but dont mind what db I have to use as long as it can handle the scalability requirements. If Amazon WS offers such a product or some other cloud solution then even better cause thats how we are planning to host this app. Google maps will be used to visualize the data.
Some requirements:
Be able to plot all data for a given map rectangle (a common google
map interface with markers representing the logging actions)
Be able to zoom in/zoom out and get relevant data for the new map rectangle
Be able to "group" markers in one bigger marker if data are very close. For instance, if point A is 1 km away from point B and I am seeing a map of 10 km radius then I should see two independent points, A and B. But if I zoom out to 500 km radius then point A and B are too close to each other so I would like to group them in one marker. Hopefully that's possible.
If SQL Server is not a good solution then a free, very cheap or cloud-based storage solution should be recommended (no I cant afford an Oracle).
All the queries above should be able to come back within milliseconds or somehow to be cached. Queries will be of the kind: Get me all analytics data for the given map window with zoom of the given rectangle latitude/longitude.
Thanks,
Yannis
If I undertsood correctly, there are 2 sides to your question:
1- A Database System, that supports storage and lookup of spatial data. Many of the free/open source RDBMS have spatial extensions: MySQL and Postgres (PostGIS) in particular. Spatial data are stored like any other data with the addition of spatial geometry attribute, which describes the shape of your data instance (point, rectangle, polygon, ellipse, ...). You can query spatial data entities, with spatial filters. And of course, spatial queries support joints and unions and almost all kind of SQL constructs.
2- A client/server api that would support rendering of spatial data (with the usual functions such as zoom in, zoom out, pan, etc.), caching and drill-down. As far as I know, there isn't one api that support all these features together, out the box. But there are some interesting apis that you might want to investigate.
Hope this helps.
I'm looking APIs to get some geographical information. Particularly, I'm interested in GIS layers that depict some semantics like roads, vegetation, buildings. If there is an API to get the type of vegetation, it'd be awesome.
Thanks.
I am not exactly sure what you mean. But if you are looking to download GIS data, I recommend using Quantum GIS (open source - http://qgis.org/). Then use the OpenLayers or OpenStreetMap plugin to download data such as roads, buildings, and possibly vegetation. It is going to depend on how much information exists on that area.
What is the approach for storing the latitudes and longitudes for multiple addresses as a one time set up. I need to find the nearby stores using Google Maps and I have to get the latitudes and longitudes of all the available stores. As the data is huge and may increase or change in future, can anyone suggest an approach taking performance and maintenance into consideration.
Thank you.
Can you consider a GIS aware/spatially aware database? Many are these days, but some are specialized for GIS. That ought to make your life easier.
I'm just starting to take an interest in visualization and I'd like to know where I can get my hands on some data, preferably real world, to see what queries and graphics I can draw from it. Its more of a personal exercise to create some pretty looking representations of that data.
After seeing this I wondered where the data came from and what else could be done from Wikipedia. Is there anyway I can obtain data from say, wikipedia?
Also, could anyone recommend any good books? I don't trust the user reviews on the amazon website :-)
You can download the raw Wikipedia data from http://download.wikimedia.org. There are many different views of the data available. The English Wikipedia is by far the largest database, and there isn't a current full dump available, but one is in progress. It will probably take months to finish and be available for download.
The most recent one was 18 GB compressed, which uncompressed to something like 2.5 TB.
A fantastic book is The Visual Display of Quantitative Information by Edward Tufte.