I'm trying to map google phone history locations on to a map using holoviews, datashader and bokeh. Mostly very similar to the examples given in the datashader website. But when I do the map overlay doesn't work as the lat/long gets mangled up.
import datashader as ds
import geoviews as gv
import holoviews as hv
from holoviews.operation.datashader import datashade, dynspread
from datashader import transfer_functions as tf
from colorcet import fire
hv.extension('bokeh')
> df2.head()
lat long
0 -37.7997515 144.9636466
1 -37.7997515 144.9636466
2 -37.7997369 144.9636036
3 -37.7997387 144.9636358
4 -37.7997515 144.9636466
This works to produce an image of the data,
ds_viz = ds.Canvas().points(df2,'lat','long')
tf.set_background(tf.shade(ds_viz, cmap=fire),"black")
However when I try to overlay it with a map it doesn't work,
from bokeh.models import WMTSTileSource
url = 'https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{Z}/{Y}/{X}.jpg'
tile_opts = dict(width=1000,height=600,bgcolor='black',show_grid=False)
map_tiles = gv.WMTS(url).opts(style=dict(alpha=0.5), plot=tile_opts)
points = hv.Points(df2, kdims=['long','lat'])
trips = datashade(points, cmap=fire,width=1000, height=600)
map_tiles * trips
What am I doing wrong?
It looks like your points are in lon,lat but your map is in Web Mercator coordinates, so you need to project your points into Web Mercator before you overlay them. GeoViews offers comprehensive support for projections, but for this specific case Datashader provides the special-purpose function datashader.utils.lnglat_to_meters. Something like this should work:
df2.loc[:, 'lon'], df.loc[:, 'lat'] = lnglat_to_meters(df2.lon,df2.lat)
Projecting can be slow, so you may want to save the resulting df2 to a Parquet file so that you only have to do it once.
Related
Summary
I am building a real-time dashboard that plots latitude and longitude using st.map(df).
Steps to reproduce
I was following this example here.
Code snippet:
import streamlit as st
import pandas as pd
import numpy as np
data = [[-33.71205471, 29.19017682], [-33.81205471, 29.11017682], [-34.71205471, 29.49017682]]
df = pd.DataFrame(data, columns=['Latitude', 'Longitude'])
st.map(df)
Actual behavior:
The plot works as intended but the data points are too large to distinguish between movements in latititude and longitude. Movements in lat and long have a 7 decimal place granularity (e.g. lat=166.1577634).
Expected behavior:
Here is an example in AWS QuickSight of how the points should look.
Any ideas on how to reduce the size of the map circles for each respective data point?
Thanks!
The code below when run in jupyter notebook renders a table with a colour gradient format that I would like to export to an image file.
The resulting 'styled_table' object that notebook renders is a pandas.io.formats.style.Styler type.
I have not been able to find a way to export the Styler to an image.
I hope someone can share a working example of an export, or give me some pointers.
import pandas as pd
import seaborn as sns
data = {('count', 's25'):
{('2017-08-11', 'Friday'): 88.0,
('2017-08-12', 'Saturday'): 90.0,
('2017-08-13', 'Sunday'): 93.0},
('count', 's67'):
{('2017-08-11', 'Friday'): 404.0,
('2017-08-12', 'Saturday'): 413.0,
('2017-08-13', 'Sunday'): 422.0},
('count', 's74'):
{('2017-08-11', 'Friday'): 203.0,
('2017-08-12', 'Saturday'): 227.0,
('2017-08-13', 'Sunday'): 265.0},
('count', 's79'):
{('2017-08-11', 'Friday'): 53.0,
('2017-08-12', 'Saturday'): 53.0,
('2017-08-13', 'Sunday'): 53.0}}
table = pd.DataFrame.from_dict(data)
table.sort_index(ascending=False, inplace=True)
cm = sns.light_palette("seagreen", as_cmap=True)
styled_table = table.style.background_gradient(cmap=cm)
styled_table
As mentioned in the comments, you can use the render property to obtain an HTML of the styled table:
html = styled_table.render()
You can then use a package that converts html to an image. For example, IMGKit: Python library of HTML to IMG wrapper. Bear in mind that this solution requires the installation of wkhtmltopdf, a command line tool to render HTML into PDF and various image formats. It is all described in the IMGKit page.
Once you have that, the rest is straightforward:
import imgkit
imgkit.from_string(html, 'styled_table.png')
You can use dexplo's dataframe_image from https://github.com/dexplo/dataframe_image. After installing the package, it also lets you save styler objects as images like in this example from the README:
import numpy as np
import pandas as pd
import dataframe_image as dfi
df = pd.DataFrame(np.random.rand(6,4))
df_styled = df.style.background_gradient()
dfi.export(df_styled, 'df_styled.png')
I have a folium.Map that contains custom HTML Popups with clickable URLs. These Popups open when clicking on the polygons of the map. This is a feature that doesn't seem to be possible to achieve using holoviews.
My ideal example of the final application that I want to build with holoviews/geoviews is here with the source code here, but I would like to exchange the main map with my folium Map and plot polygons instead of rasterized points. Now when I would like to create the holoviews.DynamicMap from the folium.Map, holoviews complains (of course) that the data type "map" is not accepted. Is this somehow still possible?
I have found some notebook on GitHub where a holoviews plot in embedded in a folium map using a workaround that writes and reads again HTML, but it seems impossible to embed a folium map into holoviews such that other plots can be updated from this figure using Streams!?
Here is some toy data (from here) for the datasets that I use. For simplicity, let's assume I just had point data instead of polygons:
import folium as fn
def make_map():
m = fm.Map(location=[20.59,78.96], zoom_start=5)
green_p1 = fm.map.FeatureGroup()
green_p1.add_child(
fm.CircleMarker(
[row.Latitude, row.Longitude],
radius=10,
fill=True,
fill_color=fill_color,
fill_opacity=0.7
)
)
map.add_child(green_p1)
return map
If I understand it correctly, this needs to be tweaked now in the fashion that it can passed as the first argument to a holoviews.DynamicMap:
hv.DynamicMap(make_map, streams=my_streams)
where my_streams are some other plots that should be updated with the extent of the folium map.
Is that somehow possible or is my strategy wrong?
I have created geo-dataframe using a combination of geopandas and geoviews. Libraries I'm using are below:
import pandas as pd
import numpy as np
import geopandas as gpd
import holoviews as hv
import geoviews as gv
import matplotlib.pyplot as plt
import matplotlib
import panel as pn
from cartopy import crs
gv.extension('bokeh')
I have concatenated 3 shapefiles to build a polygon picture of UK healthcare boundaries (links to files provided if needed). Unfortunately, from what i have found the UK doesn't produce one file that combines all of those, so have had to merge the shape files from the 3 individual countries i'm interested in. The 3 shape files have a size of:
shape file 1 = 5mb (https://www.opendatani.gov.uk/dataset/department-of-health-trust-boundaries)
shape file 2 = 204kb (https://geoportal.statistics.gov.uk/datasets/5252644ec26e4bffadf9d3661eef4826_4)
shape file 3 = 22kb (https://data.gov.uk/dataset/31ab16a2-22da-40d5-b5f0-625bafd76389/local-health-boards-december-2016-ultra-generalised-clipped-boundaries-in-wales)
I have merged them all successfully to build the picture i am looking for using:
Test = gv.Polygons(Merged_Shapes, vdims=[('Data'), ('CCG_Name')], crs=crs.OSGB()).options(tools=['hover'], width=550, height=700)
Test_2 = gv.Polygons(Merged_Shapes, vdims=[('Data'), ('CCG_Name')], crs=crs.OSGB()).options(tools=['hover'], width=550, height=700)
However, I would like to include these charts in a shareable html file. The issue I'm running into, is that when I save the HTML using:
from bokeh.resources import INLINE
layout = hv.Layout(Test + Test_2)
Final_report = pn.Tabs(('Test',layout)).save('Map_test.html', resources=INLINE)
I generate a html file that displays the charts, but the size is 80mb, which is far to large, especially if I want include more polygon charts and other charts in the same html.
Does anyone know of a more efficient way, from a memory perspective, I can store my polygon charts within a HTML file for sharing?
You can make the file smaller by rasterizng or by decimating the shapes. For rasterizng you can call hv.operation.datashader.rasterize(obj), and I think there is something in Shapely or GeoPandas for simplifying the shapes.
I want to know given a latitude and longitude if a coordinate is land or sea
According to https://gis.stackexchange.com/questions/235133/checking-if-a-geocoordinate-point-is-land-or-ocean
from mpl_toolkits.basemap import Basemap
bm = Basemap() # default: projection='cyl'
print bm.is_land(99.675, 13.104) #True
print bm.is_land(100.539, 13.104) #False
The problem is that basemap is deprecated. how di perform this with cartopy?
A question which deals with point containment testing of country geometries using cartopy can be found at Polygon containment test in matplotlib artist.
Cartopy has the tools to achieve this, but there is no built-in method such as "is_land". Instead, you need to get hold of the appropriate geometry data, and query that using standard shapely predicates.
import cartopy.io.shapereader as shpreader
import shapely.geometry as sgeom
from shapely.ops import unary_union
from shapely.prepared import prep
land_shp_fname = shpreader.natural_earth(resolution='50m',
category='physical', name='land')
land_geom = unary_union(list(shpreader.Reader(land_shp_fname).geometries()))
land = prep(land_geom)
def is_land(x, y):
return land.contains(sgeom.Point(x, y))
This gives the expected results for two sample points:
>>> print(is_land(0, 0))
False
>>> print(is_land(0, 10))
True
If you have access to it, fiona will make this easier (and snappier):
import fiona
import cartopy.io.shapereader as shpreader
import shapely.geometry as sgeom
from shapely.prepared import prep
geoms = fiona.open(
shpreader.natural_earth(resolution='50m',
category='physical', name='land'))
land_geom = sgeom.MultiPolygon([sgeom.shape(geom['geometry'])
for geom in geoms])
land = prep(land_geom)
Finally, I produced (back in 2011) the shapely.vectorized functionality to speed up this kind of operation when testing many points at the same time. The code is available as a gist at https://gist.github.com/pelson/9785576, and produces the following proof-of-concept for testing land containment for the UK:
Another tool you may be interested in reading about is geopandas, as this kind of containment testing is one of its core capabilities.