Consolidating and enhancing the visualization of CYMH Service Areas in Ontario

Preamble

This article was originally posted on May 14, 2016 and revised on July 28, 2016 to take account of changes in the geospatial representation of Children and Youth Mental Health Service Areas in Ontario. For more details, see … and then there were 33.

In previous posts, we provided the following:

  • method for partitioning the Ontario government’s Shapefile archive of the thirty-four MCYS Children and Youth Mental Health (CYMH) Service Areas into five groupings, corresponding to the MCYS Integrated Service Regions (ISRs)
  • a method for using the TopoJSON files to visualize the CYMH Service Areas within the separate ISRs
  • the addition of a responsive framework to ensure that our visualizations accommodate to the display capabilities of various PCs, laptops, tablets, and smart phones

In this post, we describe how to enhance the functionality of our visualization of the Integrated Service Regions and CYMH Service Areas and Integrated across the entire province of Ontario.

Consolidated Geodata Files for CYMH Service Areas in Ontario

In a previous post, we provided a set of GeoJSON and TopoJSON files for the individual CYMH Service Areas and their groupings into the five distinct ISRs.

For present purposes, we have created two geodata files – in GeoJSON and TopoJSON formats – for the entire set of CYMH Service Areas in Ontario.

Field Property of the CYMH Service Area
area_name Name
id ID
isr Integrated Service Region
color Color

Simple Maps of the CYMH Service Areas

By way of a quick review, let’s start with a simple map outlining the CYMH Service Areas in Ontario:

<!DOCTYPE html>
<meta charset="utf-8">
<style>

/* CSS goes here */

/* define the container element for our map */
#map {
 margin:5% 5%;
 border:2px solid #000;
 border-radius: 5px;
 height:100%;
 overflow:hidden;
 background: #FFF;
}

</style>
<body>

/* create the container element #map */
<div id="map"></div>

/* load Javascript libraries for D3 and TopoJSON */
<script src="//d3js.org/d3.v3.min.js"></script>
<script src= "//d3js.org/topojson.v1.min.js"></script>

<script> // begin Javascript for visualizing the geo data

/* define some global variables */

var topo, projection, path, svg, g;

/* 1. Set the width and height (in pixels) based on the offsetWidth property of the container element #map */

var width = document.getElementById('map').offsetWidth;
var height = width / 2;

/* Call function setup() to create empty root SVG element with width, height of #map */

setup(width,height);

function setup(width,height){

/* 2. Create an empty root SVG element */

svg = d3.select('#map').append('svg')
 .style('height', height + 'px')
 .style('width', width + 'px')
 .append('g');

g = svg.append('g');

} // end setup()

/* 3. Define Unit Projection using Albers equal-area conic projection */

var projection = d3.geo.albers()
 .scale(1)
 .translate([0,0]);

/* 4. Define the path generator - to format the projected 2D geometry for SVG */

var path = d3.geo.path()
 .projection(projection);

/* 5.0 Start function d3.json() */
/* 5.1 Load the TopoJSON data file */

d3.json("http://cartoserve.com/maps/ontario/cymhsas33_data/cymhsas_topo.json", function(error, cymhsas_topo) {
if (error) return console.error(error);

/* 5.2 Convert the TopoJSON data back to GeoJSON format */
/* and render the map using Unit Projection */

var areas_var = topojson.feature(cymhsas_topo, cymhsas_topo.objects.cymhsas_geo);

/* 5.2.1 Calculate new values for scale and translate using bounding box of the service areas */
 
var b = path.bounds(areas_var);
var s = .95 / Math.max((b[1][0] - b[0][0]) / width, (b[1][1] - b[0][1]) / height);
var t = [(width - s * (b[1][0] + b[0][0])) / 2, (height - s * (b[1][1] + b[0][1])) / 2];

/* 5.2.2 New projection, using new values for scale and translate */
projection
 .scale(s)
 .translate(t);

/* redefine areas_var in terms of the .features array, assign array to topo */
var areas_var = topojson.feature(cymhsas_topo, cymhsas_topo.objects.cymhsas_geo).features;

topo = areas_var;

/* make the map by calling our draw() function initially within the d3.json callback function */
 
draw(topo);

}); // end function d3.json


function draw(topo) {

 var service_area = g.selectAll(".area_name").data(topo);

 service_area.enter().insert("path") 
 .attr("class", "service_area")
 .attr("d", path);

} // end function draw()

</script> // end Javascript for visualizing the geo data

Giving us the following visualization of 33 Service Areas [actual web page]:

CYMH Service Areas 33
Figure 1. Basic outline of 33 CYMH Service Areas in Ontario.

Note that the Javascript for creating our map includes three functions:

  • d3.json() – a built-in function to load geo data from a TopoJSON file
  • setup() – our function to create an empty root SVG element
  • draw() – our function to render the geo data

By adding just two lines of code to our draw() function, we can colour the CYMH Service Areas:


...
function draw(topo) {
 var service_area = g.selectAll(".area_name").data(topo);
 service_area.enter().insert("path") 
 .attr("class", "service_area")
 .attr("d", path)
 .attr("id", function(d,i) { return d.id; })
 .style("fill", function(d, i) { return d.properties.color; });

...

Giving us this figure [actual web page]:

CYMH Service Areas 33 colour
Figure 2. Thirty-three CYMH Service Areas in Ontario.

Adding Tooltips

Figures 1 and 2 suffer from one obvious shortcoming: none of the CYMH Service Areas is labelled. Unfortunately, our approach to labeling the CYMH Service Areas within a single Integrated Service Region breaks down when we have to contend with the scale of Ontario taken as a whole:

cymhsas02.html w labels screenshot
Figure 3. CYMH Service Areas cluttered with fixed labels.

Our best alternative is to use a tooltips to display the name of a CYMH Service Area when the user’s mouse hovers over the corresponding area of the visualization. Adding this functionality requires two sorts of modification of our Javascript.

First, we must style the Tooltips, including:

  • styling the <div> container element corresponding to the text box within which the name of the CYMH Service Area will be displayed
  • styling the text that is displayed when the user’s mouse is hovering over a service area
  • styling the tooltip so that it is hidden when the user’s mouse is not hovering over any service area

… like so:

<style>

...

/* style of the text box containing the tooltip */

div.tooltip {
 color: #222; 
 background: #fff; 
 padding: .5em; 
 text-shadow: #f5f5f5 0 1px 0;
 border-radius: 2px; 
 box-shadow: 0px 0px 2px 0px #a6a6a6; 
 opacity: 0.9; 
 position: absolute;
}

/* style of the text displayed in text box of the tooltip when mouse is hovering over a CYMH Service Area */

.service_area:hover{ stroke: #fff; stroke-width: 1.5px; }

.text{ font-size:10px; }

/* otherwise, the text box of the tooltip is hidden */
.hidden { 
 display: none; 
}

</style>

Second, we must modify our draw() function to display/hide tooltips in response to the user’s mousemove and mouseout behaviours:


<script>

var tooltip = d3.select("#map").append("div").attr("class", "tooltip hidden");

...

function draw(topo) {

var service_area = g.selectAll(".area_name").data(topo);

service_area.enter().insert("path")
.attr("class", "service_area")
.attr("d", path)
.attr("id", function(d,i) { return d.id; })
.style("fill", function(d, i) { return d.properties.color; })
.attr("title", function(d,i) { return d.properties.area_name; });

/* define offsets for displaying the tooltips */
var offsetL = document.getElementById('map').offsetLeft+20;
var offsetT = document.getElementById('map').offsetTop+10;

/* toggle display of tooltips in response to user mouse behaviours*/
service_area
//mousemove behaviour
.on("mousemove", function(d,i) {
var mouse = d3.mouse(svg.node()).map( function(d) { return parseInt(d); } );
tooltip.classed("hidden", false)
.attr("style", "left:"+(mouse[0]+offsetL)+"px;top:"+(mouse[1]+offsetT)+"px")
.html(d.properties.area_name);
}) // end mousemove
// mouseout behaviour
.on("mouseout", function(d,i) {
tooltip.classed("hidden", true);
}); // end mouseout

} // end draw()

Giving us this sort of visualization [interactive web page]:

CYMH Service Areas 33 tooltipsr
Figure 4. Thirty-three Colour CYMH Service Areas in Ontario with Tooltips.

Next time: We’ll enhance the functionality of our visualization to allow the user to pan and zoom our map of the CYMH Service Areas in Ontario.

Geospatial Features of the MCYS Integrated Service Regions

Preamble

This article was originally posted on May 14, 2016 and revised on July 28, 2016 to take account of changes in the geospatial representation of Children and Youth Mental Health Service Areas in Ontario. For more details, see … and then there were 33.

In 2014-15, the Ministry of Children and Youth Services (MCYS) defined two administrative views of mental health services for children and youth in Ontario:

The provincial government has published geospatial data for the MCYS’s Integrated Service Regions (ISRs) and Children and Youth Mental Health Service Areas (CYMHSAs) in Shapefile format. Here, we’ll work only with the geospatial data for the ISRs.

Shapefile format

The Shapefile archive for the ISRs (mcys_integrated_regions.zip) contains four files:

  • mcys_integrated_regions.shp — shape format
  • mcys_integrated_regions.shx — shape index format
  • mcys_integrated_regions.dbf — attribute format
  • mcys_integrated_regions.prj — projection format: the coordinate system and projection information, expressed in well-known text format

For ease of reference, we rename the four mcys_integrated_regions.* files isrs.*.

Converting Shapefiles to GeoJSON format

To use d3.geo to visualize the ISRs, we first convert the Shapefiles to GeoJSON format files, using ogr2ogr. There are three steps:

  1. Determine the Spatial Reference System (SRS) used by the Shapefile
  2. Set the -t_srs switch in ogr2ogr to output the GeoJSON file using this SRS
  3. Rename variables for ease of use

The projection format file for the ISRs specifies:

GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137,298.257222101]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]]

We use Prj2EPSG, a simple online service to convert the projection information contained in the .prj file into standard EPSG codes for the corresponding spatial reference system. From this we determine that the specification contained in isrs.prj corresponds to EPSG 4269 – GCS_North_American_1983.

We now use the -t_srs switch in ogr2ogr to transform the output using the spatial reference system specified in isrs.prj:

ogr2ogr -t_srs EPSG:4269 -f GeoJSON isrs_geo.json isrs.shp

The GeoJSON format file is isrs_geo.json.

Finally, we use a text editor to give more meaningful names to a few variables in the GeoJSON files and to express them in all lowercase letters:

Original variable Modified variable
Region region_name
Order order

Converting GeoJSON files to TopoJSON format

We use topojson to convert the GeoJSON file isrs_geo.json to a TopoJSON format file:

topojson -o isrs_topo.json --id-property region_name --properties -- isrs_geo.json

The --id-property switch in topojson is used to promote the feature property "region_name" to geometry id status in the TopoJSON file isrs_topo.json.

We may now visualize the MCYS Integrated Service Regions by applying d3.geo to isrs_topo.json.

Visualizing the MCYS Integrated Service Regions Using d3.geo

Preamble

This article was originally posted on May 14, 2016 and revised on July 28, 2016 to take account of changes in the geospatial representation of Children and Youth Mental Health Service Areas in Ontario. For more details, see … and then there were 33.

Introduction

In the past few years, a wide variety of free and open source software (FOSS) tools for visualizing geospatial data have become available.  For many people like me who work in human services, coming to know how to use these tools even in rudimentary ways represents a steep learning-curve. Here, I illustrate the use of one of these tools, d3.geo, to visualize a geospatial view of children and youth services in Ontario. 1

In 2014-15 the Ministry of Children and Youth Services (MCYS) defined two administrative views of Ontario:

The five Integrated Service Regions (ISRs) combined nine previous Service Delivery Division regions and four Youth Justice Services Division regions. The ISRs are integrated with the five regional boundaries of the Ministry of Community and Social Services (MCSS).

The thirty-four thirty-three Children and Youth Mental Health Service Areas (CYMHSAs) were defined after a thorough review, including an assessment of Statistics Canada’s census divisions and projected population and children and youth.

The Ontario government has published two Shapefile archives – mcys_integrated_regions.zip and cymh_shapefile.zip and cymh_service_areas_after_march_9_2015.zip – that define the geospatial boundaries of the ISRs and the CYMHSAs, respectively. For now, we’re going to work only with the Shapefile archive for the ISRs.

Before we can visualize the ISRs using d3.geo, we need to convert the Shapefile format archive – mcys_integrated_regions.zip - to a TopoJSON format file – isrs_topo.json. (For more details of converting geospatial data from one file format to another, see Geospatial Features of the MCYS Integrated Service Regions).

A Simple Map

So, let’s use d3.geo to visualize the MCYS Integrated Service Regions in Ontario.

HTML template

In the same directory as the isrs_topo.json file, we create a file – isrs01.html – using the following template:


<!DOCTYPE html>
<meta charset="utf-8">
<style>

/* CSS goes here */

</style>
<body>
<script src="//d3js.org/d3.v3.min.js" charset="utf-8"></script>
<script src= "//d3js.org/topojson.v1.min.js"></script>

/*                                                           */
<script>
/* JavaScript for reading and rendering data goes here */
</script>

d3 supports the two main standards for rendering two-dimensional geometry in a browser: SVG and Canvas. We prefer SVG because you can style SVG using CSS, and declarative styling is easier.

There are several steps involved in reading and rendering our data in SVG:

  1. Define the width and height (in pixels) of the canvas
  2. Create an (empty) root SVG element
  3. Define the projection, beginning with a “unit projection” with .scale(1) and .translate([0,0])
  4. Define the path generator
  5. Open the d3.json callback
    1. Load the TopoJSON data file
    2. Convert the TopoJSON data back to GeoJSON format
    3. Calculate new values for .scale() and .translate() to resize and centre the projection
    4. Bind the GeoJSON data to the path element and use selection.attr to set the “d” attribute to the path data
  6. Maybe do some other stuff
  7. Close the d3.json callback

If we modify our template – isrs01.html – by adding the Javascript to load and render isrs_topo.json in SVG, we obtain:


<!DOCTYPE html>
<meta charset="utf-8">
<style>

/* CSS goes here */

</style>
<body>
<script src="//d3js.org/d3.v3.min.js" charset="utf-8"></script>
<script src= "//d3js.org/topojson.v1.min.js"></script>

<script>

/* 1. Set the width and height (in pixels) of the canvas */
var width = 960,
    height = 1160;
 
/* 2. Create an empty root SVG element */

var svg = d3.select("body").append("svg") .attr("width", width) .attr("height", height);

/* 3. Define the Unit projection to project 3D spherical coordinates onto the 2D Cartesian plane - HERE we use the Albers equal-area conic projection. */
var projection = d3.geo.albers()
    .scale(1)
    .translate([0, 0]);

/* 4. Define the path generator - to format the projected 2D geometry for SVG */
var path = d3.geo.path()
    .projection(projection);

/* 5.0 Open the d3.json callback, and
/* 5.1 Load the TopoJSON data file. */

d3.json("isrs_topo.json", function(error, isrs_topo) {
if (error) return console.error(error);

/* 5.2 Convert the TopoJSON data back to GeoJSON format */

  var regions_var = topojson.feature(isrs_topo, isrs_topo.objects.isrs_geo);
/* 5.2.1 Calculate new values for scale and translate using bounding box of the service areas */
 
var b = path.bounds(regions_var);
var s = .95 / Math.max((b[1][0] - b[0][0]) / width, (b[1][1] - b[0][1]) / height);
var t = [(width - s * (b[1][0] + b[0][0])) / 2, (height - s * (b[1][1] + b[0][1])) / 2];

/* 5.2.2 New projection, using new values for scale and translate */
projection
   .scale(s)
   .translate(t);

/* 5.3 Bind the GeoJSON data to the path element and use selection.attr to set the "d" attribute to the path data */

svg.append("path")
.datum(regions_var)
.attr("d", path);

/* 6 - 8 Some other stuff TBD later */

/* 9. Close the d3.json callback */

});

</script>

… which gives us this sort of basic rendering of the MCYS Integrated Service Regions [actual rendering]:

Figure 1. Basic rendering of the MCYS Integrated Service Regions.
Figure 1. Basic rendering of the MCYS Integrated Service Regions.

There are a few obvious improvements we can make. First, let’s draw the boundaries of the MCYS ISRs:


/* CSS goes here */

/* Define the boundary of an ISR as a 1.5 px wide white line, with a round line-join */
.isr_boundary {
  fill: none;
  stroke: #fff;
  stroke-width: 1.5px;
  stroke-linejoin: round;
}

...

/* Javascript for reading and rendering data in SVG */

...

/* 6. Draw the boundaries of the ISRs */
  svg.append("path")
      .datum(topojson.mesh(isrs_topo, isrs_topo.objects.isrs_geo, function(a, b) { return a !== b; }))
      .attr("class", "isr_boundary")
      .attr("d", path);

… giving us this sort of map [actual rendering]:

MCYS Integrated Service Regions with boundaries
Figure 2. Rendering of the MCYS Integrated Service Regions with boundaries.

… and then let’s label and colour the MCYS ISRs using the colour-scheme adopted by the MCYS:


/* CSS goes here */

/* fill the ISRs using the MCYS colour scheme */
.isrs_geo.Toronto { fill: #bd3f23; }
.isrs_geo.Central { fill: #fcb241; }
.isrs_geo.East { fill: #a083a7; }
.isrs_geo.West { fill: #e3839e; }
.isrs_geo.North { fill: #8dc73d; }

/* switch the colour of the ISR boundaries to black */
.isr_boundary {
  fill: none;
  stroke: #000;
  stroke-width: 1.5px;
  stroke-linejoin: round;
}

/* style the Region label */

.region-label {
 fill: #000;
 fill-opacity: .9;
 font-size: 12px;
 text-anchor: middle;
}

...

/* Javascript for reading and rendering data in SVG */

...

/* 7 Colour the ISRs */

  svg.selectAll(".isrs_geo")
      .data(topojson.feature(isrs_topo, isrs_topo.objects.isrs_geo).features)
      .enter().append("path")
      .attr("class", function(d) { return "isrs_geo " + d.id; })
      .attr("d", path);

/* 8 Label the Integrated Service Regions */

 svg.selectAll(".region-label")
 .data(topojson.feature(isrs_topo, isrs_topo.objects.isrs_geo).features)
 .enter().append("text")
 .attr("class", function(d) { return "region-label " + d.id; })
 .attr("transform", function(d) { return "translate(" + path.centroid(d) + ")"; })
 .attr("dy", ".35em")
 .text(function(d) { return d.properties.region_name; });

giving us this sort of map [actual rendering]:

Figure 3. Labelling and rendering of the MCYS Integrated Service Regions in colour.
Figure 3. Labeling and rendering of the MCYS Integrated Service Regions in colour.

Wrap-Up

In this post, we use d3.geo – a free and open source software tool – to visualize a publicly-available geospatial dataset related to children and youth services in five regions across Ontario. Next time, we will use another dataset to visualize children and youth services at the level of thirty-four service areas in the province.

  1. Accessing d3.geo is easy – you simply include a single call to d3 in the body of an .html web page. Installing and using some of the tools for converting geospatial data from one format to another (in order to use d3.geo) is more complicated. For some more technical details, see  Installing tools for d3.geo – 20160306.

Installing tools for d3.geo – 20160306

In Let’s make a map, Mike Bostock describes how to make a modest map from scratch using D3 and TopoJSON. Here we detail how to install the main tools on our CentOS 6x server.

Installing Tools

Geographic data files are almost always too large for manual cleanup or conversion. Fortunately, there’s a vibrant geo open-source community, and many excellent free tools to manipulate and convert between standard formats.

GDAL

The big multitool to know is the Geospatial Data Abstraction Library. Commonly referred to as GDAL, it includes the OGR Simple Features Library and the ogr2ogr binary we’ll use to manipulate shapefiles. There are official GDAL binaries for a variety of platforms – our hosted service runs on CentOS 6x.

The Enterprise Linux GIS (ELGIS) effort provides RPMs of various GIS applications, including GDAL, for CentOS and other Enterprise Linux derivatives. To upload the release RPM for CentOS 6x:

sudo rpm -Uvh http://elgis.argeo.org/repos/6/elgis-release-6-6_0.noarch.rpm

To install GDAL:

sudo yum install -y gdal

… but this generates a lot of dependency errors. In the end, the most significant issue we had to address was GDAL requirement for armadillo-3x instead of the newer (and not backward-compatible vis-a-vis GDAL) armadillo-4x.

We tracked down a copy of armadillo-3.800.2-1.el6.src.rpm, uploaded it to our /root directory, and installed:

sudo rpm -Uvh armadillo-3.800.2-1.el6.x86_64.rpm

… and then we were able to install the GDAL RPM package, as above.

Miscellaneous things we did along the way to installing GDAL

While we’re uncertain of their ultimate significance, we installed/compiled some resources that are worth noting:

Some of the missing dependencies that were flagged up to us were addressed by installing two packages:

su -c 'rpm -Uvh http://mirror.centos.org/centos/6/os/x86_64/Packages/atlas-3.8.4-2.el6.x86_64.rpm

su -c 'rpm -Uvh http://dl.fedoraproject.org/pub/epel/6//x86_64/arpack-3.1.3-1.el6.x86_64.rpm'

We also compiled proj4 and geos:

wget ftp://ftp.remotesensing.org/proj/proj-4.6.0.tar.gz
tar -zxvf proj-4.6.0.tar.gz
cd proj-4.6.0
./configure
sudo make install

cd ..
wget http://geos.refractions.net/downloads/geos-3.0.0.tar.bz2
tar -jxvf geos-3.0.0.tar.bz2
cd geos-3.0.0
./configure
sudo make install
# add lib path to ld.so.conf file
sudo vi /etc/ld.so.conf
# add this line
/usr/local/lib
sudo /sbin/ldconfig
# Add lib path to ld.so.conf file
sudo vi /etc/ld.so.conf
# add this line
/usr/local/lib
sudo /sbin/ldconfig

TopoJSON

Next you’ll need the reference implementation for TopoJSON, which requires Node.js. Fortunately, we’d already arranged for our host to install Node.js. To install TopoJSON:

npm install -g topojson

… which throws a warning: {several modules} requires inherits@’2′ but will load – even after we install pm2:

npm install pm2 -g

[note: when we were trying to merge shp files, threw error that someone suggested would be fixed by uninstalling topojson and re-installing with sudo -H npm install -g topojson]

To check the installation:

which ogr2ogr
# prints /usr/local/bin/ogr2ogr
which topojson
# prints /usr/local/bin/topojson