Datavision 2020

53. Metadatavision

2020-12-30T00:00:00+00:00

A year ago I came up with the idea of creating one data visualization per week in 2020. I wanted to learn to use some new tools, and to put into practice some of the techniques I’d read about in Claus Wilke’s book, Fundamentals of Data Visualization.

52 weeks of visualizations later, what did I learn? Here are some summary charts:

As you can see from the bar chart of visualization libraries, I grew to love d3 and it quickly became my “go to” tool. It allows you to do almost anything, but requires a lot of work. I would often use ggplot2 to do a quick analysis, then turn to d3 to present the data in exactly the way I wanted to - particularly if it involved some kind of interaction or animation. ggplot2 is great, it’s much easier to generate an off-the-shelf chart - so I used this a lot too, and spent some time trying to improve on the default presentation (shout out to cowplot here, also from Claus Wilke, and used in the charts above). Vega-lite is in third place - I didn’t reach for it as much, but I can see it becoming a useful output from other tools (like ggvis and Altair).

The second bar chart shows the distribution of the type of visualization, divided into families (irrespective of the tool used to produce it). The families are from Andy Kirk’s handy Chartmaker Directory (the same ones appear in his book too), and are intended to capture the primary role of each chart.

Most of the visualizations have a temporal dimension (how does x change over time?). I became aware of my bias here, and consciously tried to come up with visualizations that did not have a time dimension (and in fact most of my favourites were not time plots, see below). I did produce a large variety of chart types though (not shown on the chart, but the visualization type is listed on the page for each visualization) - something else that I consciously tried to do.

The third chart shows the distribution of the size of each dataset. The dataset sizes ranged over 9 orders of magnitude - from a few hundred bytes to tens of gigabytes. I didn’t pay much attention to this - my prime interest was finding interesting ways of presenting interesting data. There was no correlation in my mind between dataset size and how interesting it was - I was certainly not trying to visualize large datasets (I was quite happy to do so, but never needed more processing power than a single machine to do so).

The biggest challenge was finding interesting datasets, then turning them into a form suitable for visualization. This “data cleaning” step is notorious amongst data scientists as being slow and hard to automate. I didn’t have any special tricks here: I generally just wrote a Python script to do any pre-processing I needed. I’ve published all of the code on GitHub.

My five favourite visualizations

In the order I made them:

Hamilton songs (Week 11)

This was the first dense, interactive d3 visualization I did, and was when I really “got” d3. I like the fact you can spend time exploring the dataset with this visualization.

Programming language popularity (week 12)

This is bright and fun - and doesn’t look like it was done in ggplot2.

Voronoi football (week 21)

Not the first animation I did (that was Lots of Lotties in week 14), but possibly the most mesmerising.

Bach charts (week 25)

A piano keyboard is basically a bar chart.

Tooth cavities (week 37)

I like the simplicity of this data visualization - it conveys the data very intuitively (and my dentist approves).

Visualization type: bar charts and beeswarm plot

Data source: Datavision, CSV, 3.5 KB.

Technical notes: generated using ggplot2; code

52. Food banks

2020-12-23T00:00:00+00:00

The number of food parcels distributed by food banks in the UK has grown rapidly over the last decade. The following visualization shows data from the Trussell Trust (so it doesn’t include all food banks in the UK):

There is some evidence that increasing usage of food banks is linked to austerity.

The latest data is up to the end of March 2020, so it doesn’t include the impact of the coronavirus pandemic, which is forecast to result in a 61% increase in food parcels needed across the UK.

This time of year is particularly difficult for users of food banks, so please consider donating if you can. Thank you.

Visualization type: bar chart

Data source: Trussell Trust (and this PDF for earlier years), CSV, 236 B.

Technical notes: generated using ggplot2; code

See also: Food Banks in the UK, a House of Commons Library Research Briefing

51. Maximum daily temperatures in central England since 1878

2020-12-16T00:00:00+00:00

The Met Office has an amazing dataset of the mean daily temperature recorded in central England that goes back to 1772. The mean monthly data goes back to 1659, and is “the longest available instrumental record of temperature in the world”.

This heatmap is a visualization of maximum daily temperatures that goes back to 1878.

On the interactive version you can hover to see the recorded maximum temperature on any particular day.

You can see heatwaves quite clearly - the one in the summer of 1976, for example. You can make out the cold winter of 1962-63 too, where the maximum temperature hovered either side of freezing for weeks.

This isn’t the best visualization to see warming - the graph on the Met Office page for this dataset, and this plot by cavedave both show the warming trend over the last few decades.

Visualization type: heatmap

Data source: Met Office Hadley Centre observations datasets, space delimited text, 350 KB.

Technical notes: generated using d3; code

50. Moving house

2020-12-09T00:00:00+00:00

We moved house today. (Yay!)

Of the 23.5 million households in England in 2018-19, a little over 2 million of them moved house during that year. That’s around 8.7% of households that move in a given year.

Click on the image to see the animation. Each dot corresponds to ten thousand households.

Visualization type: animated dots

Data source: English Housing Survey 2018 to 2019, XLSX, 447 KB.

Technical notes: generated using d3 and Coördinator; code

See also: Passive houses in the UK

49. Animated gzip

2020-12-02T00:00:00+00:00

Gzip is a program for compressing files. One of the techniques used for compression is to look for duplicate parts of the file: if a particular sequence of characters has been seen before, then it can be replaced with a reference pointing back to the previous occurrence so it doesn’t have to be written in full again.

This technique (which is a part of the DEFLATE algorithm) is very effective since many files have repetitive parts - text files have many repeated words for example.

The following visualization shows the gzip representation of a Markdown file from this blog. (Markdown is a simple text-based format for writing structured text.)

Interactive version.

Each character in the text file is coloured according to how well it is compressed in the gzip file. Blue indicates better compression (the darker the better), white means that there is no effective compression (i.e. each character takes 8 bits), and orange indicates worse compression (than no compression). (This compression is achieved by a separate part of DEFLATE that does bit compression on individual characters, not the duplicate sequences mentioned above.)

So the H in the first line, for example, is coloured orange, and takes 9 bits to encode it (you can see this by hovering over the character in the interactive version).

You may wonder why H hasn’t been compressed well by the algorithm. It’s simply because it doesn’t appear very often in the document (just twice in this one, compared to dozens of occurrences of lowercase h, for example).

Some characters are in blocks of the same colour - this means that they have been recognized as duplicates as explained above. If you click on them in the interactive version, the previous occurrence is highlighted. For example, the dark blue block containing the characters Hamilton on the third line refers back to the same text at the end of the first line. This back reference means that the second occurrence can be encoded at just 1.8 bits per character.

Clicking on the “Compress” button has the effect of compressing each block visually so that it occupies the amount of space it actually takes up in the gzipped file.

Notice how Hamilton on the third line is compressed a lot more than the one on the first line.

This visualization was inspired by gzthermal, a command line tool that produces a heatmap image for gzipped files.

Visualization type: animated heatmap

Data source: Gzip compressed Markdown of the Hamilton songs blog post, gzip, 469 B.

Technical notes: generated using d3 and defdb; code, including brief instructions on how to create an animation for any gzipped file

48. Wars

2020-11-25T00:00:00+00:00

This week’s visualization is of violent conflicts from 1400 to 2000.

Interactive version.

Each line is a conflict, with the length representing the duration of the conflict. The y-axis measures the magnitude of the conflict, defined as the base-10 logarithm of the total number of deaths. So for example, a war with a million deaths would be counted as magnitude 6.

Using magnitudes like this follows Lewis Fry Richardson’s statistical treatment of wars (and other violent conflicts) in a book called “Statistics of Deadly Quarrels”, published in 1960. Brian Hayes has written a wonderful article discussing Richardson’s work.

One of Richardson’s motivations for a quantitative study of violent conflicts was to understand if war was becoming more or less frequent. He concluded that “The collection as a whole does not indicate any trend towards more, nor towards fewer, fatal quarrels.”

A more recent paper (Martelloni et al, 2018), has a similar conclusion:

After normalizing the data for the global human population, we find that the number of casualties tends to follow a power law over the whole data series for the period considered, with no evidence of periodicity.

The dataset used to draw that conclusion is from Peter Brecke’s Conflict Catalog, and is the data used for the visualization here. Brecke assigns each conflict to a region, and on the interactive version here, if you hover with your mouse over a region name it will highlight all the conflicts in that region. Be careful about reading too much into the regional distribution, however, since there are certainly biases in the data. As Brecke’s paper accompanying the dataset states:

One of the early findings of this research effort was that if one restricts oneself to the nine original data sources mentioned earlier, one discovers a strong Eurocentric bias in the data, and an especially stark bias for the period prior to 1800. The Conflict Catalog attempts to at least in part correct this disparity as it moves towards completion.

It’s interesting to see from the visualization that the 18th century has relatively fewer conflicts, something that Brecke commented upon:

The relatively pacific 18th century is a puzzle. Comparably thorough data for other regions have not yet been entered into the dataset, but my translators for the Chinese and Japanese data have without knowing these findings commented to me that the 18th century had relatively few conflicts in those two well-documented countries.

Visualization type: connected dot plot

Data source: Conflict Catalog (Violent Conflicts 1400 A.D. to the Present in Different Regions of the World), by Peter Brecke, XLSX, 790 KB.

Technical notes: generated using d3; code

See also: War and Peace, Our World in Data

47. Alternatively Powered Vehicle growth in Europe

2020-11-18T00:00:00+00:00

The proportion of new cars that are petrol or diesel is in decline across Europe. More and more new cars are “Alternatively Powered Vehicles”, that is, electric or hybrid cars (and a few that use other, non-petroleum power sources). There has been quite rapid change over the last two years:

There is a lot of variation from country to country, however. Norway stands out where over 80% of new cars are APVs. Czech Republish shows more modest growth in APVs.

Diesel is generally declining faster than petrol, and from a lower base (although Ireland - with more diesel registrations than petrol one - is an exception). Across Europe as a whole, this quarter APV registrations overtook diesel ones for the first time ever.

Many European countries have passed laws to introduce bans on new petrol and diesel cars. This page on Wikipedia has the details. In Norway, for example, this will be from 2025. The UK government announced today that it would be bringing forward the ban on new fossil fuel vehicles from 2040 to 2030, in line with many other countries, such as Germany and Ireland.

Visualization type: faceted line chart

Data source: Alternative fuel vehicle registrations from the European Automobile Manufacturers’ Association (ACEA), XLSX, 7.7 MB.

Technical notes: generated using ggplot2; code

46. Postcodes

2020-11-11T00:00:00+00:00

We tend to think of postcodes as boring things that don’t change much. It turns out that there has been a large growth in the number of active postcodes over the last 40 years, as this chart shows:

(The interactive version allows you to hover to get information about the postcode area.)

The area with the largest number of postcodes is BT (Belfast), which covers all of Northern Ireland, and which overtook the next largest, B (Birmingham), in the 1990s. The one with the smallest number is GIR, a non-geographic code for Girobank. ZE (Lerwick) in Shetland is the geographic area with the smallest number of postcodes.

There are some interesting things buried in the data that this visualization brings out (click on the interactive version to see for yourself).

In 1999, postcodes for the Wirral Peninsula that previously were in the L (Liverpool) postcode area were transferred to the CH (Cheshire) area. The jumps for both these areas are very visible on the chart.
There are other jumps in the chart that are less easy to explain. In BS (Bristol) in 1997 the number drops dramatically for a few months, before coming back to about its previous level. According to the data, over 8000 postcodes were changed in August 1997 and in December 1997, but I can’t find any external reference to this change. It could be that it’s an administrative artifact, a blip that doesn’t mean this number of postcodes were actually removed, just that the change was spread over two separate entries. (This is just speculation though, and I could be wrong about this.)
NPT was a non-standard area code for Newport (NP) (since it had no district number) that was phased out in 1984. (Oddly, there doesn’t seem to be a corresponding jump in NP.)

It’s to be expected that when new houses are built more postcodes will be allocated, but there’s actually a lot more change happening than that. The chart doesn’t show it (except for a few examples like Bristol above) - but there is also a significant amount of recoding within an area - when a large number of postcodes are changed even though the number stays about the same.

For example, in Aberdeen (AB) in August 1990 the area was completely recoded, by changing the district number from 1 digit to 2 digit codes (so, for example, AB1 changed to AB1x). This change is not visible on the chart.

I spent longer than usual trying to come up with a postcode visualization. I wanted to do something that wasn’t just showing postcodes on a map - something more about the coding system itself, ideally. One early attempt was a sunburst diagram, but the sheer number of postcode areas and districts meant that it was a blur and conveyed very little information.

I think it would be interesting to visualize postcode density - HS (Hebrides) has very few postcodes over a large area, while cities like Birmingham (B) have a large number of postcodes concentrated in a relatively small area. Or perhaps area names that are most out of proportion to the size of the area they cover - some cases include SY (Shrewsbury) that extends all the way to the coast of Wales, or the small town of Llandudno that gives its name to the LL postcode area that covers much of north Wales.

Visualization type: multi-line chart

Data source: National Statistics Postcode Lookup (August 2020), Office for National Statistics (CSV, 191 MB, compressed)

Technical notes: generated using d3 and its multi-line chart; code

See also:

List of postcode areas in the United Kingdom, Wikipedia
How often do postcodes change? - inspiration for this visualization
Postal addresses: a little history and a lot of photos

45. Occupations in Crickhowell in 1881

2020-11-04T00:00:00+00:00

Do you know what an ostler is? There were two in Crickhowell in 1881.

We know this because a census has been carried out in England, Scotland, and Wales every ten years since 1801 (except in 1941 during the Second World War). The census records basic information, including occupation, about everyone resident at every address in the country on one night of the year.

Eliane has been going through the census data for Crickhowell in 1881, so I thought it would be interesting to visualize people’s occupations at the time.

The graphic is a treemap, where each box is a separate occupation, scaled according to the number of people with that occupation (the number is printed in the box too). Different colours show different categories of occupation.

(The interactive version allows you to hover to see the category and the full text in each box.)

The main categories are Service (blue - top left) and Trade (orange - bottom left), followed by Manual, Construction, and Agriculture (green, red, lilac - top right). Despite being on the edge of a major industrial area, there were a relatively small number of people in the Industrial category (grey - bottom right). I was also surprised that the proportion of people working in agriculture was so small for a rural area.

To answer the question from above, an ostler is someone who looks after horses for people staying at an inn. However, my favourite job title from 1881 has to be Inspector Of Nuisances. Can you find it on the visualization?

Visualization type: treemap

Data source: 1881 Wales Census, Ancestry.co.uk (CSV, 151 KB)

Technical notes: generated using d3 treemap; code

See also: Census records, The National Archives

44. UK tree planting

2020-10-28T00:00:00+00:00

The number of trees planted per year in the UK has declined over the last 50 years.

This visualization shows the percentage area of each country that has been used for tree planting each year, over that period:

Scotland has had the largest planting rate, and Wales the smallest.

I was surprised that Wales has had and continues to have such a low rate, given that it has so much space for tree planting. This is all the more reason to support organizations like Stump Up For Trees, which is aiming to plant a million trees in the Brecon Beacons. (You can donate here.)

The dashed line shows the target rate recommended by the Committee on Climate Change to meet the net-zero target. This rate translates to planting trees on approximately 1/8 of a percent of the area of the UK every year by 2024 until 2050. This is more than double the rate of planting in the year up to 31 March 2020, and is not at a level seen since 1989.

What are the rates in terms of land area and numbers of trees? The next chart shows the same data, but the y-axis measures millions of trees (on the left), and thousands of hectares (on the right). The 2024 target is 30 thousand hectares.

I assumed a planting density of 2250 trees per hectare (from Cumbria Woodlands), which corresponds to 2 metre spacing between trees. The actual numbers planted are almost certainly different, but it gives a rough idea of what we are aiming for: about 70 million trees a year.

Visualization type: line chart

Data source: Forest Research, XLSX, 98 KB.

Technical notes: generated using ggplot2; code