"Reading is equivalent to thinking with someone else's head instead of with one's own" - Arthur SchopenhauerSo the long awaited library consultation report to the neighbourhood scrutiny committee was published on Thursday. It is a mammoth 164 page document of 9MB in size, which is why I've been somewhat quiet. Unsurprisingly, I've been doing what all good library users do. I've been engrossed reading it before the weekend. For what it is worth, I am shocked at the poor quality of the analysis and together with the rest of the campaigns around Manchester, I am shocked at the rhetoric.
For those who've got to know me over this campaign period, they will immediately know I am a numbers guy. Whilst a lot of people tend to bracket "lies, damn lies and statistics" together, the truth is that statistics do not actually do the lying. It's the interpreters of those statistics that do. This document is an example of just such a thing. Of course, being a guy who's not afraid of an intellectual 'punch up', I am going to give my reasons bit by starting with the below, but suffice to say that even the work submitted by the Save Burnage Library campaign is more solid than the MCC analysis conducted here, even without MCC giving the campaign all the information it needs.
Viability and Needs Analysis (or as I like to call it "This analysis isn't Viable, we Need better analysis!")Appendix 7 of the document is probably one of the worst examples of a document that's full of statistical biasing I have ever come across. Here are some reasons as to why (the list of things wrong with this report and the corroborating data is so large, I'd never get it out in one go, hence the splitting of it into chunks).
The report consistently refers to taking account of the Charteris enquiry. Note, these variates (the columns across the top) are not actually specified within Charteris at all. 6.17, 6.26, 6.27 show the sorts of evidence that Charteris would have expected to occur within an effective assessment of local need. Manchester City Council's assessment (Appendix 7) is shown below (click the image for a larger version)
|fig 1 - MLIS Viability and Needs Analysis (click to zoom)|
- Ranking system (notoriously inaccurate, since there may be a massive disparity between actual values, but they will not necessarily rank that far apart. Hence, some will 'over perform' overall)
- High/Low ranking - Where the values were ranked differently across the board. In itself this isn't a problem until you consider...
- The combined score is simply a sum of the ranks and a selection of the lowest ranking performers by this combined value. Given the 'flipping' between ranking high/low based upon this can be unsound.
I constructed a correlation matrix between the factors on the columns at the top of their analysis. In simple terms, a correlation matrix is a grid where each cell effectively tells you how similar the factors along the columns are, relative to the factors along the rows. High positive or negative numbers mean that the factors are very definitely related to each other. High positive numbers mean that as a variable (in this case rank) increases so the other variable increases. A negative number means if the ranking increases, the other value decreases. This sort of matrix can tell you a tonne of stuff, including how similar columns actually are to to each other The assumption that MCC put to us in this report is that it was a fair assessment and as such, 'Total library visits', '...user ranking' and PC usage (in hours) were not correlated to the combined score (which was the important factor). The image below shows my correlation matrix snipped from Excel. Again, click the image to expand it.
|fig 2 - Correlation matrix (click to zoom)|
Correlations and anti-correlations? Who are they?
Well, what do you know[?]
- Population of the library catchment area is strongly correlated with the total number of library visitors, Total active user ranking, PC usage and building performance.
- Population of the library catchment area, Total number of library visitors, Total active user ranking and PC usage are basically the same variable! This is corroborated because of the clump of red right in the middle, which shows the variables are also almost identical to each other, never mind just the catchment area population. For example, you will have a high active user rank if you have a high number of users that's basically common sense. If it looks like a duck, walks like a duck and quacks like a duck...
- The impact of this same variable on the results effectively weights the analysis massively in favour of the larger library catchment areas and wholly against the neighbourhood libraries.
- Neighbourhood libraries then get a further kick in the groin because they don't have access to venues to hold sessions or events. This causes a selection bias in the sample, since you are automatically excluding libraries from the top rankings which can't host events due to H&S legislation.or otherwise having no room. Note in the link in this point it states "If the selection bias is not taken into account then certain conclusions drawn may be wrong."
- This is corroborated when you look at the correlation between the combined score and the population of the library catchment area. This is the kick in the teeth, an 88.77% correlation! You pretty much cant get any closer than that in this sample size.
- Barlow Moor
- Moss Side Powerhouse
- New Moston
- Newton Heath
This is yet another example to me of MCC conducting a poor analysis. It brings back into question statements made previously by cllr Sue Murphy about how they planned for closing lunchtime opening hours and were surprised by the drop in service uptake. From this analysis alone, I am not convinced they have the statistical know-how to make that sort of decision in the first place and come to that conclusion.
So what should the table look like?
|fig 3 - Quadrant adjustment for population catchment area size (click to zoom)|
*** EDIT: 11/05/2013 ***
In theory, when you see a variable have such a massive influence, this needs to be adjusted across the board. The Save Burnage Library campaign reports never claimed a requirement for blanket coverage across the city (as stated on page 21, point 8.5). The adjustments were conducted to be able to compare like-with-like based on catchment population size. Indeed, I would go so far as to say column 1 in Appendix 7 the MCC report should have no direct bearing on the final result, since the information it brings is implicitly included in the analysis as part of the adjusted columns mentioned above. Following on from the fig 3 above, to turn this into a table:
|fig 4 - The change in ranking when direct effects of catchment area size are removed.|
In total, some libraries have moved 11 places up the rankings! Miles Platting (11), Crumpsall (6), Brooklands (5), Northenden (5) and Fallowfield (4) in particular have moved significant steps up the ranks. In this arena, of the two libraries in "Newton Heath and Miles Platting" ward, they are closing the one which is better used per head of population.
*** END OF EDIT ***
SummaryAs a numbers guy, I am shocked! Truly shocked! Reading along the bottom row (or rightmost column) to determine what factors MCC really used and to put it in political rhetoric that MCC should understand, the low correlation of the combined score with any other factors aside from the positively correlated ones, are truly saying to the people of Manchester, when it comes to libraries:
- MCC don't care about our young people
- MCC don't care about local deprivation
- MCC they don't care about libraries existing within 1.5 miles (that was just a decoy) and
- MCC don't care about how much each library visit costs