The Ethics of Data

As we pay attention to the sea of evidence before us, I believe we need to consider the larger questions of purpose, intent and the ethics behind the data we gather and the conclusions we draw. In education, we are overloaded with evidence of multiple forms and yet, too often I see us forming conclusions based on thin explorations of the question. It is critical that in our inquiries, we are going deeper before we make quick judgements, comparisons or rankings that really have limited basis in fact or, even worse, can do damage to our school cultures. This blog offers suggestions for an ethics of data before we leap to such conclusions and attempts to further open the conversations about how we can avoid misuse of data and analytics in education.

Some things to consider

• Qualitative evidence matters. At the heart of qualitative evidence is often teacher judgement. We need to get to a world where teacher judgement is seen on an even playing field with quantitative data. I know that the person who knows the most about my child’s learning in the context of the classroom setting is the classroom teacher. I also expect, and know, that the classroom teacher has dozens, if not hundreds of samples at any given time of my child’s evidence of learning that back up their judgement. That judgement is a blend of their own set of qualitative and quantitative evidence. Expanding upon that, it is the educators within a school that likely know the most about the context and culture of the community they serve.
• Quantitative measures should act as a mirror and guide in support of qualitative analysis. Numbers, patterns, and statistics are all important. They mean something but too often we see people treat the numbers as both a means and ends unto themselves. It is imperative that we continue, particularly in social sectors to look at mixed-method research into how our systems are performing.
• We must not sit in judgement. It is critical to collaborate with those included in the evidence in your analysis because, as stated earlier, it is the very people who work on the ground day-to-day that likely have the deepest insights into the context. All data emerges out of context and it’s our job to understand that context and background.
• All data analysis likely has bias. Bias is neither a bad no good thing in my mind. The important thing is to explore and unearth the bias inherent in any analysis so you can talk about it. We need to explore how our own bias is part of the process and work to be as objective as humanly possible.
• Rankings are problematic. Too often I see rankings doing ourselves harm. We routinely accept the rankings we like and oppose the rankings we dislike. The best example is PISA where we consistently accept that it’s ok, and in fact we celebrate the results. Let’s remember that this is comparing Canada to 71 other countries including places like Algeria, Malta, Albania and the United Arab Emirates. We celebrate because we’re at the top. However, when the same ranking publications are used by the Fraser Institute to rank our neighbourhood schools, we pan those results. Same outcomes, ranking institutions and systems, but different methodologies for sure. The most important part of this story, to me, is that if we accept PISA because of its superior methodology, then why don’t we use that same methodology for our own data? Or is it that we simply accept and embrace PISA because it has us at or near that top? If we were to drop 20 slots tomorrow what would be our response? In our explorations, we believe that parents want to know that their child’s school is a good school rather than how it ranks to schools across the district.
• It is all about the best questions, it isn’t about simplistic answers. The more time I spend exploring the evidence behind our system, the more I realize the critical role of pursuing the right questions. We can come up with almost any statistical answer we want to a question but it’s really about what exactly you are trying to know and what evidence can you bring to bear on the pursuit of that question. For example – how literate are your grade 5 students and, overall as a district (or school), is your literacy improving? That is a complex question and requires an equally sophisticated approach to seeking an answer. The pursuit of this question must be a collaborative and ethical process.
• We must become data literate. There is a base level of understanding of evidence and analysis that we need to have. The reason we need to have this is to start to be part of the conversations coming our way from a variety of institutions. I fully believe we will continue to see a vast expansion of data and analytics and we need to be part of these conversations and processes. Separate from that, the evidence we have can help support the compelling narratives about how we are doing and where we are going.
• Data matters. We cannot ignore the evidence in our system. In fact we don’t – we gather it at incredible rates and in enormous volumes. Class size, class composition, student designations, resource levels, funding ratios and models, staffing, allocations of hours, student attendance, and minutes of instruction are but a few of the varied data sets that exist. We collect data all the time and we use it in a myriad of ways. To say that we don’t use evidence is folly. In fact, particularly in the analysis of class size and class composition, these are the most robust data sets we have and these data are used by all in the system. We are swimming in a sea of data but the question is what are the purposes of using the data that we have?

Finally, the above list is far from exhaustive and I would love to hear other peoples’ suggestions and thoughts. The intent is to start a conversation. I simply wanted to put down a few things that quickly come to mind.

How would we begin?

Most importantly, we need an ethics of data specifically around student achievement and school success. That ethics should emerge from conversations that explore:

• What data will we collect?
• How will it be used?
• How can we use the tools and methodologies of today to shed light upon and support the incredible work our schools do as opposed to sit back in judgement and suggest that something is wrong?
• How can we move to a place where evidence is part of our ongoing work in support of the success of all?
• How can we ensure that we are publicly accountable for the expenditure of the tax dollars we receive in a way that does not punish or shame those who are struggling?

All these questions can be pursued, but that pursuit begins with a local and larger conversation about the ethics of data and what we are intending to do in support of schools and children.

I know we are beginning this journey and are in various places in various schools and districts and I look forward to see where we land in service of all children.