Providing a basis for query design for a systematic review

The past few weeks I’ve been working on the query design for my systematic review on algorithmic accountability. I encountered two problems:

  1. ‘Algorithmic accountability’ is a relatively new term, whereas the problems, themes, concepts which are connected with it, are ofcourse touched upon in earlier work in various disciplines;
  2. I wanted to find a systematic way to approach the query design, which also accounts for the diversity of the fields and terms used to discuss matters related to algorithmic accountability.

I eventually settled on the following approach, using computational methods. Out of the material that was identified as relevant prior to the review, only the articles (27) which included keywords were selected for this exploration.From these articles the collocations of the keywords were extracted.

Collocation mapping
In charting the keywords’ collocations, first the individual keywords were related to the other keywords of the article. For instance if the keywords of an article are ‘big data’, ‘algorithms’, and ‘accountability’, then the relations would be mapped as follows:

big data          –> algorithms
big data          –> accountability
algorithms      –> big data
algorithms      –> accountability
accountability –> big data
accountability –> algorithms

After the relations were prepared, these collocations were mapped in Gephi.

The nodes with the most incoming/outgoing connections (degree >= 35) were then filtered out.

This value of mapping these keywords, is that it gives some perspective on what terms are used in what fields (or, more accurately: with what other kinds of terms, thereby hinting at the field). Four (very rough) clusters could be detected by modularity: one revolving around governance (e.g. government, governance, accountability – though there are also smaller nodes refering to, for instance, journalism). The second cluster deals mainly with legal aspects (e.g. GDPR, right to explanation), the third deals with more general data-related issues (e.g. regulation, automation, surveillance). The last is predominantly dealing with ethics. The interesting thing about this last cluster is that aside from the ethics node, all other nodes in this cluster are from 1 paper (this paper had a lot of keywords, thereby constituting its own cluster) – which also hints at the limitations of this method on its own.

While the mapping provides some insight into which terms are used in what kinds of debates, it doesn’t really point as of yet to what combinations might be fruitful for query design. Thus, subsequently, the edges table was exported from Gephi, and the edge weight was used as a measure to determine the strength of the relations between keywords. The double relations (a –> b / b –> a) were resolved and their edge weight was added together.

Now, I have a systematic basis for deciding upon my query, for I can demonstrate which terms seem to be more strongly connected. Which doesn’t mean that likely it’s still going to be hard, but atleast I have some more grounding!