Professor Susan Halford and Professor Dimitra Simeondiou have led the development of BDFI since 2019. Here Susan reflects on what has made the endeavour special to her as she announces that she is standing down as academic co-Director.  Susan will continue to work with BDFI supporting its activities and mission and as the key link to the new ESRC Centre for Sociodigital Futures.

It has been a remarkable privilege to co-lead BDFI in these formative years and I am very proud indeed of what Dimitra, myself and the team have achieved during this time.

What started as a hugely ambitious aim in 2018 – to do digital futures differently – is now a unique and lively Institute, located in beautifully refurbished Victorian ‘sheds’ which will house state-of-the-art innovative research facilities for academics from across the university and our partners in industry, government and civil society.

Breaking the linear model of innovation

During the last four years I’ve been continually impressed by the commitment of our partners and academics to transform the way we do digital innovation. This calls on us all to break the linear model of innovation – where technology comes first, and impact is examined later – and to develop sociotechnical thinking, methodologies and innovation practices. This is far from trivial, and we have learned a lot along the way, but BDFI is uniquely placed to succeed because we are truly interdisciplinary, starting with the leadership and reaching right across the Institute.

Strong foundations

BDFI now sits at the heart of a truly impressive stream of sociotechnical research and innovation at the University of Bristol.

We have laid the foundations for a step-change in how we think about and do digital innovation. With ten outstanding new academic appointments already in place, and ten more to come, there is no doubt that BDFI will go from strength to strength. I look forward to remaining actively involved as the institute grows and drives digital innovation for fair, sustainable and prosperous futures.

Dr Marisela Gutierrez Lopez has been collaborating with BDFI partner LV=General Insurance to explore opening up processes behind AI decision making. How will this benefit organisations and people who are affected by automated decisions?

Sociotechnical methods are helping us to create more inclusive ways of AI discussion across public, private and academic sectors. They are therefore crucial to investigate how people shape and are shaped by AI systems, and explore the interrelations between people, algorithms, data, organisational procedures and other factors that constitute these systems. For this purpose, we integrate social and technological expertise from across the University of Bristol, and our partners in industry and communities to empirically examine what makes AI explainable from a sociotechnical perspective.

In July 2020, the Explaining AI project was started with the vision to examine the concept of “Explainable AI” (or XAI) in machine learning approaches used in decision making. Our aim was to move beyond technocratic perspectives where explanations are framed as technical challenges towards more inclusive approaches that consider what AI might mean for diverse data publics – particularly those not usually included in discussions about AI or explainability.

Working collaboratively with LV= General Insurance (LV= GI), a leading insurance provider in the UK, we are investigating the different levels of explanation of the decision-making processes informed by machine learning models and their outcomes. In addition to our investigations at a commercial setting, we have also teamed up with two local partners – Black South West Network and Knowle West Media Centre – to explore the types of explanations that would make machine learning intelligible and actionable to these communities.

Reaching out to local communities

The community strand of our project is underpinned by design justice as a framework for reconstructing Explainable AI in collaboration with those at the margins of innovation. We avoid positioning ourselves as outsiders that tell communities what AI is or why it matters. We are not aiming to solve to the black-box problem. Instead, we start from the “bottom-up”, exploring community interests and concerns as a first step.

We are co-producing community-led XAI initiatives with our community partners to ensure machine learning decisions are communicated in relatable and actionable ways. This has given our partners ownership over the project and its outcomes. For example, each community partner is shaping up their initiative by defining their research questions and the focus of their community engagements.

These community-led initiatives allow for open and speculative conversations that generate knowledge (in opposition to traditional forms of XAI), moving from individual to community understandings of what constitutes AI, and shifting the focus of attention from the past and present to possible futures. The next steps of our project involve supporting community engagements by the community groups to reach into their local areas and produce new XAI approaches that empower and give agency to different data publics.

Embedding our research at LV= GI

For the organisational strand, we set up a participatory ethnography where BDFI researchers are embedded in the LV= General Insurance data science team. As a result, the project offers a unique opportunity to closely analyse organisational practices and ways of working between data science and other business functions.

This project allows us to collectively explore ways to explain machine learning models beyond providing technical accounts of data and complying with legal requirements. It shifts the perception on what makes AI explainable with an enhanced understanding of how machine learning is shaping the organisation. Moreover, it has given us, both the research team and LV= GI practitioners, space to form deep connections, share co-working spaces, and expand our partnership even further.

Putting together industry and community – XAI perspectives 

Our project responds to the current ethical turn in AI by disrupting the concept of explainability, moving away from a purely technical solution to explaining the practice of AI rather than the principle itself. Sociotechnical methods are helping us to make research results actionable, where outputs are not abstract or distant but directly applicable in the context of each project partner.

The knowledge and dynamics generated using these methods are also helping us to connect the outcomes of the organisational and community strands of the project. Putting together cross-sector collaborations in XAI involves mutual learning, where the perspectives of all partners are equally important, and we learn from each other strengths. Additionally, it requires flexibility to adjust our priorities and facilitate two-way conversations. These conversations will become crucial in the last year of the project as we reconstruct Explainable AI together, in consideration of the findings from each place of inquiry. This will allow us to create more inclusive processes for the development of machine learning in the future.

Michael Rumbelow and Professor Alf Coles lead one of our seedcorn-funded projects that aims to help boost children’s confidence in maths.

Using an AI driven app, the interchange between learning is explored through traditional use of blocks.  Here they discuss how digitising this learning aid could benefit teacher classroom assessment and the challenges of developing novel technologies as education specialists.

In 1854, the first English-speaking Kindergarten opened in London, based on the play-based pedagogy of Friedrich Froebel (1782-1852), who designed his Kindergarten curriculum around play activities with wooden blocks. Later plastic versions of Froebel’s blocks were developed, which evolved into Lego – now the world’s largest toymaker – as well as into interlocking plastic cubes for primary mathematics classrooms – which the characters in the popular CBeebies cartoon series Numberblocks are made of. And more recently, free play with digital cubes became the basis of Minecraft, the most popular video game of all time.

Clearly, block play is a popular activity among children. And in schools there has also been a resurgence in the use of physical blocks in primary mathematics classrooms, following the government’s policy since 2016 of promoting so-called ‘Asian mastery’ approaches to teaching maths, as used in Singapore, China, South Korea and Japan, which make extensive use of physical blocks as concrete models of abstract mathematical concepts, such as counting, addition, multiplication etc. We were interested in researching children’s interactions with physical blocks from a mathematics education perspective, and one of the key challenges was how to capture data on children’s interactions with blocks for analysis.

Previous studies of block play have focused on gathering data variously through sketching or taking photos or videos of children’s block constructions, or embedding radio transmitters in blocks which could transmit their positions and orientations. Recently developments in computer vision technology offer novel ways of capturing data on block play. For example, photogrammetry apps such as 3D Scanner can now create 3D digital models from images or video of objects taken on mobile phones, and AI-based object recognition apps are increasingly able to detect objects they have been trained to ‘see’.

We felt there might be an opportunity to detect and digitise the positions of wooden or plastic cubes on a tabletop directly through a webcam, so that the coordinates of the corners could be used to create virtual animated models of stages of block constructions which could then be explored in various ways, such as in immersive virtual 3D environments, by both researchers and students. This abstracted coordinate data would also enable patterns of real-world block constructions to then be analysed statistically, for example using AI pattern recognition algorithms.

 

 

 

 

 

Figure 2: A sketch of 8 cubes being used to model a garden seat in an 1855 Kindergarten guide (left); a photo of a reconstruction of the sketched model with wooden cubes (centre); and a screenshot of a prototype 3D model generated from the reconstruction with photogrammetry app 3D Scanner (right). (The 3D model is viewable here: http://3d-viewer.xplorazzi.com/model-viewer/index.html?modelId=629e943a3aaf2b171525a9b5 )

With funding from the BDFI we were able to form a small project team of two researchers in the School of Education, and a software developer and the head of a local primary school, in order to develop an app to trial with children in the school.

Technical Challenges

The problem of capturing positions and orientations of blocks digitally almost immediately became more challenging than we had anticipated. Initially we had hypothesised that detection of straight edges would be a relatively simple computer vision task, however in practice traditional edge-detection algorithms proved unreliable in detecting the edges and extrapolating cubes positions, with multiple confounding issues including lighting, shadows, orientation, variations in perspective and vertical position, variations in wood texture and colour, and hidden edges under stacked blocks. One approach we attempted was to paint each block in a different colour to aid recognition, but this too was unsuccessful.

Finding ourselves stuck in terms of successful block recognition, we decided on two radical changes in direction: (a) to move from traditional edge-detection to AI-based computer vision algorithms, such as Mask-RCNN, and (b) to drastically simplify the recognition problem by focusing on Cuisenaire rods – standard classroom manipulatives which are 1 cm to 10 cm long, each coloured in a distinct colour, and typically arranged flat on the table, avoiding the issue of stacked blocks (Figure 3).

Our developer found that a gaming laptop equipped with a GPU processor was powerful enough to run Mask-RCNN, and with sufficient training on approximately 150 images, could detect the positions of Cuisenaire rods in an image from a live webcam feed within 2-3 seconds of processing time, which we felt was acceptable from a usability point of view.

With a feasible solution now successfully implemented for rod detection, the developer could now relatively easily add code which generated images and sounds associated with each rod, such as displaying a graphical image of it on screen, and speaking its colour or length. We trialled the app with Year 1 children in a local primary school, and produced a paper about the trial for the British Society for Research into Learning Mathematics.

Lessons learnt

As educational researchers with little experience of developing apps such as this, we have learned many lessons. One is the value of iterative, so-called ‘Agile’ approaches which enable rapid experimentation and pivoting of direction in order to solve problems that inevitably arise in developing novel technologies.

Another is the value of the ecosystem of open-source libraries, shared expertise and documentation which grows over time around any novel technology, and in particular complex open-source AI algorithms and tools such as Google’s Tensorflow, and Facebook’s Detectron. Occasionally, a novel technology we tried looked attractive in terms of affordability – for example the OAK-D camera with built-in AI camera – but was so novel at the time that the supporting knowledge eco-system had not yet developed which effectively made it unfeasible to develop for in the short term.

And a third lesson learned is the critical importance of training data for AI computer vision algorithms/  or example, to recognise blocks placed on a school desk in daylight, the algorithm should be trained on images from as similar an environment as possible, but randomised sufficiently to avoid ‘overfitting’. This process of training AI algorithms also provided us with rich insights, from an educational conceptual perspective, into current neural network models and neuroscientific theories of how human brains learn – as well as some of the power and limitations of these theories.

Future challenges

With a prototype now delivered which can successfully recognise Cuisenaire rods, running on a GPU-equipped laptop and webcam, we are now looking towards potential future phases of development.. We’d like to revisit recognising plain cubes, and to make the app accessible on other devices like low-spec computers or mobile phones, allowing us to gather data on block play more widely from schools, as well as enabling children and their families to use the app at home.

We would also like to develop an AI app to analyse the block play data and recognise patterns, for example symmetries in constructions, or commonalities and differences across settings or over time, or compared with digital block play. Currently assessment of children’s activities in pre-school is often, like the curriculum, very different from primary school, and an app that could gather and showcase a portfolio of children’s real-world block play – potentially in virtual worlds if they wish – might enable more continuity in formative assessment across the transition from pre-school to primary.

Expanding the remit

We are also interested in the applications of a simple set of physical blocks as an interface, for example for playing musical notes, or modelling language, or atomic reactions in climate science, as well as for children with visual impairments who may not be able to see touch screens easily. And there also is the potential to translate the digital 3D models of children’s physical block constructions into current 3D online block metaverses such as Minecraft, to bridge the two worlds.

We are keen to work with partners across creative and technical disciplines who are interested in exploring opportunities to augment physical block play with multi-modal digital experiences. If you would be interested in learning more or a chat about the project please get in touch with us: alf.coles@bristol.ac.uk

BDFI academic Xenofon Vasilakos, lecturer of AI for Digital Infrastructures with the Dept. of Electrical and Electronic Engineering and a member of Smart Internet Lab at the University of Bristol, discussed the current orchestration systems intelligence gap when devising 6G network services at the IEEE International Conference on Communications this week. Below, he explains a Reinforcement Learning model-based orchestration approach tested with Bristol’s 5G city testbed and a real use case, which tries to address this intelligence gap while aiming at multi-objective optimisation goals. Further, Xenofon explains how this work poses a basis for integrating and supporting sociotechnical aspects in the future, such as fair resource consumption by users and services.

Network softwarisation in the fifth and future sixth generation of wireless networks (5G, 6G) is characterised by significant flexibility and agility as a result of adopting the concepts of Software Defined Networking (SDN) and Network Function Virtualization (NFV). The former have enabled scalable vertical industry services with strict performance requirements that need to be addressed by MANagement and Orchestration (MANO) systems. Nonetheless, today’s state of the art in MANO systems faces fundamental challenges regarding the highly complex problem of optimal user service function placement. MANO systems still lack Machine Learning (ML) intelligence while remaining largely dependent on rule-/heuristic-based solutions focusing exclusively on system-level resources after predefined policies.

The above approach neglects critical technical aspects such as network dynamics and system-wide service-level performance objectives of both verticals and infrastructure providers as expressed by Key Performance Indices (KPIs) such as service latency or balanced resource utilisation. In addition, it neglects the potential of including Key Value Indicators (KVIs) such as user access fairness to deployed services to avoid user starvation.

To address these gaps, we propose and present our latest work entitled “HELICON: Orchestrating low-latent & load-balanced Virtual Network Functions” to the IEEE International Conference on Communications, in May 2022, Seoul, South Korea (https://icc2022.ieee-icc.org/), during the “QoE And Network Systems” leg of the technical symposium of “Next-Generation Networking and Internet (NGIN)”.

HELICON stands for “Hierarchical rEinforcement LearnIng approach for OrChestratiNg” low-latent and load-balanced services. Though targeting purely technical KPI-based objectives in the current stage, HELICON paves the way for introducing also KVI-based objectives into the MANO equation, thus setting the necessary technical background for supporting socio-technical aspects in current and future 6G MANO operations. In brief, HELICON:

• Poses a novel distributed hierarchical Reinforcement Learning (RL) approach that can serve as a stand-alone online service placement solution as well as a module-based extension for the current state of the art.
• Tackles a computationally/analytically difficult problem (NP-Hard) with a tunable and lightweight Q-Learning scheme that besides KPIs can also support KVIs in the future such as fair access to resources by both users and services. In its current pilot version, HELICON optimises either or both of (i) end-to-end service delay or (ii) load balancing among hosting service nodes.
• Last, we provide a real-life testbed implementation and use case-driven validation, and specifically, practical experimental results upon a realistic 5G Smart City Safety (SCS) use case conducted over a Bristol’s 5G city testbed assuming an e2e application video transcoding service.