Welcome to September’s Algorithm Governance Roundup!

This month I spoke to Victoria Ivanova, the Research & Development Strategic Lead at Serpentine. We spoke about Serpentine’s new exhibition The Call by Holly Herndon and Mat Dryhurst, which developed a choral AI model and experimented with a range of ‘data trust’-like legal structures with the aim of empowering the individuals involved in the training process. We were joined by Alex Lawrence Archer, a lawyer at AWO, who advised the Serpentine throughout the project.

The Call is open from 04 October - 02 February. Make sure to catch it in London!

As a reminder, we take submissions: we are a small team who select content from public sources. If you would like to share content, please reply or send a new email to algorithm.newsletter@awo.agency. Our only criterion for submission is that the update relates to algorithm governance, with emphasis on the second word: governance.

We would love to hear from you!

Many thanks and happy reading!
Esme Harrington

In Europe, Finland’s Henna Virkkunen has been announced as the European Commission’s executive vice-president for tech sovereignty, security and democracy. Her portfolio will cover digital policies, including enforcement of platform regulation. Commissioners will be evaluated for potential conflicts of interest by the Parliament’s Committee on Legal Affairs before the relevant policy committees start their more wide-ranging hearings.

The European Commission has hosted the first meeting of the AI Board to discuss the development and uptake of AI in the EU and the AI Act. The Board is comprised of high-level representatives from the Commission and Member States. It plans to produce an AI governance framework to facilitate participation of the Member States in the implementation of the AI Act.

The Commission has announced that over a hundred companies have signed the EU AI Pact, including multinationals and SMEs across the IT, telecoms, healthcare, banking and automative sectors. The Pact supports industry’s voluntary commitments to begin applying the principles of the AI Act ahead of its entry into force. The ‘core’ commitments involve conducting an AI governance strategy, a high-risk AI systems mapping, and promoting AI literacy amongst staff. It is reported that future of the Pact is uncertain following the resignation of Commissioner Thierry Breton.

The Commission has announced the Chairs and Vice-Chairs of the four Working Groups of the Code of Practice for general-purpose AI under the AI Act. The Working Groups focus on transparency and copyright, risk identification and assessment, technical risk mitigation, and internal risk management and governance of providers.

The European Parliament has set up a cross-committee working group to monitor the implementation of the AI Act. The Internal Market and Consumer Protection (IMCO) and Civil Liberties, Justice and Home Affairs (LIBE) committees set up the working group following concerns with the transparency of the AI Office’s staffing and the role of civil society in the implementation process.

The European Parliament Think Tank has published an impact assessment on the proposed directive on adapting non-contractual civil liability rules to AI (the AI Liability Directive, AILD). The study identifies issues with the European Commission’s impact assessment and proposes the AILD should extend to general purpose and other high-impact AI systems, as well as broadening to become a software liability regulation.

The European Commission has sent requests for information to YouTube, TikTok and Snapchat under the Digital Services Act. The platforms must provide information about the design and functioning of the platforms’ algorithmic systems, and their role in amplifying systemic risks including the mental health of users, the dissemination of harmful content, the protection of minors, and the potential for recommender systems to contribute to addictive behaviour.

In France, for the first time, the Commission Nationale de l’Informatique et des Libertés (CNIL), the Data Protection Authority, has consulted the Autorité de la Concurrence, the Competition Authority, on a draft recommendation on mobile applications. The Autorité has suggested that some of CNIL's recommended privacy measures may go beyond what is strictly required by the GDPR and may hinder or distort free competition. In particular, the Autorité raises concerns that an accumulation of data can entrench market power of certain actors and notes that the level of data protection offered can be a competitive parameter.

In Germany, the Advisory Board of the German Digital Services Coordinator (DSC) met for the first time. The Board is an independent body of experts from academia, industry and civil society that supports the DSC. It aims to ensure effective implementation of the Digital Service Act and raise academic questions, in particular relating to the handling of data.

In Ireland, the An Coimisiún um Chosaint Sonraí, the Data Protection Commission (DPC), has formally requested an opinion on AI training from the European Data Protection Board (EDPB). The opinion invites the EDPB to consider the extent to which personal data is processed at various stages of the training and operation of an AI model, including first- and third-party data, and considerations to take into account when assessing the legal basis relied upon by the data controller.

Ireland’s DPC has also opened a statutory inquiry into whether Google has conducted a data protection impact assessment (DPIA) prior to processing personal data for the development of its foundation AI model, PaLM 2.

In the Netherlands, the Autoriteit Persoonsgegevens, the Data Protection Authority, has published its report on AI and algorithmic risks. The report explores how information provision in democracy may be threatened by AI, challenges in the democratic control of AI, and the risks of profiling and selecting AI systems.

In the UK, the Information Commissioner’s Office, the Data Protection Authority, has published a final call for evidence on allocating controllership across the generative AI supply chain. It also shares the ICO’s analysis on generative AI and allocating responsibility for data processing.

The Department for Science, Innovation and Technology’s Responsible Technology Adoption Unit has updated its portfolio of AI Assurance Techniques with new case studies across a range of sectors, including communications, manufacturing, agriculture and public administration.

In the US, the Californian Governor Gavin Newsom has vetoed the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, which would have placed safety requirements on developers of large frontier AI models. Governor Newsom said the legislation was "well intentioned" but would have applied "stringent standards to even the most basic functions". He plans to consult with external experts and the state legislature to develop alternative guardrails.

The Attorney General of New Mexico has filed a lawsuit against Snap alleging its’ policies and design, including ephemeral content and recommendation algorithms, facilitate child sexual exploitation. It also alleges that Snap has designed its platform to be addictive which has led to poor mental health outcomes amongst young users.

The Office of the New York Attorney General has adopted a guide on protecting citizens from AI-generated election misinformation.

The United Nations hosted the Summit of the Future where 193 Member States negotiated a Global Digital Compact. The Compact aims to strengthen international data governance and govern AI, including by establishing an international scientific panel on AI.

The United Nations’ AI Advisory Body has published its final report on Governing AI for Humanity. Check out our April 2024 interview with the United Nation’s Office of the Secretary General’s Envoy on Technology which supported this work.

Audiovisual Generative AI and Conflict Resolution: Trends, Threats and Mitigation Strategies, Witness

Buying AI: Is the public sector equipped to procure technology in the public interest?, Anna Studman, Mavis Machirori, Ada Lovelace Institute

Codes of Conduct in the Digital Services Act, Rachel Griffin

Code of Practice on Disinformation: A Comparative Analysis of the Prevalence of Misinformation and Sources of Disinformation across Major Social Media Platforms in Poland, Slovakia, Spain, and France, TrustLab

Competition in Generative AI and Virtual Worlds, Klaus Kowalski, Cristina Colpin, Zsalt Zambari, European Commission

DSA Guidelines: Due diligence obligations for intermediary services, Netherlands Authority for Consumers & Markets

From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems, Suzanne Snoek, Isabel Barberá, Rhite, Radboud Universiteit

Fundamental Rights Impact Assessment (FRIA) in the AI Act: Roots, legal obligations and key elements for a model template, Alessandro Mantelero

Guidelines for Participatory and Inclusive AI, Partnership on AI

Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI, Gaël Varoquaux, Alexandra Sasha Luccioni, Meredith Whittaker

Public AI: Making AI work for everyone, by everyone, Mark Surman, Nik Marda, Jasmine Sun, Mozilla

Safety by Design for Generative AI: 3-Month Progress Report on Civitai and Metaphysic, Rebecca Portnoff, Thorn

Should AI systems behave like humans?, UK AI Safety Insitute

The problem of algorithmic bias in AI-based military decision support systems, Ingvild Bode, Ishmael Bhila, ICRC

Tech Policy Press is calling for applications for its 2025 Fellowship Program. It will support five fellows working on AI policy, governance and regulation with a stipend of $10,000.
Application deadline is October 15.

Princeton Laboratory for AI is seeking AI Policy Fellows to support US federal agencies during the implementation of several government AI policy initiatives. Placements could include the Departments of Commerce, Energy, Homeland Security, Justice, and State, and the US AISI, and will last for one year.
Application deadline is 24 October.

The Autoriteit Persoonsgegevens, the Dutch Data Protection Authority, is calling for input on the prohibition of manipulative and exploitative AI systems under the EU’s AI Act, including specific criteria to categorise the systems.
Submission deadline is 17 November.

Victoria Ivanova is Research & Development Strategic Lead at Serpentine. We spoke about Serpentine’s new exhibition The Call by Holly Herndon and Mat Dryhurst, which involved the development of  choral AI models and experimented with a range of ‘data trust’-like legal structures with the aim of empowering the individuals involved in the training process. We were joined by Alex Lawrence Archer, a lawyer at AWO, who advised the Serpentine throughout the project. The Call is open from 04 October - 02 February.

What is Serpentine, and can you introduce your upcoming AI-driven exhibition?
Victoria: Serpentine, located in Kensington Gardens in London, is a public cultural institution that showcases contemporary art practices. As a non-collecting institution, the remit of our work has changed over time. From 2010, as digital technologies and platforms began to exert more influence over our daily lives, we began collaborating with artists who work with emerging and advanced digital technologies. 

At this point we launched the digital technologies programme, led by my colleague Ben Vickers, our Chief Technology Officer — the first ever CTO at an arts organisation. We worked on several projects where artists developed their own tech stacks to produce artworks, including commissions using AI and augmented reality. In these projects, the artist approached technical development as a form of artistic intervention and technical crafting. The production of these artworks revealed the potential for culture to generate insights for society concerning the ways in which technology could be developed, as well as cross-cutting questions regarding agency, responsibility, and the balance of innovation and public good in relation to advanced technologies. 

Our current project with Holly Herndon and Mat Dryhurst, The Call, involves the creation of  choral AI models. Holly and Mat are musicians and artists whose practice tries to test the boundaries of AI and decentralised technologies, viewing them as co-collaborators rather than mere tools. 

The Serpentine exhibition will present the new project in an experientially compelling way, opening a window into the production process and the larger philosophical ideas it explores. This includes the concept of choral and collective voice as a form of communication technology, which is a theme running through Holly and Mat’s practice. Like a choir, where many individual voices become a collective, the artists propose that AI can augment the transformation from the individual to the collective. 

The exhibition will also present our immediate findings concerning our experiment to create a ‘data trust’ to protect the individual voices used in the model’s development. This work explored what it means to collectivise voices, the forms of agency that can emerge from this collectivisation, and the models that could secure that agency, including legal structures.

How was this work developed and why was it important to create a ‘data trust’ to promote the rights of the individuals involved in the training and development of this project?
Victoria: To develop the choral AI models, it was necessary to produce a curated dataset of choral singers. This was created by recording 15 choirs across the UK, performing a piece specifically composed by Holly and Mat for the purpose of training the AI models. This humanised the often dehumanising and automated process of collecting training data. The performers were actively involved in and aware that the recording would be used for training purposes, unlike the widespread scraping of creative works from the internet. We worked with the French public research institute IRCAM to train a ‘raw’ AI model on this dataset. Holly and Mat then adapted this model, deploying various treatments and techniques to reflect their artistic practice and signature to create their version of the choral AI model.

To protect and promote the data rights of the choirs involved in training the choral AI models, we wanted to set up a form of ‘data trust’ or intermediary. At the inception of The Call, I was working on a parallel project to develop a license agreement called Partial Common Ownership with a colleague Matt Prewitt, who leads RadicalxChange. This organisation specialises in mechanism design, technology and tools to transform political economies. In 2023, they published a report on value circulation and infrastructure in relation to data and AI, which shaped our thinking for this project. A major problem with current data economies is the huge asymmetry of power between individual data subjects and AI developers. Unfortunately, we lack sufficient intermediation infrastructures which can scale to allow data subjects to exert agency. Therefore, we explored creating such an infrastructure for The Call.

This focus on legal R&D stemmed from previous work done at Serpentine through labs collaborating with partners across academia, industry and policy. For example, the Legal Lab, led by my colleague Alana Kushnir, explored an apparent lack of legal support, provisions and infrastructures for experimental art and technology work. The art we produce often pushes the law in uncomfortable ways, revealing gaps and areas that may require policy development. 

How have you protected individuals’ data rights and promoted equitable participation, governance, and ownership in practice?
Victoria: To implement the data trust, we hired Jennifer Ding, at the Alan Turing Institute, to be our data intermediary. Her role involved organising collective deliberative and educational sessions, helping the choirs to understand their data preferences and empower them to make informed decisions about their data rights. We also worked with AWO on the content of the applicable data rights and experimented with collectivising these data rights by mandating them to an intermediary. 

We found that solo singers could benefit most from mandating their data rights to a third party, allowing them to be exercised collectively and without direct involvement. For the other performers there was limited scope for data rights due to the nature of choral data and questions of data subject identifiability. Instead, intellectual property law and performance rights were more applicable to govern the downstream uses of the choral recordings in the training dataset. Implementing this data trust was challenging, and its legal enforceability is somewhat unclear. 

What are some key learnings for other stakeholders interested in data trusts for AI systems?
Alex: At the root of generative AI are people. They have a range of conflicting interests, from artistic to commercial, which raise governance issues. This project revealed that the legal mechanisms to appropriately resolve these governance issues are not easily established. At the outset of this project, there was an assumption that a data trust would be an appropriate governance solution. In practice, this quickly dissolved into a plethora of questions: What are the rights? Whom do they belong to? Can they be mandated or assigned? If so, should they be mandated to a natural person or some other legal entity? Can it be a charity? Can such arrangements meet the requirements for a trust in law? 

It was challenging to create a ‘data trust’ because the answers were very uncertain. For this project, the best we could do was create an approximation to a data trust through other mechanisms. There was too much legal uncertainty for a true legal trust over data rights to be created. For example, it was unclear what the substantive obligations on any ‘trustee’ would be, or whether/how they would be remunerated.

This project offers a case study of the development of a model using a curated dataset with collaboration between the various parties involved. This is quite different from the large internet crawler datasets which are used for large foundation models. It is likely that the difficulties we encountered would be magnified in a situation with opposing interests, which raises additional questions about the scale of the challenge.

This project also revealed that the future of generative AI governance lies at the complex interplay of data protection and intellectual property law. Our project, with a focus on choral music, meant that there were relatively few data rights (i.e. rights under the UK GDPR) involved, and intellectual property law was more applicable. This will be different in other projects.

There is a need for stakeholders in the data governance space to be more precise in their language, as ‘data trust’ is being used to encompass a range of legal mechanisms and institutions posited as solutions to a wide range of issues. For example, some people use the term to reference a charitable trust that involves contractual rights related to datasets, whereas others are more focused on the collective assignment of data subject rights in relation to processing.

Many people rightly believe that we need to create governance mechanisms to empower the originators of content in generative AI models. Going forward, we need more projects and case studies across various contexts and sectors to explore how to empower collective groups of data subjects and creatives. It is not clear whether the law will be able to solve this governance issue, but experimentation is key.