Welcome to AWO’s Algorithm Governance Roundup. We’re back after a short(ish) break whilst I’ve been exploring the wonderful Brazil. In the meanwhile, implementation and enforcement of AI and platform regulation has continued apace in the EU and UK. We’ve got plenty of updates below, opportunities to contribute to the European Commission, alongside some super research to add to your summer reading list! Also do check out my colleague Ravi Naik’s contributions to the UK’s Joint Committee on Human Rights session on Human Rights and the Regulation of AI.

This month, I had a great conversation with Professor Didar Zowghi who leads the Data61 team on Diversity and Inclusion in AI at Australia’s CSIRO, the national science agency. We spoke about developing a robust definition and guidelines for diversity and inclusion in AI, and working with industry and public bodies on specific AI use cases in recruitment, media and sports.

As a reminder, we take submissions: we are a small team who select content from public sources. If you would like to share content, please reply or send a new email to algorithm.newsletter@awo.agency. Our only criterion for submission is that the update relates to algorithm governance, with emphasis on the second word: governance.

We would love to hear from you!

Many thanks and happy reading!

Esme Harrington (and thanks to Lucie Audibert for helping to draft this issue)

In Austria, the Austrian data protection authority has received a formal complaint about Bumble’s new AI feature, submitted by Austrian privacy group noyb. The complaint alleges that Bumble’s “icebreaker” feature shares user profile data with OpenAI’s ChatGPT without valid consent. The complaint alleges that Bumble misleads users with a consent-like prompt, but the processing actually occurs under ‘legitimate interest’.

In the EU, in relation to the Digital Services Act, the European Commission (EC) has adopted the delegated act on access to data. This outlines rules for granting access to qualified researchers to internal data of Very Large Online Platforms and Search Engines (VLOP/VLOSEs) to research systemic risks and mitigations. It clarifies the procedures to share data, including data formats and requirements for data documentation.

The EC has presented a revised draft of the guidelines on the protection of minors to the Digital Services Coordinators in a closed-door meeting. This follows the EC receiving over 400 comments on its draft guidelines on the protection of minors which outline a non-exhaustive list of privacy-by-design measures. The EC is also developing its own age-verification app, with the beta version and technical specifications published on Github.

The EC issued its preliminary view that TikTok’s ad repository breaches the DSA by failing to provide necessary information on, or enable the public to search about, the content of ads, who purchased them, and which users were targeted. In addition, the EC is investigating TikTok’s recommender system amid growing concern over “SkinnyTok”, which may promote harmful body image and eating content to minors.

The EC also sent X a request for information concerning its acquisition of xAI and is interested in how the acquisition could impact DSA-related investigations and potential fines.

The EC is considering whether to designate ChatGPT as a Very Large Online Platform/Search Engine under the DSA. OpenAI reported a rise in monthly active EU users from 11.2 million in October to 41.3 million in March, nearing the 45 million threshold for such classification. Read AWO's 2023 analysis on GenAI products and the DSA.

The EC has proposed to change the supervisory fee regime for the DSA. This will change the rules underpinning the fees paid by Very Large Online Platforms to fund the enforcement of the DSA. Several VLOPs, including Google, TikTok and Meta have appealed the current fee structure, alleging that the fee calculation method and the fees are not fair nor proportionate.

Amazon has challenged the EC’s decision to designate its online marketplace as a VLOP. It argues that classification is disproportionate on the basis that online marketplaces do not pose systemic risks and the VLOP-specific rules cannot prevent the dissemination of illegal goods.

On the EU’s AI Act, the EU AI Office presented its final version of the Code of Practice on General Purpose AI (GPAI) to stakeholders. This followed reports that the final publication of the Code of Practice would be delayed until August. The related AI Act provisions come into force on 02 August and the EC’s Executive Vice-President for technology has not ruled out delaying the implementation of the AI Act if standards and guidance are not ready by the end of August. Several member states support delaying implementation or even simplifying the AI Act, including Czechia, Spain, and Poland. The U.S. administration has also been lobbying against the draft Code of Practice for GPAI.

The EC launched a public consultation on the implementation of rules on high-risk AI systems under the AI Act. The consultation aims to collects practical examples and clarify issues relating to high-risk AI systems. Deadline for consultation responses is 18 July.

The EC's Joint Research Centre published a report examining generative AI's impact within the EU. The report considers its potential for innovation, productivity, and societal change, including the risks of bias and over-reliance. It also outlines the relevant regulatory framework in the AI Act and data legislation.

In other EU news, the EC has opened a consultation on the Consumer Agenda 2025-2030 and its accompanying action plan on consumers in the Single Market. The Agenda includes the Digital Fairness Act which aims to tackle manipulative and unethical practices online, including dark patterns, misleading influencer marketing, addictive design features and problematic personalisation that falls outside the DSA and DMA. Deadline for consultation is 11 August.

The EC has fined Apple and Meta for breach of the Digital Markets Act (DMA). Apple was fined €500 million for breaking the DMA’s rules on app stores, which require that app developers be able to advertise offers outside the app store. Meta was fined €200 million for its “pay or consent” model, as the Commission found that it did not give users the choice to opt for a service that uses less of their personal data but is otherwise equivalent to a service with personalised ads.

The EDPB has published its 2024 annual report on the Board’s work on cross-regulatory cooperation, opinions, guidelines, engagement with stakeholders and educational materials. This includes reflections on its opinions on the use of facial recognition at airports, use of personal data to train AI models, and statements on the DPAs role under the AI Act.

The EC has proposed the fourth Simplification Omnibus package. This introduces a new category of companies, small mid-caps (SMCs), which have fewer than 750 employees and either up to €150 million in turnover or up to €129 million in total assets. SMCs will access certain existing SME benefits including specific derogations under the GDPR.

In France, President Macron has committed to banning social media for children under 15, following a fatal stabbing of a teaching assistant. At the same time, Greece’s digital minister submitted a proposal to the EU to limit the amount of time teenagers can spend online. It is supported by France, Spain and Denmark amongst others.

The Commission Nationale de l’Informatique (CNIL), the data protection authority, has published its third AI regulatory sandbox recommendations focused on AI projects aimed at improving public services.

CNIL has also published recommendations on AI developer’s use of legitimate interests as legal basis under the GDPR. It provides concrete examples of the guarantees that developers must provide, and the circumstances in which legitimate interests can be used.

In Germany, a regional court has allowed Meta to start training AI models with German user data. The court rejected an urgent injunction request by the North Rhine-Westphalia Consumer Advice Centre, after Noyb issued a cease-and-desist letter alleging that Meta may be in breach of the GDPR and DMA.

The Datenschutzkonferenz, Germany’s data protection authorities, issued joint guidance on AI and data protection. It aims to help AI developers, cloud service providers and healthcare practitioners handle personal data responsibly. Additional documents address secure cloud use and data-compliant online appointment booking in medical practices. A new procedure for private sector fines was also introduced.

The Datenschutzkonferenz has also notified Apple and Google that the DeepSeek app qualifies as illegal content under the DSA, requiring them to assess whether to block the app from their German app stores. This follows DeepSeek failing to act on a request from the DPA to remove its app from the app stores on the basis that DeepSeek failed to comply with data protection law.

In Hungary, the EC requested clarification on Hungary’s potential use of facial recognition to identify attendees at Pride events. Hungary banned Pride gatherings and amended existing laws to allow police to use facial recognition to identify attendees. If the police are using ‘real-time’ systems, Hungary could breach the AI Act prohibitions.

In Italy, Garante, the data protection authority, has fined Replika.ai for violating data protection laws. The company was fined 5 million euros for lack of legitimate legal basis for processing user data and failure to implement age verification mechanisms to prevent minors from accessing the service. In addition, Garante has launched a new investigation into Replika’s model training under the GDPR.

In Japan, the Fair Trade Commission published a report on its investigation into the generative AI market. Several respondents raised concerns about existing Big Tech companies integrating their generative AI models into their existing digital services, which could constitute “tie-in sales” or “private monopolisation” if its purpose is to hinder competitors’ or raise barriers to entry.

In the Netherlands, the Autoriteit Persoonsgegevens, the data protection authority, held a consultation on the responsible development and deployment of generative AI. The report includes an overview of the technology, current trends, outlines future scenarios, alongside legal interpretation.

In the UK, Parliament has passed the Data (Use and Access) Bill. It aims to facilitate data sharing across the private and public sector. It requires Ofcom to introduce a mechanism for researcher access to social media data, alongside limited commitments on copyright and AI requiring developers to report on AI training practices.

Ofcom has published its consultation on additional safety measures under the Online Safety Act. These are a further set of targeted safety measures that include reducing the spread of illegal content through recommender system changes and crisis response protocols, proactive technologies for known illegal images, new user sanctions, and stronger protections for children. Deadline for consultation is 20 October.

Ofcom has also published its strategic approach to AI which will prioritise combatting AI-enabled risks on online platforms, including deepfakes and recommender system-driven harms, through the enforcement of the Online Safety Act.

The Information Commissioners’ Office (ICO), the data protection authority, has also published its AI and biometrics strategy. It prioritises oversight of high-risk use cases, namely the development of foundation models, use of automated decision making in recruitment and facial recognition by police forces. The ICO will also publish a new statutory code of practice for AI, and update guidance on profiling and automated decision making.

The Digital Regulation Cooperation Forum (DRCF) has published a study on consumer use of generative AI. This will inform regulatory approaches at each of the constituent regulators of the DRCF – the Competition and Markets Authority, the Financial Conduct Authority, the ICO and Ofcom.

Ofcom has opened nine investigations into services under the Online Safety Act. This includes investigating 4chan for failing to complete a suitable and sufficient illegal harms risk assessment, seven file-sharing services for possibly hosting CSAM, and a pornography site in relation to age verification rules. All the services failed to respond to Ofcom’s statutory information requests.

The Wikimedia Foundation has brought a legal challenge to the Online Safety Act Categorisation regulations. It argues the criteria in the Regulations are too broad and vague, focusing on the number of visitors to a website. As a result, Wikipedia is concerned it could be categorised as a Category 1 service and face the most onerous obligations.

A judge in the High Court of England and Wales issued a formal warning on the use of LLMs following two recent instances where lawyers submitted fictitious cases generated by AI tools. The judge cautioned that AI outputs can be wrong and must be verified, and submitting fictitious AI-generated cases could lead to criminal charges.

Getty Images has dropped the copyright infringement allegations in its lawsuit against Stability AI. The lawsuit was one of the first challenges of generative AI training on IP-protected works available online in the UK. During closing arguments, Getty decided to only pursue claims for trademark infringement, passing off and secondary infringement of copyright. The case was complicated by the fact that the AI training was done predominantly in the U.S.

In the US, the Senate almost unanimously voted against the proposal for a 10-year moratorium on state regulation of AI. This would have prevented states from passing legislation that imposes burdens on the use of AI.

In San Francisco, a federal judge ruled that Anthropic did not infringe copyright when it trained Claude on millions of copyrighted books on the basis that it was ‘fair use’ because it was ‘quintessentially transformative’. However, the judge ruled that Anthropic must face trial over alleged theft of works from training on pirated books. A separate court ruled that Meta did not breach copyright law by training Llama on books without the authors’ permission. The judge stated that the authors had not presented enough evidence that Meta’s AI would cause ‘market dilution’ by flooding the market with similar work, as a result the training was determined to be ‘fair use’.

Professor Didar Zowghi leads the Data61 team on Diversity and Inclusion in AI at Australia’s CSIRO, the national science agency.

Didar: The Commonwealth Scientific and Industrial Research Organisation (CSIRO) is Australia’ national science agency. CSIRO aims to solve Australia’s greatest data-driven challenges through innovative science and technology. Data61 is the data and digital specialist arm working across AI, robotics, cybersecurity, modelling and analytics.

The Data61 team on Diversity and Inclusion in AI was created to pioneer research on diversity, equity and inclusion in AI. It was created in the light of the increasing number of incidents of AI harm arising from biases, discrimination and marginalisation across the globe. The vast majority of published or widely reported incidents have occurred in either the USA or Europe. When I began this work in 2022, I was interested to identify any incidents in the Australian context. Initially, the only instance concerned Australian supermarkets’ use of facial recognition technology, which prompted such a backlash that the systems were discontinued. At the time, there was limited exposure in other industries such as healthcare and recruitment. More recently, we’ve started to identify a few incidents through AI incident databases, such as the OECD’s recently published database. However, there continue to be relatively few reports in Australia, likely due to our relatively small population and slower adoption of AI technology.

Our team sits within a larger Data61’s Software Systems Research Group (SSRG). Our software engineering for Responsible AI researchers have developed a Pattern catalogue to operationalise responsible AI from a systems perspective. The SSRG has also developed a question bank and metric catalogue to guide businesses in conducting concrete risks assessments of AI systems. There has also been a lot of work on AI safety from a systems perspective. For example, we have developed reference architecture for designing multi-layered run-time guardrails for foundational models. Recently, the SSRG partnered with an investor company to develop an integrated ESG and AI framework to help investors identify and manage emerging AI risks and opportunities.

Didar: We began by attempting to define diversity and inclusion in relation to AI. In 2022, when I started this work, the field lacked a convincing and overarching definition. We normatively define it as the ‘inclusion’ of humans with ‘diverse’ attributes and perspectives in the data, process, system, and governance of the AI ecosystem. This relates to our framework of five pillars that cover what diversity and inclusion in AI mean. The first and overarching pillar is humans because humans are the core part of AI and indeed any systems. The second pillar is data, often the most or only concept discussed in relation to AI inclusion, and the third and fourth pillars are processes and systems, the latter referring to algorithms and models. The fifth and overarching pillar is governance. Without an effective governance mechanism and methodology, you cannot claim to achieve diversity and inclusion in data, processes, or systems.

Based on this foundational research, we developed a collection of 48 guidelines for achieving diversity and inclusion in AI, which are available in this textbook chapter. This was based on a review of relevant academic and grey literature. Each of our guidelines offer comparatively generic advice across the five pillars, and not all may be applicable in every use case.

After we developed the definition and the guidelines, we engaged with different public bodies and industries to test its’ usability and effectiveness in specific scenarios. This includes a range of smaller start-ups, medium sized- and larger companies. For instance, we worked with SEEK, the largest recruitment company in the Asia-Pacific region, to co-design workshops on specific use cases in recruitment. Through this engagement, we developed a methodology to enable organisations to tailor and operationalise the guidelines in a specific sector and use case, including a question bank of over 200 yes and no type questions. We also developed a user story template to capture Diversity and inclusion in AI requirements.

Of course, companies are made up of a range of stakeholders. There is a common misconception of the ‘unicorn’ AI expert that can do and know everything, when this is a multi-faceted and multidisciplinary area of work and research. Providing actionable guidelines for all stakeholders has been an on-going learning process for us. Our first version focused primarily on the designers, people responsible for monitoring and evaluation, and data scientists preparing the data pipelines, with less guidance for the engineers developing the models. This is something we are building on in our second version.

Benchmarks for inclusive indicators is a fundamental question for our field. As a society, we are not doing an excellent job at diversity and inclusion. This creates fundamental problems with assessing or evaluating an AI system against a set of external benchmarks. Inclusion is not a mathematical formula you can simply mitigate. Do we have a dataset that shows inclusivity benchmark for a sector, e.g. recruitment? And if we do within a specific sector, do we want the AI system to behave in this way?

Therefore, we decided to create a question bank which can be tailored to a specific sector and use case. It serves two purposes: to educate organisations by identifying knowledge gaps across the five pillars, and to help them assess their maturity or readiness for applying diversity and inclusion considerations in AI development and deployment. If an organisation cannot answer the questions relevant to its use case affirmatively, it indicates there is room for improvement. The next piece of work we are engaging with is whether we can empirically try to allocate some scoring system to these questions.

We also have a project on AI incidents. Our team has manually analysed documented incidents to determine what proportion can be attributed to neglecting diversity and inclusion principles. Based on this work, we have developed a decision tree to identify AI incidents that could be caused by neglecting these principles. Our paper on this topic has been accepted for publication in the journal of AI Research.

First of all, our 48 guidelines have been adopted into the National AI Assurance Framework, which provides guidance on how government agencies should assess AI systems that they develop, procure and deploy.

We have also worked with several public bodies to implement our guidelines using our methodology. This includes leading co-design workshops with the Australian Broadcasting Corporation (ABC), working with journalists on specific use cases to identify the relevant guidelines and tailor them to the media industry. We are also working with the Australian Sports Commission on guidelines for organisations using AI systems in sports, which will be one of the first frameworks in this sector.

More recently, we have become involved in a bilateral project between the Australian and Indian governments on ‘inclusive AI by design’ in the healthcare sector. I recently joined a cohort travelling to India to discuss and consult on this concept and how it can be achieved in healthcare, which is a highly regulated area that also has a lot of incidents of bias, unfairness and discrimination.

On national AI governance projects, we have worked with the National AI Centre (NAIC), to develop Australia’s voluntary AI Safety Standards. The first version was released in 2024, and we are currently developing the second version.

The Network of Media Structures is calling for abstracts for a workshop on Epistemic governance of platforms: A solution towards public interest-oriented media platforms. The workshop will be held in German on 23 - 24 October at Johannes Gutenberg University Mainz.

The European Commission is seeking independent experts for the AI Scientific Panel, focusing on General Purpose AI Systems (GPAI). The Panel will advise the AI Office and national authorities on systemic risks, model classification, evaluation methodologies, and cross-border market surveillance, including alerting to emerging risks. The EC is seeking 60 members for a renewable 24-month term. Experts must have a PhD or equivalent experience and be independent of any AI provider.

A Hazard to Human Rights: Autonomous Weapons Systems and Digital Decision-Making, Brian Stauffer, Human Rights Watch

AI as Normal Technology, Arvind Narayanan and Sayash Kapoor, Knight First Amendment Institute at Colombia University

AI automation is enclosure: the case for data rent modelled on ground rent, Jai Vipra, European Law Open

AI Governance Profession Report 2025, Richard Sentinella, Evi Fuelle, Ashley Casovan and Joe Jones, IAPP and Credo AI

AI Supply Chain Impact Framework, AI + Planetary Justice Alliance

Algorithmic Gatekeepers: Impacts of LLM Content Moderation on Civic Space and Human Rights, European Center for Not-for-Profit Law

Antiqua et Nova: Note on the Relationship Between Artificial Intelligence and Human Intelligence, Dicastry for the Doctrine of the Faith and Dicastry for Culture and Education, The Vatican

Artificial Power: 2025 Landscape Report, Kate Brennan, Amba Kak and Dr Sarah Myers West, AI Now Institute

Beyond Release: Access Considerations for Generative AI Systems, Irene Solaiman, Rishi Bommasani, Dan Hendrycks, Ariel Herbert-Voss, Yacine Jernite, Aviya Showran and Andrew Trask

Creative Industries and GenAI: Good Work Research Report, Michael Katell, Mhairi Aitken, Kester Brewin, Abigail Gilbert, Peaks Krafft, David Leslie, Mia Leslie, Alex Mehta Brown, Aoife Monks, Claddagh NicLochlainn, Antonella Perini, Vjosa Preniqi, Elona Shatri, Magdalena Soffia, Anna Thomas, Institute for the Future of Work

Coded for privileged access: How Big Tech weakens rules on advanced AI, Corporate Europe Observatory

Children & AI Design Code, 5Rights Foundation

Crowdsourced AI benchmarks have serious flaws, some experts say, Kyle Wiggers, TechCrunch

Digital Decade: How the EU Shapes Digitalisation Research, edited by Rita Gsenger and Marie-Therese Sekwenz

Designing with Uncertainty: LLM Interfaces as Transitional Spaces for Democratic Revival, Sylvie Delacroix

Dramatic rise in publicly downloadable deepfake image generators, William Hawkins, Chris Russell and Brendt Mittelstadt, Oxford Internet Institute

Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers, Jared Moore, Declan Grabb, William Agnew, Kevin Klyman, Stevie Chancellor, Desmond C. Ong and Nick Haber

Five Findings from an Analysis of the US Department of Homeland Security’s AI Inventory, Paromita Shah, Tech Policy Press

Gendered Disinformation as Infrastructure: How Tech Billionaires Shape Political Power, Carine Roos, Tech Policy Press

Guidance for Inclusive AI: Practicing Participatory Engagement, Partnership on AI

How We Investigated Shadowbanning on Instagram, Tomas Apodaca and Natasha Uzcátegui-Liggett, The Markup

Learn fast and build things: Lessons from six years of studying AI in the public sector, Imogen Parker, Anna Studman and Elliot Jones, Ada Lovelace Institute

Making Recommender Systems Work for People: Turning the DSA’s Potential into Practice, Alissa Cooper and Peter Chapman, Knight-Georgetown Institute, DSA Observatory

“One day this could happen to me”: Children, nudification tools and sexually explicit deepfakes, Children’s Commissioner

Potential unreached: challenges in accessing data for socially beneficial research, Bartosz Maj and Valentina Pavel

South Korea’s New AI Framework Act: A Balancing Act Between Innovation and Regulation, Sakshi Shivhare, Future of Privacy Forum

Submission to the United Nations Secretary-General on AI in the Military Domain, International Committee of the Red Cross

Technical Tiers: A New Classification Framework for Global AI Workforce Analysis, Siddhi Pal, Catherine Schneider, Ruggero Marino Lazzaroni, Interface

The Dormant Danger: How Meta Ignores Large-Scale Inauthentic Behaviour Networks of Malicious Advertisers, Aleksandra Atanasova, Reset.Tech

The DSA’s Systemic Risk Framework: Taking Stock and Looking Ahead, Magdalena Joźwiak, DSA Observatory

The Ethics of AI Value Chains, Blair Attard-Frost and David Gray Widder

The Role of Expertise in Effectively Moderating Harmful Social Media Content, Nuredin Ali Abdelkadir, Tianling Yang, Shivani Kapania, Meron Estefanos, Fasica Berhane Gebrekidan, Zecharias Zelalem, Messai Ali, Rishan Berhe, Dylan Baker, Zeerak Talat, Milagros Miceli, Alex Hanna and Timnit Gebru

Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling, Victor Ojewale, Ryan Steed, Briana Vecchione, Abeba Birhane and Inioluwa Deborah Raji

Toward an Evaluation Science for Generative AI Systems, Laura Weidinger, Deb Raji, Hanna Wallach, Margaret Mitchell, Angelina Wang, Olawale Salaudeen, Rishi Bommasani, Sanmi Koyejo,

Understanding the Impacts of Generative AI Use on Children, The Alan Turing Institute

What we learned from tracking AI use in global elections, Khadija Alam, Rest of World

When Guidance Becomes Overreach: How the forthcoming Code of Practice Threatens to Undermine the EU’s AI Act, Martin Ebers, Verfassungsblog

Where Cloud Meets Cement: A Case Study Analysis of Data Center Development, Hanna Barakat, Chris Cameron, Alix Dunn, Prathm Juneja and Emma Prest

Understanding Consumer Use of Generative AI, Thinks Insight & Strategy for the Digital Regulation Cooperation Forum