The AI Act is finally here
AI and bias. The Ukrainian military startup.
Digital Conflicts is a bi-weekly briefing on the intersections of digital culture, AI, cybersecurity, digital rights, data privacy, and tech policy with a European focus.
Brought to you with journalistic integrity by Guerre di Rete, in partnership with the University of Bologna's Centre for Digital Ethics.
New to Digital Conflicts? Subscribe for free to receive it by email every two weeks.
N.5 - 20 March 2024
Authors: Carola Frediani and Andrea Daniele Signorelli
Index:
- The European Union Has Its Own Artificial Intelligence Law
- AI Researchers: Where is AI talent coming from and where is it going?
- LLM and Racial Bias
- The Ukrainian incubator of military technology
- In Brief
AI ACT
The European Union Has Its Own Artificial Intelligence Law
On 13 March, the European Parliament approved the AI Act: the European regulation on artificial intelligence. Here's what you need to know.
After the final formalities, the AI Act will officially enter into force by May or June.
Its provisions will gradually come into effect:
- Within six months: countries will be required to prohibit banned AI systems.
- Within one year: rules for general-purpose AI systems will begin to apply.
- Within two years: most of the AI Act will apply.
- Within 3 years: the obligations for high-risk systems come into effect.
Fines for non-compliance could be up to €35 million or 7% of the company's annual global turnover.
Prohibited:
- Exploitation of vulnerabilities of individuals or groups based on age, disability, or socio-economic status.
- Manipulative and deceptive practices, such as systems that use subliminal techniques to materially distort a person's decision-making ability.
- Biometric categorisation systems based on sensitive characteristics, which classifies individuals based on biometric data to infer sensitive information such as race, political opinions, or sexual orientation (exceptions are provided for law enforcement activities).
- Social scoring (evaluating individuals or groups over time based on their social behaviour or personal characteristics).
- Untargeted scraping of facial images from the internet or CCTV footage to create facial recognition databases.
- Emotion recognition in workplaces and educational institutions (with exceptions for medical or security reasons).
- Predictive policing (when it is based solely on profiling a person or assessing their characteristics)
Real-time biometric identification (RBI) in public spaces by law enforcement is not completely banned, but limited. Identification is permitted in defined circumstances (permitted uses include searching for a missing person or preventing a terrorist attack), subject to judicial or independent authority approval.
Biometric identification systems post-facto (“post-remote RBI”) is considered a high-risk use case, requiring judicial authorisation being linked to a criminal offence.
The following areas are not prohibited but are considered “high risk". Systems operating in these areas will be assessed both before being placed on the market and during their lifecycle. Citizens will be able to submit complaints to national authorities.
High-risk areas include not only critical infrastructures or security components but also education (for determining access or admission, assigning individuals to institutions or training programs at all levels, assessing the learning outcomes of individuals, assessing the appropriate level of education for an individual and influencing the level of education they can access, monitoring and detecting prohibited student behaviours during tests); employment (for hiring and selecting individuals, making decisions about terms and conditions of employment contracts, assigning tasks based on individual behaviours, traits, or personal characteristics, and monitoring or evaluating individuals); essential services including health care, social security benefits, social services, and credit rating; administration of justice (including alternative dispute resolution bodies); migration and border management (such as examining applications for asylum, visa, and residence permits and related claims).
General purpose AI (GPAI) systems and the models on which they are based (including large generative AI models) will have to comply with a number of transparency requirements, such as: disclosing that the content is generated by AI, ensuring that models do not generate illegal content; compliance with EU copyright law and publishing detailed summaries of the content used for training.
The most powerful models that could pose systemic risks will also have to comply with other obligations, such as conducting model assessments, assessing and mitigating systemic risks, and reporting incidents.
EU countries must create and make available at the national level regulatory sandboxes and real-world testing to allow SMEs and startups to develop AI systems before they enter the market.
(Compiled from the final text, the European Parliament document, and comments by Luiza Jarovsky and Barry Scannel)
Reactions
There is a myriad of reactions to the AI Act, many positive and celebratory, but for now I'll only mention a couple of statements from those who wanted a stricter AI Act in protecting certain rights.
"While the AI Act may have positive aspects in other areas, it is weak and even enables the use of risky AI systems when it comes to migration", writes the #ProtectNotSurveill coalition.
The AI Act, according to the NGO Access Now, "fails to properly ban some of the most dangerous uses of AI, including systems that enable biometric mass surveillance".
AI RESEARCHERS
Where is AI talent coming from and where is it going?
The United States remains a “net importer” of artificial intelligence researchers. While the USA is the main destination for high-level AI talent to study and work, China is gradually increasing its national pool. In addition to the United States and China, the United Kingdom and South Korea, along with continental Europe, have slightly increased their status as work destinations for top AI researchers. France and Germany stand out among the European countries that produce a significant number of top AI researchers, and are partially successful in retaining or attracting them.
These are some of the recently updated data from the Global AI Talent Tracker 2, based on the analysis of job profiles.
According to the Tracker, the "leading countries of origin of the most elite AI researchers (top ~2%, based on undergraduate degrees)" are distributed as follows: USA 28%, China 26%, India 7%, France 5%, Germany 4%, Canada 2%, Others 28%.
But if we look at where they work, the distribution changes: the USA grows to 57%, China decreases to 12%, the UK appears at 8%, Germany and France remain almost stable at 4%, Canada at 3%, and Others at 12%.
AI AND BIAS
LLM and Racial Bias
Bias in algorithmic systems of various kinds has been studied for years. Even the latest language models (such as GPT-4) have been the subject of these studies. Recently, a paper has been published (by various researchers from the Allen Institute, Oxford, Stanford, LMU Munich, Chicago) that examines racial bias against African Americans, focusing on the most subtle and hidden forms. It used linguistic forms typical of African American communities, and compared them with texts written in standard American English (SAE).
The research would demonstrate, to quote its authors, that "Americans hold raciolinguistic stereotypes about speakers of African American English". These stereotypes are also present in the linguistic models but show "covert stereotypes that are more negative than any human stereotypes about African Americans ever experimentally recorded, although closest to the ones from before the civil rights movement. By contrast, the language models' overt stereotypes about African Americans are much more positive".
In practice, there would be a discrepancy between what linguistic models openly say about African Americans and what they less visibly associate with them. Furthermore, the authors argue, not only do linguistic models suggest that those who speak African-American English get less prestigious jobs or are more likely to be convicted of crimes, but existing methods to mitigate racial bias in linguistic models, such as human feedback training, do not mitigate such linguistic bias. In fact, they can exacerbate the discrepancy between covert and overt stereotypes by teaching linguistic models to superficially hide racism that they continue to perpetuate at a deeper level.
The risk, writes on Twitter Valentin Hofmann, one of the authors, is that "users mistake decreasing levels of overt prejudice for a sign that racism in LLMs has been solved, when LLMs are in fact reaching increasing levels of covert prejudice".
Commenting on the paper, Margareth Mitchell, Chief Ethics Scientist at Hugging Face, writes that "as developers continue to prioritize post-training techniques to handle racism, as opposed to pre-training work, it will be harder and harder to identify how systems that have already encoded racism will disproportionately harm marginalized subpopulations once deployed".
Bias and Work
There is a widespread misconception that AI tools are less biased than humans because they work on a broader set of data, said Abeba Birhane, senior AI accountability advisor at the Mozilla Foundation. The problem is that this assumption is rarely verified, and models are not carefully vetted and tested. But there is growing evidence that "these systems stereotype," Birhane said.
This comment appears in an investigation by Bloomberg, which looked at a different aspect of the presence of bias in LLMs, for example in relation to their potential use in recruitment. According to Bloomberg's analysis, GPT 3.5 (OpenAI's language model) "systematically produces biases that disadvantage groups based on their names". The issue is complex, and I encourage you to read the details directly here.
Gender Bias Research
In recent days, a UNESCO and IRCAI study into bias against women and girls in large language models has been published. The study analyses three models – GPT-2 and ChatGPT from OpenAI, and Llama 2 from Meta – and argues that bias emerges in the generated text through gender-related word associations.
For example, British men were associated with a wide range of roles, from driver to teacher, highlighting the diversity of their occupational representations. Conversely, British women were often associated with more stereotypical roles – and I quote from the paper – "such as prostitute, model, and waitress, appearing in approximately 30% of the total texts generated".
WAR AND TECH
The Ukrainian incubator of military technology
The Russian invasion has given a strong boost to the Ukrainian technology sector. The BRAVE1 incubator aims to transform Ukraine into an advanced weapons producer.
"This activism has led the American magazine Wired to compare the organization of a startup to the way Minister of Digital Transformation Fedorov conducts war in his field of expertise. The comparison is fitting because, over the past two years, innovation in the defence sector has been driven mainly by private companies that have adopted an agile approach to the task, far removed from the top-down approach that Ukrainian military institutions have inherited from their Soviet heritage. This is the context in which the BRAVE1 defence technology cluster was born”.
Read the full article here [Italian version only]
IN BRIEF
AI AND THE ENVIRONMENT
Karen Hao writes on the Atlantic: "Microsoft’s own environmental reports show that, during the initial uptick in the AI platform’s growth, the company’s resource consumption was accelerating. In fiscal year 2022, the most recent year for which Microsoft has released data, the tech giant’s use of water and electricity grew by about a third; in absolute terms, it was the company’s largest-ever reported increase, year to year".
PROMPT ENGINEERING
New research suggests that prompt engineering is better done by the model itself, not by a human – Spectrum.ieee
JOURNALISM AND SOCIAL MEDIA
LinkedIn focuses on news as social media rivals retreat - Axios