Prohibitions & obligations under the AI Act

^ AI literacy

Concept of AI literacy

AI literacy in the AI Act is about gaining awareness and making informed decisions: about the benefits and risks of AI (in general, not just for a particular system), possible harm it can cause and associated safeguards, and in turn when deploying AI systems. It includes skills, knowledge and understanding, both on AI (technical knowledge, experience, education, training) and on the AI Act (knowledge of their rights and obligations), and the ability to relate them to the specific conditions of an AI system (the context in which it is to be used, the persons or groups of persons on whom it is to be used).

This concerns providers, deployers, affected persons, as well as any actor in the AI value chain (upstream provider, importer…), with different targeted skills associated to each role: technical knowledge relative to AI system development, measures applicable to their use and the use of the output, understanding the impact of an AI-assisted decision on one’s life, ensuring compliance, etc.

AI literacy obligation in Article 4

Providers and deployers have an obligation to ensure that their staff or other persons they involve in the AI system operation has a sufficient level of AI literacy and is able to apply it on the particular AI systems they are concerned with. This can be achieved through dedicated training, skills assessment, or various forms of communication and awareness-raising.

This obligation applies to the providers and deployers of all AI systems (except for those that benefit from specific exemptions from the AI Act), but it does not apply to individuals who deploy or use an AI system as a personal activity.

Support measures

The European Commission and the Member States are expected to directly support the increase of AI literacy throughout the EU. This includes actions for public awareness, promotion of tools and existing practices, and assisting in the drawing up of voluntary codes of conduct.

See also:

  • The European Commission Q&A on AI Literacy provides extensive information on the expectations on AI literacy, how to comply with Article 4 and how it is enforced, and points to various resources that can help in increasing AI literacy.

  • Examples of AI literacy practices are listed on the European Commission’s website, based on surveys conducted by providers and deployers. Following these practices does not guarantee compliance with Article 4 and additional measures can be needed in one’s particular context, but this can provide useful inspiration.

^ Prohibited AI practices (Article 5)

8 cases of prohibition

  • Harmful manipulation and deception: AI for subliminal, manipulative or deceptive purposes, intentionally pushing persons to make non-informed harmful decisions

  • Harmful exploitation of vulnerabilities: AI exploiting vulnerabilities such as age, disability or socio-economic situation, to distort a person's behaviour in a harmful way

  • Social scoring: AI analyzing behaviour or personality for social scoring, leading to bad treatment of individuals in a contextually unrelated or disproportionate manner

  • Individual criminal offence risk assessment and prediction: AI for profiling-based assessment of a person's risk to commit criminal offence, except in ongoing investigations with supporting facts and human assessment

  • Untargeted scraping to develop facial recognition databases: AI to feed facial recognition databases with large-scale scraping from the internet or CCTV

  • Emotion recognition: AI for emotion recognition at work or in education, except for medical or safety reasons

  • Biometric categorisation: AI to infer from biometric data the race, political opinions, trade union membership, religious or philosophical beliefs, sex life or sexual orientation of individuals, except for the sake of labelling or filtering datasets related to law enforcement

  • Real-time remote biometric identification: AI for remote biometric identification by law enforcement authorities, in real time and in public spaces

Some of these practices correspond in fact to cases of high-risk AI systems, but due to certain characteristics of the AI system they become prohibited. For instance, the same AI system for emotion recognition can be high-risk or prohibited depending on where it is deployed. A credit scoring system can be high-risk or prohibited depending on the specific information it uses about the person (behaviour that is broader than purely financial information, such as their type of expenses). An AI system can be considered as deceptive solely due to its design, because it has learned deceptive techniques as a means of optimisation, even if it was not intended as such by the developer.

Exceptions to ‘real-time’ remote biometric identification

The prohibition does not apply when that identification is strictly necessary:

  • for targeted search of abducted or missing persons, human trafficking or sexual exploitation,

  • for preventing an imminent substantial threat to life or terrorist attack,

  • or for locating criminal suspects to investigate or prosecute them for certain specific criminal offences.

These exceptions are subject to a range of conditions such as fundamental rights impact assessment, independent judicial or administrative authorisation, and declaration to relevant authorities.

See also:

^ Obligations for high-risk AI system providers

High-risk AI systems

In the AI Act, most obligations related to high-risk AI systems are for the provider. This page summarises the main obligations, but more detailed information is provided in the High-risk AI page.

Technical requirements on the AI system

Coming soon!

Procedures and record keeping

Compliance relies on two complementary processes:

  • The quality management system that the provider has to put in place. In EU legislation, such quality management systems differ from quality management systems in the sense of ISO 9001. In this case, the quality management system corresponds to the processes that are specifically established by the provider to prepare and ensure the compliance of its AI systems. If properly applied, it is not supposed to be possible that a non-compliant AI system is released. The AI Act includes specific obligations for that quality management system due to the context of AI and the particular requirements in the AI Act, but also additional measures to facilitate the implementation of that quality management system by SMEs and startups (and especially microenterprises), as well as cross-legislation compatibility for regulated sectors (Annex I).

  • The conformity assessment procedure, that effectively verifies that the AI system is compliant. Depending on the use case and on whether harmonised standards are used for compliance, that procedure can be performed either by a third party (notified body) or by the provider itself (internal control). It is based on the technical documentation of the AI system and on the documentation of the quality management system, but can also involve testing for a more precise assessment of certain requirements. Conformity assessment has to be done before placing the AI system on the market or putting into service, but the procedure also includes some surveillance after that (with potential on-site audits, mandatory communication and assessment of any change, and a full reassessment in case of major change).

Once conformity is confirmed, the provider has to write an EU declaration of conformity. This document is a formal statement whereby the AI system is in conformity with the AI Act (and any other EU legislation that also requires such a declaration, as well as the GDPR and its related legislation in case of personal data processing), together with an indication of any harmonised standard used for compliance, information on the conformity assessment procedure if performed by a notified body, and some general identification information about the AI system and the provider (or authorised representative).

The provider then has to clearly indicate on the AI system itself (or if not possible, its packaging or accompanying documentation) their identity, contact details, and a CE marking that formalizes this compliance (including identification of the notified body if any). In the case of high-risk AI systems from non-regulated sectors (Annex III, except for critical infrastructure), the provider also has to register the AI system on a special EU database maintained by the European Commission. Different conditions of registration and publicity of the information (detailed in Article 49) apply depending on the use case.

The provider is expected to keep relevant documentation for 10 years after the AI system has been placed on the market or put into service. This corresponds to the technical documentation of the AI system, the documentation of the quality management system, the EU declaration of conformity and (if assessed by a notified body) the decisions and other documents issues by the notified body, as well as the documentation of the changes approved by the notified body.

In case the provider has access to the logs of the AI system, the provider also has to keep these logs, but usually for six months (more if the use case warrants it, less if another legislation prevents from keeping them for six months, typically in case of personal data).

Non-compliance, incidents and interaction with authorities

Coming soon!

^ Obligations for high-risk AI system deployers

Proper use, human oversight and operation monitoring

Coming soon!

Obligation of information

Deployers of certain high-risk AI systems have to inform the affected persons that they are subject to the use of that AI system. This concerns:

  • High-risk AI systems from non-regulated sectors (Annex III) that make or assist in making a decision related to natural persons. For the specific case of law enforcement, the conditions remain those from the Law Enforcement Directive.

  • High-risk AI systems used at the workplace, when the deployer is the employer. In that case the deployer has to inform both the affected workers and the workers’ representatives, before putting the AI system into service or using it.

  • Emotion recognition systems and biometric categorisation systems, with specific conditions given by Article 50 (see below).

Procedures and record keeping

The only procedure applicable to deployers concerns deployers that are public authorities or Union institutions, when putting into service or using a high-risk AI system from a non-regulated sector (Annex III, except for critical infrastructure). In this case, the deployer has to register that use in the EU database maintained by the European Commission, and on which the provider is supposed to have already registered the AI system (if not, then the AI system cannot be used).

In case the deployer has access to the logs of the AI system, the deployer has to keep these logs, usually for six months (more if the use case warrants it, less if another legislation prevents from keeping them for six months, typically in case of personal data).

Non-compliance, incidents and interaction with authorities

Coming soon!

Personal data, biometrics and fundamental rights impact assessment

Coming soon!

^ Obligations for importers and distributors

Ensuring compliance

Before placing (or making available) a high-risk AI system on the market, importers and distributors have to verify that it bears the required CE marked, is accompanied by the EU declaration of conformity and by instructions for use.

Importers have to verify the compliance of the technical documentation, and that the conformity assessment procedure has been carried out. Distributors have to verify the compliance of the quality management system.

Importers also check that the (non-EU) provider has appointed an authorised representative in the EU.

Like providers, importers are required to indicate their identity and contact details on the AI system itself (or if not possible, its packaging or accompanying documentation). Distributors check that both the provider and the importer have done so.

While the AI system is under their responsibility, importers and distributors must ensure that its storage or transport conditions do not render it non-compliant with its technical requirements.

Responsibilities in case of non-compliance

Importers and distributors are required to not place (or make available) a high-risk AI system on the market if they have doubts on its compliance, but to wait until it has been brought into conformity. Importers in particular are expected to specifically account for the possibility of falsified documentation.

In case the high-risk AI system presents a risk that is beyond acceptability, they have reporting obligations. The importer has to inform the provider, the authorised representative, and authorities. The distributor has to inform the provider or the importer if any.

When these doubts on compliance arise after the AI system has been made available on the market, distributors also have a duty to take themselves corrective actions (including withdrawing or recalling the AI system if necessary), or ensure the relevant operator takes them, so that the AI system can effectively be brought into conformity. In that case the reporting has to be immediate, to include authorities, and to provide further details on the non-compliance and on the corrective actions.

Interaction with authorities

Importers and distributors of a high-risk AI system are required to cooperate with authorities in any action taken by these authorities in relation to the AI system, especially for the sake of risk reduction and mitigation.

For importers, this includes keeping certain documentation for 10 years after the AI system has been placed on the market or put into service: the EU declaration of conformity, the instructions for use and a copy of the certificate issued by the notified body if any. Then upon request from authorities they have to communicate them that documentation, but also any other information and documentation needed to demonstrate conformity of the AI system, including ensuring they can access the technical documentation of the AI system.

For distributors, the information and documentation to communicate is limited to the distributor’s own actions around verifying, ensuring and preserving compliance.

^ Transparency on AI interactions & generated content

Information on the processing and interaction: Article 50(1) and (3)

Providers have transparency obligations in the case of AI systems interacting directly with natural persons. If not already obvious for a reasonably well-informed person given the use case, the provider shall design and develop the AI system in such a way that it informs these natural persons that they are interacting with an AI system.

Deployers have transparency obligations in the cases of AI systems for emotion recognition and for biometric categorisation (which are two of the three ‘biometrics’ use cases considered as high-risk, with the third one being remote biometric identification). Deployers shall inform natural persons when they are exposed to the operation of these systems. In these cases, obligations from the GDPR and related legislation (GDPR equivalent for Union institutions, Law Enforcement Directive) also apply to the processing of personal data.

Note: This obligation only applies to certain subcases of the high-risk use cases listed in Annex III. It is complementary to the broader obligation that deployers have from Article 26(11) to inform natural persons when they are subject to an AI-made or AI-assisted decision, for all high-risk AI systems listed in Annex III.

In both cases the information is expected to be clear, distinguishable, provided by the time of the interaction, and is subject to accessibility requirements.

Generated or manipulated content: Article 50(2) and (4)

AI systems that generate or manipulate audio, image, video or text content are subject to additional transparency obligations so that the content cannot be mistaken as authentic content. This includes general-purpose AI systems but also any other AI system performing such generation or manipulation.

Providers of these AI systems have an obligation to mark the outputs in a machine-readable format and to ensure that the outputs are detectable as artificially generated or manipulated (and remain so whatever their use, e.g. through watermarking). While it is acknowledged that this is not always technically feasible (or at a reasonable cost) and can depend on the type of content, providers are expected to use commonly recognized methods (e.g. from technical standards) that are effective, interoperable, robust and reliable. This obligation does not apply to purely assistive functions for standard editing, or when the output does not substantially differ from the input or its semantics.

Deployers of these AI systems have an obligation to disclose the artificial (generated or manipulated) origin of:

  • texts meant to inform the public on matters of public interest, except if they go before publication through a human review or editorial control, and editorial responsibility is clearly held by a natural or legal person;

  • deep fakes, i.e. image, audio or video content that sufficiently resembles what exists (persons, objects, places, entities or events) to be mistaken as authentic content.

In artistic or similarly creative cases, provided it is obviously so, the disclosure obligation on the deployer is limited to informing on the existence of such artificial content, with no need to inappropriately hamper the display or enjoyment of the work.

The marking and disclosure of artificial origin are expected to be clear, distinguishable, provided by the time of the exposure, and are subject to accessibility requirements.

Law enforcement exceptions

The obligations on information and content marking do not apply to AI systems used to detect, prevent, investigate or prosecute criminal offences.

In the specific case of AI systems made available to the public for reporting criminal offences, the obligation on the provider to design and develop them in a way that it is clear for natural persons that they are interacting with an AI system is maintained.

In the specific case of AI systems for prosecuting criminal offences, the obligation on the deployer to inform natural persons that they are exposed to an emotion recognition or biometric categorisation system is maintained.

Support to compliance

The European Commission is due to publish guidelines providing further insights on the transparency obligations from Article 50, in order to facilitate their practical implementation by providers and deployers.

In addition, and specifically for the marking and detection of generated or manipulated content, a code of practice is being prepared (by stakeholders but with AI Office support) that will provide practical technical means to comply with these obligations. If successful, this code of practice will be given legal validity by the European Commission as an endorsed way to comply with Article 50(2) and (4).

See also:

^ General-Purpose AI models and systems

Concepts of GPAI models and GPAI systems

Coming soon!

Obligations

Coming soon!

Compliance

Coming soon!

Supervision by the AI Office and the scientific panel

Coming soon!

See also: