The last requirement “Accountability” is derived from the meta-principle of fairness and this requirement is probably the most important one for business leaders and governance officers, because it delivers clear KPIs for AI systems.

It is important to note that accountability entails the complete life cycle of the AI system. From development to deployment and use. And as in blockchain technologies, the accountability should be retraceable at every given point in time. This means that a company, which deploys a third party AI technology will still be held responsible for the complete accountability life cycle.

Honestly I am curious once liability charges hit the court rooms. And I am sure that companies will try to “hand down” the liability to the maker of the system. But for the sake of the user, each company, which deploys a system, should be held responsible in the face of a user and the law. Otherwise citizens in the EU will have to battle companies all over the world.

The requirement of accountability is divided into four categories:

Auditability

Minimisation and reporting of negative impacts

Trade-offs

Redress

Auditablity

Today, companies offering and running critical infrastructure or critical products are being audited all the time. Medical and pharma companies are governed with the so called GxP – framework and are tightly watched by governmental institutions. Also stock traded companies are being audited continuously and are legally required to do so. Hence, once time has passed and society has caught up with AI, these organizational and governmental structures and procedures will also probably apply to those AI systems which touch “fundamental rights, including safety-criticial applications”. The auditing will surpass the single auditing of the algorithm, but will probably also include auditing of data, the design process and other system and organization related entities. I would assume that special AI auditing companies will see the light of day within the next few years.

Minimisation and reporting of negative impacts

This category focuses on two abilities in order to minimize negative impacts: the ability to report about AI systems and the ability to respond to risks and threats. On the one hand the ability to report includes systems which ensure that employees are able to report issues without being cancelled from the organization (e.g. whistle blower hotlines), on the other hand the ability to respond will need to give employees the pure ability to actually fix the problem. This may seem harder at times and is tightly connected to other requirements like transparency, technical robustness and safety.

However, it is noted by the HLEG that “[risk] assessments must be proportionate to the risk that the AI system pose”. That means (luckily) that small, AI driven Excel Add-ons used by SMEs will probably not be assessed while AI supported elevators or plane guidance systems will probably feel the full force of governmental control.

Trade-offs

As we all learned in life: we cannot please everyone. Thus when trying to implement the seven requirements, some might not be fully met by the AI system. This is per se not bad .But all trade-offs need to be documented well and discussed properly. Furthermore the company leaders will have to acknowledge them in writing and decide if the AI system should be run despite existing trade-offs. This document will then be part of the auditing process as well.

Redress

Redress is closely tight to the ability to respond. Redressing means to “set right again”. So, employees shall be able to actually fix a faulty AI system, if errors are reported.

And this concludes the discussion of the four principles and seven requirements of trustworthy AI. In the next chapters we will discuss which technical and non-technical methods the HLEG suggests to “realise Trustworthy AI”.

This requirement extends the previous requirement “diversity, non-discrimination and fairness” and is deducted from the overall principle of fairness and prevention of harm. It desires to include the broader society into the circle of stakeholders. What does this actually mean?

Sustainable and environmentally friendly AI

First and foremost, the complete supply chain of developing, deploying and using the AI system should be environmentally friendly. What CO₂ footprint does the supply chain have? How much energy is consumed by running the servers? Are there more environmental better choices? For example could developers use different processors instead of just “scaling up the machines”.

Social impact

Since billions of people communicate via text messengers or other social media channels, communication between people has changed. The attention span has deteriorated, the amount of used words has decreased and the overall communication culture has changed. Some say for the worst. When you observe a group of friends sitting at a bar while everyone is looking at their own phone, you will know what I mean.

As we see with social media, the human-machine interaction changed the human-human interaction. Thus we have to make sure that AI systems which are socially interacting with humans will not have a negative affect on humans. Just imagine a human-like voice assistant who is smarter than your spouse, always in a good mood and always there for you. How do you think will this effect inter-human relationships? Or just imagine how human-human interaction will change, if AI systems will never feel pain, regret, sadness or discomfort when interacting with humans. How is a child’s development influenced when AI tools become better friends than other kids. Would these children be able to develop properly?

Society and Democracy

All AI systems at a certain scale, like today’s social media systems, will have effects on societies and democracies. It is obvious how for example the five big tech companies of the USA (Facebook, Amazon, Twitter, Google, Apple) influence societies and the complete democratic process of countries. The previous election in the USA is a good example of this. At a certain size of a company or software program, it becomes a gate keeper to otherwise free democratic and societal processes.  This needs to be addressed thoroughly to avoid non-democratic and unfree processes in otherwise free and democratic societies. It is always a good brain teaser that Facebook, a privately run company, has more “citizens” than any other country in the world. According to oberlo.com (16^th of Feb, 2021) Facebook has 2.80 billion monthly active users and it also has 1.84 billion users that are visiting the social networking site on a daily basis while China in 2021 only has 1.44 and India 1.39 billion citizens.

Before we discuss today’s fifth requirement, we need to define certain termini to not create confusion.

Discrimination

Discrimination has been defined by the EU Council Directive 2000/78/EC of 27 November 2000 which established a general framework for equal treatment in employment and occupation. The Directive’s purpose was “to lay down a general framework for combating discrimination on the grounds of religion or belief, disability, age or sexual orientation as regards employment and occupation”. As we see here, it has a very defined scope of definition (just religion, disability, age or sexual orientation) and people should not be treated differently based on religion, belief, disability, age or sexual orientation (Article 2,2.a).

Article 2a & 2b define further direct and non-direct discrimination as follows:

(a) direct discrimination shall be taken to occur where one person is treated less favourably than another is, has been or would be treated in a comparable situation, on any of the grounds referred to in Article 1;

(b) indirect discrimination shall be taken to occur where an apparently neutral provision, criterion or practice would put persons having a particular religion or belief, a particular disability, a particular age, or a particular sexual orientation at a particular disadvantage compared with other persons unless:

(i) that provision, criterion or practice is objectively justified by a legitimate aim and the means of achieving that aim are appropriate and necessary

As we see above, the HLEG uses the word discrimination not in a global sense as data science does, but in a very specific context. Furthermore it allows indirect discrimination if it is “objectiviely justified by a legitimate aim”. Thus denying someone a 500.000 Euro loan because the person is 85 would be objectively justified by risk management and thus would not violate the principle of non-discrimination.

Avoidance of unfair bias

There is a real risk that the use of historic data to train an AI system can lead to unwanted bias and thus unintended indirect prejudice and discrimination (as defined above). However, the HLEG extends this unfair bias interestingly also to the exploitation of the user’s biases by “engaging in unfair competition, such as the homgenisation of prices”. For further reading I suggest following paper from the European Union: #Big Data: Discrimination in data-supported decision making

Accessibility and universal design

This addresses the fact that systems should be designed to be used by people of all ages, gender, abilities or characteristics. Specifically people with disabilities should not be excluded when designing AI software.

Stakeholder Participation

Like in any good project management framework, stakeholder management is key to ensure broad acceptance and ethical validation from all groups. Thus stakeholders from all avenues of life, who are potentially effected by the AI system, should be consulted and heard.

In the next chapter we are going to discuss the sixth requirement “Societal and environmental well being”.

Today we are talking about the fourth requirement: Transparency. Transparency can be directly linked to the principle of explicability, which we talked about earlier.

Transparency of AI should not only cover transparency of the model, but also of the complete data stack and data processing pipeline (data engineering, filtering, aggregation), the set up of the underlying system, the model and most importantly the business model: How is the company generating money? What assets of a user are they selling?

Tracebility

I very much support the HLEG’s idea to extend transparency also towards data and data processing. It would be very interesting to know, which data the companies are actually using. How are they “enriching” the data they collect from your user behavior. There is a whole ecosystem of data-enriching companies out there, which many people have not heard about. It is not enough for a company to tell people which data they record on their website, but also with what data they are enriching it as it is the combination of important data points which makes information inferences so critical.

Furthermore it is important to know, how data is filtered and aggregated before it is used in the AI model. Imagine a bank aggregates the data of credit scores into just two groups before it is infused into the AI model. A person with a credit score above 100 (fictive value) gets a 1 and a person with a credit score below 100 gets a 0. This is a very critical decision of the bank and happens outside of the actual AI model.

Explainability

Explainability does not just cover the explainability and interpretability of the AI model. It also covers the complete pipeline. Futhermore, and this is a wishful thought, it should also cover the non-technical part of the AI system. What decisions are the humans taking based on the suggestion of the AI model. Are there intermediary human steps, are humans in or on the loop? By looking at AI driven application software, I would love to know how that actually works in a company. Does a human actually check which applicants are being rejected and why?

The most important aspect for the topic of explainability is that the people who need to understand it (compliance officers, board, stakeholders, users) will be presented with a format that actually is understandable for them. So AI-inherit-language needs to be translated into a format which is written for everybody. I am excited to see which new job profiles will emerge solely focusing on this topic.

Communication

As AI systems become more and more humanlike, it is important that they identify clearly as AI systems. Many website chats do not make clear, if it is AI or human operated. The HLEG makes clear that users should not only be informed but also given a chance if they want to communicate with an AI system or rather prefer a real human.

In the next chapter, we are looking into the fifth requirement of “diversity, non-discrimination.

Privacy and data protection

Today we are talking about the third requirement: privacy and data governance. Privacy is always closely linked to the previous requirement “Technical robustness and safety” while data governance focuses on the quality and integrity of the data used. The true power of artificial intelligence is the inference of traits, behaviors, sexual orientation, political views out of sometimes unrelated data. Thus the privacy protection needs to not only extend to the data provided, but also towards the inferred information created!

Quality and integrity of data

It can not be said enough, but if you feed a smart system with bad, false or outdated data, no AI system will be able to create a correct answer out of these data. “Bad data in – bad data out” is a standard saying in the data science community. For that reason I have created a project management framework, where data quality and data engineering are prominently discussed to avoid these problems. If you have questions in this regard, please let me know.

Access to data

This topic is also part of the governance strategy of a company. Mostly companies focus more on access of folders and groups than access of data. At first sight this might seem the same, but reality shows that data is easily moved from one folder to another or linked in the backend or shared via local storages or via private cloud storages. In order to prevent this, a company wide data storage policy needs to be in place first. Secondly all existing data needs to be viewed and then grouped according to its criticality. It is surprising how little companies know about their existing data and the level of criticality.

The General Data Protection Regulation (GDPR) started a valuable discussion and forced companies to look into the data they have, create and store.

However, we see that companies which are not following a unified communication or unified communication as a service approach, often also have problems to pinpoint all sources which create data. Thus focusing on a unified communication approach first, will help to solve the problem of accessibility of data later.

Let me give an example what I mean. Just imagine an employee is about to send a list of all employees bank information to an external client via a non-encrypted email channel. According to internal guidelines this is forbidden. But since you have no unified communication protocol in place, no system detects this error and damage could be done. However, with a unified communication system in place, you would always know

1. Where your data is created
2. Where your data is stored
3. Who can access this data
4. How this data can be used

We will talk about unified communication as a mean of data governance in a later chapter in more detail.

In the next chapter we will discuss the fourth requirement of transparency.

In the last chapter we talked about the first requirement “Human agency and oversight”. Today we will look into the second requirement “Technical robustness and safety”. This requirement is concerned with risk assessment and prevention of harm.

Resilience to attack and security

As with any other system, any AI system needs to be resilient to hostile attacks from the outside and inside. AI hacking extends the normal range of cyber security measures as also the model and the incoming data needs to be protected. Protection against data poisoning and model leakage is normally not familiar for cyber security teams and thus they need to actively extend their processes and procedures to protect AI systems against those AI specific threats. We discussed Microsoft’s chat bot which had been deliberately hacked by data poisoning. Futhermore you see that this requirement contradicts the first requirement in some aspects. As model leakage can lead to threats to the system, it should be avoided. But if you see it as model transparency, it should be sought after.

Risk Management / General Safety

As AI systems are used more and more in critical tools and products, risk management needs to be woven into the general project management of the AI development and should be enforced by general company governance.

Accuracy, reliability and reproducibility

Psychological tests, medicine and AI systems have a lot in common. They all are measured and judged by accuracy and reliability. Accuracy is defined as the value of how close the final result is to the actual or actual value. It measures how big or small your overall error is. Reliability however measures how much or little the errors are spread.

I know this can be confusing, so let’s look at the famous archery example which every first semester student in experimental design classes is tortured with. Imagine you have a famous archer. He shoots ten arrows at the target.

• All ten hit the bull’s eye. Then this trial run was both accurate and reliable.
• If the archer however would have put all ten arrow at the top left corner of the target, his trial run would still be reliable, because he constantly reproduced the same result. However it would be not accurate.
• On the other side, if he would have spread them unevenly, but all close around the bull’s eye, then he would still have an accurate but unreliable result.

Highly reliable and reproducible AI systems help developers, scientists and law makers to accurately describe what the AI system does. Replication files, as suggested by the HLEG, can “facilitate the process of testing and reproducing behavior”.

In the next section, we will look into the requirement of “privacy and data

The first desired requirement for a trustworthy AI system is human agency and oversight. First and foremost, AI systems should adhere to fundamental rights as defined by European courts.

Fundamental rights

To make an informed decision, I looked at the Fundamental Rights as defined by the EU charter from 2012. It is broken down in six chapters: dignity, freedoms, quality, solidarity, citizen’s rights and justice. I would advise that AI developers hold special attention to Title III, Article 21. This article talks about non-discrimination. As we discussed in previous chapters, “discrimination” is differently defined in data science than in general language. And I would advise that employees responsible for ethical governance have the legal department look into this article and get a clear definition of discrimination. The article says that “any discrimination (or group building) based on any ground as …language, …political or any other opinion (!),..property,…, age… shall be prohibited.” Just imagine you are a bank and use AI to determine if somebody should be given a loan. Then it would definitely make a difference if this person owns 10 houses or none, or if it is 40 years or 80 years old. Furthermore “any other opinion” is a very broad term. Just imagine you are assessing the personality of an applicant as an automated step for your job assessment software. Here, the basic idea behind developing personality profiles is that teams with matching personality profiles work better together. But a personality profiles is developed by getting the applicant’s opinion on different topics. If that person then doesn’t match the target profile, the AI system would reject the person of advancing in the assessment process. According to the HLEG this already would violate fundamental rights. Is that realistic? I would advise to read the charter of fundamental rights of the European Union to make sure that this aspect of the first principle is obliged to.

Human agency

The HLEG defines human agency as to:

“ be able to make informed autonomous decisions regarding AI systems. [Users] should be given the knowledge and tools to comprehend and interact with AI systems to a satisfactory degree and, where possible, be enabled to reasonably self-assess or challenge the system.

That is a very high goal and I am curious how this would play out. I would love to see a court force the social media platforms to open their algorithms to the public to see why certain stories are not being pushed or why certain articles or people are suggested. Furthermore I find it extremely challenging to have a system which is comprehendible to the user. I think just time will tell, how these general terms will live up to reality. What is “satisfactory degree”, what is “reasonably”, what is meant by “self-assess or challenge”? Should everyone be able to self-assess the AI algorithms of Facebook? It would be lovely to see full transparency in AI algorithms of banks and insurances. But again, time will tell, if ethical desires will beat business needs in reality.

Human oversight

Human oversight can be displayed on a continuum ranging from “Human-in-the-loop (HITL)”, where human can step in every decision cycle of the AI system to “human-in-commad (HIC)”, where humans are just able to oversee the general outcome or activity of the system regardless of the activities “under the hood”. In my opinion HITL is a contradictive option to AI systems, but HIC should be approached in most cases. HIC would allow to override the decisions of the system in specific cases or it could mean to not use the system in particular situations. Insurances already use the HIC principle when separating simple claims handled by AI systems and complex claims handled by humans.

In the next chapter we will look at the second requirement of “Technical robustness and safety”.

After analyzing the four ethical principles (respect for human autonomy, prevention of harm, fairness & explicability) of the second criterium, we start to explore how the HLEG envisions Europe to move forward using AI and how to implement the four ethical principles in real life.

Three stakeholders

The HLEG identifies three different groups as stakeholders in the AI lifecycle. At the beginning are the “developers”. Developers are defined as individuals and companies who develop AI systems. The HLEG wants these companies to adhere to the four ethical principles when they design and develop new AI technologies. The second group are the “deployers”. These are individuals, groups or companies which use the AI technologies in their products and services. I can imagine that AI developing companies will need to produce a declaration of no objection, or declaration of ethical clearance in the future to sell their products to deployers and that deployers might be forced by legal governance processes to only use AI software which can produce these kind of declarations (equal to processes in data protection legalized within the GDPR framework).

The third group is the end-user and the broader society which should have the right in the future to be automatically informed about the results of the ethical evaluation and who should have the right of being informed about the reasoning of the AI system.

The HLEG tried to create a list of concrete requirements to adhere on the one hand to the dimension of ethical principles and on the other hand to the dimension of stakeholders. Each requirement in this non-exhaustive list stresses a different principle and/or different stakeholder.

Seven tangible requirements of Trustworthy AI

• Human agency and oversight
• Technical robustness and safety
• Privacy and data governance
• Transparency
• Diversity, non-discrimination and fairness
• Societal and environmental wellbeing
• Accountability

As you can imagine, each requirement is not equally important to every AI system and in certain settings these requirements might even contradict each other. And this is fully okay. If you start your journey of integrating ethical discussions of AI in “development” or “deployment” you might not find a perfect solution at the end. Even within your team or company there might be different stakeholders with different view points. The most important thing is however that your collective reasoning process should be transparent and explainable and explicable. Own your ethical discourse  within your company and use this tension to progress further rather than cancelling out voices. You might be working in a field where certain ethical requirements cannot be met fully. This might have regulatory reasons or even pure economical reasons as you are fearing other companies might use your knowledge to develop even better algorithms. Whatever it is, make your thought process transparent and explicable.

In the next section will talk about the first requirement “Human agency and oversight” and we will try to see, how you as a company can try to implement it in your every day work.

As we have seen above as mathematical models progress, they leave the realm of interpretability and enter the dimension of explainability.

Simple linear regressions or decision trees are examples of interpretable models while nowadays used more sophisticated models like convolutional neural networks or random forest models are considered black box models. These black box models need special techniques to explain how the results or outcomes or predictions have been formed.

These techniques can be either grouped by locality or specifity. Locality means that some techniques focus on the result of a specific input (e.g. to prove if a model is biased towards gender) or on the global formula itself. Specifity means that some models only work for certain groups of models (like regressions or convolutional neural networks) and other techniques are so called “model-agnostic”.

As we talked about convolutional neural networks (CNN), I would like to point your attention to the oldest method of explaining a CNN: saliency maps. Saliency maps basically re-engineer the flow of data from back to front and show which feature each layer in the network is using. Below you are seeing an example of a saliency map. As you see, the observer can determine, which feature of the picture is being used by the model to make its decisions.

Guosiyuan 123, CC BY-SA 4.0  via Wikimedia Commons

To be very honest, explainability of more and more complex neural models with millions and soon billions of neurons is not an easy task. And it will take years, if not decades when we can truly explain with a very high degree of certainty why the model is behaving the way it does. This will and should make us feel a bit uncomfortable as we feel to lose more and more control over the “reasoning” of our machines. If you look at the first principle of human autonomy or the principle of prevention of harm: Are we really able to grant autonomy, prevent harm and promise fairness, if we can’t explain the reasoning of our machines? I hope that the development of explaining techniques will have the same speed and attention as the development of bigger and bigger neural networks.

In the next chapter will start looking into how we can realize Trustworthy A.I. using the previously established four ethical principles.

Over the last days, we discussed the second criterium of Trustworthy A.I. called Ethical A.I.. Ethical A.I. is defined by four ethical principles: respect for human autonomy, prevention of harm, fairness and explicability. Today and tomorrow we will dive deeper into the last principle: explicability.

Explicability has a strong connection to the procedural dimension of fairness. The HLEG states this procedural dimension of fairness as the , “ability to contest and seek effective redress against decisions made by AI systems and by the humans operating them.” Thus in order to contest decisions made by A.I. systems, the company responsible for the decision must be “identifiable, and the decision-making processes should be explicable”.

Explicability extends on the procedural dimension of fairness. But before we are going into explicability, let me discuss the difference between explicable and explainable.

Explicable vs explainable

Many people use explainable instead of explicable as they are very similar in usage and mostly used as synonyms. I was wondering why the EU was using explicable instead of explainable and I would now reason that explainable puts the focus rather on the subject and explicable puts the focus rather on the object. I know that sounds a bit confusing.

Let me give you an example: A model which is not explainable just means that the subject or observer is not able to explain the reasoning behind the models decision although it is objectively explainable or has an explanation the person just not know. However a model which is inexplicable does not even have an explanation. For the sake of simplicity, I would like to use explainable from now on as the word is much more known to many people than explicable.

Now that we know the difference between explicable and explainable, let’s see how explainable differs from interpretable. I know this article seems a bit like English class, but when it comes to ethics, words matter.

Interpretable models vs Explainable models

An interpretable model mostly belongs to three families of model classes: decision trees, linear regression of logistic regressions.

A linear regression model for a model could have a simple form as that. Imagine you want to predict the future spending of a client based on their previous purchase. Then the future spendings are Y and the previous spendings are x; b would be a weighing factor.

A decision tree model is literally what it sounds like. You make yes/no decisions while you move down a tree of questions:

Polyextremophile, Public domain, via Wikimedia Commons

As you see in the above graphic, decision tree models are very well interpretable and it is very easy to check, if these kind of models are biased (e.g. vs ethnicity or any other attribute).

However there are groups of models, which are not interpretable anymore. Few of these groups are called random forest, neural networks or convolutional neural networks. The list of non-interpretable models grows continuously as it is subject of immense research. If a regression model might have 2, 5 or even 20 variables, convolutional neural networks can have millions of variables. Our brain has several billion neurons, while a frog brain has 16 millions. Nowadays we can already simulate frog brains in the computer. But as you see, 16 million neurons/ variables is a black box of interpretability. But this computational power is needed, if you think that a simple image task (bagel vs. puppy) with 100×100 pixels for each image creates a problem with 10,000 dimensions, where each dimension has a multitude of possibilities.

This group of non-interpretable models is called “black-box models”, which can only be tried to be explain and not interpreted anymore.

Many people find that it was not a good choice to choose “interpret vs explain” in this setting as these too verbs seem to be interchangeable. For now we might define the verb explain as “to render understandable” and interpret as “to provide meaning”.

In the following chapter will look into new methods and techniques and how “black box models” can be made explainable.