Ethics, AI, and Design in Medical Genetics

When we design an AI system for medical genetics, we are not simply designing a decision-support tool. We are designing the conditions under which that decision comes to be made: what the clinician sees and what stays hidden, what uncertainty is shown, who defined the problem and who was left out. Ethics, for this reason, is not something that can be added on; it does not live in the algorithm alone.¹ It has to be built into the design of the team, into architectural decisions, into the data, into the flows, into the kind of clinical relationship the system establishes, and into future iterations.

This was the starting point of my contribution to a panel on ELSI (ethical, legal and social implications) of AI in medical genetics, held on September 11th at ULS Santo António, as part of the session “Medical Genetics in the Age of Artificial Intelligence: From Concept to Algorithm to Clinical Decision.” The panel was moderated by Alexandra G. Rocha and included Goreti Marreiros, Heidi Carmen H., Natália Oliva Teles, Elisabete Castela and Rui Nunes. It was conceived as a panel, but ended up taking on the dynamic of a round table, with each contribution building on the last and pulling the conversation into territory none of us had necessarily anticipated.

Design is an unusual voice in these discussions, generally dominated by lawyers, clinicians and data scientists. I was invited to bring the design perspective on experience design (UX) and interface.

The conversation began at the algorithm and ended at the institution, and it is that path I try to trace here.

Co-design: team composition is already an ethical design decision

Starting from how these systems are built — the engine, the model, the computational genetics — the question put to me was: how do you design the experience and the interface to guarantee equity, rights and ethical conditions?

Ethics has to be built into the development process and into the whole experience: into how the team is composed, into how requirements are defined, into architectural decisions, into the data, into the flows, into the kind of clinical relationship we are establishing with AI, and into future iterations.

On one hand, we have the AI’s training data. But before that, we need to ask who takes part in defining the problem, the requirements and the experience. Are the people on the development teams thinking through and designing the system representative of the different users it will affect?

We need to create the conditions for co-design processes with the people who actually understand these processes and the real-world effects of the decisions the system will support: doctors who deal with the result in the consulting room, patients and families living with the implications of a diagnosis, and, in this particular case, geneticists who know where interpretation is most fragile.

Co-design, in this context, is not a one-off validation workshop at the end of the process. It means deciding who takes part, when they take part, and what influence they have throughout the entire design process: what questions the system needs to be able to answer, what information is genuinely useful at the moment of a clinical decision, where uncertainty needs to be made visible.

This argument connects to the idea of representativeness I brought to the session. It is not enough to ask who is represented in the training data. We also need to ask who was in the room deciding what that data should contain, and who tested whether the final experience made sense to the people who use it every day.

It is the difference between asking, at the end, “is this well designed?” and asking, from the start, “what does this system need to do?” of the people who will live with the consequences. We can take the argument further with the idea of co-creation:² involving these voices not just in the design, but in the very construction of the knowledge the system uses.

Three levels of representativeness

Co-design is not, in this context, an abstract principle. It is a practical mechanism for making representativeness real. I propose a triad of representativeness:

Data: who is represented in the training datasets, which populations the system works well for, and — by omission — who it silently fails. A genetic decision-support system trained predominantly on one population does not serve every family it will go on to serve in the same way.

Teams: who defines the problem, writes the requirements, creates the user stories, defines the personas and designs the system’s architecture. A team made up only of engineers and data scientists will ask different questions from a team that includes clinical geneticists, patients and the families living with a diagnosis — and those differences show up in what the system treats as relevant, in the pathways and narratives it anticipates, and in the people and needs it recognizes.

Experience: who takes part in testing, whether the information is understandable to someone reading it at a moment of great anxiety, whether the decision the system proposes is acceptable to whoever has to communicate or receive it, whether the designed flow supports — or hinders — a good clinical decision.

Three simple questions that are rarely asked in this order: Who is represented in the data? Who decided what that data should contain? Who tested whether the final experience makes sense to the people who use it every day?

UX as part of the decision-making mechanism

UX is not the visual layer of a system. It is part of the decision-making mechanism.

An algorithm can be technically rigorous, and the ethical decision can still be compromised if the interface presents a genetic risk without context, without a confidence interval, without explaining what the number means. In that case, the problem is not in the model — it is in the design of the experience.

An interface can make uncertainty visible or hide it. It can invite verification or produce a sense of certainty. It can support clinical reasoning or induce near-automatic acceptance of a recommendation. Designing the interface is, therefore, also designing the relationship between the professional and the system, and, indirectly, between the professional, the patient and the decision.

More-than-human: it’s no longer about “humanizing technology”

Natália Oliva Teles underlined the importance of humanizing technology. It is perhaps time to go one step further and recognize that the relationship between humans and technology can no longer be thought of as an opposition between two poles.

Clinical decision-making in genetics is no longer just a doctor-patient relationship. It is a more-than-human relationship, one involving a network of agents: humans, machines and algorithms and, in this particular case, also future generations who do not yet exist. A network mediated by data that does not decide, but shapes what becomes visible, what can be compared, and what becomes available to support a decision.³

This post-human lens, which I apply in other areas of my work, takes on a very concrete clinical urgency in medical genetics — in line with what the design field has come to call more-than-human design.⁴ Designing for this network, rather than for an isolated user, is a design responsibility distinct from that of many other domains of AI in healthcare.

Governance and Security by Design: if the system changes, who is responsible?

Prof. Rui Nunes raised a question of considerable pragmatism: if an AI system keeps learning, adapting or being updated after it leaves the company that developed it and enters production, how can that company be held accountable for a system that, by that point, is no longer exactly what it created and validated?

The question is not purely legal. It is also a question of design and governance. Responsibility does not disappear because the system changed. But it can become much harder to locate, track and enforce. If it is difficult to enforce once the system is already in production, we need to think about mechanisms capable of tracking that transformation. Perhaps we need to take a step back along the chain and strengthen governance before the system reaches that stage.

Regulating and auditing models, establishing validation processes, defining monitoring criteria and creating mechanisms to track the system throughout its lifecycle stop being peripheral tasks. They become an integral part of how trust in the system is built and maintained throughout its lifecycle.

It is no longer enough to ask whether a model works when it launches. We need to ask how we know it keeps working, for whom it keeps working, what has changed, and who has the authority to intervene when it stops working as intended.

This is where Security by Design comes in: security, auditability, accountability and governance cannot be layers added after the system is built. A system that changes needs governance capable of tracking that process and changing along with it throughout its lifecycle.

And this, at bottom, is the same logic that runs through design: neither ethics nor governance can be an afterthought. They have to be designed to move and develop together with the system.

From more-than-human to the agentic hospital

If clinical decision-making is no longer a relationship between two humans, but a network that includes people, data, machines, algorithms and other agents, it is possible to take the idea further: what happens when that network stops being confined to a single decision and starts to structure the healthcare organization itself? It was at this point that Prof. Rui Nunes introduced the notion of the “agentic hospital,” drawing on references from China and Brazil and leaving open the possibility that Portugal might explore this path.

The expression shifts the scale of the discussion. We are no longer just talking about AI tools that support doctors, or even an interface through which a professional consults a model, but about an organization in which different artificial intelligence agents can actively participate in processes of coordination, information, decision-making and execution. It is a hypothesis that raises many questions, and it is useful precisely because it forces us to formulate them.

An agentic hospital is not simply a hospital with more artificial intelligence in it. It is a potential shift in the structure of relationships, flows, responsibilities and even paradigms. If doctors, nurses, patients, autonomous AI systems and other digital systems come to participate in the same process, we need to know not only what each one can do, but who can decide, who can override a decision, who recognizes uncertainty, who can interrupt a system, and who takes responsibility when something goes wrong.

This is where governance returns, now at a larger scale. The more agents and autonomy enter the system, the more important it becomes to clearly design the relationships between them. Governance stops being a set of rules that exists outside the system and becomes part of its architecture — dynamically, able to track the evolution of relationships, behaviors and responsibilities.

From where I stood, it was particularly interesting to watch the idea of more-than-human move beyond the relationship between humans and technology and come to be applied to the very architecture of a healthcare institution, and, at the limit, of an entire health system.

The conversation traced a circle: it began at the algorithm, moved through experience and interface, arrived at the network of agents, and ended up discussing the institution itself — an entanglement, in the end, between people, data, algorithms and institutions that becomes harder and harder to pull apart into separate pieces.

Perhaps this is the most radical consequence of thinking about AI in medical genetics from a design perspective: we are not just designing systems to be used in a hospital setting. We may be designing what a hospital is going to become — and this is no ordinary organization. Peter Drucker described the hospital as “the most complex human organization ever devised.”⁵ Adding to it agents capable of interpreting, deciding and acting is, therefore, a challenge on the scale of that very complexity.

Talking about artificial intelligence in medical genetics means deciding what relationships, responsibilities and forms of decision-making we want to make possible through technology. Ethics is not a layer added on top of the system: it has to run through it and stay with it across its entire lifecycle.

All of this is Design.

___________________________________________________________________________

¹ Christopher Frauenberger, Marjo Rauhala e Geraldine Fitzpatrick, “In-Action Ethics”, Interacting with Computers, 29(2), 2017, pp. 220–236. https://doi.org/10.1093/iwc/iww024.

² Sanders, E. B.-N., & Stappers, P. J. (2008). “Co-creation and the new landscapes of design.” CoDesign, 4(1), 5–18. https://doi.org/10.1080/15710880701875068

³ Elisa Giaccardi e Johan Redström, “Technology and More-Than-Human Design,” Design Issues, 36(4), 2020, pp. 33–44. https://doi.org/10.1162/desi_a_00612. 

4 Christopher Frauenberger, “Entanglement HCI: The Next Wave?”, ACM Transactions on Computer-Human Interaction, 27(1), 2020, pp. 1–27. https://doi.org/10.1145/3364998.

5 Peter F. Drucker, Managing in the Next Society, St. Martin’s Press, 2002. Tradução minha.