Balancing Algorithmic Efficiency and Procedural Justice in Public Sector Automated Decision Systems: A Global Governance Analysis¶

1. Introduction¶

The rapid advancement and integration of automated decision systems (ADS) in the public sector have brought significant benefits, including enhanced administrative processes, reduced costs, and improved service delivery. However, these systems also introduce complex challenges, particularly in balancing algorithmic efficiency with procedural justice. The tension between these two aspects is a critical issue, as the pursuit of efficiency can sometimes undermine the principles of fairness, transparency, and accountability that are essential for maintaining democratic governance and citizen trust.

This report aims to explore the governance mechanisms that address this tension, focusing on how public sector organizations can ensure that ADS are used ethically and responsibly. The research is grounded in a comprehensive review of academic literature and government reports, which provide a historical context for the development of ADS and the ethical frameworks that guide their implementation. The role of civil society in shaping and monitoring ADS governance is also examined, along with various government initiatives aimed at enhancing transparency and accountability.

The methodology section outlines the research design, data collection methods, and selection criteria for sources, ensuring that the study is based on high-quality, relevant, and multidisciplinary evidence. The case studies section presents both successful and problematic implementations of ADS, offering concrete examples of the challenges and solutions in real-world scenarios. The discussion section synthesizes the findings, highlighting the implications and future directions for the governance of ADS in the public sector. Finally, the conclusion summarizes the key insights and their significance for researchers and policymakers.

By examining these various aspects, this report seeks to provide a nuanced and comprehensive understanding of the governance mechanisms that can effectively balance algorithmic efficiency with procedural justice, while maintaining democratic accountability and citizen trust. The findings and recommendations from this study are intended to inform the development of robust and ethical governance frameworks for ADS in the public sector, ensuring that these systems serve the public interest and uphold the principles of justice and transparency.

2. Literature Review¶

This literature review explores the existing academic and governmental discourse on the governance of automated decision systems (ADS) in the public sector. It begins by examining the historical context of ADS governance, followed by an analysis of ethical frameworks that underpin the development and implementation of these systems. The role of civil society in shaping and monitoring ADS governance is then discussed, highlighting the importance of public participation and transparency. Finally, the section reviews various government initiatives aimed at ensuring democratic accountability and citizen trust in ADS.

2.1. Historical Context¶

The evolution of automated decision systems (ADS) in the public sector has been marked by significant milestones, reflecting the ongoing tension between algorithmic efficiency and procedural justice. From the 1950s, when computers began performing basic processing tasks, to the sophisticated machine learning applications of the 2010s, the development of ADS has been driven by the promise of enhanced administrative processes and social service delivery. Early e-government initiatives in the 2000s aimed to integrate digital technologies to improve service efficiency and reduce costs, but these systems were also met with concerns about data privacy and security [6].

In the 2010s, the use of ADS became more prevalent, with applications in criminal justice, child protection, and employment services. For instance, the United States implemented risk assessment instruments (RAI) to predict recidivism, evaluate parole, and identify crime hotspots. While these tools aimed to increase consistency and reduce human error, they also raised significant concerns about systemic bias and the lack of transparency and contestability in decision-making processes [6]. Similarly, in Canada, ADS was introduced in 2014 to automate the evaluation of immigrant and visitor applications, streamlining processes but also sparking debates about the fairness and accuracy of the decisions [6].

The theoretical foundations guiding the development and implementation of ADS in the public sector emphasize the trade-offs between efficiency and innovation. While ADS offers substantial benefits in terms of consistency and cost reduction, it introduces challenges related to procedural justice, including issues of transparency, bias, and the need for human oversight. The quality and integrity of the data used in these systems are crucial for ensuring fair and just outcomes [6].

Legal and institutional challenges in governing ADS are significant, particularly due to the phenomenon of multilayered blackboxing. ADS systems often involve complex algorithms and large datasets, making them difficult to understand and audit. This opacity, known as multilayered blackboxing, poses serious legal and institutional hurdles. For example, in the Gothenburg, Sweden, public school placement scandal, the Public School Administration (PSA) and the court system ignored widespread breaches of applicable regulations and legislation, leading to the assignment of thousands of children to schools in violation of relevant rules despite massive protests [24]. The PSA's practices of withholding information and delaying responses undermined democratic accountability and eroded citizen trust [24].

In the Netherlands, the Dutch tax authorities' reliance on an AI algorithm for flagging high-risk applicants for further scrutiny, particularly those with a migration background, led to widespread public outrage and the publication of a parliamentary report titled "Unprecedented Injustice." This misuse of algorithmic decision-making ultimately resulted in the resignation of the Dutch government in January 2021, highlighting the critical tension between algorithmic efficiency and procedural justice [23].

The UK government has also faced challenges in the governance of ADS. The Department of Work and Pensions introduced four algorithmic models to assess and flag suspected fraudulent claims around Universal Credit, aiming to generate savings of around £1.6 billion by 2030-31. However, these systems have been criticized for their lack of transparency and the potential for systemic bias [18]. As of 30 August 2024, the Public Law Project's Tracking Automated Government (TAG) Register listed 55 applications of automated decision-making tools in the UK public sector, with only 1 classified as highly transparent, indicating a significant initial lack of transparency [18].

To address these challenges, researchers and policymakers have proposed several governance mechanisms. These include enhancing transparency and accessibility through transparency registers and impact assessments, ensuring that the underlying data and algorithms are transparent and accessible to the public [12]. Additionally, the UK has proposed the establishment of a dedicated Automated Decisions Tribunal within the First-Tier tribunals system to provide a clear and accessible route for citizens to challenge decisions made by these systems [18].

The Dutch case study and the Gothenburg example illustrate the need for robust legal frameworks and institutional safeguards to ensure transparency and accountability in ADS systems. These frameworks should address the unique challenges posed by ADS, including the need for transparency, accountability, and mechanisms for redress [24]. Furthermore, the involvement of civil society and academic scrutiny in the governance of ADS is crucial for identifying and mitigating biases and ensuring that these systems are used ethically and responsibly [12].

Having examined the historical context and the theoretical foundations of ADS in the public sector, the following section will delve into the ethical frameworks that guide the development and implementation of these systems.

2.2. Ethical Frameworks¶

The core ethical principles that guide the development and implementation of automated decision systems (ADS) in the public sector are multifaceted and encompass a broad spectrum of considerations. These principles include beneficence, non-maleficence, autonomy, justice, and explicability. Beneficence emphasizes the positive contribution of ADS to society, ensuring that these systems enhance the well-being of the public. Non-maleficence, on the other hand, focuses on preventing harm, both physical and psychological, that may result from the use of ADS. Autonomy respects the rights of individuals to make their own decisions and to have control over their data. Justice ensures that ADS systems are fair and equitable, avoiding biases and discrimination. Explicability mandates that the decision-making processes of ADS are transparent and understandable, allowing stakeholders to scrutinize and contest decisions [5].

To achieve these ethical principles, technical solutions for transparency and fairness are essential. Explainable AI (XAI) techniques, such as SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations), are designed to make the decision-making processes of ADS more transparent. These methods provide insights into the reasoning behind AI decisions, which is crucial for building user trust and ensuring that the systems are reliable and consistent with ethical standards [1]. Post-hoc interpretability methods, which offer explanations after decisions have been made, are also valuable for identifying and correcting biases in ADS. Routine audits, involving the verification of models and code, are another critical technical solution that helps in maintaining the integrity and fairness of ADS systems [4].

Sociotechnical approaches to governance are equally important in ensuring that ADS systems are ethically sound and trustworthy. The "humans-in-the-loop" approach, where humans are involved at various stages of the decision-making process, is a key strategy. This approach ensures that AI recommendations are vetted by human expertise, aligning with ethical standards and reducing skepticism. For example, in the international maritime trade, a human-in-the-loop configuration was found to be effective in auditing and altering algorithms to ensure their accuracy and reliability, thereby supporting procedural justice [2]. Stakeholder engagement, involving a diverse group of participants, is another sociotechnical strategy that can help in identifying and mitigating biases and ensuring that ADS systems are inclusive and responsive to societal needs [5].

Ethics-based auditing (EBA) is a structured process that assesses the behavior of ADS against predefined ethical principles and norms. EBA promotes procedural regularity and transparency, which are crucial for maintaining democratic accountability and citizen trust. The process involves various types of audits, including functionality audits, code audits, and impact audits, which are complementary and can be combined to create a comprehensive and effective auditing framework. Unlike mere codes of conduct or certification, EBA focuses on demonstrating adherence to ethical guidelines through a purpose-oriented process that includes stakeholder consultation and adversarial testing. Ensuring operational independence between the auditor and the auditee is essential to minimize risks of collusion and clarify roles, thereby enhancing the reliability and trustworthiness of ADS [19].

Despite the availability of these ethical frameworks and governance mechanisms, there are significant challenges in their implementation. The indeterminacy of ethical principles can hinder their practical application, leading to unethical behaviors such as 'ethics shopping,' 'ethics bluewashing,' and 'ethics lobbying.' Software developers often view ethics as an impractical construct, distant from their daily work, and organizations may struggle with managing the risks of ADS due to a lack of useful governance mechanisms or conflicting interests. Moreover, the gap between high-level ethical principles and practical methods for designing, deploying, and governing ADS remains a critical issue [4].

Cross-regional variations in the prioritization of ethical principles further complicate the implementation of governance mechanisms. For instance, the EU emphasizes transparency and human oversight, while the US focuses on innovation and flexibility, and China prioritizes state security and social harmony. These differences reflect the unique legal, cultural, and economic motivations of each region, making cross-regional comparisons inherently complex. The lack of clear guidelines and standards for implementing ethical requirements in real-world settings is a significant barrier to achieving trustworthy AI in the public sector [3].

To address these challenges, the literature suggests a combination of technical and sociotechnical solutions. Enhancing the quality and robustness of data, ensuring human oversight, and promoting stakeholder engagement are critical for surmounting technical limitations and maintaining transparency, fairness, and ethical decision-making. Additionally, setting industry standards for AI-based decision support systems (DSS) is necessary to maintain user trust. Standardization discriminates against systems that do not meet minimum requirements in terms of accuracy, transparency, and explainability, which are essential for ensuring that AI systems are secure, reliable, and effective [16].

The UK government's "Ethics, Transparency and Accountability Framework for Automated Decision-Making" provides a structured approach to ensure the safe, sustainable, and ethical use of ADS within the public sector. The framework consists of seven core principles, including testing to avoid unintended outcomes, delivering fair services, clarity of responsibility, data safety, enhancing user and citizen understanding, legal compliance, and building future-proof systems. These principles are designed to be used alongside existing organizational guidance and processes, offering a practical framework for policymakers and practitioners [17].

In summary, ensuring democratic accountability and maintaining citizen trust in ADS within the public sector requires a multifaceted approach that combines transparent and bias-resistant algorithm design, human oversight and intervention, and robust legal frameworks. The literature underscores the critical role of avoiding organizational ignoring and developing multilayered strategies to address the complex interplay between technology and society. Having examined the ethical frameworks and governance mechanisms, the following section will explore the role of civil society in shaping and monitoring ADS governance.

2.3. Role of Civil Society¶

Civil society organizations and private-sector actors play a crucial role in enhancing the governance of automated decision systems (ADS) through multi-stakeholder collaborations. These collaborations are essential for ensuring democratic accountability and building and maintaining citizen trust in the public sector.

Mechanisms for Democratic Accountability¶

Civil society organizations often engage in formal decision-making processes, which are state-endorsed and involve structured interactions with public and private entities (Ansell and Gash, 2007). They participate in joint planning activities, such as strategic and project planning, to ensure that the governance of ADS is aligned with community needs and values (Lee and Bae, 2019). Workshops and training sessions are frequently organized to educate stakeholders about the implications of ADS, fostering transparency and informed participation (Fliervoet and van den Born, 2017). Civil society 'shadow networks' play a crucial role by providing local and traditional knowledge, which complements professional technical expertise, ensuring a more holistic and inclusive approach to governance (Fliervoet and van den Born, 2017). Additionally, civil society organizations in the European Union are recognized as key stakeholders in multistakeholder governance, participating in official intergovernmental processes and type II conference outcomes (Wikipedia, 2023).

Building and Maintaining Citizen Trust¶

Multi-stakeholder collaborations, combining formal and informal mechanisms, are prevalent in the governance of ADS. These collaborations help to bridge the gap between technical, civic, and political realms, thereby enhancing trust (Ansell and Gash, 2007). Thematic working groups, which include representatives from civil society, the private sector, and government, are established to draw upon diverse expertise and perspectives, ensuring that the governance process is transparent and accountable (Lee and Bae, 2019). Empirical studies suggest that the best outcomes in collaborative governance are achieved when there is a mix of professional technical expertise and local, traditional knowledge, indicating that civil society involvement is essential for building and maintaining trust (Fliervoet and van den Born, 2017).

Specific Practices and Initiatives¶

The AI Now Institute has developed the Algorithmic Impact Assessment (AIA) framework and an Algorithmic Accountability Toolkit, which are designed to enhance transparency, explainability, and public oversight of algorithmic systems. These tools are used to engage with policymakers and provide practical guidance for the implementation of algorithmic accountability policies (Ada Lovelace Institute, 2024). Furthermore, the Open Government Partnership (OGP) works with civil society and other key actors in member countries to co-create and implement OGP action plans that include concrete policy commitments. These action plans are independently monitored for ambition and completion through the OGP's Independent Reporting Mechanism, ensuring that the governance of ADS is transparent and accountable (Ada Lovelace Institute, 2024).

International Examples¶

In the Netherlands, the Dutch tax authorities' reliance on an AI algorithm for flagging high-risk applicants, particularly those with a migration background, led to widespread public outrage and the publication of a parliamentary report titled "Unprecedented Injustice." This misuse of algorithmic decision-making ultimately resulted in the resignation of the Dutch government in January 2021, highlighting the critical tension between algorithmic efficiency and procedural justice (Roehl, 2024). In contrast, the Swedish Transportation Agency's semi and fully-automated system for assessing driver license applications has been successful in reducing administrative discretion, increasing consistency and quality of decisions, and enhancing procedural justice and rule-of-law (Andersson et al., 2018).

Ethical and Societal Considerations¶

The literature emphasizes the importance of integrating ethical and societal considerations into the governance of ADS. Ethical principles such as beneficence, non-maleficence, autonomy, justice, and explicability are crucial for ensuring the integrity and relevance of the governance mechanisms (Floridi and Cowls, 2019). For instance, the UK government's "Ethics, Transparency and Accountability Framework for Automated Decision-Making" provides a structured approach to ensure the safe, sustainable, and ethical use of ADS within the public sector. The framework consists of seven core principles, including testing to avoid unintended outcomes, delivering fair services, clarity of responsibility, data safety, enhancing user and citizen understanding, legal compliance, and building future-proof systems (UK Government, 2024).

Challenges and Solutions¶

While the involvement of civil society and private-sector actors in the governance of ADS is beneficial, it also faces challenges. The complexity and opacity of ADS systems can make it difficult for both decision-makers and oversight bodies to understand and control the decision-making process, leading to a lack of transparency and accountability (Meijer, 2009). To address these challenges, the literature suggests several governance mechanisms, including enhanced transparency and accessibility through transparency registers and impact assessments, ensuring that the underlying data and algorithms are transparent and accessible to the public (644b856d). Additionally, the establishment of a dedicated Automated Decisions Tribunal within the First-Tier tribunals system can provide a clear and accessible route for citizens to challenge decisions made by these systems (887c99b1).

Conclusion¶

Having examined the contributions of civil society organizations and private-sector actors in enhancing the governance of ADS, the following section will delve into government initiatives aimed at ensuring democratic accountability and citizen trust in the public sector.

2.4. Government Initiatives¶

The UK government has taken significant steps to ensure the transparency, accountability, and fairness of automated decision-making (ADM) systems within the public sector. The Ethics, Transparency and Accountability Framework for Automated Decision-Making is a comprehensive initiative designed to enhance the responsible use of AI. This framework, developed by the Government Digital Service (GDS) and the Office for Artificial Intelligence, consists of seven key points:

1. Testing to Avoid Unintended Outcomes or Consequences: The framework emphasizes the importance of rigorous testing to prevent unexpected negative impacts, including the use of quality, relevant, and diverse datasets for testing. It also recommends conducting regular impact and risk assessments, such as Data Protection Impact Assessments and Equality Impact Assessments [9].
2. Delivering Fair Services for All Users and Citizens: To ensure fairness, the framework advocates for the involvement of a multidisciplinary and diverse team in the development process to identify and mitigate biases and prejudices. It also requires the presumption that the algorithm or system can cause harm and injustice, and integrates human judgment and stakeholder engagement throughout the lifecycle [9].
3. Ensuring Clarity of Responsibility: The framework clarifies the roles and responsibilities of those involved in the decision-making process, assigning senior owners to major processes and services. It also embeds the framework into commercial arrangements with third parties to ensure compliance and responsible use of data [9].
4. Handling Data Safely and Protecting Citizens’ Interests: Adherence to the Data Ethics Framework and data protection laws is stressed, with a focus on reviewing the quality and limitations of datasets used, particularly to avoid bias and discrimination [9].
5. Enhancing User and Citizen Understanding: Providing clear and transparent information to citizens about how ADM systems impact them is a priority. The framework works under a presumption of publication, ensuring that citizens are informed in plain English when automated systems are used [9].
6. Ensuring Compliance with the Law: The framework mandates adherence to legal requirements, particularly under data protection laws, the Equality Act 2010, and the Public Sector Equality Duty [9].
7. Building Future-Proof Systems: Continuous monitoring and review of algorithms (at least quarterly) are required to ensure they remain effective and mitigate against unintended consequences. The framework also allows for user challenges and formal review points [9].

Additionally, the Public Authority Algorithmic and Automated Decision-Making Systems Bill [HL], introduced by Lord Clement-Jones, aims to regulate the use of ADM systems across the public sector. Key objectives include ensuring transparency and fairness, boosting accountability mechanisms, and providing individuals with rights of challenge and appeal. The bill also addresses the need for a consistent legal framework, as highlighted by the Robodebt Royal Commission and the Senate’s Select Committee on Adopting Artificial Intelligence [15].

In the United States, the Government Accountability Office (GAO) has developed an AI accountability framework to ensure responsible use of AI in government programs and processes. This framework is organized around four complementary principles: governance, data, performance, and monitoring. Each principle includes key practices and a set of questions for entities, auditors, and third-party assessors to consider:

Governance Principle: - Key Practices: Set clear goals, engage with diverse stakeholders, establish a governance structure, and define roles and responsibilities. - Questions for Consideration: What are the goals and objectives of the AI system? Who are the stakeholders, and how are they engaged? What is the governance structure, and who is responsible for oversight?

Data Principle: - Key Practices: Ensure data quality, protect data privacy, and manage data risks. - Questions for Consideration: What is the quality of the data used in the AI system? How is data privacy protected? What are the potential data risks, and how are they managed?

Performance Principle: - Key Practices: Validate and test the AI system, document performance metrics, and ensure continuous improvement. - Questions for Consideration: How is the AI system validated and tested? What performance metrics are used, and how are they documented? What mechanisms are in place for continuous improvement?

Monitoring Principle: - Key Practices: Conduct regular audits and assessments, monitor for unintended consequences, and ensure transparency. - Questions for Consideration: How often are audits and assessments conducted? What methods are used to monitor for unintended consequences? How is transparency maintained in the operation of the AI system?

The development of this framework involved convening a Comptroller General Forum with AI experts from the federal government, industry, and nonprofit sectors. GAO also conducted extensive literature reviews and obtained independent validation from program officials and subject matter experts, emphasizing the importance of third-party assessments and audits in ensuring accountability and responsible AI use [22].

Internationally, the governance of ADM systems varies significantly. For example, in Canada, the government has implemented AI to enhance the immigration process control system, raising concerns about power asymmetries and the potential for algorithmic bias [31]. In Poland, the government has introduced AI to optimize employment services, with a notable reliance on the algorithm and limited questioning by responsible clerks, indicating a potential issue with procedural justice [31]. In Finland, AI is used to personalize digital services, but this has also raised concerns about the centralization of sensitive data and increased surveillance [31].

The European Union (EU) has taken a proactive approach to regulating AI, particularly in high-risk sectors. The EU AI Act includes provisions for pre-release conformity certifications, transparency, and audit requirements for high-risk AI systems. The EU Digital Services Act mandates audits of large online platforms and search engines, emphasizing the analysis of algorithmic systems and generative models [26].

In Australia, the piecemeal approach of 46 pieces of ADM legislation has been criticized for its inconsistency. The Robodebt Royal Commission recommended the introduction of a consistent legal framework for automation in government services, and the Senate’s Select Committee on Adopting Artificial Intelligence has released a final report that includes a chapter on ADM, underscoring the need for a more coherent regulatory approach [21].

These international examples highlight the need for a balanced approach that leverages the benefits of ADM while safeguarding against potential harms. Key differences and similarities in approaches to transparency, accountability, and fairness in ADM systems across different countries include:

• Transparency: The UK and EU emphasize the importance of transparency, with the UK's TAG Register and the EU's audit requirements. In contrast, the U.S. focuses more on third-party assessments and audits [9][26].
• Accountability: The UK and U.S. both propose dedicated tribunals or accountability mechanisms to provide routes for citizens to challenge decisions. The EU's focus on pre-release conformity certifications and continuous monitoring also supports accountability [15][22].
• Fairness: The UK and EU frameworks include specific provisions to address fairness and equity, such as the UK's Equality Impact Assessment and the EU's anti-discrimination requirements. The U.S. framework emphasizes stakeholder engagement and continuous improvement to ensure fair outcomes [9][26].

Having examined the government-led initiatives and frameworks aimed at ensuring transparency, accountability, and fairness in automated decision systems, the following section will explore the methodologies used in this research to gather and analyze data on these governance mechanisms.

3. Methodology¶

This section outlines the research methods employed to gather and analyze data on governance mechanisms addressing the tension between algorithmic efficiency and procedural justice in automated decision systems within public sector contexts. It begins with a detailed research design, followed by an explanation of the data collection methods used. The selection criteria for sources are then described, ensuring that only relevant and high-quality academic literature and government reports are included. Finally, the data analysis techniques are presented, along with ethical considerations that guided the research process.

3.1. Research Design¶

The research design for this study is grounded in the need to address the tension between algorithmic efficiency and procedural justice in automated decision systems (ADS) within public sector contexts, while maintaining democratic accountability and citizen trust. The design principles outlined in the literature provide a robust foundation for this research, emphasizing the importance of alignment, synergy, ethical governance, and human involvement.

First, the research design focuses on alignment between the business model and organizational resources and capabilities. This principle, as described by Herath Pathirannehelage et al. (2024), emphasizes the creation of a strategic roadmap that aligns with the organization's business model. The roadmap includes measurable AI use cases, availability of domain expertise, technical feasibility, and clear goals, sub-goals, timelines, and challenges. Leadership at the organization must champion the project, remain accountable for outcomes, and gather employee support, demonstrating the importance of alignment and accountability [1].

Second, the design aims to ensure synergy in input, model, and output to guarantee business value. An iterative design approach is employed to synergize input elements (data, domain knowledge), the AI model (machine learning and natural language processing), and output (predictions, visualizations). Comprehensive datasets are merged, and various AI models are evaluated to identify the best performers. Visualizations and explanations (SHAP, LIME) are included to make the AI outcomes understandable and trustable. A three-pronged evaluation (proof of concept, proof of value, proof of usability) is conducted to demonstrate the system's value and usability, which is crucial for gaining stakeholder commitment and resources [1].

Third, the research design incorporates ethical AI governance frameworks. Ethical AI governance is essential to address issues related to security, privacy, fairness, deskilling, surveillance, and accountability. The study offers three pathways: (1) adopting established regulatory guidelines (e.g., European Commission, OECD), (2) developing own AI guiding principles consistent with user expectations, and (3) establishing an AI auditing and governance framework. AI auditing should evaluate both business value and risks, and a transparent and ethically sound design helps gain user trust, fostering fairness and engagement [1].

Fourth, the design emphasizes human involvement and engagement. Human involvement is critical in ADS systems. Domain experts provide essential tacit knowledge, which improves system performance and helps prevent unintended outcomes. Interactive user interfaces with customizable parameters are developed to integrate this knowledge seamlessly. The system is designed to augment, not automate, decision-making, leaving responsibility and accountability with humans. Explainability of AI outcomes is enhanced using SHAP, LIME, and visualizations. User feedback is used to update models, fostering stakeholder and user trust and overcoming algorithmic aversion. Designs that involve humans help preserve domain knowledge and autonomy, preventing deskilling of the workforce [1].

The research design also considers the typology of data analytics governance mechanisms. These mechanisms include structural, process, and relational approaches. Structural mechanisms focus on organizing data analytics functions and related decision rights, with three main structures identified: centralized, decentralized, and hybrid. Each structure has its advantages and disadvantages, and the choice depends on the specific context and goals of the organization. Process mechanisms ensure that daily behaviors are consistent with policies and provide feedback for decision-making. Relational mechanisms aim to develop collaboration and shared understanding among stakeholders, which is essential for fostering a collaborative environment where stakeholders can effectively communicate, participate, and resolve conflicts, thereby enhancing the overall governance of data analytics [11].

Furthermore, the research design integrates ethics-based auditing (EBA) as a structured process to assess the behavior of ADS against predefined ethical principles and norms. EBA promotes procedural regularity and transparency, which are crucial for maintaining democratic accountability and citizen trust. The process involves various types of audits, such as functionality audits, code audits, and impact audits, which are complementary and can be combined to create a comprehensive and effective auditing framework. Unlike mere codes of conduct or certification, EBA focuses on demonstrating adherence to ethical guidelines through a purpose-oriented process that includes stakeholder consultation and adversarial testing. Ensuring operational independence between the auditor and the auditee is essential to minimize risks of collusion and clarify roles, thereby enhancing the reliability and trustworthiness of ADS [4].

To address the common challenges in implementing governance mechanisms for ADS in the public sector, the research design incorporates strategies such as enhancing legal and policy frameworks, standardizing impact assessments, providing training and capacity building for public officials, and conducting public consultations. These strategies aim to balance algorithmic efficiency with procedural justice, maintain democratic accountability, and foster citizen trust in the use of ADS within the public sector [12].

Finally, the research design is informed by international case studies to provide a broader perspective on the governance of ADS. For instance, the Canadian government's use of AI in the immigration process control system, the Polish government's optimization of employment services, and the Finnish government's personalization of digital services offer valuable insights into the challenges and successes of ADS governance. These case studies highlight the importance of transparency, accountability, and human oversight in ensuring that ADS systems are ethically aligned and considerate of different perspectives and needs [31].

Having described the research design, the following section will detail the data collection methods used to gather and analyze data on these governance mechanisms.

3.2. Data Collection Methods¶

The data collection methods employed in this study are designed to gather comprehensive and high-quality information on governance mechanisms addressing the tension between algorithmic efficiency and procedural justice in automated decision systems (ADS) within public sector contexts. The methods include a systematic literature review, case studies, and stakeholder feedback, each contributing to a deeper understanding of the challenges and solutions in this domain.

Systematic Literature Review: The systematic literature review (SLR) is a cornerstone of the data collection process. It involves a structured and replicable method to identify, select, and critically appraise relevant research. The SLR was conducted using advanced AI-based tools such as TheoryOn (Li et al., 2020) and Litbaskets (Boell and Wang, 2019), which helped in setting a manageable scope by identifying relevant journals and articles. LitSonar (Sturm and Sunyaev, 2018) was used to generate journal coverage reports and translate search queries for different databases, ensuring a broad and thorough search. The review focused on studies that reflect the increasing complexities of using AI in government, with an emphasis on public sector-focused, empirical, multidisciplinary, and explanatory research [29]. The inclusion criteria were stringent, ensuring that only relevant and high-quality academic literature and government reports were selected. The data was analyzed to identify exploratory, conceptual, qualitative, and practice-driven research, which is essential for understanding the multifaceted implications of ADS in the public sector [28].

Case Studies: Case studies were selected to provide concrete examples of how governance mechanisms have been employed in real-world scenarios. The case studies included the Gothenburg public school placement scandal, where the Public School Administration (PSA) and the court system ignored widespread breaches of applicable regulations and legislation, leading to significant social and legal injustices [24]. Another case study examined the Swedish Transportation Agency's use of a semi and fully-automated system to assess driver license applications, which has been successful in reducing administrative discretion and enhancing procedural justice and rule-of-law [25]. International examples, such as Canada's Directive on Automated Decision-Making, which mandates the use of impact assessments for agencies employing ADS systems, and the Netherlands' SyRI system, which was criticized for its lack of transparency and led to a court ruling prohibiting its use, were also analyzed [12][30]. These case studies provided insights into the practical challenges and successes of implementing governance mechanisms in different contexts, highlighting the importance of transparency, accountability, and human oversight.

Stakeholder Feedback: Stakeholder feedback was gathered through various means, including workshops and feedback from key stakeholders such as parents, media, external auditors, and school principals. This qualitative data collection method was crucial in identifying the issues with the ADM system in the Gothenburg case, where organizational ignoring practices and institutional blackboxing led to significant social and legal injustices [24]. In the context of the UK's Ethics, Transparency and Accountability Framework for Automated Decision-Making, stakeholder engagement was emphasized to ensure that the governance process is transparent and accountable, and to foster trust among citizens [17]. The feedback from these workshops and stakeholders was integrated into the analysis, providing a rich and nuanced understanding of the governance mechanisms in action.

Best Practices for Data Collection: The authors of the Gothenburg case study emphasized the importance of using multiple sources of documentation to capture a comprehensive view of the ADS system's impacts and institutional responses. These sources included media statements, organizational-level nonactions in reports, and early warnings from various stakeholders. The use of grounded theory methods, such as open and selective coding, was central to developing a conceptual model that explains the multilayered blackboxing of ADS in the public sector [24]. Similarly, the UK government's framework recommends conducting regular impact and risk assessments, involving multidisciplinary and diverse teams, and using quality, relevant, and diverse datasets to mitigate against biases and ensure fairness [17].

Challenges and Considerations: The use of AI in data collection methods requires careful consideration to avoid issues such as model overfitting, biases, black box predictions, and the need for acceptance by the research community. Methodological guidelines and informed discourse are essential to address these challenges. For instance, the UK government's framework emphasizes the need for continuous monitoring and formal review points to ensure that ADS systems remain effective and transparent [17]. The Dutch case study highlights the importance of both accessibility and explainability in algorithmic transparency, as the lack of these elements can lead to decreased trust and legal challenges [30].

Having detailed the data collection methods, the following section will describe the selection criteria for sources, ensuring that only relevant and high-quality academic literature and government reports are included in the analysis.

3.3. Selection Criteria for Sources¶

The selection criteria for sources in this study were rigorously defined to ensure that only relevant and high-quality academic literature and government reports were included. The criteria emphasized the need for public sector-focused, empirical, multidisciplinary, and explanatory research. Specifically, the literature review focused on studies that reflect the increasing complexities of using AI in government, with a particular emphasis on specific forms of AI rather than general AI concepts [28].

To enhance the robustness and credibility of the selected sources, the following quality thresholds were applied: - Journal Rankings: Only journals ranked 4*, 4, or 3 on the Academic Journal Guide 2021 (ABS list) or A* or A journals on the Australian Business Deans Council 2019 (ABDC list) were included [20]. - Search Terms and Databases: The search was conducted in April 2022 using the Scopus Database, focusing on paper titles, abstracts, and keywords. Initial searches yielded 5901 articles, which were refined to 92 papers after applying quality filters, reviewing titles and abstracts, and conducting backward and forward citation tracing [20].

The selection process also incorporated advanced AI-based tools to enhance the literature search and data collection: - TheoryOn (Li et al., 2020): Enabled ontology-based searches for constructs and construct relationships in behavioral theories. - Litbaskets (Boell and Wang, 2019): Helped set a manageable scope by identifying relevant journals. - LitSonar (Sturm and Sunyaev, 2018): Provided syntactic translation of search queries for different databases and generated journal coverage reports [29].

To ensure the ethical and methodological soundness of the selected sources, the following considerations were made: - Transparency and Accessibility: Sources were required to provide clear and accessible information, facilitating independent reviews and enhancing the transparency of the research process [7]. - Ethical and Legal Scrutiny: The inclusion of sources that address the ethical and legal dimensions of AI governance, such as the non-delegation principle and the need for meaningful human oversight, was prioritized [14].

The selection criteria were designed to support the research goal of identifying governance mechanisms that balance algorithmic efficiency with procedural justice, while maintaining democratic accountability and citizen trust. By focusing on high-quality, relevant, and multidisciplinary sources, the study aims to provide a comprehensive and nuanced understanding of the challenges and solutions in this domain.

Having described the selection criteria for sources, the following section will detail the data analysis techniques used to synthesize and interpret the gathered data.

3.4. Data Analysis Techniques¶

The data analysis techniques employed in this study are designed to ensure a robust and comprehensive evaluation of the collected data, aligning with the research goals of addressing the tension between algorithmic efficiency and procedural justice in automated decision systems (ADS) within public sector contexts. The techniques include a combination of quantitative and qualitative methods, with a focus on multi-criteria decision-making (MCDM) and explainable AI (XAI) frameworks.

Key data analysis techniques used in academic literature to evaluate the balance between algorithmic efficiency and procedural justice in ADS include survey-based decision-making and MCDM methods. Specifically, the MAMCA (Multi-Actor Multi-Criteria Analysis) framework has been advanced to integrate these techniques, allowing for a more detailed and inclusive evaluation process. In the MAMCA framework, individual preferences within stakeholder groups are elicited through various MCDM methods, such as PROMETHEE, AHP, SMART, Simos, and BWM. These methods are combined to enhance the robustness and comprehensiveness of the analysis. For instance, PROMETHEE and AHP can be used together to determine the relative importance of criteria and the preferences of alternatives. The additive model is often preferred for aggregating individual criterion scores, as it is easier to understand and more broadly applicable compared to other models like geometric or harmonic means. The additive model is represented mathematically as:

This model helps in constructing a preference matrix that consolidates the preferences of all stakeholder groups without further aggregation, allowing for a clear visualization of each group's preferences. The multi-actor view, a multi-line graph, is used to display these preferences, facilitating discussions and consensus-building among stakeholders [27].

To enhance the transparency and accountability of the data analysis, the study also incorporates advanced AI-based tools for conducting literature reviews. These tools, such as programming libraries supporting thematic analyses based on Latent Dirichlet Allocation (LDA) models (e.g., Antons and Breidbach, 2017) and GUI applications for scientometric analyses (e.g., Swanson and Smalheiser, 1997), help researchers identify promising areas and verify research gaps. Advanced AI-based tools can significantly enhance the literature search process, enabling ontology-based searches for constructs and construct relationships in behavioral theories (TheoryOn, Li et al., 2020), setting a manageable scope by identifying relevant journals (Litbaskets, Boell and Wang, 2019), and generating journal coverage reports and translating search queries for different databases (LitSonar, Sturm and Sunyaev, 2018). These tools have very high potential due to their ability to automate repetitive and time-consuming tasks, thus improving the efficiency and comprehensiveness of the literature review [29].

The analysis of case studies and stakeholder feedback is another critical component of the data analysis techniques. The Gothenburg case study, for instance, utilized open coding to closely analyze empirical observations, and iterative sorting of open codes formed subcategories and categories related to the PSA's and the court's (non)acknowledgment of ADM errors and the dimensions of injustices produced and reproduced. The analysis was informed by multiple sources of documentation, including media statements, organizational-level nonactions in reports, and early warnings about the ADM system from various stakeholders. This approach ensures a comprehensive and transparent evaluation of the governance mechanisms in action [24].

To counteract algorithmic blackboxing and promote transparency, the study employs a combination of intrinsic and post-hoc interpretability techniques. Intrinsic interpretability involves designing AI models that are inherently understandable, such as decision trees, while post-hoc interpretability focuses on developing methods to explain the decisions made by complex, opaque models after they have been trained. The General Data Protection Regulation (GDPR) introduced a "right to explanation" for automated decision-making in 2018, which has significantly influenced the development and application of post-hoc interpretability techniques. Current XAI research emphasizes both intrinsic interpretability and post-hoc explainability, recognizing the importance of incorporating both forms to ensure that AI systems remain comprehensible and auditable [13].

Ethics-based auditing (EBA) is integrated into the data analysis process to assess the behavior of ADS against predefined ethical principles and norms. EBA promotes procedural regularity and transparency, which are crucial for maintaining democratic accountability and citizen trust. The process involves various types of audits, such as functionality audits, code audits, and impact audits, which are complementary and can be combined to create a comprehensive and effective auditing framework. Unlike mere codes of conduct or certification, EBA focuses on demonstrating adherence to ethical guidelines through a purpose-oriented process that includes stakeholder consultation and adversarial testing. Ensuring operational independence between the auditor and the auditee is essential to minimize risks of collusion and clarify roles, thereby enhancing the reliability and trustworthiness of ADS [8].

The data analysis techniques also include regular impact and risk assessments, such as Data Protection Impact Assessments and Equality Impact Assessments, to ensure compliance with ethical standards and enhance public trust. These assessments are crucial for identifying and mitigating potential issues, such as biased input data, flawed algorithm design, incorrect output decisions, technical flaws, usage flaws, and security flaws. The UK government's "Ethics, Transparency and Accountability Framework for Automated Decision-Making" provides a structured approach to these assessments, emphasizing the need for thorough, controlled, and staged testing before deployment [17].

Having examined the data analysis techniques, the following section will delve into the ethical considerations that guided the research process.

3.5. Ethical Considerations¶

The ethical considerations and principles guiding the research process on automated decision systems (ADS) in the public sector are multifaceted and essential for ensuring that these systems are used responsibly and ethically. The unique context of public authorities, where individuals do not have the freedom to choose service providers and are inescapably subject to administrative decisions, necessitates special responsibilities, including the protection of fundamental rights and the provision of transparent and accountable decision-making processes [7].

Ethical Principles and Benchmarks¶

Beneficence and Non-Maleficence: The primary ethical benchmarks for the automation of administrative procedures in the public sector are beneficence and non-maleficence. Beneficence emphasizes the positive contribution of ADS to society, ensuring that these systems enhance the well-being of the public. Non-maleficence focuses on preventing harm, both physical and psychological, that may result from the use of ADS [10].

Autonomy: Respecting the autonomy of individuals is crucial. This involves providing affected individuals with the ability to understand and challenge AI decisions, ensuring that they have control over their data and the processes that affect them [10].

Justice and Fairness: Ensuring that ADS systems are fair and equitable is essential. This requires the identification and mitigation of biases and prejudices, particularly in areas overlapping with special category data (race, ethnicity, sexual orientation, political or religious beliefs) [10].

Explicability: ADS must be transparent and understandable to both technical and non-technical stakeholders. Techniques such as Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP) are valuable for making the decision-making processes of ADS more transparent, which is crucial for building user trust and ensuring that the systems are reliable and consistent with ethical standards [10].

Strategies to Counteract Algorithmic Blackboxing and Promote Transparency¶

Presumption of Publication: The UK's framework recommends a "presumption of publication" for all algorithms that enable automated decision-making. This involves notifying citizens when a process or service uses automated decision-making, providing plain English explanations, and ensuring that all exceptions are agreed upon with government legal advisors before ministerial authorization [17].

Clear Responsibility and Accountability: The framework stresses the importance of making responsibility and accountability for algorithms and their outcomes clear. Organizations and individuals should be held accountable to ensure the proper functioning of artificial intelligence, promoting transparency and trust [17].

Best Practices for Involving Diverse Stakeholders¶

Multidisciplinary and Diverse Teams: The development of algorithms or systems should involve a multidisciplinary and diverse team to identify and counter biases. This includes drawing on expertise from various disciplines, such as policy, operational delivery, and data science, to reduce bias and produce more accurate outcomes [17].

Stakeholder Engagement: Engaging with a wide range of stakeholders, including those from different backgrounds and with different perspectives, is crucial to ensure that the system is developed with an inclusive and collective approach. This helps in spotting and countering prejudices, bias, and discrimination [17].

Integrating Ethics-Based Auditing into Methodologies¶

Regular Impact and Risk Assessments: The framework suggests conducting regular impact and risk assessments, such as Data Protection Impact Assessments and Equality Impact Assessments, to ensure compliance with ethical standards and enhance public trust [17].

Red Team Testing: The framework recommends performing 'red team testing,' which assumes that all algorithmic systems are capable of inflicting some degree of harm. This type of testing helps identify and mitigate potential issues, ensuring that the system is robust and ethical [17].

Key Ethical Considerations for Data Protection and Security¶

Compliance with Data Protection Legislation: The framework emphasizes the need for algorithms and systems to handle data safely and comply with data protection laws. The public sector has a responsibility to lead in citizen data handling, ensuring that data is secure and used ethically [17].

Quality and Limitations of Datasets: Researchers must review the quality and limitations of datasets used, ensuring they are accurate, representative, and free from bias. Caution and human oversight are required when using datasets for purposes they were not originally intended for [17].

Additional Context and Nuances¶

Algorithmic Risks: The framework highlights several risks associated with automated decision-making, including biased input data, flawed algorithm design, incorrect output decisions, technical flaws, usage flaws, and security flaws. These risks must be thoroughly assessed and managed to ensure ethical and effective deployment [17].

Policy Intent and Specification: Before implementing an automated system, researchers should be confident that the policy intent, specification, or outcome will be best achieved through such a system. This involves a rigorous and controlled testing process to avoid unintended outcomes or consequences [17].

Practical Steps and Resources¶

Testing Phases: Prototype and test the algorithm or system to ensure it is robust, sustainable, and delivers the intended policy outcomes. Testing should focus on accuracy, security, reliability, fairness, and explainability. Independent and red team testing are recommended to identify potential harm [17].

Relevant Resources: The framework provides resources such as the ICO’s guidance on Article 22 of the GDPR, the Data Ethics Framework, and the National Data Strategy, which can be used to guide the ethical development and deployment of automated decision systems [17].

International Comparisons and Best Practices¶

Canada: The Canadian government has faced internal and external criticism for its use of automated decision-making in the immigration system. The backlash led to increased scrutiny and the halting of some controversial practices, demonstrating the importance of civil society and academic scrutiny in ensuring responsible and ethical use of ADS [31].

France: France has established the AI for Humanity initiative, which includes a strong emphasis on ethics, transparency, and accountability. The French government has implemented a regulatory framework to ensure that AI systems are transparent and that their decisions can be challenged [31].

Germany: The German government has also emphasized the need for ethical guidelines and transparency in AI. The Federal Ministry for Economic Affairs and Energy has published a set of ethical guidelines for AI, focusing on data protection, non-discrimination, and accountability [31].

United States: The U.S. Government Accountability Office (GAO) has developed an AI accountability framework, organized around four complementary principles: governance, data, performance, and monitoring. Each principle includes key practices and a set of questions for entities, auditors, and third-party assessors to consider, ensuring that the use of AI in government programs is responsible and transparent [22].

Challenges and Solutions¶

Input Data Quality: Ensuring that datasets are accurate, representative, and free from bias is a significant challenge. This requires rigorous data review and the involvement of multidisciplinary teams to identify and mitigate biases [17].

Algorithm Design: Addressing flawed assumptions and logical biases in the algorithm is crucial. This involves thorough testing and the use of diverse and ethical datasets to ensure that the system is robust and fair [17].

Output Decisions: Preventing incorrect interpretations and ensuring fairness in output decisions is essential. Human oversight and intervention are required during testing phases to ensure the system's technical resilience and accuracy [17].

Technical Flaws: Ensuring sufficient rigour in development and testing to avoid technical issues is necessary. This includes conducting regular impact and risk assessments and using independent and red team testing [17].

Usage Flaws: Integrating the system seamlessly with existing operations and ensuring that it is used in a manner that aligns with policy intent is a challenge. This requires careful planning and stakeholder engagement [17].

Security Flaws: Protecting against deliberate manipulation and ensuring data security is a critical concern. This involves adhering to data protection laws and the Data Ethics Framework [17].

Legal Compliance: Adhering to relevant legislation, such as the GDPR and the Equality Act (2010), is essential. Early engagement with legal advisors is recommended to ensure compliance and address potential legal issues [17].

Conclusion and Transition¶

By integrating these ethical considerations and principles into the research process, the study aims to provide a comprehensive and nuanced understanding of the governance mechanisms for ADS in the public sector. The next section will explore the specific case studies that illustrate both successful and problematic implementations of these systems, offering concrete examples of how governance mechanisms can be effectively employed to balance algorithmic efficiency with procedural justice, while maintaining democratic accountability and citizen trust.

4. Case Studies¶

This section presents a series of case studies that illustrate the implementation of automated decision systems (ADS) in various public sector contexts. We will examine both successful and problematic implementations, providing insights into the factors that contribute to effective governance and those that lead to challenges. By analyzing these real-world examples, we aim to highlight the complexities and tensions inherent in balancing algorithmic efficiency with procedural justice, while also maintaining democratic accountability and citizen trust. The following subsection will begin with an exploration of successful implementations of ADS in the public sector.

4.1. Successful Implementations¶

The Swedish Transportation Agency's implementation of a semi and fully-automated decision system to assess driver license applications stands out as a successful example of balancing algorithmic efficiency with procedural justice. This system automates the collection of relevant information, identification of legal rules, evaluation of information against these rules, and the selection of a course of action. By reducing administrative discretion, the system increases consistency and quality of decisions, thereby enhancing procedural justice and rule-of-law [25].

Another notable success is the Canadian government's Directive on Automated Decision-Making, which mandates the use of impact assessments for agencies employing ADS systems. These assessments help identify potential biases or discrimination before the systems are deployed, and include mechanisms for public input and access. This approach ensures that the ADS systems are transparent and accountable, contributing to procedural justice and maintaining citizen trust [31].

In Poland, the government has implemented AI to "optimize" employment services. Civil society and academia have played a crucial role in scrutinizing these systems, particularly during the stages of goal-setting, procurement, and implementation. This scrutiny helps ensure that the systems are transparent and fair, contributing to procedural justice. Although the system is designed to be an advisory tool with a human in the loop, the involvement of external stakeholders at all stages is essential for maintaining transparency and accountability [31].

Finland has used AI to personalize digital service experiences, aiming to improve service delivery. The paper suggests that this approach has the potential to enhance citizen trust and governance, provided that the systems are designed and implemented with careful consideration of ethical and procedural issues. The Finnish government has emphasized the importance of transparency and accountability in the use of AI, which aligns with the principles of good governance [31].

These case studies illustrate that successful governance mechanisms for ADS in the public sector involve robust civil society and academic scrutiny, the creation of a common evaluation framework, and the conduct of pilot experiments to ensure transparency and mitigate risks. The involvement of a wide range of stakeholders, including the public, media, and external auditors, is crucial for maintaining democratic accountability and citizen trust. Additionally, the development of legal frameworks that address the unique challenges posed by ADS, such as the need for transparency, accountability, and mechanisms for redress, is essential for ensuring that these systems are used ethically and responsibly [31].

Having explored successful implementations of ADS in the public sector, the following section will delve into problematic implementations, providing insights into the challenges and failures that can arise when governance mechanisms are not adequately designed or enforced.

4.2. Problematic Implementations¶

The Swedish Public School Administration (PSA) in Gothenburg faced significant issues with its automated decision-making (ADM) system, which was used to assign children to schools. The ADM system resulted in widespread breaches of applicable regulations and legislation, assigning thousands of children to schools in violation of relevant rules despite massive protests from parents and community members. The PSA and the court system initially failed to address these violations, engaging in practices of organizational ignoring and institutional blindness. This led to a situation where the ADM system's errors remained uncorrected for a prolonged period, causing social and legal injustices. The first author, Charlotta Kronblad, filed a lawsuit against the City of Gothenburg to explore the legal system's ability to deal with ADM, which eventually brought some attention to the issue. However, the court system initially failed to correct the injustices, reinforcing the algorithmic errors and the production of legal injustice [24].

The Dutch tax authority's use of a self-learning algorithm to detect suspected social welfare fraud is another example of a problematic implementation of ADS. The algorithm erroneously flagged innocent families for further scrutiny, leading to thousands of families being pushed into poverty and over a thousand children being placed in foster care. The Dutch court of the Hague determined that the governmental use of the algorithm violated the right to private and family life, issuing a court order to suspend its use. However, it took six years and mounting media attention for the court to take action, highlighting the significant delays and institutional blindness that can occur in the governance of ADS [24].

In the fall of 2019, the PSA in Sweden also encountered issues with its ADM system for school placements. Despite repeated warnings from the software firm, PSA employees, and school principals about potential irregularities and violations of national and local regulations, the PSA proceeded with the implementation and communication of the ADM-generated placement decisions without thorough review. This lack of engagement and scrutiny by the PSA led to 1,400 children (13%) failing to get any of their preferred school choices, a stark increase from the 4% rate under the manual process the previous year. The administrative court system further compounded the problem by failing to assess the ADM system and provide an avenue for legal recourse, effectively maintaining a state of institutional blackboxing. The court's inability to scrutinize the ADM system and address the wide range of errors made by the PSA resulted in social and legal injustice, making it practically impossible for affected public service recipients to seek redress [24].

These case studies highlight the need for robust legal frameworks and institutional safeguards to ensure transparency and accountability in ADS systems. The failure to implement such frameworks can lead to significant social and legal injustices, undermining democratic accountability and eroding citizen trust. The involvement of civil society and public protests played a crucial role in eventually bringing the issues to light and prompting corrective action, but these mechanisms alone are insufficient to ensure long-term accountability and justice. Structural changes in legal and procedural norms are necessary to prevent and address algorithmic injustices effectively [24].

Having examined the problematic implementations of ADS in the public sector, the following section will delve into the regulatory and governance mechanisms that can help balance algorithmic efficiency with procedural justice, while maintaining democratic accountability and citizen trust.

4.3. Regulatory and Governance Mechanisms¶

The regulatory and governance mechanisms employed in the case studies of automated decision systems (ADS) in the public sector reveal both effective strategies and significant shortcomings in maintaining democratic accountability and citizen trust. These mechanisms aim to balance the efficiency and accuracy promised by ADS with the procedural justice and transparency required by the rule of law.

In the Gothenburg public school placement scandal, the Swedish Public School Administration (PSA) and the court system engaged in practices of organizational ignoring and institutional blindness, leading to widespread breaches of applicable regulations and legislation. Despite repeated warnings from the software firm, PSA employees, and school principals, the PSA proceeded with the implementation and communication of the ADM-generated placement decisions without thorough review. This lack of engagement and scrutiny resulted in 1,400 children (13%) failing to get any of their preferred school choices, a stark increase from the 4% rate under the manual process the previous year. The administrative court system further compounded the problem by failing to assess the ADM system and provide an avenue for legal recourse, effectively maintaining a state of institutional blackboxing. The court's inability to scrutinize the ADM system and address the wide range of errors made by the PSA resulted in social and legal injustice, making it practically impossible for affected public service recipients to seek redress [24].

Similarly, the Dutch tax authority's use of a self-learning algorithm to detect suspected social welfare fraud led to significant issues. The algorithm erroneously flagged innocent families for further scrutiny, pushing thousands of families into poverty and placing over a thousand children in foster care. The Dutch court of the Hague determined that the governmental use of the algorithm violated the right to private and family life, issuing a court order to suspend its use. However, it took six years and mounting media attention for the court to take action, highlighting the significant delays and institutional blindness that can occur in the governance of ADS [24].

To address these challenges, several governance mechanisms have been proposed and implemented internationally. The UK government's "Ethics, Transparency and Accountability Framework for Automated Decision-Making" provides a structured approach to ensure the safe, sustainable, and ethical use of ADS within the public sector. The framework consists of seven key points, including testing to avoid unintended outcomes, delivering fair services, ensuring clarity of responsibility, handling data safely, enhancing user and citizen understanding, ensuring legal compliance, and building future-proof systems. These principles are designed to be used alongside existing organizational guidance and processes, offering a practical framework for policymakers and practitioners [17].

In Canada, the government requires agencies using ADS to conduct and publish algorithmic impact assessments before deploying any ADS system and to update these assessments when changes occur in the system's functionality or scope. This mechanism ensures transparency and accountability, contributing to procedural justice and citizen trust. Uruguay has established a multi-stakeholder oversight body for the government’s use of AI, alongside a new six-year AI Strategy (2024–2030) co-created with the public. This approach fosters democratic accountability and engagement [12].

The European Union mandates human intervention in high-risk AI systems, such as those used for facial recognition and evaluating eligibility for public benefits and creditworthiness, to balance algorithmic efficiency with procedural justice. The EU AI Act includes provisions for pre-release conformity certifications, transparency, and audit requirements for high-risk AI systems, ensuring that these systems operate transparently and are subject to continuous monitoring [26].

In the Netherlands, the Dutch System Risk Indication (SyRI) case study underscores the importance of both accessibility and explainability in algorithmic transparency. SyRI, an algorithm used to predict welfare fraud, was criticized for not publishing its parameters and decision rules, and for not informing investigated residents about the investigation. This lack of transparency led to protests from residents and activists, culminating in a 2020 Dutch Court ruling that prohibited the use of SyRI. The court's decision was primarily based on the system's lack of transparency, which undermined the perceived trustworthiness of the decision-making process [30].

The concept of ethics-based auditing (EBA) is introduced as a structured process for assessing the behavior of ADS against predefined ethical principles and norms. EBA promotes procedural regularity and transparency, which are crucial for maintaining democratic accountability and citizen trust. The process involves various types of audits, such as functionality audits, code audits, and impact audits, which are complementary and can be combined to create a comprehensive and effective auditing framework. Unlike mere codes of conduct or certification, EBA focuses on demonstrating adherence to ethical guidelines through a purpose-oriented process that includes stakeholder consultation and adversarial testing. Ensuring operational independence between the auditor and the auditee is essential to minimize risks of collusion and clarify roles, thereby enhancing the reliability and trustworthiness of ADS [19].

The implementation of governance mechanisms for ADS in the public sector faces several common challenges, including the phenomenon of multilayered blackboxing. According to Kronblad, Essén, and Mähring (2024), blackboxing occurs through organizational ignoring practices, which can obscure the consequences of ADS and prevent effective addressing and contesting of these issues. These practices include intra-organizational ignoring, where organizational members ignore warnings and fail to seek knowledge about the destructive consequences of ADS, and extra-organizational ignoring, where external stakeholders are unaware of or ignore the negative impacts of ADS due to a lack of visibility and understanding of the technology. Institutional ignoring, where institutional arrangements and legal systems become blind to the consequences of ADS, can lead to systemic issues where avenues for recourse and restitution are blocked, resulting in a lack of accountability and sustained injustices [24].

To overcome these challenges, the authors suggest several strategies: - Engaging a Wider Set of Stakeholders: Beyond the traditional "humans-in-the-loop" approach, involving a broader range of stakeholders (society-in-the-loop) can enhance the transparency and fairness of ADS systems. This includes engaging community members, advocacy groups, and other relevant parties in the training and monitoring processes. - Developing Institutional Safeguards: Additional layers of protection against algorithmic misconduct are necessary, particularly in the form of institutional safeguards. These safeguards should be designed to ensure that ADS systems are not only technically sound but also aligned with social and legal justice standards. - Active Interventions: Field interventions and engaged scholarship can generate critical data and insights that would otherwise remain hidden. Such interventions can help in understanding the algorithmic agency and its consequences before widespread diffusion and institutional responses are established. - Legal Frameworks: The development of comprehensive legal frameworks for ADS in the public sector is crucial. These frameworks should address the legal dimensions of algorithmic injustice, ensuring that citizens have the right to participate in key decisions, understand how ADS works, and contest decisions made by these systems.

These mechanisms collectively aim to balance algorithmic efficiency with procedural justice, ensuring democratic accountability and citizen trust in the public sector. The examples from the UK, Canada, and the EU illustrate the importance of transparent and bias-resistant algorithm design, human oversight and intervention, and robust legal frameworks. The failure to implement these mechanisms, as seen in the Gothenburg and Dutch cases, can lead to significant social and legal injustices, undermining the legitimacy and trustworthiness of automated decision systems.

Having examined the regulatory and governance mechanisms employed in the case studies, the following section will delve into the broader discussion of the implications and future directions for the governance of ADS in the public sector.

5. Discussion¶

The findings from the literature review, methodology, and case studies provide a comprehensive understanding of the governance mechanisms addressing the tension between algorithmic efficiency and procedural justice in automated decision systems (ADS) within public sector contexts. The literature highlights the historical evolution of ADS, emphasizing the ongoing trade-offs between efficiency and innovation. Ethical frameworks, such as beneficence, non-maleficence, autonomy, justice, and explicability, are crucial for ensuring the integrity and relevance of governance mechanisms. The role of civil society in shaping and monitoring ADS governance is also significant, as multi-stakeholder collaborations help bridge the gap between technical, civic, and political realms, thereby enhancing trust and accountability.

Tension Between Algorithmic Efficiency and Procedural Justice¶

The historical context of ADS in the public sector reveals a persistent tension between algorithmic efficiency and procedural justice. Early e-government initiatives in the 2000s aimed to integrate digital technologies to improve service efficiency and reduce costs, but these systems were also met with concerns about data privacy and security [6]. In the 2010s, the use of ADS became more prevalent, with applications in criminal justice, child protection, and employment services. For instance, the United States implemented risk assessment instruments (RAI) to predict recidivism, evaluate parole, and identify crime hotspots. While these tools aimed to increase consistency and reduce human error, they also raised significant concerns about systemic bias and the lack of transparency and contestability in decision-making processes [6]. Similarly, in Canada, ADS was introduced in 2014 to automate the evaluation of immigrant and visitor applications, streamlining processes but also sparking debates about the fairness and accuracy of the decisions [6].

Role of Civil Society¶

Civil society organizations and private-sector actors play a crucial role in enhancing the governance of ADS. Mechanisms for democratic accountability, such as formal decision-making processes, joint planning activities, and stakeholder engagement, are essential for ensuring that ADS systems are aligned with community needs and values [5]. For example, the AI Now Institute has developed the Algorithmic Impact Assessment (AIA) framework and an Algorithmic Accountability Toolkit, which are designed to enhance transparency, explainability, and public oversight of algorithmic systems [12]. These tools provide practical guidance for policymakers and help in identifying and mitigating biases and ensuring that ADS systems are used ethically and responsibly.

Effectiveness of Governance Mechanisms¶

The case studies provide concrete examples of both successful and problematic implementations of ADS in the public sector. The Swedish Transportation Agency's semi and fully-automated system for assessing driver license applications has been successful in reducing administrative discretion and enhancing procedural justice and rule-of-law [25]. Similarly, the Canadian government's Directive on Automated Decision-Making, which mandates the use of impact assessments and public input, has contributed to transparency and accountability in ADS systems [31]. In contrast, the Gothenburg public school placement scandal and the Dutch tax authority's use of a self-learning algorithm to detect social welfare fraud highlight significant failures in governance. Both cases involved practices of organizational ignoring and institutional blindness, leading to widespread breaches of regulations and legislation, and causing social and legal injustices [24].

Regulatory and Governance Mechanisms¶

The regulatory and governance mechanisms proposed and implemented in various jurisdictions offer valuable insights into effective strategies. The UK government's "Ethics, Transparency and Accountability Framework for Automated Decision-Making" provides a structured approach to ensure the safe, sustainable, and ethical use of ADS. This framework includes testing to avoid unintended outcomes, delivering fair services, ensuring clarity of responsibility, handling data safely, enhancing user and citizen understanding, ensuring legal compliance, and building future-proof systems [17]. The EU AI Act and the EU Digital Services Act mandate human intervention, pre-release conformity certifications, transparency, and audit requirements for high-risk AI systems, ensuring that these systems operate transparently and are subject to continuous monitoring [26]. Uruguay's multi-stakeholder oversight body and six-year AI Strategy (2024–2030) co-created with the public also exemplify effective governance mechanisms that foster democratic accountability and engagement [12].

Challenges in Implementation¶

However, the implementation of these mechanisms faces several common challenges. One of the most significant is the phenomenon of multilayered blackboxing, where the complexity and opacity of ADS systems make it difficult to understand and control the decision-making process. This can lead to a lack of transparency and accountability, as seen in the Gothenburg and Dutch cases [24]. To address these challenges, researchers and policymakers have proposed several strategies, including engaging a wider set of stakeholders (society-in-the-loop), developing institutional safeguards, and conducting active field interventions and engaged scholarship. These strategies aim to ensure that ADS systems are transparent, fair, and aligned with social and legal justice standards [24].

Integration of Ethics-Based Auditing¶

The integration of ethics-based auditing (EBA) into the governance process is another critical strategy. EBA involves various types of audits, such as functionality audits, code audits, and impact audits, which are designed to assess the behavior of ADS against predefined ethical principles and norms. Unlike mere codes of conduct or certification, EBA focuses on demonstrating adherence to ethical guidelines through a purpose-oriented process that includes stakeholder consultation and adversarial testing. Ensuring operational independence between the auditor and the auditee is essential to minimize risks of collusion and clarify roles, thereby enhancing the reliability and trustworthiness of ADS [19].

Key Insights and Implications¶

In summary, the governance of ADS in the public sector requires a multifaceted approach that combines transparent and bias-resistant algorithm design, human oversight and intervention, and robust legal frameworks. The successful examples from Sweden and Canada, and the proposed frameworks from the UK and EU, illustrate the importance of these elements in maintaining democratic accountability and citizen trust. The problematic cases from Gothenburg and the Netherlands underscore the critical need for addressing multilayered blackboxing and ensuring that governance mechanisms are effectively designed and enforced. Future research should focus on developing and testing these governance mechanisms in diverse international contexts to enhance their applicability and effectiveness. Additionally, further exploration of the ethical and legal dimensions of ADS, particularly in high-risk sectors, is necessary to ensure that these systems are used responsibly and ethically.

Having discussed the implications and future directions for the governance of ADS in the public sector, the following section will conclude the report by summarizing the key findings and their significance.

6. Conclusion¶

This report has provided a comprehensive examination of the governance mechanisms addressing the tension between algorithmic efficiency and procedural justice in automated decision systems (ADS) within public sector contexts. The literature review traced the historical evolution of ADS, highlighting the ongoing trade-offs between efficiency and innovation. It underscored the importance of ethical frameworks, such as beneficence, non-maleficence, autonomy, justice, and explicability, in ensuring the integrity and relevance of governance mechanisms. The role of civil society in shaping and monitoring ADS governance was also emphasized, with multi-stakeholder collaborations playing a crucial role in bridging the gap between technical, civic, and political realms, thereby enhancing trust and accountability.

The methodology section outlined a robust research design that aligns with the principles of ethical governance, synergy, and human involvement. The systematic literature review, case studies, and stakeholder feedback were used to gather and analyze data, ensuring a multifaceted and nuanced understanding of the governance challenges and solutions. The selection criteria for sources and the data analysis techniques, including multi-criteria decision-making (MCDM) and explainable AI (XAI) frameworks, further reinforced the methodological rigor and ethical considerations guiding the research process.

The case studies provided concrete examples of both successful and problematic implementations of ADS in the public sector. Successful implementations, such as the Swedish Transportation Agency's semi and fully-automated system for assessing driver license applications, have demonstrated the ability to reduce administrative discretion and enhance procedural justice and rule-of-law [25]. Similarly, the Canadian government's Directive on Automated Decision-Making, which mandates the use of impact assessments and public input, has contributed to transparency and accountability in ADS systems [31]. In contrast, the Gothenburg public school placement scandal and the Dutch tax authority's use of a self-learning algorithm to detect social welfare fraud highlighted significant failures in governance. Both cases involved practices of organizational ignoring and institutional blindness, leading to widespread breaches of regulations and legislation, and causing social and legal injustices [24].

The regulatory and governance mechanisms proposed and implemented in various jurisdictions offer valuable insights into effective strategies. The UK government's "Ethics, Transparency and Accountability Framework for Automated Decision-Making" provides a structured approach to ensure the safe, sustainable, and ethical use of ADS. This framework includes testing to avoid unintended outcomes, delivering fair services, ensuring clarity of responsibility, handling data safely, enhancing user and citizen understanding, ensuring legal compliance, and building future-proof systems [17]. The EU AI Act and the EU Digital Services Act mandate human intervention, pre-release conformity certifications, transparency, and audit requirements for high-risk AI systems, ensuring that these systems operate transparently and are subject to continuous monitoring [26]. Uruguay's multi-stakeholder oversight body and six-year AI Strategy (2024–2030) co-created with the public also exemplify effective governance mechanisms that foster democratic accountability and engagement [12].

However, the implementation of these mechanisms faces several common challenges. One of the most significant is the phenomenon of multilayered blackboxing, where the complexity and opacity of ADS systems make it difficult to understand and control the decision-making process. This can lead to a lack of transparency and accountability, as seen in the Gothenburg and Dutch cases [24]. To address these challenges, researchers and policymakers have proposed several strategies, including engaging a wider set of stakeholders (society-in-the-loop), developing institutional safeguards, and conducting active field interventions and engaged scholarship. These strategies aim to ensure that ADS systems are transparent, fair, and aligned with social and legal justice standards [24].

The integration of ethics-based auditing (EBA) into the governance process is another critical strategy. EBA involves various types of audits, such as functionality audits, code audits, and impact audits, which are designed to assess the behavior of ADS against predefined ethical principles and norms. Unlike mere codes of conduct or certification, EBA focuses on demonstrating adherence to ethical guidelines through a purpose-oriented process that includes stakeholder consultation and adversarial testing. Ensuring operational independence between the auditor and the auditee is essential to minimize risks of collusion and clarify roles, thereby enhancing the reliability and trustworthiness of ADS [19].

In conclusion, the governance of ADS in the public sector requires a multifaceted approach that combines transparent and bias-resistant algorithm design, human oversight and intervention, and robust legal frameworks. The successful examples from Sweden and Canada, and the proposed frameworks from the UK and EU, illustrate the importance of these elements in maintaining democratic accountability and citizen trust. The problematic cases from Gothenburg and the Netherlands underscore the critical need for addressing multilayered blackboxing and ensuring that governance mechanisms are effectively designed and enforced. Future research should focus on developing and testing these governance mechanisms in diverse international contexts to enhance their applicability and effectiveness. Additionally, further exploration of the ethical and legal dimensions of ADS, particularly in high-risk sectors, is necessary to ensure that these systems are used responsibly and ethically.

References¶

1. Savindu Herath Pathirannehelage, Yash Raj Shrestha, Georg von Krogh. (2024). Design principles for artificial intelligence-augmented decision making: An action design research study. European Journal of Information Systems.
2. Tor Grønsund, Margunn Aanestad. (2020). Augmenting the algorithm: Emerging human-in-the-loop work configurations. Journal of Strategic Information Systems.
3. https://www.tandfonline.com/doi/full/10.1080/08839514.2025.2463722. Available at: https://www.tandfonline.com/doi/full/10.1080/08839514.2025.2463722 (Accessed: August 22, 2025)
4. Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations. Available at: https://link.springer.com/article/10.1007/s11948-021-00319-4 (Accessed: August 22, 2025)
5. https://www.sciencedirect.com/science/article/pii/S0040162525001076. Available at: https://www.sciencedirect.com/science/article/pii/S0040162525001076 (Accessed: August 22, 2025)
6. Automated decision-making. Available at: https://en.wikipedia.org/wiki/Automated_decision-making (Accessed: August 22, 2025)
7. Automated Decision-Making Systems in the Public Sector – Some Recommendations. Available at: https://algorithmwatch.org/en/adm-publicsector-recommendation/ (Accessed: August 22, 2025)
8. Teresa Heyder, Nina Passlack, Oliver Posegga. (2023). Ethical management of human-AI interaction: Theory development review. Journal of Strategic Information Systems.
9. Government AI to be more ethical, transparent & accountable. Available at: https://www.government-transformation.com/data/government-ai-to-be-more-ethical-transparent-accountable (Accessed: August 22, 2025)
10. Frontiers | Transparency and accountability in AI systems: safeguarding wellbeing in the age of algorithmic decision-making. Available at: https://www.frontiersin.org/journals/human-dynamics/articles/10.3389/fhumd.2024.1421273/full (Accessed: August 22, 2025)
11. Jeroen Baijens, Tim Huygh, Remko Helms. (2022). Establishing and theorising data analytics governance: a descriptive framework and a VSM-based view. Journal of Business Analytics.
12. Digital Governance: Automated Decision-Making, Algorithms, and Artificial Intelligence. Available at: https://www.opengovpartnership.org/open-gov-guide/digital-governance-automated-decision-making/ (Accessed: August 22, 2025)
13. (PDF) Explainable AI (XAI) for trustworthy and transparent decision-making: A theoretical framework for AI interpretability. Available at: https://www.researchgate.net/publication/389783123_Explainable_AI_XAI_for_trustworthy_and_transparent_decision-making_A_theoretical_framework_for_AI_interpretability (Accessed: August 22, 2025)
14. https://www.researchgate.net/publication/334329322_Administrative_law_and_the_machines_of_government_judicial_review_of_automated_public-sector_decision-making. Available at: https://www.researchgate.net/publication/334329322_Administrative_law_and_the_machines_of_government_judicial_review_of_automated_public-sector_decision-making (Accessed: August 22, 2025)
15. https://lordslibrary.parliament.uk/research-briefings/lln-2024-0071/. Available at: https://lordslibrary.parliament.uk/research-briefings/lln-2024-0071/ (Accessed: August 22, 2025)
16. https://www.mdpi.com/2078-2489/15/11/725. Available at: https://www.mdpi.com/2078-2489/15/11/725 (Accessed: August 22, 2025)
17. Ethics, Transparency and Accountability Framework for Automated Decision-Making. Available at: https://www.gov.uk/government/publications/ethics-transparency-and-accountability-framework-for-automated-decision-making/ethics-transparency-and-accountability-framework-for-automated-decision-making (Accessed: August 22, 2025)
18. Automated Public Decision Making and the Need for Regulation. Available at: https://ppr.lse.ac.uk/articles/10.31389/lseppr.110 (Accessed: August 22, 2025)
19. Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations. Available at: https://pmc.ncbi.nlm.nih.gov/articles/PMC8260507/ (Accessed: August 22, 2025)
20. Jiaqi Yang, Alireza Amrollahi, Mauricio Marrone. (2024). Harnessing the Potential of Artificial Intelligence: Affordances, Constraints, and Strategic Implications for Professional Services. Journal of Strategic Information Systems.
21. https://www.tandfonline.com/doi/full/10.1080/10383441.2025.2477946. Available at: https://www.tandfonline.com/doi/full/10.1080/10383441.2025.2477946 (Accessed: August 22, 2025)
22. https://www.gao.gov/products/gao-21-519sp. Available at: https://www.gao.gov/products/gao-21-519sp (Accessed: August 22, 2025)
23. Human–AI Interactions in Public Sector Decision Making: “Automation Bias” and “Selective Adherence” to Algorithmic Advice | Journal of Public Administration Research and Theory | Oxford Academic. Available at: https://academic.oup.com/jpart/article/33/1/153/6524536 (Accessed: August 22, 2025)
24. Charlotta Kronblad, Anna Essén, Magnus Mähring. (2024). When Justice is Blind to Algorithms: Multilayered Blackboxing of Algorithmic Decision Making in the Public Sector.
25. Automated, administrative decision‐making and good governance: Synergies, trade‐offs, and limits - Roehl - 2024 - Public Administration Review - Wiley Online Library. Available at: https://onlinelibrary.wiley.com/doi/full/10.1111/puar.13799 (Accessed: August 22, 2025)
26. https://www.ntia.gov/issues/artificial-intelligence/ai-accountability-policy-report/overview. Available at: https://www.ntia.gov/issues/artificial-intelligence/ai-accountability-policy-report/overview (Accessed: August 22, 2025)
27. He Huang, Peter Burgherr, Cathy Macharis. (2024). A collaborative group decision-support system: the survey based multi-actor multi-criteria analysis (MAMCA) software. Journal of the Operational Research Society.
28. https://www.sciencedirect.com/science/article/pii/S0740624X21000137. Available at: https://www.sciencedirect.com/science/article/pii/S0740624X21000137 (Accessed: August 22, 2025)
29. Gerit Wagner, Roman Lukyanenko, Guy Paré. (2022). Artificial intelligence and the conduct of literature reviews. Journal of Information Technology.
30. Explaining Why the Computer Says No: Algorithmic Transparency Affects the Perceived Trustworthiness of Automated Decision‐Making - Grimmelikhuijsen - 2023 - Public Administration Review - Wiley Online Library. Available at: https://onlinelibrary.wiley.com/doi/10.1111/puar.13483 (Accessed: August 22, 2025)
31. AI governance in the public sector: Three tales from the frontiers of automated decision-making in democratic settings. Available at: https://pmc.ncbi.nlm.nih.gov/articles/PMC7164913/ (Accessed: August 22, 2025)