By Sanjay Raman Sinha
Artificial Intelligence (AI) algorithms are beginning to throw a spanner into the judicial process. The courtroom—traditionally governed by human logic, deliberation, and rigorous research—is now facing a formidable technological challenge. The concern is not merely automation, but hallucination: machine-generated fabrications that produce phantom cases, unpardonable faux pas, and potentially derail the course of justice.
This concern recently came into sharp focus in Omkara Assets Reconstruction Pvt Ltd vs Gstaad Hotels Bengaluru, where the Supreme Court encountered a troubling instance of AI-driven distortion. The counsel for Omkara Assets Reconstruction pointed out in a rejoinder that over a hundred citations relied upon by the appellant were imaginary case laws—hallucinated outputs generated by AI tools. While the apex court bench, comprising Justices Dipankar Datta and AG Masih, proceeded to hear the matter on its merits, the episode triggered serious judicial concern. The unbridled use of AI, the Court cautioned, carries unintended consequences and risks undermining justice if left unchecked.
The legal profession has witnessed an explosion of AI-powered applications that function as digital assistants—automating case management, drafting pleadings, conducting research, organizing documents, managing billing, and even connecting lawyers to clients. For practitioners overwhelmed by massive caseloads and tight deadlines, these tools are a boon. Yet, the underlying algorithms remain unreliable. In their effort to comply with prompts, AI systems often “create” legal authorities, facts, and references that do not exist. When such errors spill into court filings, they threaten to throw procedural fairness and judicial integrity out of gear. The Supreme Court has warned that reliance on these tools without advocate-on-record verification is a “terrible error”.
Recognizing the gravity of the situation, the Supreme Court has reconstituted its Artificial Intelligence Committee, chaired by Justice PS Narasimha. The Committee’s mandate is to guide the integration of technology into the judiciary while safeguarding due process. Its objectives include streamlining judicial workflows, enhancing access to justice, and promoting transparency. Comprising judges from various High Courts and technical experts, the Committee is tasked with ensuring that AI adoption remains accountable, ethical, and user-centric.
The Omkara Assets case is not an isolated incident, but part of a growing global pattern of AI hallucinations. In Mata vs Avianca (2023), lawyers were fined for citing fictitious cases generated by ChatGPT. In Noel Anthony Clarke vs Guardian News & Media (2025), an AI transcription error altered a speaker’s name, changing the legal context of the case. Such hallucinated outputs have no legal or constitutional basis and directly undermine the constitutional guarantee of a reasoned judgment.
These developments have prompted international concern. UNESCO has issued Guidelines for the Use of AI Systems in Courts and Tribunals, establishing guardrails for both institutions and individual judges. The guidelines aim to ensure ethical, human-rights-based use of AI in justice systems. They outline 15 core principles, including information security, auditability, transparency, and human oversight, and provide recommendations covering the entire lifecycle of AI systems. Crucially, they emphasize that AI must support—not replace—human judgment in the judiciary.
The Supreme Court’s e-Committee has already pioneered several notable AI initiatives. These include SUPACE (Supreme Court Portal for Assistance in Court Efficiency), which assists judges by summarizing facts and identifying relevant legal references; SUVAS and PANINI, translation tools that convert English judgments into regional languages to democratize access to justice; and LegRAA, an in-house generative AI tool trained exclusively on Indian case law to ensure domain accuracy and data security.
Acknowledging the inevitability of AI usage in the judiciary, the Supreme Court has also released a White Paper on AI and the Judiciary, proposing a structured framework for responsible adoption. It mandates disclosure of documents drafted using AI, accompanied by prominent disclaimers, and requires advocates to certify manual verification of all citations against official databases. The White Paper makes it clear that AI must remain advisory, with all discretionary and decision-making powers vested in human judges. It also recommends prioritizing in-house AI tools to prevent data leaks to foreign servers. Together, these measures aim to preserve transparency, accountability, and the integrity of the judicial process.
A recent UNESCO survey underscores the urgency of these safeguards. While nearly 44 percent of judicial operators worldwide report using AI tools, only nine percent have received formal training. An overwhelming 73 percent believe that mandatory regulations and guidelines are essential. The findings reveal a stark gap between AI adoption and preparedness, highlighting the need for standardized training and governance frameworks to ensure that AI-enhanced justice remains firmly anchored in human reasoning.
The post When Algorithms Enter the Courtroom appeared first on India Legal.