LEGAL-UQA
Pakistan's Constitution is written in English, but most Pakistanis speak Urdu or a regional language. A parallel English–Urdu benchmark for testing whether language models can answer constitutional questions in Urdu, a low-resource language.
📄 Paper (under review): LEGAL-UQA: Testing Constitutional Question-Answering Systems for Urdu Speakers with a Parallel English–Urdu Benchmark — Faizan Faisal*, Umair Yousaf*, Muhammad Hassnain, Zeerak Babar (*equal contribution).
I grew up in Pakistan, where most people speak Urdu or a regional language such as Punjabi, Sindhi, Pashto or Balochi in daily life. The Constitution — the document that sets out everyone’s rights — is written in English. Urdu is the national language, yet the authoritative text of the law is in a language many citizens do not read comfortably. For a large part of the country, finding out what the Constitution actually says about your own rights means relying on someone else to read and explain it.
Language-model assistants could help close that gap by answering legal questions in the language people actually speak. But Urdu, like many widely spoken languages, is low-resource for language technology: there is little high-quality, domain-specific data to train or test models with, and almost none for law. Without such data, there is no way to know whether an assistant answers an Urdu legal question correctly — or confidently gets it wrong.
LEGAL-UQA (Legal Urdu Question Answering) is a benchmark for exactly that question. It is built from the Constitution of Pakistan and contains 619 parallel English–Urdu question–answer pairs drawn from 305 article passages, with each question, reference answer and source passage available in both languages. The constitution was digitised with OCR and cleaned by hand, and the question–answer pairs and their translations were produced with the help of GPT.
Question
Do the provisions regarding the Holy Quran and Sunnah affect the personal laws of non-Muslim citizens?
کیا قرآن پاک اور سنت کے احکام غیر مسلم شہریوں کے قوانین شخصی پر اثر انداز ہوتے ہیں؟
Reference answer (Article 227)
No, the provisions do not affect the personal laws of non-Muslim citizens or their status as citizens.
نہیں، یہ احکام غیر مسلم شہریوں کے قوانین شخصی یا ان کی بطور شہری حیثیت پر اثر انداز نہیں ہوتے۔
GPT-5.4 — human score: 1
نہیں، اس حصے میں کسی امر کا غیر مسلم شہریوں کے قوانینِ شخصی پر اثر نہیں پڑے گا۔
English translation: No, nothing in this Part shall affect the personal laws of non-Muslim citizens.
Fine-tuned mT5 — human score: 0
نہیں، قرآن پاک اور سنت کے احکام کو صرف اس طریقے کے مطابق نافذ کیا جائے گا جو اس حصے میں منضبط ہے۔
English translation: No, the injunctions of the Holy Quran and Sunnah shall be enforced only in the manner provided in this Part.
LEGAL-UQA tests two parts of legal question answering separately:
- Answering: given an Urdu question and the relevant constitutional passage, can a model produce the correct answer in Urdu?
- Retrieval: given only the question, can a system find the right passage in the constitution?
The repository contains the dataset, the source documents, saved model outputs, human grades, and the notebooks and scripts used to build the dataset and evaluate models.
The new paper compares nine sets of Urdu answers against reference-similarity metrics and human grades, tests word-matching and embedding-based retrieval, and checks whether an automated LLM judge can stand in for human grading of Urdu answers.
The first version of LEGAL-UQA was led by Faizan Faisal and Umair Yousaf and released on arXiv in 2024.