Incomplete and Garbled Immigration Court Data Suggest Lack of Commitment to Accuracy
TRAC recently discovered gross irregularities in recent data releases from the Executive Office for Immigration Review (EOIR), the agency that oversees the US immigration court system. After attempting – unsuccessfully – to work with the EOIR to fix these problems, TRAC decided to make public our observations of the quality of the agency’s public data releases as well as express our concerns about the lack of commitment within the agency to responsible data management.
Policymakers and the public routinely put their faith in federal agencies to provide complete and accurate information about their work. The value of government transparency is even higher in the area of immigration law and the Immigration Courts, which have become topics of considerable concern for Americans from all walks of life and for all three branches of government. In the present context, TRAC views concerns about EOIR’s data inconsistencies – outlined below – as substantive, ongoing, and in need of prompt attention. Of greatest concern is the lack of commitment from EOIR to ensuring the public is provided with accurate and reliable data about the Court’s operations.
“Significant Errors” in Past EOIR Data
This is not the first time the public has identified significant inaccuracies in EOIR’s reported data. For instance, the Supreme Court of the United States relied upon figures provided by the EOIR as the basis for a major ruling affecting ICE detention practices. After the Supreme Court decided the case, the public discovered that the figures provided by the EOIR were fundamentally wrong. The EOIR did not uncover the data irregularities on its own. The EOIR’s mistakes were only recognized because the public obtained the underlying data through a Freedom of Information Act (FOIA) request and identified the relevant discrepancies.
After the public alerted the government to its inaccuracies, in 2016 the U.S. Solicitor General was compelled to issue a formal letter to the Supreme Court apologizing for providing inaccurate data. The following excerpt of the Solicitor General’s letter on August 26, 2016 attests to this error:
“This letter is submitted in order to correct and clarify statements the government made in its submissions. … EOIR made several significant errors in calculating those figures. … This Court’s opinion cites figures that ‘EOIR ha[d] calculated,’ …, and those are, in fact, the figures EOIR had calculated, albeit incorrectly. … The Court therefore may wish to amend its opinion…” (emphasis added)
This example illustrates the very real danger posed by the EOIR’s mishandling of data, as well as the value to society – and the government itself – of ongoing oversight through Freedom of Information Act (FOIA) requests. Despite the EOIR’s past data mistakes, however, the quality of the agency’s data releases has recently declined to unacceptable levels, as we discuss in the following section.
Recent Data Trouble at the EOIR
As a result of TRAC’s ongoing FOIA requests, the Executive Office for Immigration Review releases a large batch of anonymized data about Immigration Court cases every month. Statistics on the operation of the Immigration Courts largely rely on information kept in a massive database maintained by the EOIR. The EOIR records information on each matter filed with the court and tracks subsequent events as the Court processes each case. This data is central to the Court’s ability to efficiently and effectively manage its workload.
Although this data is highly valuable to policymakers and the public, the EOIR’s mishandling of the data undermines its accuracy and public value. These data problems have been occurring with increasing regularity. Severe irregularities with the September 2019 data release set a new low.
On October 9, 2019, the EOIR responded to TRAC’s FOIA request for updated case-by-case data through September 2019. TRAC promptly began processing the data in order to update TRAC’s online tools and reports, and discovered serious inconsistencies that made the data unusable. TRAC alerted the EOIR to the problems we uncovered. The chronology below summarizes the cycle of data mishandling for the September data release, and TRAC’s attempts to work with the EOIR to obtain complete and corrected data.
- Data Release, Batch 1. The initial release of the EOIR’s September data included 11 separate files of records on Immigration Court proceedings that were incorrectly formatted. The garbled data resulted in substantial confusion over the relationship between certain variables and values, with some values appearing to apply to the wrong variables in the file. If potential users were even able to read the garbled data, one could reach entirely erroneous conclusions about court events. As soon as TRAC discovered these issues, it alerted the EOIR directly. EOIR promised to look into the matter.
- Data Release, Batch 2. In response to TRAC’s notification, the EOIR replaced the first release with a second release and informed TRAC that the problems had been fixed. However, when TRAC processed the second release, it found that while the first set of problems had been fixed, an entirely new set of problems had occurred. In Batch 2, thousands of records of court proceedings and 2.8 million records on scheduled hearings – hearings and proceedings which were included in the first release – had entirely disappeared. TRAC alerted the EOIR directly to the new set of data inconsistencies. EOIR promised once again to look into the matter.
- Data Release, Batch 3. The EOIR informed TRAC that it had fixed these new problems, and that TRAC could trust Batch 3 of the EOIR’s data release. Note that EOIR doesn’t change the labels it uses for each release; the file name remains the same and hence on its face indistinguishable from any previous release. After processing millions of records contained in the series of separate tables that made up the new release, TRAC found that problems in batch three were identical to problems in batch two. We again notified EOIR that the problems remained. At first EOIR insisted that TRAC was wrong and that the problems had been fixed. It later emerged that while the General Counsel’s office of EOIR (TRAC’s point of contact) believed a third and corrected release was being supplied, the files had not been changed but were actually the same files that TRAC had received in Batch 2.
- Data Release, Batch 3 (cont.) TRAC was finally provided access to what was again billed as the corrected September release. TRAC again processed these files. This time, based on total record counts it appeared that the missing 2.8 million records on scheduled hearings had reappeared. However, some court proceedings that had been contained in Batch 1 were still missing. And there were still other puzzling omissions which we describe in more detail below.
After this series of mistakes, TRAC urged the agency to implement basic quality control procedures to ensure that the EOIR’s data releases to the public were not inadvertently garbled or incomplete. Moreover, TRAC expressed concern about the EOIR’s underlying data management practices which posed a risk to both the public and the government if left unaddressed. We conveyed these concerns to EOIR noting specifically:
“There are standard procedures that anyone in charge of maintaining databases use. The pattern of repeatedly releasing files which are either unreadable or incomplete demonstrates the agency’s standard operating procedures are woefully inadequate.
This really needs to be taken seriously. Without answers to our questions that get to the bottom of what occurred, identifying what went wrong, and implementing a plan to catch mistakes before the agency publicly distributes bad data, means that history will keep repeating itself.”
On Friday, October 25, 2019, while admitting mistakes had been made, the EOIR dug in its heels. The agency responded to TRAC’s entreaties by sidestepping the underlying issue and avoiding responsibility for its routine inaccuracies:
“[T]he FOIA [Freedom of Information Act] does not require the Agency to create records in response to your specific questions, nor to certify the accuracy of data contained in responsive documents.”
TRAC was forced to take note of the EOIR’s unwillingness to fully correct their mistakes and to work with the public to resolve the declining quality in their data releases.
The Case of the EOIR’s Disappearing Data
After recognizing the seeming inability of EOIR to produce a correct and complete data release for September 2019, TRAC began digging deeper into the problem.
Our concern about EOIR’s data was already heightened. We recently discovered that some months ago the EOIR had begun silently deleting swaths of records in their entirety from the data releases that we and other members of the public received. EOIR belatedly told us that withholding of entire records was necessary to protect immigrants’ privacy. This rationale was perplexing since these records were already anonymized and all identifying details deleted. Regardless of the EOIR’s justification for withholding the records, the agency had started making these deletions without alerting us that it was doing so, and failed to mark the data in any way to indicate the magnitude of the deletions or indicate in which files the deletions occurred.
TRAC is in a fairly unique position to examine this problem. For many years, TRAC has been regularly requesting snapshots of anonymized data from EOIR’s database as part of our mission to provide the public (and often other government agencies themselves) with access to reliable, accurate data about the Immigration Courts. Because we receive and retain these monthly snapshots, we are able to monitor changes in these releases over time and assess whether releases are incomplete or inaccurate in some other way.
Therefore, TRAC undertook a careful comparison by matching the records received in the September 2019 release against the EOIR’s release for the previous month, August 2019, and with the release we received a year ago for September 2018. This time we matched records based on unique identification numbers rather than simply comparing total record counts. This allowed us to identify records which the EOIR released in the past but were missing entirely from the current shipment.
The results of this comparison were sobering. Compared to the August 2019 release, the (allegedly-accurate) final September 2019 release was inexplicably missing more than 1,500 applications for relief that were present the previous month. We further found that 896,906 applications for relief which were present in the September 2018 release from a year ago were missing from the September 2019 files we received. This discrepancy of nearly a million records largely occurred because the EOIR appears to have started silently but systematically deleting records.
Compared to the data from August 2019, the EOIR’s files for September were also missing records on over 600 charges DHS had filed. Also missing were over 700 case and/or court proceeding records, and over 900 records on scheduled hearings. An additional 1,200 records flagging various specific types of cases were also missing. For context, this flagging system is used to identify juveniles, recently arrived families seeking asylum, and immigrants required to remain in Mexico under the Migration Protection Protocols, and other special cases.
When the records in the September 2019 release TRAC received were matched with those from the September 2018 release a year earlier, the problems we uncovered multiplied. It was clear that the problem of missing records grew by leaps and bounds with the passage of time.
EOIR Data Management: Problems and Solutions
Based on the investigation above, TRAC identified key gaps in the EOIR’s data verification procedures that lead to unreliable and inaccurate data releases.
- Unintentional data removal. The EOIR’s data is inconsistent because the agency apparently does not perform a simple yet essential data verification step: it does not compare the number of records in its source database and the number of records in its released files to ensure that no records have been lost along the way. This is not merely a best practice. It is an industry standard for agencies managing large databases, and it is a routine practice in many of the EOIR’s peer agencies that provide large data releases to TRAC.
- Intentional data removal. The EOIR also does not appear to be keeping track of intentionally deleted records. If the EOIR is screening out records for specific reasons, then the number withheld for each reason in a file should be counted and these counts provided. The number withheld plus the number released should match the total number of records read in to ensure reliability.
- Garbled data releases. The EOIR is paying insufficient attention to how data releases are produced and formatted. Columns and rows in each table need to properly line up; otherwise information becomes garbled. And since EOIR’s database consists of many closely interconnected tables, copying data in individual tables at widely separated points in time inherently means the information will be out of sync.
- Possible data deletion in master database. Deletions of the EOIR’s original source records need to be carefully tracked and procedures in place to prevent unauthorized deletions from occurring. If applied systematically, such a verification process would also pinpoint whether there were deletions made in EOIR’s original source records. Any suspicious deletions need to be investigated to ensure the integrity and completeness of this master database is maintained.
If the EOIR does not implement basic data verification procedures, the public cannot tell if records were intentionally withheld and why they were withheld, or if records were accidentally omitted during the data copying process. The failure to address these problems also means that the public has no way to test for potential problem areas in EOIR’s underlying master data files.
Accuracy, Reliability, Cooperation
Under any circumstance, maintaining a massive database of this nature is challenging. Clearly it requires the resources necessary for day-to-day operations. More fundamentally, however, it requires a commitment on behalf of the agency to provide the public with complete, accurate, and reliable data about the agency’s operations. When TRAC uncovered unexplained data issues in the past, we have brought them to the attention of the EOIR and generally found the agency to be fairly responsive and committed to ensuring accurate reporting. The recent change in posture is therefore concerning. Moreover, because EOIR’s data are relied upon as part of the official record of court filings and proceedings that have taken place, one should not expect official records to simply go missing without explanation.
It is deeply troubling that rather than working cooperatively with TRAC to clear up the reasons for these unexplained disappearances, the agency has decided to dig in its heels and insist the public is not entitled to have answers to why records are missing from the data EOIR releases to the public. TRAC urges the EOIR to take the basic steps necessary for managing any large database, especially a database of as inestimable value and relevance as the one EOIR maintains for the Immigration Courts.
TRAC is a nonpartisan, nonprofit data research center affiliated with the Newhouse School of Public Communications and the Whitman School of Management, both at Syracuse University. For more information, to subscribe, or to donate, contact trac@syr.edu or call 315-443-3563.
