Inside the Brazilian Supreme Court’s distribution of processes

On 13/09, the team of researchers from the University of Brasilia (UnB) responsible for examining the Supreme Court of Brazil’s case distribution system published an advisory opinion with the results of its evaluation. In today’s post, we interviewed Professor Henrique Araújo Costa, one of the group’s researchers, regarding the process and the team’s conclusions.

Process distribution systems and the demand for transparency

The legal regulation of the distribution of cases, which determines which judges will be responsible for what causes, establishes that this distribution must occur in an alternating, random and egalitarian manner (article 285 of the CPC). This procedure is performed through computational algorithms in many Brazilian courts, including the Supreme Court and the Superior Court of Justice (STJ). The internal functioning of these programs is rather obscure, which raises important issues related to the transparency and security of the sortitions .

When Minister Teori Zavascki died in a plane crash in early 2017, the question of who would be drawn to report on the processes involved in Operation Car Wash attracted much public interest. The definition of the rapporteur of a process largely influences its progress, which causes even greater curiosity about the system.

As cryptographer Diego Aranha warns: “Contrary to common perception, there is no guarantee that machines need to work as expected, and their correct functioning depends on both the programmer’s ability to design algorithms and their resistance to external manipulations.”

With that in mind, the Supreme Court issued a notice for an evaluation of its system in this year’s May, which was carried out by the UnB team and had the opinion as result.

The evaluation results and the areas for improvement of the system

According to Professor Henrique Costa, there is “nothing wrong” with the way the distribution is currently made. The distribution was classified by him as “very good”, although there are “areas for improvement”.

One such space would be the system documentation. According to the opinion, “Supreme Court representatives said that there are documents and spreadsheets that explain part of the solution, but that there is no complete documentation of the software and components.” In this sense, the text recommends “documenting procedures (workflows) to improve the degree of governance of the following routines: classification, assessment, and distribution of processes” as well as software development, especially in deploying new versions.

Another possible improvement in the researcher’s view would be “to mix opinions with dialogues, and to have meetings and even public hearings if necessary so that everyone can respond in a more institutional environment and we can move forward.” In the same train of thought, he pointed out the importance of alternating the institutions responsible for evaluation over time.

Why not to disclose the source code?

In the opinion, the UnB team took a position in favor of disclosing the algorithm’s source code and pointed out that the system is not limited to it. Notwithstanding this point, the researchers state that “within the perspective of a work under construction” the opening of this code would be “desirable as one of the evolutionary milestones of the distribution process.”

The Supreme Court Commission, on the other hand, claims in its report (which can be found below the opinion on the link above) that “the OPINION presented does not detail any risks and does not clarify the reasons for the suggestion of publication of the source codes; considering also that, at the option of UnB TEAM, there was no effective access to the source codes and other components that integrate the process distribution solution.” That would be an obstacle, according to the Commission, prevent them from reaching the same conclusion regarding the disclosure of the code.

In Professor Costa’s view, the question can be reversed: “If there is no vulnerability in disclosure, why not spread it?” He pondered.

“No one will find out anything wrong, because there is not, but what will happen is that people will continue to create narratives, hypotheses, and questions. A distribution will happen and everyone will say that this minister or the other minister received by induction, that it was not random, etc. So the bad thing about not divulging it is that you educate less people, and by educating less, there is more space for conspiracies and sensationalism.” he reflected.

Finally, the professor restated that this is a work under construction and reported that the UnB team is available to “continue in this work and clarify everything that has not been clear.”

The full interview is available below.

The evaluation involved meetings to present the legal, procedural and technological aspects of the distribution, as well as technical visits to the Supreme Court’s section responsible for this distribution. How did these encounters and visits take place?

As you can see, the Supreme Court Commission is a committee that had participants from both the presidency and the Judicial Secretary – a more operational part that understands the lawsuits – and from the I.T part. They had a multidisciplinary composition. When we demanded some deeper clarification into one of these fronts, they called other people. We had a series of interviews where we leveled our knowledge and checked some things and then had access to a room with security and everything else where we could see the code and ask questions. That is how the work unfolded.

In the UnB team there were people from the legal area and also people from the technological area. Was the access to the code considered satisfactory by these people in the technological are?

See, when we come in to see such a code, it’s not enough having tech people who do not have a view of the context. They need to be assisted. And there I was present and we considered that it was enough to express the opinion that we issued. Now, of course, I’ve already mentioned this in other interviews, this is a work under construction. Everything can be improved. So we were happy to have had the chance to initiate a dialogue and to have enough knowledge to recommend the disclosure of the algorithm. That’s it for now.

At the end of the meetings, the UnB team considered it more appropriate to present an advisory opinion containing several recommendations on the system than to present an exhaustive report addressing point by point (as foreseen in the Invitation to Bid). How was the group led to this choice of presentation of the results, as opposed to the report?

This is an interesting question that I can answer in a simple way. I think it escaped us because we, when we are called, our language is the language of the opinion, so we gave an  opinion. But calling a paper of opinion or report does not change its essence at all. What happens is that the notice fixed some questions and these questions we consider that not all of them would be necessary to express ourselves on the heart of the matter.

Then we manifest ourselves about what we consider to be mature and we think it is a possible way to limit ourselves to what we say and conclude what we have concluded. I remember that the Supreme Court’s response says, at some point, that it would be available for further work. And we are willing to continue this work and clarify everything that has not been made clear.

Can you give an example of any of these points that the notice suggested to be presented more exhaustively, but that would not be necessary in the evaluation of the UnB team? And for what reason?

It is good to remember that I am subject to a term of secrecy, so there are aspects that I will prefer that are brought to the Supreme Court itself, but I would say that it is just a question of identifying the core of what we need to manifest about. First, there is the issue of transparency, and we are presenting an exempt view or at least a view that is not from the institution but from a third party. And this view is in the sense that if there is no vulnerability in disclosure, why not to disclose it? I think there is a bit of that reversal happening. We do not try to justify why it should be published, we try to investigate the contrary: why they do not divulge? That is the question.

Now there are a lot of questions there that we do not evaluate because they are collateral. They are not any kind of detour. What matters is that the Supreme did something for the first time in launching this edict and we did it, but it’s something that is not so hot from the point of view of the news. The fact is that often, when you are going to carry out a job, that job can have some positive results in this dynamic of entering, auditing and manifesting. That is a possibility.

Now there is another richer possibility, which we have not yet been able to do, which is dialogue. And this dynamic of dialogue is much more flexible and allows much more progress to be made. These returns we have actually made, they appear to be ill-explained, as if we had not answered everything. With dialogue this does not happen, does it? Because you say, “Look, I think the best question is this one, this one and this one” or “The aspects we can approach with the time and resources that we have are these, and after that we will have more clarity on the subject and we will be able to deepen”. I see it this way: What an opinion can do and what the dialogue can do.

This brings up the issue of disclosing or not disclosing the source code. The team’s position in the opinion is that there are no risks involved in disclosing the code. Could there be risks involved in not disclosing it in the group evaluation? As, for example, the existence of bugs or vulnerabilities that would go unnoticed?

If it is not disclosed, everything remains as it is. No one will find out anything wrong, because there is not, but what will happen is that people will continue to create narratives, hypotheses, and questions. There will be a distribution and everyone will say that this minister or the other minister received by induction, that this was not random, etc. So the bad thing about not divulging it is that you educate people less. And by educating less, one has more room for conspiracy and sensationalism. I think that’s it.

One of the points that the group touches on in the opinion is that although the issue of whether or not to disclose the source code of the distribution software receives a lot of attention, transparency regarding the system is not limited to that. What other aspects did the team consider significant regarding the transparency of this system and for what reasons?

If you look, one of the most substantial recommendations and suggestions we’ve made is to improve the disclosure of distribution certificates. What does that mean? To improve the way the process documentation is done from the moment the process arrives at the institution until it is randomly referred to a judge.

What happens today? I’ll take the liberty of explaining something a bit more technological. The whole judicial process was built on the metaphor of the document. Paper records, documents, certificates, all this bureaucracy. And the electronic process replicates this logic, even innocently. When you begin to enter into a larger abstraction that is that of the workflow, or what the routine is like, what the data is and how it can be consulted in a massive way so that you have a view given by a subset of data, this is totally different from the approach of a pile of paper stacked metaphor.

And what we are trying to suggest is that it is important that data on processing and distribution, both the part that prepares it and the part that actually draws it, should be available in a way that every scientist or citizen had formulate their hypotheses and tests, make their charts and their queries. So one of the recommendations we make is to evolve the way we deliver information.

There is a question that concerns the preparation prior to this distribution, which refers to a change that the Supreme Court has undergone in recent years: from distributing processes grouped by class to distributing them individually. Could you comment on that?

When a lawsuit arrives in any court, the law currently says that its distribution should be immediate. That means that it can not wait for a draw, it has to come and be drawn. Imagine a situation in which any court decided to only receive new cases after judging all those who are already accepted by him. And then your process, for example, is without a responsible and you do not even have to turn to.

This is an illegal situation, however, the law no longer allows this when defining that distribution should be immediate. Then I turn to the question back to you: to say that the distribution should be immediate means that each process must be drawn in the second that arrives or we can do lots and distribute several times throughout the day?

This is a question, in fact, whose answer depends on the interpretation that is made. I understand that there is no objection to the raffles taking place in several lots throughout the day. It is not necessary to sort the case as soon as it arrives. The point is that if the raffle happens process by process, the safety precaution needs to be greater.

If the distribution happens in process blocks, for example, every hour with the processes received at that time, this security is done in an easier manner. That’s because no one can dominate a batch distribution, while someone can, in theory, dominate a lottery that is separate. That is the question. So there are choices: when choosing to make a single distribution, process by process, care has to be greater. Choosing to do a batch distribution means that no one will ever master the distribution of a single process. This is cheaper, it is easier. It is only an administrative matter to be considered, it is not the vulnerability.

In addition to the ideal of randomness in distribution, the system is also responsible for ensuring equality in the division of labor among ministers. What mechanisms are employed to ensure this, even at the technical level? And, in your view and in the opinion of the team, is there enough regulation around these compensation mechanisms?

This is an interesting question because it involves several concepts and gives rise to much speculation. The fact is: not every minister receives the same workload on the same day, because they may be ill, traveling, away or in separate functions, and the system must create compensation techniques so that some ministers, by overcoming this impediment had – let’s say for being away on a trip – so that on the return they receive a little more than the others and have the same workload.

And then the doubts arise: this compensation, does it hurt the randomness? And that is the question that is commonly asked. And the answer is this: it does not hurt randomness, what happens is that over the course of time compensation will happen, some ministers will compete with a slightly greater chance than others. Over time, this compensation will stabilize and day to day differences will be diluted over time.

This is a very ingenious mechanism that is little known, but it does exist. And compensation addresses this problem: how to distribute equal work among ministers. It has no vulnerability or problem. It’s just that people may not be ready to understand that one minister can receive more than the other and that does not hurt randomness.

It would only hurt randomness if it were compensation without any limit. Imagine you are a minister, and on the day you return from travel, all lawsuits go to you. Imagine this scenario. There would be randomness. But there are controls in the system so this does not happen. If you are a minister who leaves and is left without receiving proceedings, when you return, for some time, you will receive a little more. This is how the workload and randomness in the distribution are made compatible.

This is at the level of how the thing is accomplished. And at the level of normative regulation of these mechanisms, how do you assess the relationship between this process and the existing framework as to how compensation should be made?

The possibility of compensation is regulated. Now, and this is something that I always emphasize to never get the wrong message and be taken out of context, everything in life can be improved, even a very good thing has room for improvement. The distribution of the Supreme Court is a very good thing that still has room for improvement. And in any corporate system evaluated areas for improvement are found.

One of the areas for improvement in the Supreme Court is the documentation and justification of all changes made so that there is no margin. Our contribution is much more in the sense of improving the quality of documentation than of modifying what is in progress. It is necessary to establish a record of the information regarding justification, granularity, frequency, security, etc.

All this has to be present, not just that it works. And I would even say that in any audit conducted in any court, public agency or company, this is going to be a universal recommendation. It does not mean a vulnerability, but rather a space for improvement.

Prior to the notice, the Supreme Court had ignored and even denied requests made through the Access to Information Act to publicize the code, alleging lack of ‘normative forecast’ about it. The Superior Court of Justice has also denied disclosing its algorithm code based on allegations of security concerns. The Superior Electoral Court has adopted a similar attitude, imposing significant restrictions on researchers who performed the voting machine evaluation, such as removing entire partes from the source code during the security test to increase test difficulty, for example. How do you see this attitude of some judicial bodies facing the social demand for transparency in the use of new technologies?

It is important to stress that I am answering a question about my opinion on other courts. It’s my opinion. I’m not giving the opinion of the team or the University. And my opinion on this is that the cycle of demand for transparency that we are living will require, also from the institutions, a learning, and a preparation.

While transparency is desirable, it has burdens, and we must have, in our institutions, ways of dealing with these burdens, so that we no longer have that kind of response. This is a response that comes up, for example, in this security argument, which is not really security, but rather, of the unavailability of means, of a lack of preparation for this to occur.

It is a consequence of the demand for transparency that needs to be addressed and provided. I do not think that kind of justification will prevail in the long run.

You said that everything is under construction and everything can be improved. In your assessment of how the evaluation of the Supreme Court distribution system was carried out this time, what aspects do you see that could be improved

This question is very difficult, but I will respond with a focus on something that has distressed me greatly: false promises. We have to be aware of what is really possible and what is not yet possible. So while it is a very promising field, we can not become hostage to empty promises.

I think we will still go through a time when we will lower our expectations regarding the achievements of technology. Today we are very heated and believing that technology is the way out for many things. This leads to false expectations.

So we’ll have, I believe, a slowdown in that because we’re going to be a bit frustrated with the achievements that technology is bringing. And once we get frustrated a little, it will be great, because from then on we will begin to act more within the possibilities of a gradual and painful evolution. I think that’s it.

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Institute for Research on Internet and Society.