Last week, some University of Minnesota (UMN) security researchers kicked a hornet nest, when it was revealed that they’d tried to insert deliberately buggy patches into Linux. Greg Kroah-Hartman, the well-respected Linux kernel maintainer for the Linux stable branch, responded by banning not only them but any UMN-connected developers from contributing to the Linux kernel. Now, the researchers have sort of, kind of, apologized for their mistakes: “We sincerely apologize for any harm our research group did to the Linux kernel community.”
The apology started well enough. But, Kangjie Lu, the assistant professor in the Computer Science & Engineering Department of the UMN in charge of the product, and graduate student researchers, Qiushi Wu, and Aditya Pakki continued:
Our goal was to identify issues with the patching process and ways to address them, and we are very sorry that the method used in the “hypocrite commits” paper was inappropriate. As many observers have pointed out to us, we made a mistake by not finding a way to consult with the community and obtain permission before running this study; we did that because we knew we could not ask the maintainers of Linux for permission, or they would be on the lookout for the hypocrite patches. While our goal was to improve the security of Linux, we now understand that it was hurtful to the community to make it a subject of our research and to waste its effort reviewing these patches without its knowledge or permission.
You think? This is simply not how Red Team testing is done. At least some of the leaders of the targeted “ethical hacking attack” are made aware that an attack is coming. Otherwise, there’s no real difference between what these researchers did and ordinary, run-of-the-mill criminal hacking.
They continued, “We just want you to know that we would never intentionally hurt the Linux kernel community and never introduce security vulnerabilities. Our work was conducted with the best of intentions and is all about finding and fixing security vulnerabilities.”
They then explained, “The “hypocrite commits” work was carried out in August 2020; it aimed to improve the security of the patching process in Linux. As part of the project, we studied potential issues with the patching process of Linux, including causes of the issues and suggestions for addressing them.”
And, in any case, “This work did not introduce vulnerabilities into the Linux code. The three incorrect patches were discussed and stopped during exchanges in a Linux message board, and never committed to the code. We reported the findings and our conclusions (excluding the incorrect patches) of the work to the Linux community before paper submission, collected their feedback, and included them in the paper. [“On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits”].
The researchers continued that:
* All the other 190 patches being reverted and re-evaluated were submitted as part of other projects and as a service to the community; they are not related to the “hypocrite commits” paper.
* These 190 patches were in response to real bugs in the code and all correct–as far as we can discern–when we submitted them.
* We understand the desire of the community to gain access to and examine the three incorrect patches. Doing so would reveal the identity of members of the community who responded to these patches on the message board. Therefore, we are working to obtain their consent before revealing these patches.
What is this message board? We don’t know. It appears to not be one of the many official Linux developer mailing lists, nor the main Linux mailing list: The Linux kernel mailing list (LKML).
Their apology went awry though in their last point:
* Our recent patches in April 2021 are not part of the “hypocrite commits” paper either. We had been conducting a new project that aims to automatically identify bugs introduced by other patches (not from us). Our patches were prepared and submitted to fix the identified bugs to follow the rules of Responsible Disclosure, and we are happy to share details of this newer project with the Linux community.
Kroah-Hartman didn’t see these recent patches as being in the least bit valuable or even trustworthy. As he wrote about those April 2021 patches:
You, and your group, have publicly admitted to sending known-buggy patches to see how the kernel community would react to them and published a paper based on that work. Now you submit a new series of obviously incorrect patches again, so what am I supposed to think of such a thing?
They obviously were _NOT_ created by a static analysis tool that is of any intelligence, as they all are the result of totally different patterns and all of which are obviously not even fixing anything at all. So what am I supposed to think here, other than that you and your group are continuing to experiment on the kernel community developers by sending such nonsense patches?
When submitting patches created by a tool, everyone who does so submits them with wording like “found by tool XXX, we are not sure if this is correct or not, please advise.” which is NOT what you did here at all. You were not asking for help, you were claiming that these were legitimate fixes, which you KNEW to be incorrect.
A few minutes with anyone with the semblance of knowledge of C can see that your submissions do NOT do anything at all, so to think that a tool created them, and then that you thought they were a valid “fix” is totally negligent on your part, not ours. You are the one at fault, it is not our job to be the test subjects of a tool you create.
Our community welcomes developers who wish to help and enhance Linux. That is NOT what you are attempting to do here, so please do not try to frame it that way.
Our community does not appreciate being experimented on, and being “tested” by submitting known patches that are either do nothing on purpose or introduce bugs on purpose. If you wish to do work like this, I suggest you find a different community to run your experiments on, you are not welcome here.
As for their apology and request to “rebuild the relationship with the Linux Foundation and the Linux community from a place of humility to create a foundation from which, we hope, we can once again contribute to our shared goal of improving the quality and security of Linux software.” Kroah-Hartman simply stated:
As you know, the Linux Foundation and the Linux Foundation’s Technical Advisory Board submitted a letter on Friday to your University outlining the specific actions which need to happen in order for your group, and your University, to be able to work to regain the trust of the Linux kernel community.
Until those actions are taken, we do not have anything further to discuss about this issue.
What are these actions? We don’t know. I’ve asked the appropriate people to comment on the Linux community’s demands.
In the meantime, as for the code itself, Kroah-Hartman declared: “Because of this [issue], all submissions from this group must be reverted from the kernel tree and will need to be re-reviewed again to determine if they actually are a valid fix. Until that work is complete, remove this change to ensure that no problems are being introduced into the codebase.”
Kees Cook, Google Linux Kernel Engineer and member of The Technical Advisory Board wrote the “Board is taking a look at the history of UMN’s contributions and their associated research projects. At present, it seems the vast majority of patches have been in good faith, but we’re continuing to review the work.”
On Twitter, Cook added, “I spent a fair bit of time today going through each of the recent UMN research papers and mapping them to commits. They appear to all be in good faith. There are a small handful of mistakes that got later fixes, but given the volume of commits, that’s expected.”
As for UMN, Department of Computer Science and Engineering Mats Heimdahl, Department Head, and Loren Terveen, Associate Department Head, issued a statement in which they stated they’d learned on April 21st, only after Kroah-Hartman brought the matter to the developer world’s attention. They added that they’d learned “about the details of research being conducted by one of its faculty members and graduate students into the security of the Linux Kernel. The research method used raised serious concerns in the Linux Kernel community and, as of today, this has resulted in the University being banned from contributing to the Linux Kernel.”
The leaders continued, “We take this situation extremely seriously. We have immediately suspended this line of research. We will investigate the research method and the process by which this research method was approved, determine appropriate remedial action, and safeguard against future issues if needed. We will report our findings back to the community as soon as practical.”
Looking from above, Linux creator Linus Torvalds is reported to have said, “I don’t think it has been a huge deal _technically_, but people are pissed off, and it’s obviously a breach of trust.”
Stay tuned. While there appear to be no serious security problems as a result of the UMN blunders, trust lost is not easily regained. And, in the Linux kernel community, where trust is everything, that’s no small matter.
Related Stories:
