The murderer I emailed with is still in prison. And the software that got him pissed off at me still runs, so I ran it. Now here I am to pass on the history and then go all geeky. Here’s the tell: If you don’t know what a “filesystem” is (that’s perfectly OK, few reasonable adults need to) you might want to stay for the murderer story then step off the train.

Filesystems are one of the pieces of software that computers need to run, where “computers” includes your phone and laptop and each of the millions of servers that drive the Internet and populate the cloud. There are many flavors of filesystem and people who care about them care a lot.

One of the differences between filesystems is how fast they are. This matters because how fast the apps you use run depends (partly) on how fast the underlying filesystems are.

Writing filesystem software is very, very difficult and people who have done this earn immense respect from their peers. So, a lot of people try. One of the people who succeeded was named Hans Reiser and for a while his “ReiserFS” filesystem was heavily used on many of those “Linux” servers out there on the Internet that do things for you.

Reiser at one point worked in Russia and used a “mail-order bride” operation to look for a spouse. He ended up marrying Nina Sharanova, one of the bride-brokerage translators, and bringing her back to the US with him. They had two kids, got divorced, and then, on September 3, 2006, he strangled her and buried her in a hidden location.

To make a long story short, he eventually pleaded guilty to a reduced charge in exchange for revealing the grave location, and remains in prison. I haven’t provided any links because it’s a sad, tawdry story, but if you want to know the details the Internet has them.

I had interacted with Reiser a few times as a consequence of having written a piece of filesystem-related software called “Bonnie” (more on Bonnie below). I can’t say he was obviously murderous but I found him unpleasant to deal with.

As you might imagine, people generally did not want to keep using the murderer’s filesystem software, but it takes a long time to make this kind of infrastructure change and just last month, ReiserFS was removed as a Linux option. Which led to this Mastodon exchange:

(People who don’t care about filesystems can stop reading now.)

Now, numbers

After that conversation, on a whim I tracked down the Bonnie source and ran it on my current laptop, a 2023 M2 MacBook Pro with 32G of RAM and 3T of disk. I think the numbers are interesting in and of themselves even before I start discoursing about benchmarking and filesystems and disks and so on.

Bonnie says:

1. This puppy can write 3.7 GB/second to a file, and read it back at 6.1GB/sec.

2. It can update a file in place at 2.8 GB/sec.

3. It can seek around randomly in a 64GB file at 33K seeks/second.

4. Single-threaded sequential file I/O is almost but not quite CPU-limited.

I wonder: Are those good numbers for a personal computer in 2024? I genuinely have no idea.

Bonnie

I will shorten the story, because it’s long. In 1988 I was an employee of the University of Waterloo, working on the New Oxford English Dictionary Project. The computers we were using typically had 16MB or so of memory (so the computer I’m typing this on has two thousand times as much) and the full text of the OED occupied 572MB. Thus, we cared really a lot about I/O performance. Since the project was shopping for disks and computers I bashed out Bonnie in a couple of afternoons.

I revised it lots over the years, and Russell Coker made an excellent fork called Bonnie++ that (for a while at least) was more popular than Bonnie. Then I made my own major revision at some point called Bonnie-64.

In 1996, Linux Torvalds recommended Bonnie, calling it a “reasonable disk performance benchmark”.

That’s all I’m going to say here. If for some weird reason you want to know more, Bonnie’s quaint Nineties-flavor home and description pages are still there, plus this blog has documented Bonnie’s twisty history quite thoroughly. And explored, I claim, filesystem-performance issues in a useful way.

I will address a couple of questions here, though.

Do filesystems matter?

Many performance-sensitive applications go to a lot of work to avoid reading and/or writing filesystem data on their critical path. There are lots of ways to accomplish this, the most common being to stuff everything into memory using Redis or Memcached or, well, those two dominate the market, near as I can tell. Another approach is to have the data in a file but access it with mmap rather than filesystem logic. Finally, since real disk hardware reads and writes data in fixed-size blocks, you could arrange for your code to talk straight to the disk, entirely bypassing filesystems. I’ve never seen this done myself, but have heard tales of major commercial databases doing so.

I wonder if anyone has ever done a serious survey study of how the most popular high-performance data repositories, including Relational, NoSQL, object stores, and messaging systems, actually persist the bytes on disk when they have to?

I have an opinion, based on intuition and having seen the non-public inside of several huge high-performance systems at previous employers that, yes, filesystem performance still matters. I’ve no way to prove or even publicly support that intuition. But my bet is that benchmarks like Bonnie are still relevant.

I bet a few of the kind of people who read this blog similarly have intuitions which, however, might be entirely different than mine. I’d like to hear them.

What’s a “disk”?

There is a wide range of hardware and software constructs which are accessed through filesystem semantics. They have wildly different performance envelopes. If I didn’t have so many other hobbies and projects, it’d be fun to run Bonnie on a sample of EC2 instance types with files on various EBS and EFS and so on configurations.

For the vast majority of CPU/storage operations in the cloud, there’s at least one network hop involved. Out there in the real world, there is still really a lot of NFS in production. None of these things are much like that little SSD slab in my laptop. Hmmm.

Today’s benchmarks

I researched whether some great-great-grandchild of Bonnie was the new hotness in filesystem benchmarking, adopting the methodology of typing “filesystem benchmark” into Web search. The results were disappointing; it doesn’t seem like this is a thing people do a lot. Which would suggest that people don’t care about filesystem performance that much? Which I don’t believe. Puzzling.

Whenever there was a list of benchmarks you might look at, Bonnie and Bonnie++ were on that list. Looks to me like IOZone gets the most ink and is thus probably the “industry-leading” benchmark. But I didn’t really turn up any examples of quality research comparing benchmarks in terms of how useful the results are.

Those Bonnie numbers

The biggest problem in benchmarking filesystem I/O is that Linux tries really hard to avoid doing it, aggressively using any spare memory as a filesystem cache. This is why serving static Web traffic out of the filesystem often remains a good idea in 2024; your server will take care of caching the most heavily fetched data in RAM without you having to do cache management, which everyone knows is hard.

I have read of various cache-busting strategies and have never really been convinced that they’ll outsmart this aspect of Linux, which was written by people who are way smarter and know way more than I think I do. So Bonnie has always used a brute-force approach: Work on a test file which is much bigger than main memory, so Linux has to do at least some real I/O. Ideally you’d like it to be several times the memory size.

But this has a nasty downside. The computer I’m typing on has 32GB of memory, so I ran Bonnie with a 64G filesize (128G would have been better) and it took 35 minutes to finish. I really don’t see any way around this annoyance but I guess it’s not a fatal problem.

Oh, and those numbers: Some of them look remarkably big to me. But I’m an old guy with memories of how we had to move the bits back and forth individually back in the day, with electrically-grounded tweezers.

Reiser again

I can’t remember when this was, but some important organization was doing an evaluation of filesystems for inclusion in a big contract or standard or something, and so they benchmarked a bunch, including ReiserFS. Bonnie was one of the benchmarks.

Bonnie investigates the rate at which programs can seek around in a file by forking off three child processes that do a bunch of random seeks, read blocks, and occasionally dirty them and write them back. You can see how this could be stressful for filesystem code, and indeed, it occasionally made ReiserFS misbehave, which was noted by the organization doing the benchmarking.

Pretty soon I had email from Reiser claiming that what Bonnie was doing was actually violating the contract specified for the filesystem API in terms of concurrent write access. Maybe he was right? I can’t remember how the conversation went, but he annoyed me and in the end I don’t think I changed any code.

Here’s Bonnie

At one time Bonnie was on SourceForge, then Google Code, but I decided that if I were going to invest effort in writing this blog, it should be on GitHub too, so here it is. I even filed a couple of bugs against it.

I make no apologies for the rustic style of the code; it was another millennium and I was just a kid.

I cheerfully admit that I felt a warm glow checking in code originally authored 36 years ago.

The “simple” 38 step journey to getting an RFC

The Internet is built on the mutual understanding of network protocols and practices, and most of those protocols are defined using Request For Comments (RFC) or Best Common Practices (BCP) documents.

by Jeremy Kohler

ProPublica is a nonprofit newsroom that investigates abuses of power. Sign up for Dispatches, a newsletter that spotlights wrongdoing around the country, to receive our stories in your inbox every week.

One month after Missouri voters approved a constitutional amendment guaranteeing the right to abortion, Republican lawmakers in the deeply red state are already working to overturn it — or at least undermine it.

One measure would ask voters to amend the state constitution to define life as beginning at conception, declaring that embryos are people with rights to life, liberty and the pursuit of happiness.

The result would be to classify abortion as an unlawful killing.

Another proposal, aimed at repealing the abortion rights amendment, would ask voters to ban gender transition procedures for minors, tying the two issues together, despite the fact that the amendment did not address gender surgery and gender-affirming care for transgender children is already illegal in Missouri.

Other proposed amendments include stricter abortion limits, such as restricting access to cases of rape, incest, medical emergencies and fetal anomalies. These measures would impose additional requirements, such as mandating that rape survivors file police reports to obtain an abortion.

GOP lawmakers have also introduced a measure to raise the threshold for amending the state constitution through voter initiatives, which could make it harder to pass similar measures in the future.

The legislative moves follow the Nov. 5 election, in which the amendment to put abortion rights in the state constitution won by a 51.6%-48.4% margin. Starting Thursday, the right to abortion will be constitutionally guaranteed up to the point of fetal viability, while restrictions on post-viability abortions will remain in place.

In other states where voters approved abortion rights measures last month, there were no signs yet that lawmakers would also try to counter those measures.

Even before votes in Missouri had been counted, proponents of Amendment 3, as the measure was called, had anticipated that a victory would be met with efforts to somehow undercut abortion rights.

“These people will continue to rail against abortion,” said state Rep. Deb Lavender, a Democrat from the St. Louis suburbs.

Although Missouri already has a law recognizing life as beginning at conception, stating that unborn children have “protectable interests in life, health, and well-being,” the proposed constitutional amendment would go further. It would effectively elevate this principle to the state constitution and potentially complicate not only abortion rights but the legality of in vitro fertilization and the handling of embryos.

Several states have laws recognizing fetal personhood, but Missouri would be the second — after Alabama — to enshrine it in its constitution. That could create legal and ideological confusion or even conflicts, experts say.

“You could see voters saying, ‘I support a right to abortion,’ but also saying, ‘Life begins at conception,’ without understanding that you can’t have both of those things at the same time,” said Jamille Fields Allsbrook, a professor at St. Louis University School of Law and a former policy analyst for Planned Parenthood Federation of America.

The author of one of the personhood measures, Rep. Justin Sparks, a Republican from the St. Louis suburbs, said he was emboldened by the narrow margin of the abortion rights vote.

“A clear mandate has not been achieved,” he said. While the amendment had strong support in metro St. Louis and Kansas City and in the county that’s home to the University of Missouri, “the vast majority of the rest of the state voted in a different direction,” he added. “So I think it’s fair to again bring the question up.”

But state Sen. Tracy McCreery, a Democrat also from the St. Louis suburbs, noted that Sparks was going against the will of voters in the St. Louis area. “I find that even more disrespectful of the voters,” she said. “It wasn’t just voters that tend to vote Democratic that voted yes on Amendment 3. It was also Republican voters and independent voters, and I think that’s getting lost in this discussion.”

The measure to link abortion and transgender rights reflects the campaign before the election, when abortion opponents conflated these topics. Critics said this strategy seeks to distract from abortion rights, which had strong voter support, by capitalizing on voter discomfort with transgender issues.

While GOP lawmakers push these measures, the legal landscape around abortion in Missouri is already shifting. On Wednesday, a Jackson County Circuit Court heard arguments in a lawsuit brought by Planned Parenthood and the American Civil Liberties Union of Missouri that seeks to strike down Missouri’s near-total abortion ban and other laws that regulate abortion. The lawsuit followed the passage of Amendment 3. Planned Parenthood said if it wins in court it plans to resume abortion services in St. Louis, Kansas City and Columbia on Friday.

Missouri Attorney General Andrew Bailey has acknowledged that the amendment will legalize most abortions when it goes into effect, but he has said he intends to enforce remaining restrictions, such as a ban on abortions after fetal viability, a 72-hour waiting period and parental consent for minors.

Lawmakers are also pushing to raise the bar for passing constitutional amendments. Now, a simple majority is enough; that has allowed Missouri voters to bypass the legislature and pass progressive amendments that lawmakers oppose. A new bill would ask voters to pass a constitutional amendment requiring not just a statewide majority but also a majority of voters in five of the state’s eight congressional districts — a change critics argued would give disproportionate power to rural areas over urban voters. It would then be harder for voters to approve measures that don’t align with the priorities of the conservative politicians they tend to elect.

Earlier this year, a similar effort to make it harder to amend the constitution failed after Democrats in the Senate filibustered it.

Sparks criticized the Republican leadership in the General Assembly for allowing the failure, pointing to a Republican supermajority in both houses that could have passed the measure.

“We hold all the power,” Sparks said. “We hold all the procedural levers of power, and we can shut down debate in both houses any time, any day, for any bill we choose to.”

Florida shows how a higher threshold for voter initiatives might play out. In 2006, the state raised the bar for constitutional amendments to 60%. This year, a majority of voters — 57% — supported an abortion rights amendment, an even bigger margin than in Missouri, but not sufficient in Florida.

It’s not clear yet, though, whether any of the measures have enough support in Missouri’s General Assembly.

Lavender said that the campaign supporting abortion rights significantly outraised its opposition during the election. “It’s going to be difficult to overturn,” she said. “You’ll have the same money that supported it now going up against you.”

Willis explained, Trump "seems to genuinely like YMCA and he's having a lot of fun with it," and "I simply didn't have the heart to prevent his continued use of my song in the face of so many artists withdrawing his use of their material. [...]

Willis also acknowledged the financial benefit, noting that he stands to make "several million dollars" due to Trump's continued use of the song.

As for the idea that "YMCA is "somehow a gay anthem," Willis said that "is a false assumption based on the fact that my writing partner was gay, and some (not all) of Village People were gay, and that the first Village People album was totally about gay life."

"This assumption is also based on the fact that the YMCA was apparently being used as some sort of gay hangout and since one of the writers was gay and some of the Village People are gay, the song must be a message to gay people. To that I say once again, get your minds out of the gutter. It is not. [...]

And when I say, "hang out with all the boys" that is simply 1970s black slang for black guys hanging-out together for sports, gambling or whatever. There's nothing gay about that.

As always, The Onion got there first. "Why Do All These Homosexuals Keep Sucking My Cock?"

Previously, previously, previously, previously, previously, previously.