Software developer at a big library, cyclist, photographer, hiker, reader. Email:
17553 stories

Better JIT Support for Auto-Generated Python Code

1 Share

Performance Cliffs

A common bad property of many different JIT compilers is that of a "performance cliff": A seemingly reasonable code change, leading to massively reduced performance due to hitting some weird property of the JIT compiler that's not easy to understand for the programmer (e.g. here's a blog post about the fix of a performance cliff when running React on V8). Hitting a performance cliff as a programmer can be intensely frustrating and turn people off from using PyPy altogether. Recently we've been working on trying to remove some of PyPy's performance cliffs, and this post describes one such effort.

The problem showed up in an issue where somebody found the performance of their website using Tornado a lot worse than what various benchmarks suggested. It took some careful digging to figure out what caused the problem: The slow performance was caused by the huge functions that the Tornado templating engine creates. These functions lead the JIT to behave in unproductive ways. In this blog post I'll describe why the problem occurs and how we fixed it.


After quite a bit of debugging we narrowed down the problem to the following reproducer: If you render a big HTML template (example) using the Tornado templating engine, the template rendering is really not any faster than CPython. A small template doesn't show this behavior, and other parts of Tornado seem to perform well. So we looked into how the templating engine works, and it turns out that the templates are compiled into Python functions. This means that a big template can turn into a really enormous Python function (Python version of the example). For some reason really enormous Python functions aren't handled particularly well by the JIT, and in the next section I'll explain some the background that's necessary to understand why this happens.

Trace Limits and Inlining

To understand why the problem occurs, it's necessary to understand how PyPy's trace limit and inlining works. The tracing JIT has a maximum trace length built in, the reason for that is some limitation in the compact encoding of traces in the JIT. Another reason is that we don't want to generate arbitrary large chunks of machine code. Usually, when we hit the trace limit, it is due to inlining. While tracing, the JIT will inline many of the functions called from the outermost one. This is usually good and improves performance greatly, however, inlining can also lead to the trace being too long. If that happens, we will mark a called function as uninlinable. The next time we trace the outer function we won't inline it, leading to a shorter trace, which hopefully fits the trace limit.

Diagram illustrating the interaction of the trace limit and inlining

In the diagram above we trace a function f, which calls a function g, which is inlined into the trace. The trace ends up being too long, so the JIT disables inlining of g. The next time we try to trace f the trace will contain a call to g instead of inlining it. The trace ends up being not too long, so we can turn it into machine code when tracing finishes.

Now we know enough to understand what the problem with automatically generated code is: sometimes, the outermost function itself doesn't fit the trace limit, without any inlining going on at all. This is usually not the case for normal, hand-written Python functions. However, it can happen for automatically generated Python code, such as the code that the Tornado templating engine produces.

So, what happens when the JIT hits such a huge function? The function is traced until the trace is too long. Then the trace limits stops further tracing. Since nothing was inlined, we cannot make the trace shorter the next time by disabling inlining. Therefore, this happens again and again, the next time we trace the function we run into exactly the same problem. The net effect is that the function is even slowed down: we spend time tracing it, then stop tracing and throw the trace away. Therefore, that effort is never useful, so the resulting execution can be slower than not using the JIT at all!


To get out of the endless cycle of useless retracing we first had the idea of simply disabling all code generation for such huge functions, that produce too long traces even if there is no inlining at all. However, that lead to disappointing performance in the example Tornado program, because important parts of the code remain always interpreted.

Instead, our solution is now as follows: After we have hit the trace limit and no inlining has happened so far, we mark the outermost function as a source of huge traces. The next time we trace such a function, we do so in a special mode. In that mode, hitting the trace limit behaves differently: Instead of stopping the tracer and throwing away the trace produced so far, we will use the unfinished trace to produce machine code. This trace corresponds to the first part of the function, but stops at a basically arbitrary point in the middle of the function.

The question is what should happen when execution reaches the end of this unfinished trace. We want to be able to cover more of the function with machine code and therefore need to extend the trace from that point on. But we don't want to do that too eagerly to prevent lots and lots of machine code being generated. To achieve this behaviour we add a guard to the end of the unfinished trace, which will always fail. This has the right behaviour: a failing guard will transfer control to the interpreter, but if it fails often enough, we can patch it to jump to more machine code, that starts from this position. In that way, we can slowly explore the full gigantic function and add all those parts of the control flow graph that are actually commonly executed at runtime.

Diagram showing what happens in the new jit when tracing a huge function

In the diagram we are trying to trace a huge function f, which leads to hitting the trace limit. However, nothing was inlined into the trace, so disabling inlining won't ensure a successful trace attempt the next time. Instead, we mark f as "huge". This has the effect that when we trace it again and are about to hit the trace limit, we end the trace at an arbitrary point by inserting a guard that always fails.

Diagram showing what happens in the new jit when tracing a huge function until completion

If this guard failure is executed often enough, we might patch the guard and add a jump to a further part of the function f. This can continue potentially several times, until the trace really hits and end points (for example by closing the loop and jumping back to trace 1, or by returning from f).


Since this is a performance cliff that we didn't observe in any of our benchmarks ourselves, it's pointless to look at the effect that this improvement has on existing benchmarks – there shouldn't and indeed there isn't any.

Instead, we are going to look at a micro-benchmark that came out of the original bug report, one that simply renders a big artificial Tornado template 200 times. The code of the micro-benchmark can be found here.

All benchmarks were run 10 times in new processes. The means and standard deviations of the benchmark runs are:

Implementation Time taken (lower is better)
CPython 3.9.5 14.19 ± 0.35s
PyPy3 without JIT 59.48 ± 5.41s
PyPy3 JIT old 14.47 ± 0.35s
PyPy3 JIT new 4.89 ± 0.10s

What we can see is that while the old JIT is very helpful for this micro-benchmark, it only brings the performance up to CPython levels, not providing any extra benefit. The new JIT gives an almost 3x speedup.

Another interesting number we can look at is how often the JIT started a trace, and for how many traces we produced actual machine code:

Implementation Traces Started Traces sent to backend Time spent in JIT
PyPy3 JIT old 216 24 0.65s
PyPy3 JIT new 30 25 0.06s

Here we can clearly see the problem: The old JIT would try tracing the auto-generated templating code again and again, but would never actually produce any machine code, wasting lots of time in the process. The new JIT still traces a few times uselessly, but then eventually converges and stops emitting machine code for all the paths through the auto-generated Python code.

Related Work

Tim Felgentreff pointed me to the fact that Truffle also has a mechanism to slice huge methods into smaller compilation units (and I am sure other JITs have such mechanisms as well).


In this post we've described a performance cliff in PyPy's JIT, that of really big auto-generated functions which hit the trace limit without inlining, that we still want to generate machine code for. We achieve this by chunking up the trace into several smaller traces, which we compile piece by piece. This is not a super common thing to be happening – otherwise we would have run into and fixed it earlier – but it's still good to have a fix now.

The work described in this post tiny bit experimental still, but we will release it as part of the upcoming 3.8 beta release, to get some more experience with it. Please grab a 3.8 release candidate, try it out and let us know your observations, good and bad!

Read the whole story
Share this story

Ethiopia: From Nobel laureate to global pariah, how the world got Abiy Ahmed so wrong - CNN

1 Share
Read the whole story
Share this story

Child-care workers are quitting rapidly, a red flag for the economy - The Washington Post

1 Comment
Read the whole story
Share this story
1 public comment
23 hours ago
Imagine if we spent money like we valued families?
Washington, DC

Young climate activists channel anxiety about floods and wildfires - The Washington Post

Read the whole story
Share this story

jwz: Murderbots

I swear, 10% of the car traffic on SOMA streets these days is composed of single occupancy "self-driving" cars, plastered with their performatively-spinning greeblies and logos, testing this week's git pull of the new "let's see if we know how to not swerve into the bike lane yet" code on me without my consent.

I'm getting used to seeing the bored, dead-eyed stare of the hourly contractors sitting in these murder boxes, wasting fuel by driving in endless loops around my neighborhood, all day and all night long. It's disgusting.

Here's a clip from a video of some asshole "testing" his self-driving car by putting strangers into mortal danger. He starts his video by saying, "I just want to keep doing it for science, and see how it reacts, let's just roll." Fuck you entirely, you monstrously irresponsible piece of shit. After his murderbot almost mows down a crosswalk full of people, he says, "Not perfect! A big improvement, though."

I'm not linking to the original source because you shouldn't give this deadly troll his ad views. Don't reward the kind of person who never saw a Trolley Problem lever he didn't want to wildly yank back and forth.

Also apparently the Musk Defense Crew keep doing DMCA take-downs on Twitter of anyone who reposts it: "This media has been disabled in response to a report by the copyright owner."

Previously, previously, previously, previously, previously.

Tags: bike, computers, conspiracies, corporations, doomed, firstperson, mad science, mpegs, robots, security, sf, sprawl

Read the whole story
23 hours ago
Share this story

Paris is taking space back from cars. Here’s how.

1 Comment and 3 Shares

Over the past six years, Paris has done more than almost any city in the world to take space back from cars. Mayor Anne Hidalgo has opened linear parks in the old highways along the Seine, phased out diesel cars in the city, opened bus lanes, raised parking meter prices, and plowed bike lanes down hundreds of streets. When COVID hit, Paris eliminated cars from the Rue de Rivoli, its major crosstown thoroughfare. Plans are in the works to pedestrianize the Champs-Elysées and plant thousands of trees to green, clean, and cool the city.

As the adjunct mayor for transportation and public space, David Belliard is the point man for many of these endeavors. His latest projects include establishing car-free zones outside schools and enforcing the capital’s new speed limit of 30 kilometers per hour—a notch below 20 mph.

Earlier this month, I met him in his office to talk about Paris, COVID, and cars. Our conversation has been edited and condensed for clarity.

David Belliard: You want an overview?

Henry Grabar: Sure.

OK, quickly: At the start of the 20th century, in the ’20s, ’30s, the car asserts itself as a travel mode in urban centers, which are transformed. Paris is clearly an old city with many centuries of history with an urban fabric. Even though it was transformed by Haussmann in the 19th century, it has an extremely dense urban fabric with a lot of small streets and a configuration a priori not adapted to the auto. When the car arrives, we transform what we can call public space, and this public space becomes automobile space, with the logical system of the car imposing itself in Paris. And public space is completely devoured, eaten away, and in a certain way privatized to one single, unique use.

Very quickly we see the limits of “total car” in Paris, even in the ’60s and ’70s. We try to say, “How can we preserve this city?” Well, by putting cars underground. So we construct parking, even whole highways, under Paris. But there’s opposition to the highway on the Seine. There were protests. When we did the parking under Notre-Dame, there was a lot of opposition, because they were going to graze the crypt underneath.

They did that?

They didn’t, but the crypt was closed in a case of concrete to protect it from the parking that was installed alongside.

Starting in the ’90s, the negative externalities become more and more obvious, in terms of deaths and injuries on the road, danger for children and older people, and air pollution. The right-wing mayor started creating bike paths. But when I arrived in 2001, 2002, I bought a bike at once, and it was war. It was really difficult to ride in Paris, and I never felt secure. They built bike paths, and we called them “death paths.”

You mention that he was right wing to show it was not a political project?

It was a general movement. We elected officials on this proposal to give up less space to cars. Reserved bus lanes were an absolute scandal. Bike lanes really started in 2015. We’re in a situation where climate change is accelerating, which manifests in heat waves of 107, 109 degrees, and 122 degrees in 10 or 20 years. That means we won’t be able to live in Paris if we don’t do anything, because the city is too vulnerable right now.

Parisian public space is rare, precious, and very useful. It belongs to everyone and it can’t be captured by one unique usage, which is the automobile. Today, still, 50 percent of public space in Paris is consecrated to the car, whether it’s on the road or parked. That represents just 10 percent of trips.

I know it’s a movement with a long history, but how has COVID changed your approach? It’s given you the chance to do projects more quickly, perhaps, but at the same time, has it made you rethink the idea that Paris might always be the central business district for a million people every day?

First, COVID permitted us to accelerate certain things, especially with respect to the bicycle. We created a lot—in two years, since [Hidalgo’s reelection in June 2020], more bike lanes than in the whole preceding term. There’s a strong taste for biking we’ve seen rising for several years, but COVID was a kind of electroshock.

The question that’s posed with COVID, climate, social pressures, all that, is: What is a city? Can we still think of cities like we did a century ago? Can we still have big cities that capture so many economic and cultural resources, requiring hundreds of thousands of people to move from the periphery to the center to work, consume, etc.?

You don’t think there’s an element of social exclusion? Paris is a small part of the metropolis, it’s the most expensive place to live, so by limiting parking and the entry of private cars, you favor people who already live in Paris, while people who live in the banlieue might find access to Paris more limited.

Who lives in Paris? Nothing against you, but you presupposed that everyone in Paris is rich, which isn’t true. In Paris, like many urban centers, you have great social inequalities. More than 20 percent of the 19th arrondissement [a large, outer district on the city’s eastern edge] lives below the poverty line. Who uses their car today? Generally, and especially when you take out businesses, it’s the rich. One in three Parisians has a car. In the 20th, the level falls toward 15 percent. On the other hand, in the 16th, a neighborhood that’s much more bourgeois, it’s closer to one per household. It’s the same thing in the suburbs.

We always say “Madame Michoux, a nurse at the Saint-Antoine Hospital, who lives at the edge of Paris, when she works nights she must take her car, the poor thing, she’s got to park!” First of all, parking is free at night. Madame Michoux, I know her, my mom was Madame Michoux. She takes public transit. Eighty percent of trips between Paris and the suburbs are by public transit. And for the other 20 percent, a lot is tied to small businesses.

The redistribution of public space is a policy of social redistribution. Fifty percent of public space is occupied by private cars, which are used mostly by the richest, and mostly by men, because it’s mostly men who drive, and so in total, the richest men are using half the public space. So if we give the space to walking, biking, and public transit, you give back public space to the categories of people who today are deprived.

In the U.S., for projects that prioritize pedestrians, a lot of the opposition comes from small business. Have you tried to convince them that this is in their interest?

It’s counterintuitive. A lot of them believe they’re going to lose business because there will be less traffic. All the figures show the opposite. Every time we do experiments, we see that first, a big majority of their clients don’t use a car to come to their place, and then, people who use bikes and on foot consume more than people in cars. So really, what we’re trying to do is not convince, but argue by proving it, showing that it works.

When I listen to debates, when I see social media, I see right-wing elected officials talk about a Paris that doesn’t exist. I have the impression that everyone is living in the ’80s, with their big Jeeps, bringing their kids to school, big trucks delivering sofas. The city I see is parents using cargo bikes—that’s exploding—the post office which is making most of their fleet electric, a parking lot near Notre-Dame transformed into a warehouse and distribution center for groceries.

Anne Hidalgo is running for president. Is there a national element, something for people who don’t live in Paris, in her accomplishments here?

I’m from the country. I did my studies in high school in Vesoul—18,000 inhabitants, east of France, biggest employer was Peugeot. It’s a city marked by the car. Forty kilometers away you have a bigger city, Besançon. It’s a pretty city, I invite you to visit, it’s very cool.

These two cities were tied by a railroad, which disappeared in the ’30s. Replaced by what? By a road, which has been enlarged substantially, to four lanes. The mayor of Besançon is écolo; we were saying, “Can we keep doing this? Big highways through the countryside?” The question we ask in the urban centers about the place of the car can be asked about everywhere, and especially in rural places. Obviously you’re not going to do bike paths between Vesoul and Besançon, but frankly, can we not put the money we put into the route into a railroad line to create a public service? That’s a question that can be asked everywhere. France has an incredible rail network, in terms of density, and we closed dozens, hundreds of lines. In the presidential campaign, that should be a challenge—the reopening of those lines.

Are you suggesting there’s something in these anti-car politics that could be attractive even to someone who supports the gilets jaunes, the yellow-vest protesters who took over French cities two years ago?

The gilets jaunes is what? People saying, “We’ve had enough of gas prices going up.” Frankly they’re right. When I was in Vesoul, I was obliged to have a car, the car was not an object of emancipation but of servitude. I could do nothing without my car. So evidently, we are asking them to pay ever more for something they are required to use. The question of alternatives is the fundamental question.

Read the whole story
23 hours ago
Share this story
1 public comment
1 day ago
“Fifty percent of public space is occupied by private cars, which are used mostly by the richest, and mostly by men, because it’s mostly men who drive, and so in total, the richest men are using half the public space. So if we give the space to walking, biking, and public transit, you give back public space to the categories of people who today are deprived.”
Washington, DC
Next Page of Stories