JPEG Decoding with the Best

Last week, we talked about how you can use the Core Animation instrument to find sources of stalls in your app's feed. That's all fine and good, but what if that instrument is giving your feed a clean bill of health, but you're still seeing stutters? If that's the case then you might have missed something when you were using Time Profiler and you might want to jump back in.

One big source of main thread work that can be easy to miss is JPEG decoding.

But I don't remember decoding any JPEGs...

The fact of the matter is, if your app shows any images downloaded from the web, they're probably JPEGs and you're already decoding and rendering them, even if you don't realize it.

The easiest way to decode and render a JPEG is:

That's right, any time you have a UIImage created from a JPEG, assigning it to be displayed by an image view means that an implicit CATransaction will be created and your image will be copied, decompressed, and rendered on the next display pass triggered by the run loop.

This all happens on the main thread. If your app isn't very image heavy, you can probably get away with this approach, but if you're dealing with a lot of images, especially in a feed, then you may find this decoding adding up to a significant amount of your app's main thread work.

If you don't believe me, here's the stack trace to prove it.

This stack trace is from a pretty straightforward sample app that renders some images and text in a feed.

The panel on the right shows the heaviest stack trace. If you look at the line I've hightlighted, you'll see that there's a function called applejpeg_decode_image_all that's being called. If you follow the stack trace up you'll also notice some copy functions.

The tricky part about this being a big bottleneck in your app is that literally none of the code in the stack is code you wrote directly. If you turn off "System Libraries" in the Call Tree options, this code won't show up at all.

Even if you aren't hiding system libraries, it can be easy to accidentally ignore Apple code in traces since, let's be honest, it can be a little intimidating to look at and it's easy to subconsciously try to find code we're already familiar with.

To counter this tendency, let's take a quick detour and go over what's happening in this stack trace at a high level.

Core Animation Overview

We've talked about how the UI is rendered via the Render Server before, but let's go into a little more detail.

Anything that happens to a CALayer (or UIView by extension) goes through a pipeline of 5 stages to get from the code you've written to showing something onscreen.

1. Under the hood, a CATransaction will be created with your layer hierarchy and committed to the Render Server via IPC.
2. On the Render Server, the CATransaction will be decoded and the layer hierarchy re-created.
3. Next, the Render Server will issue draw calls with Metal or OpenGL depending on the phone.
4. Then, the GPU will do the actual rendering once the necessary resources are available.
5. If the render work is done before the next vsync, the rendered buffer will be swapped into the frame buffer and shown to the user.

An important thing to take note of is the fact that number 1, aka the CATransaction phase, is the only phase that runs in-process. Everything else happens on the render server!

This means it's the only phase that you can see in your stack traces as well as the only one you can directly influence.

CATransaction Overview

The transaction phase itself is broken down further into 4 stages (I know, too many stages to remember right?). The first two should be pretty familiar to you already.

1. Layout: Do all the calculations necessary to figure out layer frames. -layoutSubviews overrides are called at this point.
2. Display: Do any necessary custom drawing via CoreGraphics. -drawRect: overrides are called and, interestingly, all string drawing happens here aka UILabels and UITextViews. 
3. Prepare: Additional Core Animation work happens here including image decoding when necessary.
4. Commit: Finally, the layer tree is packaged up and sent to the render server. This happens recursively and can be expensive for complex hierarchies.

Now that we have the high level overview of what Core Animation will do, let's take a look at that stack trace again.

Starting from the top, we see that main() kicks off the run loop by calling [UIApplication run]. Then, in the "do observers" phase of the run loop run we have a CATransaction being committed with the CA::Transaction::commit() function. I've highlighted the section that corresponds to this commit.

If you scan through, you'll see a function called prepare_commit() which we can reasonably assume is that "prepare" phase I mentioned a moment ago. Scan a little more and you'll see that this preparation consists of copying the image into a CGImageProvider and then decoding it.

If you take nothing else away from this, just remember that there's a good amount of work that is happening under the hood when you use a UIKit component and there ain't nothin in this world for free.

Removing This Bottleneck

So if you have found this to be a problem in your app, how the heck can you fix it? After all, you do have to decode jpegs to be able to show them, so it's not like you can just stop doing that.

Well you can't make UIImageView do its decoding more quickly, but if you're so inclined, you are able to preemptively render the jpeg yourself!

Using this function, you can pass in a regular UIImage backed by a JPEG and it will return a UIImage that's backed by an already decompressed and rendered version.

Couple of things to note here: 

First, there is, unfortunately, no API that's been exposed for you to tell whether a given UIImage has already been decoded or not. This means that if you're using this trick, you'll want to make sure to carefully cache and re-use these images when you can since they take up a lot more memory than their compressed counterparts.

Second, this particular method is technically "offscreen rendering" in that it's no longer hardware accelerated, but it's the ok-when-it's-useful kind of offscreen rendering, not the kind that makes your GPU stall.

This is where the idea of "perceived performance" otherwise known as "responsiveness" comes into play. This method of decoding can actually be slower than what the image view will do by default, but the win is that its under your control and you can jump to a background thread to do the decoding while keeping the main thread free from blockers.

Now you may see an empty imageView for a moment while scrolling, but you've traded a moment of "placeholder" state for not dropping frames if the decoding takes a while.

Conclusion

Hopefully this medium-depth dive into what's happening under the surface of the humble UIImageView has given you a better appreciation for what UIKit does for you and has also made you a little more comfortable with looking at an intimidating Time Profiler trace.

As far as other resources for looking at JPEG decoding go, this is a topic that's been explored many times in the iOS community.

For instance Texture (previously AsyncDisplayKit) is basically a responsiveness engine that does everything we talked about (measurement, layout, decoding, and drawing) off of main for you.

Fast Image Cache has a really interesting method of getting around this problem where it decompresses JPEGs up front and stores them on-disk in their decompressed form so that the only thing you need to do is retrieve them from disk when they're needed instead of doing all the decompression work on the fly.

Have you decided to use a method like this in your app? If so, or if you have any questions, let me know in those comments!

References:

Advanced Graphics and Animations for iOS Apps WWDC 2014

Blog RSS

Friends Don't Let Friends Render Offscreen

Have you ever been using an app and a certain feature just seemed to be overwhelmingly sluggish compared to the rest of the app? I'm not here to judge (I totally am), but a lot of times scroll view animations can really suffer in apps due to developers using techniques that are less than ideal.

If the app in question is your app, then I wouldn't be a good friend if I didn't point out that you have a problem.

How to Catch Your App in the Act

Sometimes you can bust out Time Profiler and do some profiling to figure out what exactly is going wrong, but a lot of times the problem won't be super obvious from your stack traces.

If this is the case, then it might be time to reach for the Core Animation Instrument. 

This instrument is nifty because, not only does it give you a (decent) real-time readout of your app's fps, but it also provides you with a set of render debug options that will show you when your code is indirectly causing problems for the GPU.

They can all be pretty useful, but one of the most handy options is the Color Offscreen-Rendered Yellow option.

One fun thing about these options is that you can turn them on in Instruments, and then switch to any app on your phone to see how it's doing.

In this example you can see that the native Twitter app is doing some pretty aggressive offscreen rendering while loading the stretchy profile header. The video is (unfortunately) a real-time capture of the animation on my iPhone 5. 

As you can probably guess, this is not an ideal experience for people using your app.

So What Exactly is Offscreen Rendering?

This concept is one of the more confusing ideas when considering performance on iOS, so I'll do my best to try to help clear up some of that confusion.

On the CPU: Offscreen rendering, or offscreen drawing can just mean what happens when you need to draw a bitmap in memory using software instead of rendering directly to the screen.

For example, writing your own draw method with Core Graphics means your rendering will technically be done in software (offscreen) as opposed to being hardware accelerated like it is when you use a normal CALayer. This is why manually rendering a UIImage with a CGContext is slower than just assigning the image to a UIImageView.

This type of offscreen rendering can be advantageous at times but should be undertaken only alongside careful testing and measurement.

On the GPU: There's a very specific type of offscreen rendering that can occur when you ask a CALayer to draw something, but haven't given it enough information.

This means that when the Render Server goes to render your layer hierarchy it will get to a layer subtree that it doesn't fully know how to render yet.

This forces it to stall and switch contexts from its normal "onscreen" rendering to "offscreen" rendering in order to fully figure out how to draw what it needs to. Once its done, it can switch back to onscreen drawing and proceed as normal.

The real damage is done by the two context switches! These can really add up if you have many views being animated that force this kind of GPU stall.

This is what the Core Animation Instrument is trying to show us when it colors layers yellow.

There's a few common scenarios where this will happen and pretty straightforward fixes for each.

Drawing (View Shaped) Shadows

When drawing shadows, it's really easy to use the CALayer shadow properties. You can technically draw a shadow by using just the offset, color, radius and opacity properties.

Unfortunately, if you do, you'll force the type of very-bad-for-your-app offscreen rendering we're trying to avoid because the GPU won't automatically know the shape of the shadow.

In this case, just make sure you always set the shadowPath property to tell the layer what shape your shadow will be. If it isn't possible to use this property, then don't use any of these properties at all.

But wait, there's more!

While that strategy isn't bad, one thing to keep in mind is that the drawn shadow is dynamic.

The vast majority of the time, you won't be animating a view's shape and expecting the shadow to animate along with it.

When this is the case, you can just have your designer do the hard work and provide you with a good shadow image.

This way, they can make the shadow look exactly the way they want, and you can apply it to your view by creating a stretchable version of the image in code and then using a UIImageView that you can place directly behind the view you want to give a shadow to. 

Make sure to use the .stretch option as the resizing mode.

This will take whatever image you were given, and define how far in the non-stretched edges go.

The middle will be stretched, but this shouldn't matter since you only care about the edges sticking out from under your view.

Ideally, the image you use will be as small as possible to reduce the amount of computation necessary to load and render the image.

As far as text goes, both of these methods can be used to draw generic blob shaped shadows, which is fine as long as your designer approves.

Drawing (Text Shaped) Shadows

Alternatively, if you're trying to draw drop shadows behind text, then you've got a little more work to do.

The most straightforward option is to opt for using an NSShadow along with an NSAttributedString.

Drawing Round Corners

This can be another big source of offscreen rendering in your feed. Luckily, you can once again avoid this offscreen render pass by using a UIBezierPath to round your corners.

To do so, you can just write a UIImage extension to get pre-drawn round images like so:

Like I mentioned earlier, this is technically "offscreen" in some sense, but there shouldn't be a back and forth with GPU stalls to worry about, thus the corner rounding becomes inconsequential.

Using the .shouldRasterize Property

I mentioned this in a earlier post, but the tl;dr is that you need to be really sure it will help your app.

Odds are it won't and instead it will introduce an offscreen render pass on every frame of any animation involving that layer.

Conclusion

At the end of the day, performance is a critical aspect of how enjoyable your app is and with a little extra thought you can make sure your app isn't slowing down unnecessarily.

Have you found any other nifty ways of avoiding offscreen rendering? Let me know in the comments!

Blog RSS