Masking in Photoshop CS6

If you’re constructing icons or user interface elements with a decent amount of complexity, there’s a good chance you’re going to need some kind of masking. Here’s a rundown of what’s available, using only vector shapes.

If you need portions of a shape layer to be see-through, multiple paths with different boolean path operations can be used. In the example below, the larger circle is set to combine shapes, and the smaller circle is set to subtract front shape. The sequence you draw the shapes matters, because that’s the order the boolean path operations are applied.

You can reorder after drawing, using the path arrangement dropdown to the right of the path operations icon. The path operations and path arrangement drop down menus are in the options bar.

That’s a great way to create simple shapes that are a single colour. Using the same method isn’t particularly good for more complex elements that require more than one layer.

If you wanted to colour a portion of the circle red, creating two separate shape layers and using boolean operations to mask the red area separately would be a poor choice.

The top red shape is antialiased at the edge of the light grey circle, so edge pixels are a combination of partially transparent red pixels and partially transparent grey pixels. That’s a bad thing, giving the right edge a grey halo. This is because the two shapes are masked separately.

A better way to construct the same thing would be with a clipping mask. Clipping the red circle to the light grey circle means the red circle is rendered in its entirety, then composited with the light grey circle, then the light grey circle’s mask is applied. It means the red circle extends through the antialiased areas around the edge of the visible shape — the right edges on the shape are entirely red, with varying transparency, which is what we want.

To create a clipping mask, right click on the top layer’s name in the layers panel and choose Create Clipping Mask. Or, option-click on the line between layers.

You can clip as many layers as you’d like to another layer.

In Photoshop CS6, you can even clip layers to a group. Here’s the same result, with a very different layer set up.

Another advantage to clipping masks is how they work with layer styles. Layer styles are applied on top of clipped layers, so you can add effects that treat the clipped layers and clipping layer as one. It’s important to consider the implications of the rendering sequence though — using a colour fill layer style, gradient layer style or pattern layer style that’s completely opaque will mean you can’t see the clipped layers, because the layer style is covering it.

Using Photoshop CS6’s group layer styles means we can set up an identical element using a group with a mask and layer styles. Identical result, different layer setup.

This opens up some new opportunities. We can now use additional clipping masks for the layers contained within the group.

If you created the same thing using just shape layers and boolean path operations, it would be noticeably worse quality. Editing is also more cumbersome, because the paths used to mask the blue and red circles would have to be kept in sync with the light grey circle.

If you’re after a soft edge, shapes can have a feathered mask, as can shapes that are being used to clip other layers.

Groups can also have feathered masks. By nesting layers inside a group and using feathering for the group’s mask, a wide range of effects can be achieved without resorting to bitmap masks.

From there, it’s possible to nest everything inside another group, with yet another mask. This technique can be used for really complex results, just be mindful of the 10 group nesting limit.

Be a little careful when scaling documents that use mask feathering, because the feathering value isn’t scaled with the document. You’ll have to edit them all by hand after scaling, or run one of the handy Scale Mask Feather scripts. Fun.

Being entirely vector and generated effects-based, all the above techniques can be infinitely scaled without losing quality.