When rotating layers in Photoshop to exactly 90 or 270 degrees, the quality of the outcome is determined by the artwork size. If your layer is an even width and even height, you’ll be fine. If it’s an odd width and odd height, you’ll also be ok. But if it’s odd by even or even by odd, you’ll see something similar to the results below.
In this case, the artwork is 20×9 pixels—even by odd dimensions. The results are different for bitmap layers to vector layers, but they both produce unacceptable results because the origin of rotation doesn’t fall on an exact pixel boundary. This is similar to what happens when artwork is pasted from Illustrator.
We know that even by odd or odd by even dimensions are the enemy. We need a way to ensure the contents of the layer are an odd by odd or even by even size. Probably any method you can think of will solve this problem, be it adding square bitmap mask to a layer, or adding more content to the layer you’re rotating. You can also draw a square on another layer and rotate both at once.
As long as the dimensions for the layer or layers are even by even or odd by odd, it’ll be fine.
The method I use is a little quirkier. I wanted something that was actionable. Something that works when one or multiple layers are selected. The best way to do this seems to be grouping the selected layers, then creating a bitmap mask for the group. This forces the artwork to rotate around the origin of the group’s mask (assuming the mask is larger than the artwork).
It probably sounds a little more complex than it is in practice.
Here’s what I do: Select the layer or layers I want to rotate in the layers palette. Make a square selection with the marquee tool that’s larger than the artwork. Run the action to rotate 90 or 270 degrees.
If you’d like to use my actions, grab them from this article: Photoshop Actions & Workflows
Pratik Ringshia suggested changing the origin of rotation to the top left (or any other corner). This works brilliantly and is the simplest solution yet. I've edited the actions so they now use this method. Thanks Pratik.
Photoshop CS6 fixed the issue noted here, so I’ve deprecated the rotation actions. They’re still part of the package, but have been moved to Other/Bjango Actions (Old).atn.
Bitmap and vector masks are affected by this issue as well, so please take care. However, this advice is only for rotating layers, either via Free Transform or Transform under the Edit menu. Rotating your entire canvas via Image > Image Rotation is fine—the problem only affects rotating layers.