When using really large stroke sizes, layer style strokes often end up with stepped curves. This isn’t the case for vector strokes, which are perfectly formed. I assume this is because vector strokes are created from the original vector shape data, whereas layer style strokes are built from the bitmap mask for the shape. Wildly different techniques, when you think about it.
Does this matter for smaller strokes sizes? Maybe not. The difference is subtle.
However, vector strokes can be non-integer sizes, so you can use a 0.5 pixel stroke in a non-Retina document and it’ll scale up to precisely 1 pixel for Retina sizes. Layer style strokes have to be integer sizes, so scaling a document can cause rounding errors.
When a shape is blurred using mask feathering, vector strokes are also blurred. Layer style strokes are not. That may be a positive or negative, depending what you’re trying to achieve.
Due to their construction, layer style strokes are always rounded. Vector strokes can be square, rounded or beveled. I can’t see myself using the beveled kind very often, but square corners are definitely useful.
Layer style strokes max out at 250 pixels. Vector strokes can be slightly larger, with a maximum size of 288 pixels. You can even cheat to achieve larger vector strokes by applying a stroke then scaling the document up (tested to 1600px before giving up).
As well as various path cap options, vector strokes also respect open paths. Layer style strokes do not.
Vector strokes can be dashed. Dashed vector strokes can be used in combination with mask feathering. And they can also be filled with gradients. And the gradients can include transparent or semi-transparent colours. Things get pretty crazy. Dr Seuss would be proud.
Look, sir. Look, sir. Mr Nack, sir.
Let’s do tricks with strokes and dots, sir.
Let’s do tricks with masks and caps, sir.
And, just when you thought you were going to use vector strokes for absolutely everything, layer style strokes prove they’re no slouch — say hello to the rather well hidden shape burst gradient style. Shape burst is only available as a layer style stroke gradient.
Shape burst gradients follow the path’s shape, creating effects that would be difficult otherwise.
A small update after some discussion with Matt Kelsh and Edward Sanchez: Vector strokes dont’t have separate opacity control. This means you can’t set a stroke to be more transparent than a fill, unless you use a gradient fill for the stroke and set the two ends of the gradient as the same colour with the desired opacity. Possible, but a bit messy.
Another minor win for layer style strokes.