The other thing to worry about is cross-layer shorts in the holes. The copper pads around the drill also act to keep etchant and other gunk away from the planes. But that's mostly for internal planes, not surface planes.