Aside from the performance disadvantage, is there ever a reason to not choose a larger neighborhood search window?

It seems as though smaller search windows would lead to irregularities that sometimes might be more interesting to use as a texture.