Wind? Definitely not. 1x4 pieces would not be able to stand straight up more than 2-3 high if this was the case, and I've definitely stacked them higher.
The force of the blocks? It shouldn't be. They seem to be lined up perfectly, then somewhere along the way they unalign. Unevenly. It's annoying.
Proof? Try stacking ONLY 2x2 blocks directly where they spawn. You can do 1x4 if they're put down longways (4x1), but I still recommend only 2x2 pieces. Somewhere it'll magically start to wobble. If there's no wind and the only force is a pseudo-gravity pulling the blocks down, something is amiss.