Nb: the post below was my first attempt at understanding this - there is a more streamlined version here:
I've borrowed the image below from an interesting website that connects the Ulam Spiral with the Gann Square (the latter being a voodoo technique used by chartist stock gamblers). It has a pretty good account of how laying numbers out in this way results in the diagonals describing parabolas. The grey areas are primes, and the thing that fascinated Ulam is the way that these tend to fall on particular diagonals.
I just want to note that this can also be seen in the light of composite odd numbers always being expressable as the difference between two squares. (something I've looked at in previous posts).
It is fairly obvious that the square numbers fall on two diagonals of this chart, running from bottom left to top right. The equations for the diagonals are given in the linked article as quadratic polynomials/parabolas. Take the diagonal to the North East corner, which contains a high concentration of primes. The numbers are 5, 17, 37, 65, 101, 145, 197, 257, 325, 401.
If we look at how these can be expressed in terms of the nearest square numbers above them, we can rewrite the list in terms of N^2 - X thus:
3^2 - 4
5^2 - 8
7^2 - 12
11^2 - 20
13^2 - 24
15^2 - 28
17^2 - 32
19^2 - 36
21^2 - 40
(Edit - I should also note that this can be expressed as (2N+1)^2 - 4N)
Of these, there are three cases where that can be rewritten as N^2 - M^2 (all composite except for 3^2 - 4, where the difference between N and M is 1). Of the others, all are prime except for 145 (which can be expressed as 17^2 - 12^2).
Another diagonal is 31, 59, 95, 139, 191, 251.
These can be expressed as:
6^2 - 5
8^2 - 5
10^2 - 5
12^2 - 5
14^2 - 5
16^2 - 5
Different diagonals can be expressed in similar ways as N^2 - X, where X is either a constant value or increasing by a constant value at each stage.
The point about this is not that any diagonal is going to be guaranteed to contain a high number of primes. But odd numbers that can't be written as N^2 - M^2 are always prime, and it's instinctively fairly obvious why some sequences will contain a higher proportion of such numbers than others.
The more important point is that there are diagonal squences which will never contain primes. Obviously the even diagonals contain no primes (other than 2). And there are odd diagonals that will also be prime-free.
For instance, look at the sequence 21, 45, 77, 117, 165, 285, 357, 437. This can be rewritten as N^2 - 4 for all odd numbers from 5 upwards. So this sequence will contain zero primes as it continues upward.
For me this is a first step to understanding why the Ulam spiral produces those diagonal patterns of primes and composites. As the pattern expands there are more and more odd diagonals which cannot contain primes. The other odd diagonals must contain all the primes therefore the pattern focuses on these spirals. Of course there is a deeper complexity to the pattern but some of this can also be explained using the difference between two squares.
For instance look at the pattern drawn by N^2 - 16. 9 and 33 are on a straight line, but then 65, 105, 153, 209, 273 run down an oblique diagonal (then the pattern jags back to the left down a simple diagonal). As we deal with larger squares, these kinds of oblique diagonal runs of composites will become longer.
Finally there are straight lines like 3, 33, 95, 189, 315, 473 (odd numbers only) which can be expressed thus
2^2 - 1^2
7^2 - 4^2
12^2 - 7^2
17^2 - 10^2
22^2 - 13^2
27^2 - 16^2
There is an obvious mathematical sequence here, N^2 - M^2 with N increasing by 5 and 3 respectively each time.
People have often looked at the Ulam spiral in terms of the formulas that generate a high proportion of primes. I think it maybe makes more sense to look at how composite numbers will cluster in particular odd diagonals, thus leaving the other diagonals to be more densely populated with primes.
I'll elaborate slightly on this in the next post...