A caveat: the following questions are so obvious that they have surely been asked elsewhere, and possibly given much more convincing answers. References welcome!
The Sato-Tate conjecture implies that the normalized trace of Frobenius \(b_p \in [-2,2]\) for a non-CM elliptic curve is equidistributed with respect to the pushforward of the Haar measure of SU(2) under the trace map. This gives a perfectly good account of the behavior of the unnormalized \(a_p \in [-2 \sqrt{p},2 \sqrt{p}]\) over regions which have positive measure, namely, intervals of the form \([r \sqrt{p},s \sqrt{p}]\) for distinct multiples of \(\sqrt{p}.\)
If one tries to make global conjectures on a finer scale, however, one quickly runs into difficult conjectures of Lang-Trotter type. For example, given a non-CM elliptic curve E over \(\mathbf{Q},\) if you want to count the number of primes p < X such that \(a_p = 1\) (say), an extremely generous interpretation of Sato-Tate would suggest that probability that \(a_p = 1\) would be
\(\displaystyle{\frac{1}{4 \pi \sqrt{p}}},\)
and hence the number of such primes < X should be something like:
\(\displaystyle{\frac{X^{1/2}}{2 \pi \log(X)}},\)
except one also has to account for the fact that there are congruence obstructions/issues, so one should multiply this factor by a (possibly zero) constant depending one adelic image of the Galois representation. So maybe this does give something like Lang-Trotter.
But what happens at the other extreme end of the scale? Around the boundaries of the interval [-2,2], the Sato-Tate measure converges to zero with exponent one half. There is a trivial bound \(a_p \le t\) where \(t^2\) is the largest square less than 4p. How often does one have an equality \(a^2_p = t^2?\) Again, being very rough and ready, the generous conjecture would suggest that this happens with probability very roughly equal to
\(\displaystyle{\frac{1}{6 \pi p^{3/4}}},\)
and hence the number of such primes < X should be something like:
\(\displaystyle{\frac{2 X^{1/4}}{3 \pi \log(X)}}.\)
Is it at all reasonable to expect \(X^{1/4 \pm \epsilon}\) primes of this form? If one takes the elliptic curve \(X_0(11),\) one finds \(a^2_p\) to be as big as possible for the following primes:
\(a_{2} = -2 \ge -2 \sqrt{2} = -2.828\ldots,\)
\(a_{239} = -30 > -2 \sqrt{239} = -30.919\ldots,\)
\(a_{6127019} = 4950 \le 2 \sqrt{p} = 4950.563\ldots,\)
but no more from the first 500,000 primes. That's not completely out of line for the formula above!
Possibly a more sensible thing to do is to simply ignore the Sato-Tate measure completely, and model \(E/\mathbf{F}_p\) by simply choosing a randomly chosen elliptic curve over \(\mathbf{F}_p.\) Now one can ask in this setting for the probability that \(a_p\) is as large as possible. Very roughly, the number of elliptic curves modulo \(p\) up to isomorphism is of order \(p,\) and the number with \(a_p = t\) is going to be approximately the class number of \(\mathbf{Q}(\sqrt{-D})\) where \(-D = t^2 – 4p;\) perhaps it is even exactly equal to the class number \(H(t^2 – 4p)\) for some appropriate definition of the class number. Now the behaviour of this quantity is going to depend on how close \(4p\) is to a square. If \(4p\) is very slightly — say \(O(1)\) — more than a square, then \(H(t^2 – 4p)\) is pretty much a constant, and the expected probability going to be around \(1\) in \(p.\) On the other hand, for a generic value of \(p,\) the smallest value of \(t^2 – 4p\) will have order \(p^{1/2},\) and then the class group will have approximate size \(p^{1/4 \pm \epsilon},\) and so one (more or less) ends up with a heuristic fairly close to the prediction above (at least in the sense of the main term being around \(X^{1/4 \pm \epsilon}).\)
But why stop there? Let's push things even closer to the boundary. How small can \(a^2_p – 4p\) get relative to \(p?\) For example, let us restrict to the set \(S(\eta)\) of prime numbers p such that
\(\displaystyle{S(\eta):= \left\{p \ \left| \ p \in (n^2,n^2 + n^{2 \eta}) \ \text{for some} \ n \in \mathbf{Z} \right.\right\}}.\)
For such primes, the relative probability that \(a_p = \lfloor \sqrt{4p} \rfloor = 2n\) is approximately \(n^{\eta}/p \sim n^{2 \eta – 1}.\) So the expected number of primes with this property will be infinite providing that
\(\displaystyle{\sum \frac{n^{3 \eta}}{n^2 \log(n)}}\)
is infinite, or, in other words, when \(\eta \ge 1/3.\) So this leads to the following guess (don't call it a conjecture!):
Guess: Let \(E/\mathbf{Q}\) be an elliptic curve without CM. Is
\(\displaystyle{\liminf \frac{\log(a^2_p – 4p)}{\log(p)} = \frac{1}{3}?}\)
Of course, one can go crazy with even more outrageous guesses, but let me stop here before saying anything more stupid.
This is a long long way from the Guess, but note that (at least) one can prove, for a non CM-curve E, that $latex \liminf |a^2_p -4p| = \infty$. If $latex a^2_p – 4p = – D,$ then the eigenvalue of Frobenius generates the order $latex \mathbf{Z}[\sqrt{-D}].$ But, for fixed $latex D,$ there are only finitely many j-invariants $latex j_D$ in $latex \overline{\mathbf{Q}}$ with CM by this order, and there cannot be a congruence $latex j_E \equiv j_D \mod p$ for infinitely many primes unless there is an equality, which would contradict the assumption that E has CM.
I think the same method can be used to give some explicit bound like $latex \lim\inf \frac{ a_p^2 -4p}{\log (p) \log \log \log(p)} >0$ by showing that the norm of $latex j_E – j_D$ is at most exponential in $latex D \log \log D$ by looking explicitly at how the j invariants are distributed on the modular curve.
Good point! (I didn’t check the precise bound, but I your main point is that one *can* give such explicit bounds).