The “ISW mystery” deepens considerably (II)

[... continued from yesterday]

[Note added April 29: Through correspondence with some of the authors from what I label as the "French group" below I have learned that the density threshold was actually applied in their work as well. This makes things rather confusing as it means that their methods and the methods of "the DHB" are much more similar. However, they have also updated their paper to reflect new knowledge about the void catalogues and see a slightly more significant signal, similar to what Planck find (see note below). Everything is rather confusing right now. Again, once the dust has settled, I will write a post clearing everything up.]

[Note added March 21: Wow, sometimes science moves quickly. Today Planck released its data. They appear to confirm the anomalous spots in the original "Granett" (Hawaiian) result. They also appear to confirm the new anomalous result that was present in the paper that is now retracted (see the note below from March 19), albeit with a slightly reduced significance. It is unclear exactly what is going on, but it is clear that it is something interesting. I will keep you informed as things progress.]

[Noted added March 19: The paper described in the second half of this post (I called its authors the DHB) has been withdrawn from the journal it was submitted to (see the new abstract at this link: http://arxiv.org/abs/1301.6136). It is unclear whether the problems that the authors found in their analysis will affect their conclusions. However, I suggest you are cautious regarding how you interpret the conclusions I have drawn below based on this paper. I will keep you informed as/when things progress.]

A really neat figure from arXiv:1301.5849 showing the locations and sizes of the various catalogues of voids being examined. A larger redshift means the void is further away from us and one Megaparsec (Mpc) corresponds to three million light years. The purple "Granett et al." box is the original catalogue used by the Hawaiian group back in 2008. 

Isn't this just "a posteriori" statistics?

There is another possible explanation for the mystery. The probability of ZOBOV picking out these lines of sight at random is exceedingly small (less than 0.003), but it isn't zero. Might this have just been a crazy fluke?

Suppose 100 different groups of physicists look for unexpected, but interesting, signals in cosmological data. Then, even if each group is very careful you still expect one of them to find something that would seem to them to be unlikely. Unfortunately, they would be the only ones to publish their results. So we wouldn't see one “detection” paper and ninety-nine papers consistent with no detection. We would just see the one “detection” paper.

The best way to determine whether this is what happened is to look for the signal in other surveys.  If the original measurement was a fluke, it won't show up anywhere else. But, if it does show up again, then the chances that it was a fluke will significantly diminish.

The Friday before last a paper appeared that did exactly this. A French group took two catalogues of voids (so no over-densities), which have been produced by applying ZOBOV to a new catalogue of galaxies (these ones are closer to us). The French group then did more or less the same thing as the Hawaiians did. They examined images of the CMB along the lines of sight of these voids, averaged the temperature in all the images and checked whether the resulting signal could have happened at random.

They found no significant result.

This was quite sobering to read on the day. The paper did verify the significance of the original measurement, but not finding it in the new catalogues was highly suggestive that the story I painted above of a sort of community wide “look elsewhere effect” was true.

Hold on though!

Things at this date in time did look bad for the anomaly, but there was one important piece missing from the French group's analysis. The Hawaiians only used the most extreme over and under-dense regions in their analysis. ZOBOV found many more than 50 regions for them and if they had used all of them, they also wouldn't have obtained a statistically significant signal. This was always a crucial part of their analysis because we already knew from other observations that the observed ISW effect from most of the universe is as small as the predicted signal.

What would the French group have seen if they had only examined the most extreme voids?

A new observation

Apparently it is a rule of thumb for observers, that the more interesting your observation is, the more boring you are meant to make your title. These guys probably deserve a promotion. The paper is here.

Three days later (last Monday) a mixture of physicists from Durham, Hawaii and Baltimore (the DHB) released a paper. It answered the question posed above. For anybody interested in finding new physics, the answer is very exciting.

In their paper, the DHB repeated almost exactly the same analysis as the French group. Except, they made the crucial additional step of removing all the voids that weren't below a certain density threshold.

Figure C. This is what ZOBOV should be seeing. On the left is the average ISW temperature shift around simulated voids selected by ZOBOV. Note the clear cold spot, surrounded by a hot ring. On the right is the average radial profile for both the density contrast of the selected voids and the ISW temperature shift. Taken from arXiv:1301.6136.

The DHB also did something else important. They applied ZOBOV to simulations of the universe with accompanying simulated ISW maps. Therefore, they were also able to finally obtain accurate predictions for the expected signal, rather than just conservative over-estimates. This verified that the conservative over-estimates are indeed very conservative as the accurate expected signal is much smaller. But also, the DHB were able to use these simulations to determine the angular size of CMB patch that should give an optimal ISW signal for a given void. This corresponded to about three fifths of the radius of each void.

On to the exciting bit.

How did they analyse the data? Firstly, they took one of the same catalogues of voids used by the French group. Then they kept only the voids that have a number density less than 0.2 of the average number density of galaxies in the survey. Then, they isolated patches of the CMB along the lines of sight of the remaining voids. Finally, they rescaled each patch of the CMB proportionally to the radius of the void that selected it. A stacked image of all of these patches can be seen in figure D. When compared to the same image obtained from simulations (figure C) it actually looks quite similar (don't forget this one also has statistical noise in it), except look at the scales. It might look the same, but its magnitude is significantly larger.

Figure D. One of the more exciting images on this blog. The average temperature in CMB patches centred around voids found through observation. Even by eye, a cold spot can be made out with a surrounding hot ring. Just like from simulations, but note the different scale! The right panel is (close to) the data equivalent to the red curve in figure C. The profile is quite similar, but the magnitude definitely is not. Taken from arXiv:1301.6136.

The real test is the probability that this spot could have arisen randomly. To determine this, the group first rescaled each CMB patch to three fifths of the radius of the void that selected it (i.e. the theoretical optimal radius). They then sampled patches of the CMB along the same number of randomly chosen lines of sight and rescaled these by the same factors. They repeated this random sampling many times. Finally, they calculated the average temperature in each set of rescaled patches. This allowed them to compare the patches with voids behind them to the randomly selected patches. The result is in figure E. The top, coloured, curves are the observation (as a function of number of voids). The dashed curve shows the observational uncertainty, determined from the randomly selected patches. And the curve right at the bottom is the theoretical expectation for an ISW effect, obtained from their simulation. The lower panel of the figure shows the total “signal to noise” of the observation. You can see that the observed signal is consistently three standard deviations larger than the random fluctuations.

Figure E. Possibly the most exciting figure on this blog. In the top panel, the three coloured curves are the observation, the dotted line is the observational uncertainty and the dashed line is the theoretical prediction. The different colours correspond to different frequency bands. The fact that all three are almost identical suggests this is unlikely to be the result of foregrounds. The bottom panel shows the ratio of the signal to noise, which is consistently \(\sim 3 \sigma\). Taken from arXiv:1301.6136.

The probability of one “\(3\sigma\)” fluctuation is less than 0.003. However, this measurement is not the only measurement of this type. It is, in fact, a verification, from a different part of the universe, of the previous observation made by the Hawaiian group. That previous measurement was even less likely than this one (i.e. a “\(4\sigma\)” fluctuation). The chances of this being a fluke have greatly diminished. In fact, if you want to combine the significances, this is perilously close to, if not beyond, the magical (and completely arbitrary) “\(5\sigma\)” mark required by some to declare a discovery of new physics.

I don't know about you, but I find this incredibly exciting. Many anomalous measurements show up in science (especially cosmology). Most often it turns out that one of the following is true: we didn't understand our own models well enough; someone made a simple mistake; or it was a fluke and it goes away. As I've argued in this post, for this anomaly, all of these possibilities have now been rigorously checked. Firstly, the ISW effect cannot be this big. Secondly, the original measurement has been verified by a number of groups, so the Hawaiian group did not make a mistake. Finally, the signal has been found somewhere else, with strong significance, so the first measurement was not a fluke.

The one lingering concern is that this is a “fake CMB” being emitted by galaxies, or perhaps that something very, very subtle is being missed in the analysis (if you think you know what that might be, please leave a note in the comments). However, as I've argued above, there are certain properties a “fake CMB” would be likely to have and this signal does not seem to have them. In fact, it actually looks very, very similar to what we would expect from an ISW effect, except for the fact that it is just far too big.

So, if this is new physics, what does it mean?

That's a very good question and I don't know the answer. The signal is so much bigger than expectations that it is difficult to reconcile these observations with many well motivated “new physics” models. But we are trying to find out. The most important thing now is to explore this signal in as many ways as possible in as many regions of the universe as possible. If we can understand what makes it bigger; what makes it smaller; how it changes with void size and with void depth; etc, then we can get clues as to what is causing it (whether that is new physics or fake CMB).

Maybe we are on the verge of a new discovery. I'll keep you posted, it should be exciting times...

…...........................
Questions on any of the above are very much welcomed. If you don't understand anything, or think I/we've missed something obvious, please leave a comment. Maybe your question/observation will show the way to resolving the mystery.

Twitter: @just_shaun
................................

References:

Yan-Chuan Cai, Mark C. Neyrinck, Istvan Szapudi, Shaun Cole, & Carlos S. Frenk (2013). A Detection of the Cold Imprint of Voids on the Microwave Background Radiation arXiv arXiv: 1301.6136v1
S. Ilic, M. Langer, & M. Douspis (2013). On the detection of the integrated Sachs-Wolfe effect with stacked voids arXiv arXiv: 1301.5849v1