Friday, September 19, 2014

Comparing Planck's noise and dust to BICEP2

In case anyone reading this doesn't recall, back in March an experiment known as BICEP2 made a detection of something known as B-mode polarisation in the cosmic microwave background (CMB). This was big news, mostly because this B-mode polarisation signal would be a characteristic signal of primordial gravitational waves. The detection of the effects of primordial gravitational waves would itself be a wonderful discovery, but this potential discovery went even further in the wonderfulness because the likely origin of primordial gravitational waves would be a process known as inflation which is postulated to have occurred in the very, very early universe.

 The B-mode polarisation in the CMB as seen by BICEP2. Seen here for the first time in blog format without the arrows. Is it dust, or is it ripples in space-time? Don't let Occam's razor decide!

I said at the time, and would stand by this now, that if BICEP2 has detected the effects of primordial gravitational waves, then this would be the greatest discovery of the 21st century.

However, about a month after BICEP2's big announcement a large crack developed in the hope that they had detected the effects of primordial gravitational waves and obtained strong evidence for inflation. The problem is that light scattering of dust in the Milky Way Galaxy can also produce this B-mode polarisation signal. Of course BICEP2 knew this and had estimated the amplitude of such a signal and found it to be much too small to explain their signal. The crack was that it seemed they had potentially under-estimated this signal. Or, more precisely, it was unclear how big the signal actually is. It might be as big as the BICEP2 signal, or it might be smaller.

Either way, the situation a few months ago was that the argument BICEP2 made for why this dust signal should be small was no longer convincing and more evidence was needed to determine whether the signal was due to dust, or primordial stuff.

Planck

The best measurement of the dust signal comes from the Planck satellite. Planck doesn't measure the CMB with the same sensitivity as BICEP2, but it has measured the CMB over the whole sky and at many different frequencies. The fortunate situation is that the amplitude of a dust B-mode signal would increase at larger frequencies. Therefore, the hope is that, if this signal is due to dust, then Planck will be able to see it at the larger frequencies. In fact, it was estimates from unreleased Planck data that indicated that perhaps the dust signal is of the same amplitude as the BICEP2 signal.

The problem is that the expected amplitude of signal in Planck's larger frequency measurements, if the BICEP2 signal is dust, is right on the verge of Planck's sensitivity. Therefore, even though Planck can tell us something about the likelihood that this is or isn't dust, noise is still a big issue. In the long run we need to wait for BICEP2 level of sensitivity at multiple frequencies at which point it will be easy to tell dust from primordial stuff. In the medium run, Planck and BICEP2 are now, apparently, collaborating and will be looking, carefully, to see whether Planck's high frequency measurements look like BICEP2's low frequency measurement. If they do, that's bad news, because within BICEP2's field of vision Planck high frequency measurements are only sensitive to dust. If they don't look similar, this doesn't necessarily mean that BICEP2 haven't measured dust, because Planck could just be noise dominated. All of these tricky subtleties are being worked out and hopefully, before the end of 2014, some sort of quantitative (though perhaps still not conclusive) statement about the probability that BICEP2 has seen dust will arise.

In the meantime, in the "short run", cosmologists are going to be impatient and will try to extract as much information as they can from any available data. I like this attitude. I think it's a sign of a healthy curiosity and passion for knowledge. However, one should be careful about what confidence one places in any results obtained. The reason Planck and BICEP2 are taking a long time to say anything is not just because Planck is a large group and getting agreement takes many meetings, conference calls and emails. It is also because there are many effects that need taken into account and understanding each of them takes time. If one doesn't take that time, one might miss something.

With that set of caveats out of the way I'll discuss this interesting paper from a few days ago.

Digitising pdfs, the new way to do cosmology

 The B-mode polarisation in the CMB as seen by Planck. Are the features the same as in the BICEP2 plot? That's the crucial question. Well, that and whether this actually corresponds to the CMB as seen by Planck, given that it was digitised from a slide presentation in 2013. Still, not long now for the real thing (i.e. a paper from which to digitise images!)

Neither Planck, nor BICEP2 have released their B-mode polarisation data (i.e no file was released giving the B-mode polarisation signal associated with each line of sight analysed on the sky). Instead, they've released images of the signal on the sky, mostly in pdf format, with a colour bar indicating the signal.

The sneaky thing various groups have been doing, while waiting for actual data, is to digitise these images. That is, to use the colour scale in the image and convert this to a set of signal amplitudes at the various lines of sight being analysed. In fact, even BICEP2 did this, to a Planck image, in their first manuscript. Today another group has analysed a digitised version of Planck's maps, as well as BICEP2's map.

What this group did is conceptually similar to what Planck and BICEP2 are (so the rumours say) doing behind the scenes. That is, to essentially look at the two maps and measure how similar they are.

I won't go into any additional details regarding how the digitising was done. It is described in the paper. The main obstacles come from a bunch of arrows on the BICEP2 image that need to be removed and replaced with estimates of the signal, and from removing the small and large scale fluctuations from the Planck image (because BICEP2 did this to their image and one needs to compare like for like). This process is a little messy and we shouldn't forget that the Planck map being used is the same one from 2013 that has been used in the past and only ever appeared in a slide during a conference talk! However, without the data itself, it's the best people can do, so why not? It's better than nothing (or so I think).

What they saw

With these digitised images they performed a number of tests. The first test basically amounts to counting the numbers of hot spots in the image that pass a certain hotness threshold and subtracting the number of cold spots colder than the equivalent threshold (the "genus statistic"). One can compare this result as a function of the threshold to what is expected from Gaussian statistics. BICEP2 (or, at least, the digitised data from BICEP2's images) appears consistent with Gaussianity under this test. The Planck data does too. At least, this is true after removing the large scales and the small scales from the image. It is worth noting that without this removal, Planck's data seems highly non-Gaussian by this test, not surprising for dust.

 The dotted line is the expectation for something that has a Gaussian distribution. The solid line is the BICEP2 data. It seems BICEP2 passes this Gaussianity test. I wonder if this rules out any inflationary models that predict a freaky strong amount of tensor non-Gaussianity? Someone should write a paper!
They then compare the amplitude of the genus statistic for each experiment. Here they find that BICEP2's value is larger than Planck. The interpretation of this is that the fluctuations BICEP2 see are more prevalent on small scales and less prevalent on large scales, compared to Planck. This is actually what one would expect if Planck was seeing noise+dust (i.e. a more flat spectrum) and BICEP2 was seeing the effects of primordial gravitational waves (i.e., a spectrum that, over the considered scales, is growing larger towards smaller scales). However, as they point out, this isn't new. One can already see this from plots of the angular power spectrum in BICEP2's own paper (i.e. the fluctuations are larger on smaller scales). Also, in an earlier paper it was found that primordial gravitational waves are a marginally better fit to BICEP2 alone than dust is, if the amplitude of each is allowed to be free.

 The cross-correlation of the previous two images. The blue spots are where both images were blue, the red spots are where both were red, the green is where one image was blue and the other red. The white haze is where either plots was not particularly blue or red (i.e. no positive or negative correlation). I can definitely see more red/blue than green, I think. Is this enough correlation to explain all of BICEP2's measurement? Is this consistent with randomness?
The next test they did, that I'll discuss, is a cross-correlation of the two data sets. This essentially amounts to statistically examining whether Planck's data is showing positive and negative B-modes along the same line of sight as BICEP2's. A large cross-correlation would indicate that when Planck is positive, so is BICEP2 and when Planck negative, so is BICEP2. A value close to zero would indicate that there is no relation, when Planck is positive, BICEP2 is just as likely to be positive as negative. A negative value would be incredibly surprising and would indicate that when Planck is seeing a positive signal, BICEP2 is more often than not seeing negative (and vice versa).

They do see a small, positive, cross-correlation. Now, remember, that what Planck is seeing is likely some combination of dust and noise. Their noise couldn't possibly correlate with BICEP2's (completely different instruments at different locations). Therefore, if there is some correlation, it will be coming from the dust. This positive correlation therefore indicates that at least some of BICEP2's signal is probably coming from dust. The crucial question is how much?

The answer in the paper is probably not all. They estimate the amount of correlation between Planck and BICEP2 that would be needed to fully account for BICEP2's signal and it is more than what they observe. This, also, isn't really particularly new. In fact, BICEP2 did a similar analysis in their original submission, using the same conference talk Planck data, and came to a similar conclusion.

Anyway, after accounting for this correlation, and estimating the remaining signal in BICEP2, the obtained value for "$$r$$" (which essentially measures the amplitude of primordial gravitational waves) is $$0.1 \pm 0.04$$. This is not the "$$5\sigma$$" initially claimed by BICEP2 (i.e. $$r\simeq 0.2$$), but, if everything else that led to this value can be trusted, it is still non-zero evidence for primordial gravitational waves. Curiously, this smaller value for $$r$$ is actually much easier to align with Planck's temperature data and inflation (for example it would alleviate what I called a second "cosmological coincidence problem").

Where now?

Now, we continue waiting. This paper hasn't really said anything that hasn't already been said or wasn't already known. It has just said and shown these known things in different ways. Any day now we are to expect Planck's paper revealing the non-conference-talk maps of the high frequency polarisation signal along BICEP2's line of sight. These will just be images though, not raw data. The word on the street/corridor is that a fully written draft exists and has clearance to be submitted and nobody I've spoken to knows why it hasn't been. The sort of phrases I've heard about what to expect from this is that "it will clarify a lot of things", but "it won't be conclusive". The safe bet is that it will show that Planck has seen some dust along this line of sight and some noise and that some of BICEP2's signal is almost certainly dust, but that, for now, precisely how much isn't certain.

When mentioning things like $$r=0.1 \pm 0.04$$ to Planck people in the past they've essentially shrugged their shoulders and said something like "yeah, that's probably possible"; however, one should keep in mind that a $$2.5\sigma$$ deviation of noise alone in BICEP2 would "probably be fine", so that doesn't really say much.

What we really crave is a cross-correlation analysis, similar in spirit to the one in the paper discussed above, but using the actual data. With the data not being public, only BICEP2 and Planck can do this, and they are. Results from this are expected "before the end of the year" (though which year is unclear).

What we really, really crave is more data, at more frequencies, with BICEP2 or better level of precision. This will also come in time.

1. I find it fascinating how different are the publication conventions in different branches of science. It seems crazy to me that anyone would extrapolate data from a pdf figure and then re analyse them in their own paper. If you tried to do that in life sciences you would never get it published! If I needed data from another paper I would just contact the authors and they would almost certainly send them to me, if they weren't included in the original paper to begin with. Is this kind of protectiveness common in physics/cosmology or is there something unique to this situation that demands it? I wonder whether it has to do with how physicists seem to be more open with their results before publication than life scientists, and therefore can't be as open with the raw data as they don't yet have the safety net of formal publication? Nonetheless, particularly given how cooperative and open the physics community appears to an outsider, this looks very strange indeed!

On a separate note - I was wondering roughly how many major questions in cosmology (and physics in general) are in a similar boat to this BICEP2-Planck debate - i.e. you really just need better technology to get a definitive answer?

1. It's not that different a situation in physics really. Digitising somebody's pdf slide is really bad science and you will not get your result published anywhere. The BICEP2 result was of course published in a prestigious journal, but clearly they got slapped down a bit by the referees and the version that finally got published was much toned down from the pre-print version we saw in March.

Anyone who is trying to publish a result entirely based on digitised slides without a BICEP2- level of ground-breaking science discovery to form the primary part of the paper is I think going to find it very hard to get a journal to accept it. The typical comment about this paper is "this kind of study is fine to satisfy your own curiosity, but why would you put it in a paper?" Which I mostly agree with (though Shaun's blog post about it is nice).

On the other hand, peer review in physics is sometimes of a pretty lax standard and all sorts of rubbish gets published, so maybe I'm wrong.

2. Also, in general I don't think it is true that physicists are more protective of their data*. The reason the Planck dust/polarization data have not been made public yet is because they is not ready yet - there's a whole lot of validation and verification work going on in the collaboration to actually produce data that is fit for science. Which makes nicking preliminary versions of that data even worse.

*Some major experiments do of course have a "first-use" embargo on the data until a certain date before it is made public, which is a precaution to ensure that they get the most bang for their rather considerable investment buck in acquiring it. But the data is always made public eventually. This isn't even that kind of a case.

3. For the record, I agree with Sesh regarding the protectiveness of physicists with their data. I think this is just a consequence of people using data before the experiment considered the data "ready", rather than because the experiment was hoarding the data. This wasn't even data taken from an unpublished, but ready for public dissemination pre-print; it was from a slide at a conference!

Regarding whether digitising someone's slide is good or bad science I kind of disagree with Sesh. People should be willing to use as much information as they can. The talk slide was additional information and digitising it was the best way for people outside Planck to extract data from it. And, in fact, it wasn't the digitisation procedure that BICEP2 got wrong, it was knowledge about what was actually in the slide! They digitised it fine.

I think the boundary for whether something should be publishable or not (though publishing seems to me personally to be an outdated 20th century or earlier method of doing science) I would just want it to add considerable insight. If BICEP2 hadn't digitised this slide, the best dust models would have predicted dust that wasn't of the right magnitude to mimic BICEP2's signal. So BICEP2's result would have looked just as strong. If another group had then come along, having been the first to digitise this slide and had correctly interpreted what it was showing and had thus said "hey hang on, maybe dust can explain this signal" I would say that is definitely "publishable", and would have a significant impact.

All results have caveats and approximations. those obtained from digitised slides would have their own, but so do even purely theoretical results. Over time the caveats are ironed out, the same has and will happen with these digitised images.

2. "Planck intermediate results. XXX. The angular power spectrum of polarized dust emission at intermediate and high Galactic latitudes", "(Submitted on 19 Sep 2014)", "submitted to A&A":

"... Extrapolation of the Planck 353GHz data to 150GHz gives a dust power ℓ(ℓ+1)CBBℓ/(2π) of 1.32×10−2μK2CMB over the 40<ℓ<120 range; the statistical uncertainty is ±0.29 and there is an additional uncertainty (+0.28,-0.24) from the extrapolation, both in the same units. This is the same magnitude as reported by BICEP2 over this ℓ range, which highlights the need for assessment of the polarized dust signal. The present uncertainties will be reduced through an ongoing, joint analysis of the Planck and BICEP2 data sets."

[ http://arxiv-web3.library.cornell.edu/abs/1409.5738?context=astro-ph ]

1. Heh, thanks Torbjörn. I unfortunately had my week of free evenings in the wrong week it seems, so I was able to blog about the non-mainstream paper and not the main Planck result.

Having said that, the Planck result didn't really say anything we didn't already know from the work in this paper by Flauger, Hill and Spergel, so I wasn't super motivated to write about it.

As we knew then, the dust does seem to have a comparable amplitude to BICEP2's signal. We now need to know whether it has the same shape, which is the next step.

3. Thanks for this update Shaun. I have to admit that it is a little complex for this lay person, however I did understand 60% of it 
Have to admit I was wondering why everything had gone so quiet on this so it is interesting to know it is being worked on in the background.

1. Thanks for the feedback. Sorry for the late reply from me (I guess you'll miss it). Work is definitely going on behind the scenes. You might have missed that on the day you wrote your comment there was a paper by Planck about the amplitude of polarisation from dust in the sky, and in particular in BICEP2's field of vision.

What's more, there are numerous B-mode polarisation experiments going on right now measuring this stuff, so it definitely hasn't gone away for good.

Either a detection will be made and confirmed, or the upper limits on the possible size of this signal will start to drop considerably over the next five or so years.