Reports for this article are below. If you have any questions or comments about these reports, please
Quality rating: E1E - The work contains errors and they can't be fixed without writing a completely new paper.
1 - Newsworthy. This could interest anyone in the world, not just researchers.
The discovery of two new kinds of radio transients, Fast Radio Bursts (FRBs) and Perytons, has electrified the field of radio astronomy over the past decade. FRBs and Perytons were discovered at roughly the same time in pulsar surveys with the Parkes telescope in Australia.
FRBs have all the characteristics of traditional fast radio transients (e.g., millisecond pulse widths consistent with scattering by astrophysical media, dispersion sweep with inverse frequency squared delay law), but are so dispersed that they can only originate far outside of our Galaxy; such a transient would be extremely valuable to probe the intergalactic medium and potentially as a new kind of cosmological probe. Perytons, by contrast, have irregular spectral structure, wider pulse widths, and appear to be extended on the sky (appearing in many beams of the Parkes multi-beam receiver). These features are more consistent with some terrestrial phenomenon, perhaps even man-made interference. Since Hippke et al was published on the arxiv, a physical model for the origin of Perytons has been proposed (Petroff et al., 2015), but that is beyond the scope of this review.
The differences in observed properties between these two classes of event suggested they were not related to each other. However, since these events were observed with large, single-dish telescopes, it has been difficult to localize them on the sky to identify multi-wavelength counterparts. Rapid follow-up observing campaigns have also failed to find optical and X-ray counterparts that would be expected for many classes of progenitor (e.g., supernovae or gamma-ray bursts). For these reasons, FRBs have been under a cloud of suspicion, despite their huge potential.
Since that time, the Arecibo observatory has found another likely FRB and the Bleien observatory has found a few more Perytons. This has shown that both events are global phenomena, but has not entirely settled the question as to their origin.
Hippke et al (2015) comes to the FRB and Peryton field with a more phenomenological approach by asking what can we learn by studying their observed properties. This is useful in the sense that one could imagine tying the phenomena together in order to find a common origin (as suggested by Kulkarni et al 2014, ApJ) or finding evidence for nonphysical patterns in the data.
I read Hippke et al with some interest, given that I am actively conducting my own program to search for FRBs. In fact, my first serious effort to find an FRB failed and in doing so, forced a downward revision in their published rate of occurrence. This has placed me as something of a skeptic in the field, though I am actively continuing the hunt. My initial reaction to Hippke et al was one of serious concern (see the Twitter thread at http://goo.gl/SpRiwy; the present report is based on version 2 of the Hippke et al on the arxiv).
However, upon closer analysis Hippke et al suffers from the problems associated with poorly-defined hypotheses. For weak or open hypotheses (e.g., "is there something strange in these data?"), robust statistical analysis is extremely difficult. This is a serious challenge to studies of efficacy in pharmaceuticals or psychological studies. John Ionnidis raised this point in a landmark publication titled "Why Most Published Research Findings Are False" (Ioannidis, 2005, PLoS Med). A central point is that for an arbitrarily large number of hypotheses to test, there will always be some that exceed some threshold of significance. Without an understanding of the class of hypothesis and its scope, a standard p-value significance is not meaningful. Physicists do not typically worry about this, as our hypotheses tend to be very specific (e.g., is there significant emission from that source?). Other fields, such as in medicine, have many tools to deal with the question of significance beyond simple p-value tests.
The hypothesis of Hippke et al falls into dangerous territory when asking "is there structure to the FRB DM distribution?". Is the question about periodicity or clustering? Would DM values coincident with a fundamental physical constant (e.g., pi or sqrt(2)) be considered significant? Are ratios of cluster DMs anomalous?
Independent of that point, I repeated the analysis presented in Hippke et al and failed to confirm the significance of the "fundamental DM". The analysis is presented in an IPython notebook at http://goo.gl/4Phqgy. By my estimate, the DM structure in question can be produced roughly 1-2% of the time by simulating random DM distributions with 10 objects (as with the sample FRBs discussed here). This p-value is still unusual, but far more common than the 0.05% chance quoted by Hippke et al; neither of these p-values are correctef for the broadness of the hypothesis, as discussed above. As noted by Hippke et al, future FRB DM values will ultimately show whether this is a significant clustering.
In the second revision of the paper, Hippke et al raise a more important question by asking whether FRBs and Perytons *share* a DM periodicity or structure. My analysis confirmed that both distributions have a similar "fundamental DM" according to the Rayleigh test statistic (or a periodogram, as used by Hippke et al.). However, this hypothesis is also poorly defined, since it isn't clear how Perytons should be counted. One paper published a large number of Perytons that all occurred within a few minutes of each other. They presumably are associated, but without a physical model or specific test, the significance of their coincidence with FRBs is unclear.
Finally, v2 of the paper added an analysis of the arrival times of FRBs. This perhaps suffers from the poor-hypothesis problem most severely. First, FRBs are dispersed pulses, so they have different times of arrival as a function of frequency. Which one to use? Typically astronomers calculate the true time at infinite frequency, which has no dispersion delay. Hippke et al calculate the arrival time at 1500 MHz without justification. Second, and most significantly, this draft of the paper claims that FRBs and Perytons are associated by the fact that the mean arrival time of each population is similar relative to the integer second. However, the mean of the modulus of a periodic value is always near the phase midpoint. Values such as the mean and standard deviation are not meaningful after calculating the modulus. Circular statistics (such as in the Rayleigh test) must be used. I confess that I have not attempted to reproduce the KS test as presented in Hippke et al.
The collection of these flaws raises serious concerns about Hippke et al.. While FRBs are a hot topic deserving of scrutiny, this paper only makes one modestly useful contribution by noting the coincidence of FRB and Peryton DM "periodicity". If this holds up, it could indicate that both classes are caused by man-made interference. However, the significance of that point is undercut by the rest of the paper, which presents a wide range of poorly justified hypotheses.
About the refereeReview written by Casey Law, UC Berkeley
Expertise: I am an expert in the subject of this paper
Familiarity with the preprint: I have read and understood the paper thoroughly
Disclosure of conflicts of interest:
The referee has not reported any conflicts of interest.
Date and time of submission: 2015-04-23 18:19:19