Data Formats & the Ecosystem¶

Radio data passes through a chain of formats on its way from antenna to result: raw voltages → channelised spectra → calibrated visibilities → images. Knowing the formats — and the tools that read them — is half of practical radio astronomy. This page is the reference behind Chapter 16 and the jansky.formats helper module.

flowchart LR
  V["Raw voltages<br/>GUPPI raw · SigMF"] --> C["Channelised<br/>filterbank · HDF5/UVH5"]
  C --> S["Spectra<br/>.sps · .spd"]
  C --> Vis["Visibilities<br/>UVFITS · Measurement Set"]
  C --> P["Pulsars<br/>PSRFITS"]
  Vis --> Img["Images<br/>FITS"]

The format landscape¶

Format	What it holds	Produced by	Read with	Spec / reference
GUPPI raw (`.raw`)	Raw complex voltages, 8-bit, `[chan][time][pol]`	Green Bank / Breakthrough Listen backends	`jansky.formats`, `blimpy`	Estévez: writing GUPPI with GNU Radio
SIGPROC filterbank (`.fil`) / HDF5	Channelised power (waterfall)	`rawspec`, pulsar/FRB backends	`blimpy`	SIGPROC / Breakthrough Listen
PSRFITS	Folded/search-mode pulsar data	pulsar backends	`astropy.io.fits`, `your`, PSRCHIVE	PSRFITS standard (see Ch 13)
Measurement Set (MS)	Calibrated interferometer visibilities	CASA	CASA, `pyuvdata`	casacore (see Ch 12)
UVFITS / UVH5	Visibilities (interchange)	correlators, `pyuvdata`	`pyuvdata`	RASG / pyuvdata
SigMF (`.sigmf-meta` + `.sigmf-data`)	SDR recordings: JSON metadata + raw samples	any SDR	`jansky.formats`, `sigmf`	sigmf.org
SPS (`.sps`)	Radio-Sky Spectrograph spectra	Radio-Sky Spectrograph / Radio JOVE	`jansky.formats.read_sps`	Radio JOVE · data: radiojove.net
SPD (`.spd`)	Radio-SkyPipe strip-chart	Radio-SkyPipe	`jansky.formats.read_spd`	radiosky.com

Where to find SkyPipe / SPS data

Actual .spd/.sps observations live in the Radio JOVE Data Archive (radiojove.net) and the MASER/VESPA collection at Paris Observatory — see the Amateur & Radio JOVE / SkyPipe data table in Resources.

Using `jansky.formats`¶

The helper module implements the formats we can follow to a public specification, and is honest about the ones we cannot yet verify byte-for-byte.

GUPPI raw¶

from jansky import formats
import numpy as np

volts = (np.random.randint(-20, 20, (4, 16, 2))        # (nchan, ntime, npol)
         + 1j*np.random.randint(-20, 20, (4, 16, 2)))
formats.write_guppi("demo.0000.raw", volts, header={"OBSFREQ": 1420.0, "TBIN": 1e-6})

header = formats.read_guppi_header("demo.0000.raw")     # parse the ASCII cards
for hdr, block in formats.iter_guppi_blocks("demo.0000.raw"):
    ...                                                 # block: complex (nchan, ntime, npol)

The header is a stack of 80-byte ASCII cards (KEYWORD = value, FITS-like) ending in END, followed by BLOCSIZE bytes of interleaved int8 I/Q ordered [channel][time][polarisation]. Run rawspec to reduce raw → filterbank, then turboSETI (the seti extra) for a Doppler-drift search — the Breakthrough Listen pipeline.

SigMF (portable SDR recordings)¶

formats.write_sigmf("capture", samples, sample_rate=2.4e6, center_freq=1.42e9)
samples, meta = formats.read_sigmf("capture")

Talking to Radio-Sky Spectrograph over the network¶

Radio-Sky Spectrograph (RSS) accepts a live TCP feed — the same path RASDR uses. The protocol (documented in "How to Talk to Radio-Sky Spectrograph" and the RASDR socket commit): RSS listens on 127.0.0.1:8888; the client sends an ASCII config F <Hz>|S <Hz>|O <Hz>|C <nchan>|, then streams each sweep as nchan little-endian ("LoHi") uint16 samples (12-bit data), highest channel first, ending each sweep with the terminator bytes 0xFE 0xFE. No timestamps, no acknowledgements.

jansky.formats ships both a client and an in-process mock server, so you can exercise the full wire format with nothing installed:

server = formats.MockRSSServer(); server.start()
with formats.RSSClient(center_hz=21_000_000, bandwidth_hz=5_000_000,
                       n_channels=256, host=server.host, port=server.port) as rss:
    for sweep in spectra:          # each: 256 channel powers
        rss.send_sweep(sweep)
server.join()
assert server.config["C"] == 256   # the decoded sweeps are in server.sweeps

To feed the real application instead, point RSSClient at 127.0.0.1:8888 with RSS running and its Options → Radio → RTL Bridge/TCP input selected.

Reading .sps / .spd files

formats.read_sps() and read_spd() parse the Radio-Sky binary layout (a 156-byte little-endian header, a 0xFF-delimited notes block, then big-endian uint16 sweeps for SPS or int16/float64 time samples for SPD) into a Spectrogram. read_sps is validated byte-for-byte against a real Radio JOVE recording — fetch it with python -m jansky.data --fetch radiojove-sps (AJ4CO/Typinski, a dual-polarisation 16–32 MHz Jupiter dynamic spectrum). For live streaming data, use the RSS protocol above instead.

The wider ecosystem¶

RASG — Radio Astronomy Software Group — pyuvdata (visibility interchange: MS ↔ UVFITS ↔ UVH5), pyradiosky, and RFI tooling. Install the formats extra (uv sync --extra formats) for pyuvdata.
Breakthrough Listen / SETI — blimpy (filterbank/HDF5 I/O), rawspec, turboSETI (seti extra). See Chapter 16 and the SETI material in Field Notes.
VIRGO — a single-dish HI/continuum package (the engine behind PICTOR); install the hi extra. A zero-hardware route to a real hydrogen-line observation, complementing Chapter 6.
CASA — calibration & imaging of Measurement Sets; already covered in Chapter 12 and Resources.

Bundled real starter datasets¶

So you can work with real bytes — not just simulation — jansky.data registers a handful of small (< 2 MB) real files served from stable raw-GitHub URLs, cached on first use into data/:

python -m jansky.data --list                     # see them all (small first, then opt-in large)
python -m jansky.data --fetch pint-ngc6440e-par  # download one

The starter set includes a real search-mode PSRFITS and SIGPROC filterbank (from the your test suite) and the real NANOGrav timing model + TOAs for PSR J1748−2021E in NGC 6440 (NGC6440E.par/.tim, the PINT example). The 576 MB HI4PI all-sky map is kept as an opt-in "large" entry, off the default path; offline, jansky.data.synthetic_hi_cube() stands in. Every URL is checked by scripts/check_dataset_urls.py.

See also: Projects, Kits & Hacks for the hardware that produces these files, and the Bibliography for the science.