New Horizons sub-frame WINdow SUMMary
Background: The New Horizons (NH) Spacecraft saves high-speed instrument
data in the on-board Solid State Recorder (SSR). High-speed data are
never telemetered to Earth, but instead either are packetized without
compression, or are compressed using either a lossless or lossy algorithm
into low-speed data. Low-speed data can be scheduled to be telemetered
back to Earth. When that telemetry reaches a Science Operations Center
(SOC), they are decommutated and converted back into a data format
suitable for the original instrument and detector. For some imaging
instruments (LORRI and RALPH/LEISA and RALPH/MVIC), LOSSLESS data may be
sub-framed i.e. a rectangular window that is a subset of the complete
observation is compressed and telemetered to Earth. For each single
observation, up to eight independent sub-frames (a.k.a. windows) may be
defined. The pipeline processes each window independently, because each
one comes in a single packet set, producing one FITS file per window. A
packet set is a sequence of packets: a start packet; zero or more
intermediate packets; a stop packet. An alternate packet set is a single
packet that is both a start and stop packet. Before March, 2015, each
window ended up in a separate FITS file; so only one set of windowing
parameters was needed in the FITS header to keep track of the original
compression settings.
As of March, 2015, the pipeline started to merge each window (packet set)
with previously processed windows from the same observation. The
implementation keeps a running history of the merge process in a BINTABLE
in the FITS file. The history contains three items:
1) the merged data
1.1) non-missing values in new data overwrite old data
2) a history of data keys in the PDU header
3) history mismatches detected when merging data, as a BINTABLE with
EXTNAME = 'WINDOWS_MISMATCHES'
3.1) the location of each mismatch
3.2) the numerical difference of each mismatch, new-old
3.3) the data key of each mismatch (from the new data)
Each data key comprises seven items, normally stored in each FITS PDU
header using the following keywords:
WINDOWX - Integer, left-edge offset to sub-frame, compr. params X_1
through X_8
WINDOWY - Integer, bottom-edge offset to sub-frame, compr. params Y_1
through Y_8
WINDOWW - Integer, sub-frame width, compr. params X1_len through
X8_len
WINDOWH - Integer, sub-frame height, compr. params Y1_len through
Y8_len
APID - String, Application Process IDentifier (e.g. '0x630')
GRT - Float, Ground-Received Time of packet(s)
ARCHDATE - String, packet telemetry filename,
e.g. '2008/290/2008290.tlm'
The data key history, for each packet, set is stored as FITS keywords, in
the PDU header, similar to the following:
WINX_000 - Original WINDOWX value
WINY_000 - Original WINDOWY value
WINW_000 - Original WINDOWW value
WINH_000 - Original WINDOWH value
APID_000 - Original APID value
GRT__000 - Original GRT value
ARCH_000 - Original ARCHDATE value
- the common _000 suffix indicates these values all came from the same
packet set. Later suffixes (_001, _002, _003, etc.) indicate
separate packet sets.
- N.B duplicate data keys are removed; that is, each data key will
appear only once in the header
- N.B the _nnn suffix does NOT indicate the order in which the packets
sets were processed; rather the sub-frame parameters are sorted, and
re-sorted each time a new packet set is merged with the data from
previous packet sets.
In addition to the summary history in the PDU FITS header, the merging
implementation creates a BINTABLE FITS eXTENSION of the history of the
window processing, plus the locations of any mismatches in the pixel
values loaded. Each window added to the data will generate at least one
entry (row) in the table, with a location using -999 as the offsets and
zero as the delta. There will be ten or eleven columns for 2D or 3D data
respectively:
MMNAXIS1OFFSET offset along NAXIS1 of data
MMNAXIS2OFFSET offset along NAXIS2 of data
MMNAXIS3OFFSET offset along NAXIS3 of data (for 3D data only)
DELTA delta from current pixel value to pixel being merged
WINDOWX Window parameter from data being merged
WINDOWY Window parameter from data being merged
WINDOWW Window parameter from data being merged
WINDOWH Window parameter from data being merged
APID Window parameter from data being merged
GRT Window parameter from data being merged
ARCHDATE Window parameter from data being merged
N.B. Each merged window will have at least one entry with the OFFSETs
set to -999 and DELTA set to zero. Note that such entries are a
superset of the data key list in the PDU header, and that these
entries with -999 as the offsets do not indicate any anomalies in
the merge process, and most if not all files will have multiple
entries with the -999 values.