--------------------------------- Page 155 --------------------------------- Deep Impact - Calibration Pipeline Summary Excerpt from the Deep Impact Instrument Calibration Document By Klaasen, et al., (2006) 6.0 Pipeline Processing 6.1 Standard Steps For each image, there is a standard set of procedures and settings applied in our pipeline processing in order to calibrate the images automatically (see Figure 100). In general, these default settings are the best the science team has been able to derive for the data set as a whole and thus do not necessarily reflect the best possible processing for any particular image. However, there are some observations around encounter, especially with the IR spectrometer, that contain very valuable scientific information but are not processed optimally by the default settings. For these cases, the automated pipeline has the ability to specify special settings for particular observations. --------------------------------- Page 155 --------------------------------- Figure 100 - A flowchart describing the image processing pipeline used to calibrate Deep Impact images. Note that some modules are not applied to all instruments. The standard pipeline begins by decompressing the image if it was compressed on the spacecraft. Images can be compressed using one of four 14-bit to 8-bit lookup tables optimized for different types of exposures. To uncompress the images, a reverse lookup table is used which maps each 8-bit value to the average of all corresponding 14-bit values. All saturated pixels are flagged in the quality map (Section 6.2). Then an IR image is linearized using the correction described in Section 5.3.1. A VIS image does not need this step because the instrument responds linearly (Section 4.3.1). Next, a dark frame is subtracted from the image. If a dark frame has been created by the science team for the specific observation, then it is subtracted. Otherwise, a dark model is used to generate the frame (Sections 4.3.3-VIS, 5.3.3-IR). --------------------------------- Page 156 --------------------------------- After the dark subtraction, a VIS image undergoes a few extra processing steps not taken by every IR image. First, the electrical crosstalk (Section 4.3.7.5) is removed by subtracting a derived ghost frame. Each quadrant in this frame is a linear combination of rotated versions of the other three quadrants. Next, the image is divided by a flat field (Section 4.3.6) in order to account for variable responsivity across the detector. A flat field is only applied to unbinned IR images because the best binned-mode flat field does not seem to provide any noticeable improvement in SNR (and in the data products published as PDS version 1, unbinned IR images are not flat fielded either). Lastly, VIS CCD transfer smear is removed using the parallel overclock rows if the image was taken in modes one through six or a column averaging approximation if the image is in modes seven or eight (Section 4.3.4). After bad pixels are flagged, the image is radiometrically calibrated to produce a radiance image in W/[m^2 sr um] and an I/F image. For a VIS image, this is simply done by dividing the image by integration time and then multiplying by the appropriate conversion factor derived in Section 4.3.5 for the given filter and desired output. For an IR image, the procedure is more complicated as the absolute calibration is wavelength dependent, which in turn is temperature dependent. First, the wavelength and bandwidth for each pixel are calculated as described in Section 5.3.4. Then, each pixel is multiplied by the appropriate wavelength-dependent calibration factor (Section 5.3.6) and divided by integration time and the pixel's spectral bandwidth. Once this radiance image is created, a copy is converted to I/F by dividing by the solar spectrum at the target's distance from the sun and then multiplying by pi. At this point, two reversible data products have been created, one radiance image and one I/F image, and copies are run through the rest of the pipeline, which performs a series of non-reversible steps. First, the data are interpolated over the bad pixels and gaps. For a VIS image, this interpolation is performed using thin plate splines anchored by the valid data around the edges of each hole. For an IR image, a linear interpolation is performed in the spatial dimension only. Next, a despiking routine is applied in order to remove cosmic rays. This routine performs a sigma filter by calculating the median of each NxN box, where N is odd, and then replacing the central pixel with the median if it is more than M median deviations from the median. By default, both M and N are set to 3. The median deviation of a set S is defined as: Med(| S - Med(S) |). Lastly, a VIS image is deconvolved using the methods described in Section 4.2.3. This is especially important for the HRI-VIS instrument which is out of focus. 6.2 Calibration Quality Map Along with each calibrated image, a byte map is created that defines the data integrity for every pixel. For each byte in the map, representing one pixel, each bit acts as a flag that is set to 1 if the given criterion is met for that pixel. These flags are: --------------------------------- Page 157 --------------------------------- MSB LSB +-------------------------------+ | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +-------------------------------+ 0. Bad Pixel - This pixel is a known bad pixel. 1. Missing - The data for this pixel was not received from the spacecraft. 2. Despiked - This pixel was modified by the despiking routine. 3. Interpolated - This pixel has been reclaimed by interpolating from its neighbors. 4. Some Saturated - The raw value for this pixel is above the point where some pixels are full-well saturated. For VIS instruments, this occurs at 11,000 DN, while for the IR spectrometer, this occurs at 8,000 DN. 5. Most Saturated - This raw value for this pixel is above the point where most pixels are full-well saturated. For VIS instruments, this occurs at 15,000 DN, while for the IR spectrometer, this occurs at 11,000 DN. 6. ADC Saturation - The ADC was saturated for this pixel. 7. Ultra Compressed - The pixel was in a compression bin so large that the value contains very little information. For example, if the pixel is bad and has been reclaimed by interpolation, the decimal value in the quality map will be 2^0 + 2^3 = 9. In the normal FITS format for the calibrated image, this map exists as the first image extension. 6.3 Signal-to-Noise Ratio Map In order to provide more information to the end user, the last extension of the image contains a map estimating the signal to noise ratio for each pixel. The signal is taken to be the dark-and bias-subtracted image value in 14-bit DN, while the noise estimate consists of the root-sum-squared of three different noise sources: shot noise, read noise and quantization noise. The shot noise in 14-bit DN is defined as: N_s = Sqrt ( (Raw - Bias) / K ) where K is the gain in electrons/14-bit DN and is dependent on the instrument and mode, and Raw and Bias are in 14-bit DN. For the IR spectrometer, Bias is 0 by definition except in Mode 6. The quantization noise is defined as: N_q = Q / Sqrt(12) where Q is the quantization step in 14-bit DN. For uncompressed data, Q depends on the ADC performance of the instrument (see Sec. 4.3.7.3 and Sec. 5.3.8.3), while for compressed data, Q is set to the bin size in the decompression lookup table that the pixel used or to the uncompressed Q value, whichever is larger. The parameter values needed for the noise calculation were determined from ground-based testing of the instruments and are shown in Table 20. --------------------------------- Page 158 --------------------------------- Table 20 - Noise parameters determined in ground tests of all instruments. 6.4 Spectral Registration Maps In an IR image product, the second and third extensions are pixel-by-pixel maps of the spectral registration for the image. The second extension contains the effective wavelength of the pixel, while the third extension contains the spectral bandwidth. The calculations of these values are described in Section 5.3.4. 6.5 Optional Steps Beyond the automated calibration pipeline described in Section 6.1, a manual calibration can be performed where the user can specify his/her own settings and calibration files for each step. Also, any processing module can be disabled, and there are two extra ones that can be enabled. The first such module is a noise-reduction module that is applied after the despiking routine. This applies the BayesShrink wavelet thresholding algorithm (Ref. 22) with a robust mean noise estimator (Ref. 22) to remove some of the noise. The other step that can be enabled applies a rubber sheet geometric distortion correction. This is not normally applied as the optical distortion though the telescope is minimal.