VIRTIS-H CALIBRATION ===================== Prepared by S. JACQUINOD JM REESS F. HENRY Approved by S. ERARD Authorized by JM REESS 2. Scope of the document ----------------------- This document describes how the science data of VIRTIS-H are calibrated from raw measurements. 3. Applicable documents ----------------------- None 4. Reference documents --------------------- - [RD1] RO-VIR-UM-001: VIRTIS Experiment User Manual, issue 3, July 2003 - [RD2] VIR-INAF-IC-007: VIRTIS To Planetary Science Archive Interface Control Document (EAICD) - [RD3] VVX-DLR-NC-002: Non-conformance report: H-PEM shutter synchronization - [RD4] VVX-LES-SW-2264: The VIRTIS PDS/IDL software library 5. List of acronyms ------------------- ADU - Arbitrary Digital Units IAS - Institut d'Astrophysique Spatiale (Orsay, France) IAPS - Istituto di Astrofisica e Planetologia Spaziali (Roma, Italy) FM - Flight Model LESIA - Laboratoire d'Etudes Spatiales et d'Instrumentation en Astrophysique (Observatoire de Paris, Meudon, France) ME - Main Electronics PDS - Planetary Data System SCET - SpaceCraft Elapsed Time TC - TeleCommand TM - TeleMetry TF - Transfer Function 6. Virtis-H instrument ---------------------- VIRTIS-H is a cross-dispersion spectrometer using 8 orders of a diffraction grating [RD1], [RD2]. The 8 spectral orders are spread over the entire surface detector matrix. Each order covers 432 x 5 pixels, i.e., 432 spectral channels spread on 5 pixels each. In the spatial dimension, the image of the slit only covers 3 pixels, but we integrate on 5 pixels in order not to loose signal. Thus overall only 15% of the 438x270 pixels matrix surface is used. To reduce the overall data rate and volume, H uses the so-called Pixel Map which gives the exact location of the spectra over the H-IR detector. 6.1 Detector size ----------------- The Virtis-H detector is a matrix of 438x270 pixels. During development, subsystem tests of Virtis-H were performed without the ME, and the data obtained are full-frame 2d images of 438x270 pixels. When operated with the ME, only a window of 432x256 pixels of the detector is processed. The location of this window on the detector is described by the parameters Xwin and Ywin, which remained constant during operations (Table 1). |---------|------|------| |Detector | Xwin | Ywin | |---------|------|------| | FM | 3 | 7 | |---------|------|------| Table 1: window position on Virtis-H detector 6.2 Virtis-H pixelmap --------------------- To locate the illuminated pixels on the detector a pixel map is used. This pixel map is the 8 x 3-coefficient table of the 2-degree polynomial function per order that identify the pixels that are illuminated on the detector. This pixel map has to be checked and redefined from time to time. The pixel map is defined using the internal calibration sessions. This kind of session is a set of 7 acquisitions, all made with the cover closed. In this mode, the full frame is read and transferred, in order to identify the pixels illuminated through the spectrometer. For each order, a set of 3 coefficients is computed to best fit a 2-degree polynomial function. The process has to be applied to each order. The Pixel map is created as an ASCII file with the 8 x 3-coefficient table. Whenever it changes, the new parameters are uploaded to the ME as a TC. The most recent version is used on board to process data acquired in nominal mode. The pixel map was regularly checked and proved to change very little after launch. Only the Mars flyby data were acquired with an on-board pixel map differing from the actual one. The Mars flyby data acquired in nominal mode can therefore only be partly recovered and calibrated in flux. 6.3 Acquisition modes --------------------- VIRTIS-H has 3 acquisition modes. 6.3.1 Nominal mode ------------------ In nominal mode, the downloaded data are spectra, a set of 432x8 pixels. A spectrum can be defined as a composition of the 8 orders imaged on the H-IR detector; the spectrum is extracted from the two-dimensional detector by using the onboard pixelmap. As H has no spatial resolution the 5 pixels of the slit image are averaged together, thus the final endproduct in the nominal acquisition mode is a 3456 (or 432x8) pixels spectrum representing the full spectral range of the instrument from 1.88 through 5.03 micron. 6.3.2 Backup mode ----------------- The full frame can also be downloaded in Backup mode, which is used to check the state of the detector and possible shifts of the pixel map. In this case all 432x256 pixels are sent. The extraction of the spectra with the pixelmap is thus performed on ground. 6.3.3 Internal calibration mode ------------------------------- A sequence of internal calibration of Virtis-H is operated as follows onboard: - Slit_spectral_calibration : 3 images (H_Image_Slice) with Cover closed, H-Shutter closed then S-lamp switched on, using functional param. integration time H_INT_SPECT_S - Telescope_spectral_calibration : 2 images (H_image_slice) with H-shutter closed, then T_lamp switched on, using functional param. integration time H_INT_SPECT_T - Image_slice_radiometric_calibration : 2 images, one H-shutter closed, then R-lamp switched on, using functional param. integration time H_INT_RADIO - Spectrum_Radiometric_Calibration : same, using the H_spectrum mode (dark with shutter closed, and then R-lamp on) The total of an internal calibration is 7xH_Image_Slice + 2 H_spectrum. The purpose of the internal calibration is the verification of the pixel map and wavelength pixel map (through telescope spectral calibration), the radiometric evolution of the VIRTIS-H chain (lamps+spectrometer+detector), which can be followed along the mission, and the functionality of the on board spectral reconstruction by comparing the H_spectrum mode to Image_Slice mode on the same radiometric calibration image. The slit calibration using the S-lamp is redundant with the T-lamp, but provides in addition a check of the integrity of the detector, by illuminating a large part of the detector (by scattered light), to follow the bad pixels evolution. 7 VIRTIS-H data --------------- 7.1 File formats ---------------- A qube is a PDS formatted file that contains data [RD2]. A qube is associated to sideplanes providing extra information, in particular selected housekeeping parameters. It can be read with the IDL library virtispds or similar software. The output of this routine is a structure, as described in [RD2] and [RD4]. 7.1.1 Raw Nominal qubes ----------------------- A raw Nominal qube contains raw data, acquired in Nominal mode. Relevant fields of the structure are: - qube: 3d integer array containing the data. Dimensions are spec_dim x 64 x n_frame - spec_dim is the size of one spectrum, and is equal to 8 orders x 432 pixels = 3456 - 64 is the number of spectra per frame o - n_frame is the number of frame in the qube Note: The reason of this grouping is that Virtis-H spectra are compressed and processed onboard by packets of 64 spectra [RD1] - suffix: 3d array containing the housekeeping parameters 7.1.2 Dark current qubes in nominal mode ---------------------------------------- In nominal acquisition mode, the dark spectra are sent and stored separately. The dark current qube only contains spectra acquired with the shutter closed. 7.1.3 Raw Backup qubes ---------------------- A raw Backup qube contains raw data acquired in Backup mode. In this mode, the whole detector is read. Dimensions are 432 x 256 x n_acquisition. The fields of the structure are the same as those of the raw Nominal qube. The dark current data are also contained in the qube. 7.1.4 Calibrated qubes ---------------------- A calibrated qube (nominal or backup) contains calibrated data. Units are W/m2/sr/micron. Relevant fields of the structure are: - qube: 3d float array containing the data. Dimensions are 3456 x n_acquisitions - n_acquisitions is 64 x n_frame - suffix: 3-columns array containing the time in SCET coded on 3 unsigned integers. One can converts into a real double precision value with the IDL routine v_scet(s0, s1, s2). - table: 3-columns array containing, for each pixel : - The wavelength, in micron - The spectral full width at half maximum (FWHM), in micron - This column should contain the 1- error on the radiance, but it's currently not computed. This column is filled with the error code -999.999d. 7.1.5 Internal calibration sequences ------------------------------------ Internal calibrations produce a backup qube and a set of two spectra (see 6.3.3). Relevant fields are those of raw qubes. Dimensions of the field 'qube' is always 432 x 256 x 7, that is to say seven 2d images of 432 x 256 pixels. The two spectra are stored in a different file. 7.2 Acquisition sequences ------------------------- 7.2.1 Observations and darks patterns ------------------------------------- One telecommand (TC) sent to virtis-H leads to one session of acquisitions. Observations and dark acquisitions are interleaved, following this pattern: 1 dark h_dark_rate x 1 observations 1 dark ... Where h_dark_rate is specified at the TC stage. In order to reduce the data volume, Virtis-H can sum acquisitions onboard. The parameters h_sum = yes and h_n_sum_frame = n lead to sum n acquisitions. Another way to reduce data volume is to use the parameter h_n_frame = p. In this case, only 1 acquisition among "p" is sent. Usages of h_n_sum_frame and h_n_frame are mutually exclusive. We describe here the usual observation modes. Details of H observing modes are provided in [RD1]. 7.2.2 Acquisition_id -------------------- Each acquisition has its own acquisition_id during a session, which is stored in the associated housekeeping parameters. The first acquisition (which is always a dark) has the number 1. The following h_dark_rate x 1 observations have the numbers 2 to (h_dark_rate + 1). In nominal mode, the reported acquisition_id is the one of the last spectrum of the current 64-spectra packet. Then all the 64 spectra of the same frame have the acquisition_id of the last one. Dark current measurements are sent independently each time they are acquired. Thus each dark has its own acquisition_id reported in the housekeepings. In backup mode, each acquisition is sent with its own acquisition_id. 7.3 Filenaming convention ------------------------- All data product files throughout different VIRTIS data sets are named using the same file naming convention [RD4]. Data qubes are named according to the suffix indicating the channel, the spacecraft clock reset number and the acquisition SC_CLOCK_START_COUNT (integer part). The possible suffix values are related to the data transfer mode: - H: H image transfer mode (backup observation mode) - S: H single spectrum transfer mode (including dark current files in nominal mode) - T: H "64-spectra frame" transfer mode (nominal mode) For example, acquisitions starting at SPACECRAFT_CLOCK_START_COUNT = 1/0432992043.44607 (1/ is the clock reset number) are named: T1_00432992529.QUB for a nominal spectrum The extension of raw files is .QUB The extension of calibrated qubes is .CAL Interpolated dark acquisitions are stored in files with extension .DRK 8. Radiometric Calibration -------------------------- 8.1 Calibration process ----------------------- The calibration process follows several steps described in the flow diagram hereafter (Image.10). We will describe those steps in details. 8.1.1 Nominal and Backup modes ------------------------------ As said in 6.3 the raw data are acquired in two modes: Nominal and Backup. In Nominal mode, the data are stored in spectrum format. In Backup mode, the data are stored as 2D images. For these data, we first need to transform the collection of 2d images into a collection of spectra. We compute for each pixel of each order the mean of the 5 pixels around the relevant position given by the pixel map. All 432-channels orders are concatenated, beginning with order 0. The resulting 3456 elements array is returned (same format as the Nominal spectra). For data from tests at sub-system level, the pixel map is applied starting from position [xwin,ywin] of the detector. Image.10: Flow diagram describing the calibration process for Nominal data 12 8.1.2 Despiking and interpolation of dark spectra ------------------------------------------------- Due to cosmic particles, which sometimes go through the detector, some erratic values (spikes) are detected in the data. In a first step, we need to despike the dark spectra. We perform the despiking channel by channel along the time dimension in the dark measurement seqeunce. For each channel considered we perform a filter at 3 sigma to eliminate the spikes. The first step of the calibration is to recover an optimized dark spectrum. In Nominal acquisition mode, the latest dark acquired at the end of a frame (64 spectra) is subtracted on board from the frame's spectra (cf. 7.1). First, we re-add the subtracted dark to the frame's spectra. Then, we interpolate the dark measurements acquired between the beginning and the end of a frame to obtain one dark per spectrum. Finally, we subtract the interpolated darks to the frame's spectra. Note that for the Backup mode data, we re-add the dark image to the science images before extracting the spectra as described in 8.1.1. Dark measurements are identified from the housekeeping parameters and processed separately (We use the same method to extract dark spectra from the dark images). Then we proceed as explained above for the interpolation. 8.1.3 Detector Odd-even effect correction ----------------------------------------- The second step of the calibration process consists in correcting the odd-even effect due to different responses of even and odd pixels of the detector. We separate odd and even pixels: we interpolate the 216 even pixels responses over a 432 points vector. Idem for the odd channels. The corrected dark is the mean of these two vectors. 8.1.4 Stray light correction ---------------------------- The third step of the calibration process consists in correcting the stray light effect. We currently only apply this correction to the Backup data because the stray light contamination information is only accessible in the 2D images of the detector. We loose the information in the spectrum format. To correct this effect we proceed as described in 10.1.1.2. 8.1.5 Division by transfer function and exposure time ----------------------------------------------------- To obtain the calibrated spectra, we need to divide the spectra by the transfer function of the instrument. The current transfer function is derived from a physical modeling of Lutetia based on the observations: a first order calibration derived from ground measurements is used to get the radiance spectrum of a selected area on Lutetia; spectral reflectance and thermal emission are fit on this spectrum; the ratio of the model to the measurement provides the transfer function (the detailed procedure is described in 10.1). This two-step process provides a transfer function free from features of remaining water vapor and CO2 in the ground calibration tank, which would affect the detection of molecular species in the coma in spite of a tentative removal (10.1.1.3). Absolute radiance levels are controlled by the ground-based measurements, and are estimated to be accurate to ~20-30%. Finally we also divide the spectra by the exposure time of the acquisition (available in the PDS label) and we obtain the calibrated spectra in radiance, with unit W/m2/sr/um. 8.2 Open issues on the calibration ---------------------------------- 8.2.1 Stray light correction for nominal spectra ------------------------------------------------ A refinement of the stray light correction, including for the Nominal mode, is under study and will be included in a future version of the archive. 8.2.2 Improvement of odd/even effect correction ----------------------------------------------- A refinement of the odd/even correction is under study and will be included in a future version of the archive. 8.2.3 Strabismus ---------------- VIRTIS-H actually uses two grating with different blaze angles, so as to enlarge the range in each spectral order. These two gratings do not see exactly the same part of the slit, and therefore do not have exactly the same Field of View. Whenever high frequency structures exist (e.g., shadowed areas on the nucleus), the signals from the two gratings do not fully match. This effect is not accounted for in the present version of the calibration, and spectra acquired near illumination boundaries are affected by a visible artifact. It is essentially negligible in the coma. This effect is being studied, and a partial correction will be included in a future version of the archive. 9. Spectral calibration ---------------------- 9.1 Principle ------------- The spectral calibration consists in determining the central wavelength of each pixel. The internal spectral calibration is made by mean of a pre-calibrated Fabry-Perot with respect to the temperature. The Fabry-Perot emission (Image.11) gives between 20 and 10 lines depending of the grating order. This device allows finding back the spectral registration of the instrument. The absolute position of each known spectral lines on the detector is gauss fitted for each order and used to reconstruct the spectral registration of the instrument, which fits a 2-degree polynomial function. Those polynomial functions are then used to give for each pixel the correspondence between the pixel number and the wavelength. Image.11: Emission of the Fabry-Perot spectral lines in each order 15 9.2 The PxLMapL --------------- The PxlMapL is a table of 3 coefficients per order. Those coefficients are the coefficients of the 2-degree polynomial function giving the correspondence between the pixel number and the wavelengths. For each order, we scan the spectrum to find the FP spectral line having a level upper than 'Offset' using a gauss fit function. The peak positions returned by the Gaussian fit function of each spectral line are stored in a table. Using the FP temperature 'dT' and the initial room temperature FP thickness 'e', a table of theoretical wavelengths emitted through the FP is generated. For each order a table 'lambdamin' of first expected wavelengths at the border of the detector is determined. The maximum error tolerated is given by the difference between two consecutive FP spectral lines. If the table is false, the registration will be false by a factor that is the difference between two consecutive FP spectral lines. Nonetheless, the error will be identified immediately on a science spectrum. If so the `lambdamin' table has to be readjusted. Once the table of FP wavelengths and positions on the detector are created for each order, the 2-degree polynomial fit is used to generate the coefficients. Those coefficients are then stored in a PxLMapL file. 9.3 Accuracy of the spectral calibration process ------------------------------------------------ The sources of errors to construct the registration table are: - The accuracy of the gauss fit for each FP spectral line; - The deviation from a 2 degree polynomial function for the registration; - The initial room temperature thickness for the FP. The addition of all the errors gives a maximum estimated global error for the registration of 1 pixel. The Image.12 and Image.13 estimate the maximum drift of the registration in spatial and spectral directions as a function of spectrometer and detector temperatures, using all the internal calibrations since the Steins Flyby (Sept. 2009). In the spatial direction (Y dimension of the detector) the maximum drift is estimated to 0.09 pixel, in spectral direction (X dimension of the detector), the maximum drift is estimated to 0.41 pixel. Image.12: Registration drift estimation in spatial direction using all internal calibrations since Steins Flyby Image.13: Registration drift estimation in spectral direction using all internal calibrations since Steins Flyby These errors can be decreased by using science spectra having well known spectral lines and by adjusting the parameters of the function. We have done this to improve the spectral calibration in orders 0 to 4 using data from the Earth and Mars flybys, and from the Escort Phase data of comet 67P/Churyumov-Gerasimenko (MTP019/STP070) based on theoretical modeling of those lines (see example on Figure 5). This results in particular in a much better match of overlapping orders. Image.14: Optimization of the spectral registration in order 4, using ESCORT phase data and theoretical model 9.4 Channels spectral profiles ------------------------------ 9.4.1 Definition ---------------- The spectral registration given by the PixelMapL table does not take into account the spectral profile within the image of the slit. This profile is the spectral response of the instrument. Owing to the optical design, the spectral response varies significantly along the spectrum. No dedicated on-ground calibration was able to measure this spectral response. Nonetheless, as VIRTIS-H is almost diffraction limited, this profile can be estimated by the diffraction theory using the optical parameters of the spectrometer. 9.4.2 Spectral profile calculation ---------------------------------- The spectral response is calculated for each wavelength in each order using the diffraction theory. It is the convolution of the diffraction profile by the width of the slit image. The formula used to calculate the spectral response are given in Image.15 with: - l is the wavelength - k is the order number - a is the incident angle on the grating - n is the groove density of the grating - pxl is the pixel size - L is the width of the grating - f is the objective focal length The following Image.16, Image.17 and Image.18 show the spectral response at two different wavelengths in two different orders. The blue line shows the geometrical slit width. Image.16: Spectral response at 2 micron, order 8 Image.17: Spectral response at 2 micron, order 0 Image.18: Plot of the spectral profile through order 0 20 10. Annexes ----------- 10.1 Derivation of the transfer function ---------------------------------------- To recover the calibrated spectra from raw data, a transfer function (TF) is used. The transfer function is usually computed from measurements done during the ground calibration phase. In our case, due to different problems in the on-ground calibration data (traces of H2O and CO2 in the vacuum chamber, odd-even effect...) and due to variations during launch and the lack of calibration during the Cruise Phase, we followed a new approach and defined a new method to compute the TF. This method is composed of two steps described below. 10.1.1 STEP 1: ground-based calibration --------------------------------------- The first step consists in building a first "BETA" transfer function, using on-ground calibration blackbodies data as input. As said previously, those data contains some flaws that we need to correct first: odd / even effect, stray light, H2O and CO2 contamination. 10.1.1.1 Odd/even effect correction ----------------------------------- This effect is due to a difference of gain and offset between the output voltage of the odd and even columns of the detector. It is only present in the X direction of the matrix (cf. Image.19) It depends of the charge (ADU) and decreases with the charge (non-linear effect). We use detector calibration measurements performed at LESIA on the YACADIRE bench to correct this effect. Image.19: Odd/even effect on the Virtis-H matrix detector (Up: Backup image, Down: extraction of a pixel values in X direction) In Image.20, an example of the correction curves we obtain using YACADIRE data. We interpolated those curves to obtain correction factors that we apply on odd channels of raw data. Image.20: YACADIRE correction curve 10.1.1.2 Stray light correction ------------------------------- Stray light is present in all on-ground data. This effect is polluting the right bottom corner of the detector matrix by adding dirty flux to the useful signal (cf. Image.21). This effect can be 2D-fitted to obtain a map of the stray light. Thus we subtract this map to the 2Draw images to correct this effect on data acquired in backup mode. Image.21: a) Backup raw image (2D) - b) map of the stray light - c) Backup raw image corrected from the stray light effect 10.1.1.3 Correction of H2O and CO2 contamination ------------------------------------------------ During on-ground calibration, blackbody stimuli bench and setup are placed under nitrogen environment whereas VIRTIS instrument is placed in vacuum chamber. A small quantity of H2O and C2O vapor remains. This leads to the detection of H2O and CO2 transmission spectral lines in the calibration blackbodies data. To remove those spectal lines, a simple low frequency filter is not sufficient, as it will distort the transfer function. We use the HITRAN (http://hitran.org) database, adjust the molecules concentration and convolve by the VIRTIS-H instrument profile. We optimize the correlation coefficient between measurement and model by adjusting the molecule concentration. In Image.22, an example of HITRAN database fitting to correct H2O transmission lines. Image.22: H2O vapor correction by HITRAN database fitting 10.1.1.4 Creation of a beta transfer function --------------------------------------------- After applying all the corrections described above to the on-ground blackbody calibration data, the final step to create the BETA transfer function consists in adjusting the radiometric level by fitting calibration black bodies. The complete procedure of the ground-based calibration is described in the flow diagram below (Image.23). Image.23: Flow diagram of the BETA Transfer Function creation 10.1.2 STEP 2: Lutetia as a calibrator -------------------------------------- The transfer function derived from ground measurements proved to be inconsistent when applied to flight data, which was ascribed to small shifts between optical parts occurred during the launch with Ariane 5. Besides, the remaining gas absorptions in the calibration tank are difficult to remove at a level not perturbing the measurements in the coma. Finally, no point source could be targeted during cruise with a single pixel instrument and provide enough flux to derive a new calibration. The second step of the transfer function derivation therefore consists in using the asteroid Lutetia observed during Cruise Phase as a calibrator. The process is described in the flow diagram below (Image.24). A selected area on Lutetia is integrated to provide a "Reference spectrum", both in raw data and calibrated with the above procedure. The radiance reference spectrum of Lutetia is modeled as the sum of reflected and emitted light. The ratio of the Raw Lutetia spectrum and the model provide the Final Transfer Function used on flight data. Image.24: Flow diagram of the Final Transfer Function creation 10.2 Corrupted darks -------------------- 10.2.1 Description of the problem --------------------------------- A serious problem was identified in 2006 on the model of VIRTIS-H flown on VenusExpress: some dark current acquisitions contained science signal. After investigation, this appeared to be due to an incorrect synchronization of the shutter in the onboard software, and the same problem affected the Rosetta software (see RD3). Image.25 shows an example: the shutter opens during dark current acquisition (dotted line). A growing fraction of the exposure is performed with the shutter open, until it cycles from the start; the solid line represents a correctly interpolated dark current, which in this case decreases with instrument temperature. Image.25: Example of dark corruption 10.2.2 Workaround ----------------- It is of course impossible to remove a science signal mixed in the dark current. The only solution is to identify the corrupted dark acquisitions and to skip them during data processing. Because this identification process is partly automatic, some corrupted darks may be still present. The onboard software has been modified and uploaded onto the spacecraft on the 27th of August 2009 (before the Steins flyby), which fixed the issue permanently.