New Horizons SOC to Instrument Pipeline ICD September 2007 SwRI Project 05310 Document No. 05310-SOCINST-01 Contract NASW-02008 New Horizons SOC to Instrument Pipeline ICD SwRI Project 05310 Document No. 05310-SOCINST-01 Contract NASW-02008 Prepared by: Joe Peterson 23 February 2007 Contributors: ALICE specifics prepared by: Maarten Versteeg, Joel Parker, & Andrew Steffl LEISA specifics prepared by: George McCabe & Allen Lunsford LORRI specifics prepared by: Hal Weaver & Howard Taylor MVIC specifics prepared by: Cathy Olkin PEPSSI specifics prepared by: Stefano Livi REX specifics prepared by: Ivan Linscott & Brian Carcich SDC specifics prepared by: David James SWAP specifics prepared by: Heather Elliott General Approval Signatures: Approved by: ____________________________________ Date: ____________ Hal Weaver, JHU/APL, Project Scientist Approved by: Leslie Young via email 1 December 2005 SwRI, Deputy Project Scientist Approved by: John Andrews via email 4 December 2005 SwRI, SOC Project Manager Instrument-specific Signatures: (Science Theme Teams members) Approved by: ____________________________________ Date: ____________ Jeff Moore, NASA, GGI Science Theme Team Lead (applies to MVIC, LORRI) Approved by: /s/ Dale Cruikshank via email 6 March 2006 NASA, COMP Science Theme Team Lead (applies to LEISA) Approved by: Randy Gladstone via email 6 December 2005 SwRI, ATM Science Theme Team Lead (applies to ALICE, REX) Approved by: /s/ Fran Bagenal via email 23 January 2006 CU, P&P Science Theme Team Lead (applies to SWAP, PEPSSI, SDC)) (Instrument PIs & Scientists) Approved by: Alan Stern via email 5 December 2005 SwRI, ALICE Instrument PI Approved by: Dave Slater via email 2 December 2005 SwRI, ALICE Instrument Scientist Approved by: Alan Stern via email 5 December 2005 SwRI, ALICE Instrument PI Approved by: Dennis Reuter via email 5 December 2005 NASA, LEISA Instrument Scientist Approved by: ____________________________________ Date: ____________ Andy Cheng, JHU/APL, LORRI Instrument PI Approved by: ____________________________________ Date: ____________ Hal Weaver, JHU/APL, LORRI Instrument Scientist Approved by: Alan Stern via email 5 December 2005 SwRI, ALICE Instrument PI Approved by: Dennis Reuter via email 5 December 2005 NASA, LEISA Instrument Scientist Approved by: ____________________________________ Date: ____________ Ralph McNutt, JHU/APL, PEPSSI Instrument PI Approved by: ____________________________________ Date: ____________ Stefano Livi, JHU/APL, PEPSSI Instrument Scientist Approved by: /s/ Len Tyler via email 8 March 2006 Stanford, REX Instrument PI Approved by: /s/ Ivan Linscott via email 8 March 2006 Stanford, REX Instrument Scientist Approved by: ____________________________________ Date: ____________ Mihaly Horanyi, CU, SDC Instrument PI Approved by: ____________________________________ Date: ____________ Mihaly Horanyi, CU, SDC Instrument Scientist Approved by: Dave McComas via email 12 December 2005 SwRI, SWAP Instrument PI Approved by: Heather Elliot via email 8 March 2006 SwRI, SWAP Instrument Scientist Release to Document Control: Approved by: ____________________________________ Date: ____________ L. D. McCullough, SwRI Project CM TABLE OF CONTENTS Page 1. SCOPE 2. SIGNATURES, AUTHORSHIP, AND RESPONSIBILITY 3. APPLICABLE DOCUMENTS 3.1 New Horizons Documents 4. INTRODUCTION 5. INTERFACE DESCRIPTION 6. REQUIREMENTS 6.1 Level 2 (output) Files 6.2 Pointing Information 6.3 The Code 6.4 Calibration Data 6.5 Documentation 6.6 Error Conditions and Integration Troubleshooting 6.7 PDS Archiving 6.8 Configuration Management 6.9 Pipeline Updates 6.10 Acceptance Review 6.11 Longevity 7. ALICE INSTRUMENT DESCRIPTION 7.1 PERSI-Alice Instrument Overview 7.2 "Raw" Data Specifics 7.2.1 Data Format 7.2.1.1 Histogram FITS file: 7.2.1.2 Pixel list FITS file: 7.2.2 Data Sources (High/Low Speed, CCSDS, ITF) 7.2.3 Definition of an "Observation" 7.2.4 S/C Housekeeping Needed in Level 1 Files (for Calibration) 7.3 "Calibrated" Data Specifics 7.3.1 Algorithm for Pipeline 7.3.1.1 Deadtime Correction 7.3.1.2 Dark Correction 7.3.1.3 Effective Area 7.3.1.4 Flat Field 7.3.2 Format of Calibrated Data 7.3.2.1 Histogram 7.3.2.2 Pixel List 7.3.3 Scientific Units 7.3.4 Additional FITS and PDS Keywords Added 7.3.5 Hardware/OS Development Platform 7.3.6 Language(s) Used 7.3.7 Third Party Libraries Required 7.3.8 Predicted Execution time 7.3.9 Contact/Support Person(s) 8. LEISA INSTRUMENT DESCRIPTION 8.1 Overview 8.2 Raw Data Specifics 8.2.1 Data Format 8.2.2 Data Sources (High/Low Speed, CCSDS, ITF) 8.2.3 Definition of an "Observation" 8.2.4 Housekeeping Needed in Level 1 Files (for Calibration) 8.2.5 Science Data and/or Housekeeping Requirements 8.3 Calibrated Data Specifics 8.3.1 Algorithm for Pipeline 8.3.2 Dataflow Block Diagram 8.3.3 Data Format 8.3.4 Extra FITS Extensions (planes) and Their Definitions 8.3.5 Scientific Units 8.3.6 Additional FITS and PDS Keywords Added 8.3.7 Hardware/OS Development Platform 8.3.8 Language(s) Used 8.3.9 Third Party Libraries Required 8.3.10 Calibration Files Needed (with Quantities) 8.3.11 Memory Required 8.3.12 Temporary File System Space Needed 8.3.13 Predicted Size of Output File(s) 8.3.14 Predicted Execution time 8.3.15 Contact/Support Person(s) 8.3.16 Maintenance Schedule (Code/Data Updates, Documentation) 9. LORRI INSTRUMENT DESCRIPTION 9.1 Overview 9.2 Raw image Specifics 9.2.1 Data Format 9.2.2 Data Sources (High/Low Speed, CCSDS, ITF) 9.2.3 Definition of an "Observation" 9.2.4 Housekeeping Needed in Raw Image Files (for Calibration) 9.2.5 Raw Science Data and/or Housekeeping Requirements 9.3 Calibrated Image Specifics 9.3.1 Algorithms for Pipeline Calibration Process 9.3.1.1 Bias Subtraction 9.3.1.2 Smear Removal 9.3.1.3 Flat-Fielding 9.3.1.4 Absolute Calibration (Conversion from corrected DN to physical units) 9.3.1.5 Pointing Information 9.3.1.6 Conversion of instrument housekeeping items to engineering units 9.3.2 Instrument Characterization 9.3.3 Special Processing 9.3.4 Dataflow Block Diagram 9.3.5 Data Format 9.3.6 Extra FITS Extensions (planes) and Their Definitions 9.3.7 Scientific Units 9.3.8 Additional FITS and PDS Keywords Added 9.3.8.1 Reading FITS file contents using IDL 9.3.9 Hardware/OS Development Platform 9.3.10 Language(s) Used 9.3.11 Third Party Libraries Required 9.3.12 Calibration Files Needed (with Quantities) 9.3.13 Memory Required 9.3.14 Temporary File System Space Needed 9.3.15 Predicted Size of Output File(s) 9.3.16 Predicted Execution time 9.3.17 Contact/Support Person(s) 9.3.18 Maintenance Schedule (Code/Data Updates, Documentation) 9.4 References 10. MVIC INSTRUMENT DESCRIPTION: 10.1 Overview 10.2 Level 1 Specifics 10.2.1 Data Format 10.2.2 Data Sources (High/Low Speed, CCSDS, ITF) 10.2.3 Definition of an "Observation" 10.2.4 Housekeeping Needed in Level 1 Files (for Calibration) 10.2.5 Raw Science Data and/or Housekeeping Requirements 10.3 Level 2 Specifics 10.3.1 Algorithm for Pipeline 10.3.1.1 Remove bias and flat-field pattern 10.3.1.2 Geometric Correction 10.3.1.3 Conversion to Physical Units 10.3.1.4 Error Propagation 10.3.1.5 Data Quality Flags 10.3.2 Dataflow Block Diagram (to do - add distortion correction to tree) 10.3.3 Data Format 10.3.4 Scientific Units 10.3.5 Additional FITS and PDS Keywords Added 10.3.6 Extra FITS Extensions() and Their Definitions 10.3.7 Hardware/OS Development Platform 10.3.8 Language(s) Used 10.3.9 Third Party Libraries Required 10.3.10 Calibration Files Needed (with Quantities) 10.3.11 Memory Required 10.3.12 Temporary File System Space Needed 10.3.13 Predicted Size of Output File(s) 10.3.14 Predicted Execution time 10.3.15 Contact/Support Person(s) 10.3.16 Maintenance Schedule (Code/Data Updates, Documentation) 11. PEPSSI INSTRUMENT DESCRIPTION 11.1 Overview 11.1.1 PEPSSI Investigation 11.1.2 PEPSSI Sensor Description 11.1.3 PEPSSI Electronics Description 11.2 Introduction to PEPSSI Data 11.3 Level 1 Specifics 11.3.1 Data Sources (High/Low Speed, CCSDS, ITF) 11.3.2 Definition of an "Observation" 11.3.3 Header with Keywords 11.3.4 Spacecraft Housekeeping Needed in Level 1 Files (for Calibration) 11.3.5 Raw Science Data and/or Housekeeping Requirements 11.4 Level 2 Specifics 11.4.1 Algorithm for Pipeline 11.4.1.1 IDL L1 to Pre-L2 step 11.4.1.2 MIDL L1 to Pre-L2 step 11.4.2 Dataflow Block Diagram 11.4.2.1 L1 to Pre-L2 11.4.2.2 Pre-L2 to L2 processing 11.4.3 L2 Data Format 11.4.3.1 PHA_HIGH_ION 11.4.3.2 PHA_ELECTRON 11.4.3.3 PHA_LOW_ION 11.4.3.4 PHA_DIAG 11.4.3.5 N1 and D_N1 11.4.3.5.1 Rate Box Definitions 11.4.3.6 N2 and D_N2 11.4.3.7 (D)_N(1/2)_STATUS 11.4.4 Bad Time Intervals (BTIs) 11.4.5 L3Data Format 11.4.5.1 Primary HDU: Rate Weighted 2-D Histogram 11.4.5.2 Quick Look Spectrograms 11.4.5.3 FLUX 11.4.5.3.1 FLUX Calibration Procedure 11.4.5.3.2 Derivation and Explanation of Calibration Table Values 11.4.5.4 PHA Data 11.4.6 Memory Required 11.4.7 Temporary File System Space Needed 11.4.8 Predicted Size of Output File(s) 11.4.9 Predicted Execution time 11.4.10 Contact/Support Person(s) 11.4.11 Maintenance Schedule (Code/Data Updates, Documentation) 12. REX INSTRUMENT DESCRIPTION 12.1 Overview 12.2 Raw Data Specifics 12.2.1 Raw Data Format 12.2.1.1 PDU Content 12.2.1.1.1 ROF ID byte 12.2.1.1.2 ROF Status byte 12.2.1.1.3 Integrated Radiometry values 12.2.1.1.4 Time Tag values 12.2.1.1.5 I & Q value pairs 12.2.1.2 PDU Data layout: Interleaving 12.2.1.2.1 Layout of ID & Status bytes 12.2.1.2.2 Layout of Radiometry 12.2.1.2.3 Layout of Time Tags 12.2.1.2.4 Layout of I & Q values 12.2.1.3 Layout of ROF 12.2.2 Data Sources (High/Low Speed, CCSDS, ITF) 12.2.3 Definition of an "Observation" 12.2.4 Housekeeping in Raw FIT files' extensions 12.2.5 Raw Science Data and/or Housekeeping Requirements 12.3 Calibration Specifics 12.3.1 Calibration Algorithms 12.3.1.1 Calibrating the REX filter output: In-phase & Quadrature-phase values 12.3.1.1.1 Combine the unsigned bytes into a two's complement 16-bit signed integer filter value 12.3.1.1.2 Scale the filter value by the calibrated reference voltage 12.3.1.2 Calibrating the REX Radiometry 12.3.1.3 Calibrating the REX Time Tags 12.3.2 Calibrated FITS file data format 12.3.2.1 Extension Data Unit (EDU) 1: I & Q values 12.3.2.2 EDU 2: Radiometry & Time Tags 12.3.2.3 Additional FITS Extensions (planes) and Their Definitions 12.3.2.4 Scientific Units 12.3.2.5 Additional FITS and PDS Keywords 12.3.2.5.1 Radiometry calibration provenance added to PHDU 12.3.2.5.2 FITS keywords added to EHDU 1 12.3.2.5.3 Scaling parameters 12.3.3 Hardware/OS Development Platform 12.3.4 Language(s) Used 12.3.5 Third Party Libraries Required 12.3.6 Calibration Files Needed (with Quantities) 12.3.7 Memory Required 12.3.8 Temporary File System Space Needed 12.3.9 Predicted Size of Output File(s) 12.3.10 Predicted Execution time 12.3.11 Contact/Support Person(s) 12.3.12 Maintenance Schedule (Code/Data Updates, Documentation) 13. SDC INSTRUMENT DESCRIPTION 13.1 Overview 13.2 Raw Data Specifics 13.2.1 Data Format 13.2.2 Data Sources (High/Low Speed, CCSDS, ITF) 13.2.3 Definition of an "Observation" 13.2.4 Housekeeping Needed in Level 1 Files (for Calibration) 13.2.5 Raw Science Data and/or Housekeeping Requirements 13.2.6 Notes about Raw Data 13.3 Calibration 13.3.1 Pre-Flight Calibration Procedure- Charge --Charge Calibration File-- 13.3.2 Calibration - Mass 13.3.2.1 Pre-Flight 13.3.2.2 Mass Production 13.4 Calibrated Data Specifics 13.4.1 Algorithm for Pipeline 13.4.2 Dataflow Block Diagram 13.4.3 Data Format 13.4.4 Extra FITS Extensions (planes) and Their Definitions 13.4.5 Scientific Units 13.4.6 Additional FITS and PDS Keywords Added 13.4.7 Hardware/OS Development Platform 13.4.8 Language(s) Used 13.4.9 Third Party Libraries Required 13.4.10 Calibration Files Needed (with Quantities) 13.4.11 Memory Required 13.4.12 Temporary File System Space Needed 13.4.13 Predicted Size of Output File(s) 13.4.14 Predicted Execution time 13.4.15 Contact/Support Person(s) 13.4.16 Maintenance Schedule (Code/Data Updates, Documentation) 14. SWAP INSTRUMENT DESCRIPTION 14.1 Overview 14.2 Electronics and Flight Software 14.3 SWAP Data Types 14.4 Raw File (Level 2) Specifics 14.4.1 Data Format 14.4.2 Definition of an "Observation" 14.4.3 Housekeeping Needed in Level 2(Raw) Files (for Calibration) 14.4.4 Raw Science Data and/or Housekeeping Requirements 14.5 Calibrated (Level 3) File Specifics 14.5.1 Data Format 14.5.2 Further Algorithm details for Pipeline 14.5.3 Real-time science data processing 14.5.4 Errors 14.5.5 Quality Flags 14.5.6 Thruster Firings 14.5.7 SPICE Orbit and Attitude Calculations 14.5.8 Summary and Histogram Files 14.5.9 Calibration 14.5.10 Background Subtraction 14.6 Operations 14.7 Observation Examples 14.7.1 SWAP Science Goals 14.7.2 Dataflow Block Diagram 14.7.3 Extra FITS Extensions and Their Definitions 14.7.4 Scientific Units 14.7.5 Additional FITS and PDS Keywords Added 14.7.6 Hardware/OS Development Platform 14.7.7 Language(s) Used 14.7.8 Third Party Libraries Required 14.7.9 Memory Required 14.7.10 Temporary File System Space Needed 14.7.11 Predicted Size of Output File(s) 14.7.12 Predicted Execution time 14.7.13 Contact/Support Person(s) 14.7.14 Maintenance Schedule (Code/Data Updates, Documentation) 14.8 Packet Description REVISION NOTICE Draft 1: October 2005 Draft 2: February/March 2006 Initial Issue: February 2007. Updated Sections 10 (MVIC), 11 (PEPSSI), 13 (SDC) and 14 (SWAP). 1. SCOPE This document specifies the interfaces between the New Horizons Science Operations Center (SOC) and the instrument pipeline, which process data from Level 1 to Level 2. The purpose is to define the various aspects of the interfaces in sufficient detail to establish a clear understanding between the SOC and the instrument team to allow for a parallel pipeline development. 2. SIGNATURES, AUTHORSHIP, AND RESPONSIBILITY This document is unusual in that many parties took part in its writing. Specifically, sections 1 through 6 were written by the document author, whereas the bulk of the instrument-specific sections (7 through 14) were written by representatives of the instrument described. Because of this, a few words will be said regarding signatures. Each instrument team will have a person or persons responsible for their section. If changes are made to that section, only the person(s) responsible need to sign the new revision. If, however, changes are made to sections 1 through 6, all parties need to sign. The title(s) of the person(s) responsible for each instrument section are given in the signature section above. 3. APPLICABLE DOCUMENTS 3.1 New Horizons Documents - New Horizons SOC Data Pipeline Guide (SwRI Doc. No. 05310-SOCPL-G-01) - New Horizons SOC Level 1 Data Formats (SwRI Doc. No. 05310-SOCL1DATA-01) - New Horizons SOC Pipeline User Manual (SwRI DOc. No. 05310-SOCPLUM-01) - New Horizons Data Management and Archiving Plan (SwRI Doc. No. 05310-DMAP-01) - New Horizons Longevity Plan (APL Doc. No. 7399-9009) - Definition of the Flexible Image Transport System (FITS) (ftp://legacy.gsfc.nasa.gov/fits_info/fits_office/fits_standard.pdf) 4. INTRODUCTION The New Horizons SOC is part of the ground system that processes data returned from the New Horizons planetary spacecraft. Data downlinked from the spacecraft in raw packetized form is retrieved by the SOC from the Mission Operations Center (MOC) along with navigation and related ancillary data. The SOC generates the higher level (more refined) data products used by the instrument teams and science teams. In addition, the SOC performs archiving of data (but not data products, such as maps) to the Planetary Data System (PDS). The science data processing component of the SOC is called the SOC pipeline. The Level 2 pipeline (called the "instrument pipeline" and also known as the "calibration pipeline") software is a segment of the SOC Pipeline. The instrument pipeline for each instrument is written by the appropriate instrument team. It is run on the SOC processing station to transform Level 1 decommutated data into Level 2 calibrated science data. The instrument pipeline creates PDS standard, Level 2 provisional products in Flexible Image Transport System (FITS) format, which is described in the document referenced herein (Definition of the Flexible Image Transport System). The SOC pipeline is divided into two main parts: the Level 1 pipeline segment and the Level 2 (instrument) pipeline segment. Pipeline processing is carried out sequentially. Results of the Level 1 pipeline are provided as inputs to the instrument Level 2 pipeline segment. More information about the formats of Level 1 data can be found in the "New Horizons SOC Level 1 Data Formats" document (SwRI Doc. No. 05310-SOCL1DATA-01). The instrument pipeline generates Level 2 results that the SOC forwards to the PDS archiving process. The SOC obtains science data from the Mission Operations Center (MOC) through automated network transfers. Low-speed housekeeping and high-speed science data from the spacecraft are provided in packetized Supplemented Telemetry Packet (STP) form. Navigation data, ephemerides and spacecraft pointing information, is provided in SPICE format from the MOC and is used in Level 1 processing. Upon getting packets (housekeeping and science data) from the MOC, the data is decommutated in the SOC and written to an SQL database. Housekeeping from the database and science data are associated by MET time and other methods, such as by using meta data inserted in the high-speed telemetry. Data for an observation are combined to create the Level 1 uncalibrated data file. A PDS detached header file (a separate file containing the PDS header with a pointer to the data file itself) is also created for each file. The header of the Level 1 file contains most of the necessary information about the particular observation needed by the instrument pipeline (an exception is detailed pointing, which will be calculated during calibration). The instrument pipeline segment creates the Level 2 calibrated data file from the contents of the Level 1 file and calibration data provided by the instrument team. Level 1 and Level 2 science data files are stored in FITS format. Lastly, the SOC archives pipeline data products to the PDS. SOC pipeline processing is automated under the control of the Master Data Manager (MDM), which is a collection of scripts that control the flow of the pipeline. While manual execution of the program is permitted, normal operation of the SOC pipeline is not directed by manual requests or user inputs. Pipeline segments are called by the MDM when data from the MOC or from a previous pipeline step is available. The hardware platform used for the SOC as implemented for launch and early mission is an Intel Pentium 4 processor running at 3.2GHz with 4GB RAM and a 146GB SCSI hard disk. In the case of the primary SOC (TSOC), located in Boulder, Colorado, two of these machines are used (one for pipeline processing and the other for data storage). At the secondary (backup) SOC (CSOC), only one machine is used for both purposes. The operating system used in all cases is Linux. Migration to newer machines compatable with SOC requirements is a possibility during the long flight missions. 5. 5. INTERFACE DESCRIPTION The SOC pipeline code will call the Level 2 pipeline code by executing a separate process. The name of the executable or script shall be: [instrument]_level2_pipeline (where "[instrument]" is the full instrument name (i.e. alice, leisa, lorri, mvic, pepssi, rex, sdc, or swap) in lower case). The parameters (all are character strings) passed to the Level 2 code will follow the executable name and will be in the following order (note that "full path," when stated below, means a file specification containing the filename and all directories in the file's path): - in_file - Location of input (Level 1) file (in_file) The SOC will provide the full path of the Level 1 input file to the Level 2 pipeline code. - in_pds_header - Location of input (Level 1) detached PDS header The SOC will provide the full path of the Level 1 PDS header to the Level 2 pipeline code. - calibration_dir - Location of calibration data and temporary file storage Data provided by the instrument team that is needed for calibration shall be supplied by the instrument team. The SOC will provide the root directory containing these files (and potentially, subdirectories) to the Level 2 pipeline code so it references the correct location. The structure of the directories under this directory is up to the instrument team. - temp_dir - Location for temporary storage used by code This is a place where the instrument pipeline code may write files for temporary use. The contents of this directory will be erased upon completion of the instrument pipeline. - out_status - Location of status file The Level 2 pipeline, upon completion, may write a short machine readable status file used to communicate the results of the processing to the main SOC pipeline. The location (full path) of this file will be supplied by the SOC. - out_file - Location of output (Level 2) file This is the file (full path) to which the output will be written. The SOC will provide this to the Level 2 pipeline code. - out_pds_header - Location of output (Level 2) detached PDS header This is the file (full path) to which the Level 2 PDS header will be written. The SOC will provide this to the Level 2 pipeline code. 6. REQUIREMENTS This section describes the various requirements that the instrument team shall follow with regard to the Level 2 pipelines. 6.1 Level 2 (output) Files The Level 2 data files shall be FITS files, and there shall be exactly one Level 2 file produced for each Level 1 file (in any given run of the instrument pipeline). The headers will be mostly the same, except for optional additional keywords inserted in the Level 2 files (this could include, for example, refined pointing). In other words, typically, the Level 2 header keywords will be a superset of the Level 1 header keywords. The filename of the Level 2 file will be supplied by the SOC, and it will be similar to the Level 1 filename. PDS requires a "27.3" ASCII character limit on the filenames, and the format of the names shall be as follows: - Level 1: [instrument]_[MET]_[ApID]_eng_[version].fit - Level 2: [instrument]_[MET]_[ApID]_sci_[version].fit (where "[instrument]" is the first three letters of the instrument - name (i.e. ali, lei, lor, mvi, pep, rex, sdc, or swa) in lower case, - "[MET]" is the 10-digit MET time, either from the instrument itself - (low-speed data) or from the instrument interface card (high-speed - data), "[ApID]" is the hexadecimal value of the ApID for this - observation, and "[version]" is the version stamped on this file - based in existing previous versions produced). The instrument/mode strings above will be derived from the ApID of the data, and these filenames will be supplied to the instrument pipeline (see the interface description above). Whereas the Level 1 files will be in the same "units" as the data coming from the spacecraft/instrument (i.e. same binary representation - this is partly to avoid any round-off or conversion loss), the Level 2 files shall express values in physical units useful for scientific interpretation. Whereas the Level 1 files only contain the header and data itself, the Level 2 files shall contain (when appropriate) two additional "panes" (FITS extensions): - Error (specifies error bars on the numbers - defined by the instrument team) - Quality (Indicates the quality of the data - defined by the instrument team) If it is desired to re-run the Level 2 pipeline (because of new Level 2 code or calibration data), a new version of the output file will be named as specified above when calling the Level 2 pipeline. 6.2 Pointing Information The pointing information included in the Level 1 files will be mostly non-instrument specific (except for bore-sight vector where applicable). It also may not cover the time granularity needed for calibration in the Level 2 pipelines (see the "New Horizons SOC Level 1 Data Formats" document (SwRI Doc. No. 05310-SOCL1DATA-01) for specifics). Therefore it is expected that the Level 2 pipelines may have to make use of SPICE. It is therefore the responsibility of the Level 2 pipelines to provide this functionality. SPICE kernels will be available on the SOC. 6.3 The Code The SOC defines the interface the code uses to access the required data. This includes the directory structure on the disk where the Level 1 data file can be found as well as the path (specific to each instrument) where instrument-team-supplied calibration files and other data will be stored and accessed. Also, the filename of the output file is supplied. The code shall be written in a language that meets the SOC's longevity requirements (see section 6.11). More information on this can be found in the "New Horizons SOC Pipeline Guide" (SwRI Doc. No. 05310-SOCPL-G-01). The languages allowed are as follows: - C - C++ - Fortran - IDL - Python - Java - Perl The code written by the instrument team shall not implement very time consuming iterative numerical processing to the extent that it has an impact on the daily processing done by the SOC. In other words, if the time to compute Level 2 data is so extreme that it jeopardizes the completion of each daily run of the pipeline (so ample idle time is not available between runs), the situation will need to be re-evaluated. It is expected that a daily run of the entire pipeline be complete within a few hours of its start. This gives most of the day for users to access new data before the next run is initiated. The maximum time allowed for execution of an instrument's Level 2 pipeline shall be 5 minutes (for each input file processed). The predicted actual maximum time is negotiated and specified in the instrument-specific sections. Any needed 3rd party libraries also shall meet the longevity requirements. Specifically, source code should be available and must be provided with the code unless already available to the SOC. 6.4 Calibration Data The code will most likely need calibration data in addition to the Level 1 data files themselves. This data can be anything required. The SOC will provide a directory where these files will be placed on the SOC pipeline system, and the instrument pipeline code will be able to access them there. The combination of the Level 1 file (and detached PDS label) and the data provided must be sufficient to produce each Level 2 file. If housekeeping information (instrument or spacecraft) is needed, these must be already in the header (or tables) of the Level 1 file. If continuously varying values (e.g. temperature over many seconds, etc.) are needed, a FITS table will be written into the Level 1 file with this information. 6.5 Documentation All code and data files shall be accompanied by thorough documentation. The code itself should have clear and appropriate comments throughout. Error conditions shall be documented in the code as well (see section 6.6 for more on this topic). Documentation and code shall be written assuming that it will be read by someone years from now who is unfamiliar with the system. Understanding of the documentation should not rely on special scientific knowledge or specific domain knowledge. 6.6 Error Conditions and Integration Troubleshooting If there are any reasons the code might abort processing, these shall be defined, and the resulting action should be specified. Also, if such an abort happens, the reason should be noted in the status file written ("out_status" file described above). A contact person shall be specified who will be responsible for helping the SOC operators when unexpected errors occur. This person should be able to help with debugging and should also be available to respond and help in two days or less for consultation during the pipeline integration process. 6.7 PDS Archiving The Level 2 (output) files, as well as Level 1 files, will be archived to PDS. Therefore the format of the files shall meet PDS requirements. This includes size requirements set forth in the New Horizons Data Management and Archiving Plan (#05310-DMAP-01). PDS detached labels for the Level 2 files shall be generated by either the instrument pipeline code or by the SOC using a translation table (from FITS to PDS keywords). Which method is appropriate will be determined on an instrument by instrument basis. If generated by the SOC, "in_pds_header" and "out_pds_header" can be ignored in the instrument pipeline code. 6.8 Configuration Management All code, documentation, and calibration files will be put under configuration management at the SOC. Also, the necessary keywords shall be inserted into the Level 2 headers by the Level 2 pipeline code to specify the version of the code and data used to produce the Level 2 files. This ensures that data is traceable back to the correct code version. In addition, the Level 2 pipeline code shall insert, using header keywords, the calibration files used and the versions thereof (if applicable). 6.9 Pipeline Updates Updates to the instrument pipeline (including code, documentation, and calibration data) are to be delivered to SOC personnel for integration; all such updates will require appropriate documentation and will fall under SOC CM. The code will be checked in to the SOC configuration management after regression tests are run. Any special instructions or changes should be communicated to the SOC personnel, and a file containing release notes (called "RELEASE_NOTES") should accompany the update. The SOC personnel will notify the instrument team when the new update is in place and active. 6.10 Acceptance Review The instrument pipeline (including code, documentation, and calibration data) shall be subject to an acceptance review. 6.11 Longevity The code and all third party libraries and data files used shall meet the longevity requirements as specified in the New Horizons Longevity Plan (#7399-9009). Also, development, documentation, and testing of the pipeline shall adhere to these requirements. 7. ALICE INSTRUMENT DESCRIPTION 7.1 PERSI-Alice Instrument Overview PERSI-Alice is a UV spectrograph that is sensitive to Ultraviolet (UV) light in the range of 520-1870 Angstroms. The instrument consists of a telescope section with an off-axis parabolic mirror, and a spectrograph section that includes a diffraction grating and a microchannel plate (MCP) detector. The MCP detector is an electro-optical device sensitive to extreme and far ultraviolet light. It consists of a photo cathode-coated MCP surface that converts UV photons to electrons, an MCP Z-stack configuration of three MCPs to amplify the signal, and a two-dimensional double delay-line (DDL) anode readout array. The first (x) dimension provides the spectral location of the detected photon and the second (y) dimension provides one-dimensional spatial information. The DDL detector system outputs to the command-and-data-handling (C&DH) electronics the pixel location for each detected photon event. The pixel location (X,Y) is given as a pair of coordinates, spectral (X axis) and spatial (Y axis). The events are processed by the C&DH electronics. The C&DH is also the controller of Alice; it receives commands from the spacecraft, acquires data from the MCP detector system, and returns telemetry to the spacecraft. Science data telemetry generation is performed by the detector hardware but the C&DH also controls this function. Alice has two acquisition modes (see below) in which the spectral/spatial data from the detector are processed by the C&DH subsystem. All following descriptions assume a nominal operating instrument. The instrument hardware also provides a basic, hardware-controlled, default pixel list acquisition mode, which is activated when the instrument control hardware detects multiple successive software failures. This mode is called the 'State Machine Mode' (SMM); for a description of this mode, the reader is referred to the instrument C&DH hardware description. Once this mode is activated any software control of the acquisition operation is disabled, until the instrument C&DH is reset either by a power cycle reset or by a hardware S/C reset. Data recorded on New Horizons are sent to the ground via the Deep Space Network. From there the data are sent to the Mission Operations Center (MOC) at the Applied Physics Laboratory (APL). The Science Operations Center (SOC) at the Southwest Research Institute (SwRI) in Boulder retrieves new data from the MOC daily. The data from the MOC are in a packetized form. Software pipelines at the SOC convert the data from these raw packets into FITS (Flexible Image Transport System) format files with scientifically useful and calibrated data. The initial processing sorts the packets into FITS files (as images and data tables) with useful header keywords. These keywords include the mode of the observation, exposure and timing information, and basic pointing information of the instrument boresight. The initial processing also adds relevant housekeeping telemetry (temperatures, voltages, etc.) in a binary table as an extension to the FITS file. The calibration pipeline then performs basic scientific calibration on these data. 7.2 "Raw" Data Specifics The term "raw" as used here refers to CODMAC Level 2 data 7.2.1 Data Format PERSI-Alice High Speed data frame formats: Science data frames consist of "raw" science data of 32768 16-bit words in size, consisting of 1 word (16 bits) of frame ID header information and a 32767-word data block. Science data frames are generated by the acquisition hardware and sent to the spacecraft via the dedicated LVDS interface. Data are transferred at a rate of 1 Mbit/sec, resulting in a frame transfer time of 557 milliseconds. The spacecraft tags the received data with a receive time (referred to as 'collect time') and stores the data in the Solid State Recorder (SSR). Note that these science frames are not identical to the CCSDS telemetry packets that are used to transfer the science data to the ground. The generation of telemetry (TM) packets from the frames stored in the SSR is handled by the spacecraft and multiple CCSDS TM packets are used to transfer a single science frame. To identify the Science frames, a single 16-bit word header is inserted into each frame. This header is generated by the acquisition hardware and includes the information listed in Table 7 1. The complete header word of the most recently generated science frame is included in the housekeeping TM packet to allow for correlation between the two data streams (low and high speed) after the data are received by the ground system. The contents of the remainder of the data frames (the 32767-word data block) depend on the acquisition mode: - Pixel List: Each word in the data block from a pixel list exposure describes a photon event or a time hack. Photon events are indicated by bit 15 having a value of zero and time hacks are indicated by bit 15 having a value of one. For a photon event, the remaining 15 data bits encode the location of the detected event consisting of a 10-bit encoded spectral location (X) and a 5-bit encoded spatial location (Y). The time hack is used to provide temporal information about the photon events. The acquisition hardware will generate and insert time hacks in the frame on a periodic basis; the frequency of the time hacks is configurable (by command) for each acquisition in a range of 4 - 512 ms. For a time hack, the remaining 15 bits contain an incrementing counter that counts the number of 4 ms periods. This value allows for verification and correction of timing in case of lost frames or packets. - Histogram: Each word in the data block from a histogram exposure is a 16-bit "counter" giving the number of photon events detected at each specific X,Y location on the detector. The format of the detector is 1024x32 pixels, which are the spectral and spatial dimensions respectively, i.e., there are 1024 spectral elements along the X-axis and 32 spatial elements along the Y-axis, giving a total of 32768 values (however, the first word always contains the header word). The counters are stored row-wise, corresponding to the spectral dimension indexing most quickly. These counters saturate at a maximum value of 65535 to indicate a completely filled counting bin, meaning that the counters do not wrap around. In addition some special data words (header cross-identification and pulse height distribution) are overlying the lower left-hand corner of the actual array in a block of 4 (spatial) x 32 (spectral) words. This usage doesn't affect the science data contents because detector events do not occur in this region. In this over-written block, the first row contains the header cross-identification word, the second and third rows contain the 64 words of pulse height information, and the fourth row is filled with zeros. A "pulse height" is the amplitude of a photon event, and this pulse height distribution (PHD) shows the number of events in a 64-bin distribution with 6-bit resolution; the value in each PHD bin gives the number of events that occurred with the particular amplitude associated with that bin. These PHD counters also saturate at a value of 65535. So a single photon event is counted both in the spectral/spatial array and in the pulse height list. The PHD is used as a diagnostic for the health and behavior of the detector. For test purposes the instrument can fill the memory with known deterministic patterns so the interfaces to the spacecraft and ground can be verified. The instrument software allows for the generation of 5 different test patterns. 7.2.1.1 Histogram FITS file: The primary data unit in the FITS file is a 2-D raw histogram frame (also referred to as an "image") consisting of 1024x32 16-bit integer numbers. Note that the Alice instrument data are unsigned 16-bit integers (giving values of 0 to 65535). The first data extension in the FITS file contains the 64-element pulse height distribution (PHD) that was acquired together with the histogram. When the Level 1 pipeline saves the PHD to this extension, it then zeros out that part of the histogram array. The second data extension contains a 141 column by t row binary table, where t is the time of the exposure in seconds, of housekeeping values recorded during the observation (the housekeeping report rate is 1 Hz). FITS File Storage Location Description Primary Data Unit (PDU) Raw Histogram image (uncorrected counts) Extension #1 Pulse Height Distribution (PHD) Extension #2 Binary Housekeeping Table 7.2.1.2 Pixel list FITS file: Upon receiving a pixel list frame, the Level 1 processing creates a ground-calculated "reconstructed histogram" from the received pixel list data and places it in the primary data unit of the FITS file; this enables an easy quick-look inspection of the pixel list data (e.g., using most FITS viewers that by default typically display the data in the primary data unit). The pixel list data itself can be hard to interpret, so this reconstructed histogram image is desirable to enable the scientist to determine, e.g., data quality, whether the target was in the field of view, etc... The first data extension contains the raw pixel list data set, which includes the full stream of photon events and time hacks. The second extension contains the count rate derived from the pixels list data. Each count rate bin shows the number of events that occurred between successive time hacks. The resolution of this count rate data set is determined by the hack rate used for the pixel list acquisition. The length of this vector is variable depending on the source flux and the hack rate. The third data extension contains a 141 column by t row binary table, where t is the time of the exposure in seconds, of housekeeping values recorded during the observation. FITS File Storage Location Description Primary Data Unit (PDU) Reconstructed Histogram image (uncalibrated counts) Extension #1 7.2.1.3 raw pixel list Extension #2 7.2.1.4 Count rate vector from pixel list data (sampled at time hack rate) Extension #3 Binary Housekeeping Table 7.2.2 Data Sources (High/Low Speed, CCSDS, ITF) 7.2.2 Data Sources (High/Low Speed, CCSDS, ITF) PERSI-Alice data are transferred via CCSDS packets that are packetized 7.2.by the spacecraft from the Solid State Recorder. The spacecraft will packetize the PERSI-Alice High Speed Telemetry data into CCSDS packages before sending the data to the ground. Different APIDs are used to distinguish the histogram and pixel list data packets (see Table 7 2). For the PERSI-Alice science frames the spacecraft will either use the "RAW" packetized format or the loss-less compressed format to transfer the data. In either case the result will be data encoded in CCSDS telemetry packets. The package APIDs listed in Table 7 2 are used to distinguish the different Alice CCSDS packets. The packet time as listed in the CCSDS packet represents the time at which the packetization operation was performed. The second time field contains the frame collection time as sent from a spacecraft perspective, meaning this represents the time at which the spacecraft received the frame from the instrument. As the instrument immediately sends the frame at the completion of the acquisition this corresponds to the time at which the acquisition was completed. Table 7-2: PERSI-Alice science APIDs Packetized "RAW" telemetry: The nominal data transfer method for the Alice pixel list science data is packetized "raw" data. Each CCSDS science packet can transfer a segment of up to 480 data bytes. In order to transfer a full PERSI-Alice frame of 32768 words (16-bits), 137 science packets are needed; the first 136 packets will all be full size segments of 480 bytes, the last packet will transfer the remaining 256 bytes. The grouping flags of the packets indicate the start and end segment within a complete frame transfer. Note the '#' marks in the following tables refer to the third digit of the APID, the valid digits for these numbers are indicated in Table 7 2. Table 7-3: PERSI-Alice CCSDS Packetized (RAW) science packet Lossless Compressed telemetry: The nominal data transfer method for the Alice histogram science data is lossless compressed data. When applied to pixel list data, the 'FAST' algorithm results in negligible compression rates, and occasionally in a 1% expansion, therefore lossless compression will generally not be used with pixel list data. The spacecraft uses the so called 'FAST' algorithm to compress the image data. The 'FAST' algorithm uses one-dimensional correlation between successive data elements to remove redundancy. Data is encoded in blocks of 16 successive science values, the first value of such a block is send in full 16 bits, the remainder of the block is encoded using successive differences, using an adaptive coding mechanism. Table 7-4: PERSI-Alice CCSDS Compressed (Loss-Less) science packet 7.2.3 Definition of an "Observation" An observation will be a single histogram image or one frame of a pixel list series. Each observation will be written to a separate FITS file. A pixel list resulting from a single exposure command may therefore produce many such frames, each of which will be saved as a separate FITS file. 7.2.4 S/C Housekeeping Needed in Level 1 Files (for Calibration) Spacecraft housekeeping that may be needed in the Alice pipeline include any temperature sensors on the spacecraft around the Alice instrument and the spacecraft-measured instrument bus voltage and power consumption on the different busses. Spacecraft measured temperatures related to Alice (ApId 0x00D and 0x08D): T_A.CDH_TEMP_ALICE_BRACK_BASE_00D T_A.CDH_TEMP_ALICE_1_00D T_A.CDH_TEMP_ALICE_2_00D PDU parameters related to Alice (ApId 0x009, 0x00a, 0x089 and 0x08a): ALICE_LVPS_A_VOLT_009 ALICE_LVPS_A_CURR_009 ALICE_LVPS_B_VOLT_009 ALICE_LVPS_B_CURR_009 ALICE_ACT_A_VOLT_00A ALICE_ACT_A_CURR_00A ALICE_ACT_B_VOLT_009 ALICE_ACT_B_CURR_009 Note that these temperatures currently are not used in the pipeline processing, but may be used in the future as the code and calibrations are revised. 7.3 "Calibrated" Data Specifics "Calibrated" data as used here refers to CODMAC level 3 data. 7.3.1 Algorithm for Pipeline Overview: The Alice calibration pipeline that is run at the SOC applies various calibrations to raw Alice data to convert the data from units of counts to flux units (photons/s/cm2). Four types of operations can be performed. They are, in order of application to the data: deadtime correction, dark correction, effective area correction, and flat field correction. These are described in more detail below. 7.3.1.1 Deadtime Correction The Alice detector electronics require a finite time to process a photoelectron pulse. As a result, if photoelectron pulses arrive too close together in time, the latter pulse(s) will not be recorded, resulting in an effective decrease in the sensitivity of the instrument that is a function of the count rate. The deadtime correction time constant is 18 microseconds. At input rates below 50 kHz, the detector electronics is non-paralyzable (fixed deadtime per event that is not re-triggerable). To calculate the detector output count rate, the following formula is used: C_out = C_in / (1 + C_in tau), where Cout is the output (i.e., detected) count rate and C_in is the input count rate. At a count rate of 1 kHz, the deadtime correction factor (tau) is approximately 1.02, while at 20 kHz, the deadtime correction factor is approximately 1.56. 7.3.1.2 Dark Correction The Alice detector electronics register photon events even when the aperture door is closed and the detector is not illuminated. The spatial distribution of dark counts is approximately uniform across the detector. However, there is some low-level 2-D structure to the dark counts. Alice observations made with the aperture door closed are summed together to create a "superdark". This superdark image is then scaled to the exposure time of an Alice science observation and subtracted from the data. During in-flight commissioning, these dark counts were measured at a rate of approximately 94 Hz across the entire detector. The primary source of dark counts is the spacecraft RTG. Dark exposures are made throughout the mission to monitor the background event rate and detector performance. 7.3.1.3 Effective Area The sensitivity of Alice to UV photons varies as a function of wavelength. It is convenient to think of the Alice sensitivity in terms of the effective area of the instrument. For a point source located at infinity, effective area is defined as the area of the surface that intercepts incident photons at the same rate as is detected by the Alice instrument. Dividing the observed count rate by the effective area yields the incident flux of photons. In general, effective area depends on the geometric size of the instrument aperture, reflectivities of the optical surfaces, sensitivity and quantum efficiency of the detector, etc. The Alice effective area curve (v003) is based on pre-flight estimates and calibration stars observed by both Alice and the IUE satellite. Above 1350 Angstroms, the effective area is derived from matching the stellar flux observed by Alice with that observed by IUE. Below 1350 Angstroms, the pre-flight effective area estimate has been scaled by a factor of 0.6 so that the long wavelength end of the pre-flight effective area estimate approximately matches the calibration derived from the stellar observations. 7.3.1.4 Flat Field When uniformly illuminated by a monochromatic source, the counts detected by the Alice instrument vary from pixel to pixel with a standard deviation of approximately 15%. This spatial variation in instrument sensitivity is the instrument flat field response. As of September 2007, no suitable observations have been made from which to derive an in-flight flat field calibration. Thus the flat field correction is currently disabled in the Alice pipeline code. Current plans are that during annual checkouts, a stellar scan along the slit could be used to generate a pseudo-flat field that could be used in the Alice pipeline. 7.3.2 Format of Calibrated Data 7.3.2.1 Histogram The primary data unit in the FITS file is a 2-D calibrated histogram frame consisting of 1024x32 array of 32-bit floating-point numbers. The units of the histogram image are photons/s/cm2. The first data extension in the FITS file is a 1024x32 array of 32-bit floating numbers containing the uncertainty in the histogram image. The second data extension is a 1024x32 element array containing the wavelength for each pixel in the histogram image. The third data extension is the 64-element PHD, identical to that in the raw data. The fourth data extension is an array containing the number of photon events per second, sampled at a rate of 1 Hz. The fifth data extension is the 141 column by t row housekeeping row as in the raw data. FITS File Storage Location Description Primary Data Unit (PDU) Calibrated Histogram image (photons/sec/cm2) Extension #1 Uncertainties in histogram data values Extension #2 Wavelength Image (Angstroms) Extension #3 Pulse Height Distribution (PHD) Extension #4 Count rate vector from HK (sampled at 1 Hz) Extension #5 Binary Housekeeping Table 7.3.2.2 Pixel List The primary data unit in the FITS file is a 2-D calibrated reconstructed histogram image consisting of a 1024x32 array of 32-bit floating-point numbers. The units of the histogram image are photons/s/cm2. The first data extension in the FITS file is a 1024x32 array of 32-bit floating numbers containing the uncertainty in the reconstructed histogram image. The second data extension is a 1024x32 element array containing the wavelength for each pixel in the reconstructed histogram image. The third data extension contains a binary table of 5 columns and rows for each photon event. The five columns are the X (spectral) position of each photon event, the Y (spatial) position of the photon event, the wavelength of the photon event, the cumulative number of elapsed time intervals (starting from 0 at the beginning of the file), and the ephemeris time (et), i.e. seconds past J2000, at the time of detection. The fourth data extension is the count rate derived from the pixels list data, showing the number of events that occurred between successive time hacks. The resolution of this count rate data set is determined by the hack rate used for the pixel list acquisition. The length of this vector is variable depending on the source flux and the hack rate. The fifth data extension contains the binary housekeeping table. FITS File Storage Location Description Primary Data Unit (PDU) Reconstructed Calibrated Histogram image (photons/sec/cm2) Extension #1 Uncertainties in reconstructed histogram data values Extension #2 Wavelength Image (Angstroms) Extension #3 Binary Pixel List Table (X, Y, wavelength, time hack #, MET) Extension #4 Count rate vector from pixel list data (sampled at time hack rate) Extension #5 Binary Housekeeping Table 7.3.3 7.3.3 Scientific Units For Histogram, units are counts (histogram), angstroms (wavelength array), and counts (PHD array). For Pixel List, units are counts (generated histogram), angstroms (wavelength array), counts, pixel location, and angstroms, and seconds (pixel list array), counts per second (count rate array). 7.3.4 Additional FITS and PDS Keywords Added Below is an example of the Mike pipeline keyword block added to the FITS header: COMMENT ============================================= COMMENT MIKE_BEG= 'Feb 15 16:12:57 2005' / START MIKE KEYWORD BLOCK MIKE_VER= '2.0 [2005 Feb 15]' /Version of Mike pipeline code K_MODE = 'ACQMODE ' / Keyword containing the mode name K_ETIME = 'EXPTIME ' / Keyword for the effective exposure time FILE_IN = 'test/ali_0000006498_0x4b3_eng_1.fit' / Input file for processing FILE_OUT= 'test/test_his.fit' / Output file after processing DIR_CAL = 'cal/ ' / Directory of calibration data DIR_DONE= ' ' / Directory to put raw data after processing BADFILE = ' ' / FITS file of bad pixel mask array BADFLAG = -1 / Bad pixel mask flag BADVALUE= -666 / Bad pixel value DEADFILE= 'deadtime/ra_dead_002.txt' / Deadtime correction file DEADFLAG= 1 / DEADTYPE= 'FUN ' / Correct using FUNction or lookup table (LUT)? DEADCORR= 'TOTAL ' / Correct by TOTAL or each PIXEL count rate? BIASFILE= ' ' / Bias image filename BIASFLAG= -1 / Bias correction flag DARKFILE= 'dark/ra_dark_001.fit' / Dark image filename DARKFLAG= -1 / Dark correction flag FLATFILE= 'flat/ra_flat_001.fit' / Flag field image filename FLATFLAG= -1 / Flat field correction flag FLATNORM= 'AVERAGE ' / How to normalize flat field WCALFLAG= 0 / Wavelength calibration flag WCALPRO = 'alice_wavecal' / IDL program to perform wavelength calibration WCALPARS= 'T_DELECC' / keywords for parameters to use for wave cal AEFFFLAG= 1 / AEFFPRO = 'alice_aeff' / IDL program to get effective area AEFFPARS= 'T_DELECC' / keywords for parameters to get effective area LOG_FILE= 'test/log.out' / Filename to save log file (default = append to LOG_MAIL= ' ' / address (if any) to e-mail log file MIKE_ERR= 1 / MIKE_END= 'Tue Feb 15 16:12:57 2005' / END MIKE KEYWORD BLOCK COMMENT ============================================= COMMENT 7.3.5 Hardware/OS Development Platform Dell Linux, Redhat 7.2; Apple G5 Power PC and PowerBook G4, OS X v10.4 7.3.6 Language(s) Used IDL 7.3.7 Third Party Libraries Required IDL Astro (http://idlastro.gsfc.nasa.gov/) 7.3.8 Predicted Execution time A few seconds per file. 7.3.9 Contact/Support Person(s) Andrew Steffl, Joel Parker, and Maarten Versteeg 8. LEISA INSTRUMENT DESCRIPTION 8.1 Overview LEISA is an infrared imaging spectrometer. The detector is a 256x256 pixel array. Spectral separation is done with a wedged optical etalon filter placed in close proximity to the detector array. The filter is made of two pieces, a high spectral resolution (lambda/delta(lambda)=580) segment and a low spectral resolution (lambda/delta(lambda)=280) segment, bonded together. The detector-filter assembly is located at the plane of focus of the Ralph telescope where a 2-D image is recorded simultaneously with the infrared spectrum of the scene. The layout for the filter assembly is shown in Figure 8-0: Layout for LVF Filter Assembly. The wavelength range of the sensor is 1.225-2.5 nm for the low resolution segment and 2.1-2.25 microns for the high resolution segment. The wavelength of transmission of the filter varies along one axis and is constant in the other. Lines of constant wavelength are aligned with the row direction of the detector array. The number of pixel-limited spectral channels is the number of rows of the detector, excluding a number of rows (4) obscured by opaque adhesive at the bond joint between the two filter segments. Figure 8-0: Layout for LVF Filter Assembly The LEISA detector array is a Rockwell PICNIC device. It is read out by the Ralph electronics in Correlated Double Sample (CDS) mode. The signal is converted to a 12-bit value using the middle 12 bits of a 16 bit analog to digital (A/D) converter. There are two data transfer modes, one in which both signal and reset level data are returned (un-subtracted mode), and the other in which the reset level is subtracted from the signal level and only the difference is returned (subtracted mode). The image cube is recorded as a series of N image frames, with N determined by the length of the scan multiplied by the frame rate. Detector frame rate is adjustable between 0.25 and 8 Hz in 1 ms steps. Each frame covers the complete range of wavelengths. LEISA is normally operated in a scanning mode, with the target moving through the image plane, row by row. Slicing the image cube along one row gives a scanned image of the target in one wavelength. Co-registering each wavelength image (removing motion and optical distortion) yields an IR spectrum of the target. Data recorded on New Horizons is sent to the ground via the Deep Space Network. From there the data is sent to the Mission Operations Center (MOC) at the Applied Physics Laboratory (APL). The Science Operations Center (SOC) retrieves new data from the MOC daily. The SOC software pipelines convert the data from the MOC archives into FITS (Flexible Image Transport) files with scientifically useful and calibrated data. The SOC first sorts the packets into image cubes of raw (12-bit) sensor counts with useful header keywords. These keywords include the recording mode of the observation, timing information and basic pointing information of the instrument boresight. The raw processing also gathers housekeeping (H/K) telemetry from the Ralph instrument into a table. Once the raw processing is complete, the SOC produced a calibrated data set for each observation. 8.2 Raw Data Specifics 8.2.1 Data Format Raw Dataset: The SOC stores the LEISA data cubes in Band Interleaved by Line (BIL) order, i.e. image frames are stored sequentially. To re-order LEISA images as received from the spacecraft, the SOC does the following to each frame of data: 1. de-interlace by quadrant 2. reverse the Y direction 3. rotate 180 degrees The resulting frames from LEISA have the (0,0,0) element of the 3-D array corresponds to the location of wavelength 2.5 microns on the LEISA filter at the minimum X axis location in the image in the first frame. The SOC raw data product is a FITS format data file and PDS detached label file. Ancillary data for an observation is placed in the primary header of the FITS data file. The 256X256XN data cube is stored in the primary data unit as an array of integers. The first FITS extension is a binary table of Ralph housekeeping data. Outline of the raw FITS file: - Primary HDU - Raw 12 bit image counts - Primary Header (FITS + pointing + observation keywords) - 256 X 256 X N integer point array - Extension 1 - Binary table of Ralph housekeeping - Ext. Header (keywords + binary table definition) - Ext. Binary Table (115 X S binary table of Ralph housekeeping data) *[N is the number of data image frames in the observation, S is the number of seconds in the observation] **[In the case of un-subtracted readout mode, frames alternate between read and reset signal levels] Image Data The primary data unit contains the raw spectral image data. Values recorded by the instrument with 12 bit precision are stored as 16 bit integers. Housekeeping Data Housekeeping data generated by the Ralph instrument is stored in Extension 1 as a binary table. The first field in each row of the table is mission elapsed tine (MET). Table entries are sorted by increasing MET. The time interval between each table entry is fixed, one second per entry, unless there is missing data. Pipeline Processing The limits of an observation are established by the SOC using information in each telemetry packet of an observation sequence. 8.2.2 Data Sources (High/Low Speed, CCSDS, ITF) Ralph housekeeping data is transmitted in the form of CCSDS packets. One housekeeping packet is produced by the Ralph instrument each time a spacecraft PPS (pulse per second) signal is received. State information is gathered, time tagged, and written to the low speed bus in CCSDS packet form. The CCSDS packets are transmitted during the next DSN pass. LEISA image data is transmitted in the form of CCSDS packets produced by the spacecraft compression/packetization routines from the data written to the high-speed bus. The LEISA detector has 4 output channels, one for each array quadrant. The first 4 elements of the data stream are the first pixels from each quadrant. The second 4 elements are the second pixels from each quadrant, and so on. One 'line' of data in this order is 4x 128 pixels long, and is not the same as a line in the final image. During the observation, image data is written to the spacecraft high speed bus. These data are not automatically transmitted. A compression/packetization routine is scheduled some time later that converts the Ralph sensor counts to a packetized form. The data can be packetized without compression, with lossless compression, or with lossy JPEG compression. The packetization routine can also process a sub-frame area of interest, or a more complicated sliding subframe that tracks the image target as the scanning observation proceeds. In un-subtracted readout mode, the spacecraft interface supports only uncompressed downlink. Regardless of windowing or compression, the SOC raw data processing reassembles the data into a full 256X256XN data cube. 8.2.3 Definition of an "Observation" Science Operation The size of the LEISA detector is 256 x 256 pixels. One read out of all 65,536 detector pixels is called an image or frame. The data content of a LEISA "frame" is consistent with the S/C definition of the term. The processing definition of "image" is consistent with the optical definition except that the order of pixel elements is different in the electronic data stream. An observation is a sequence of frames. The number of frames per observation is variable. Pixel values are recorded with 12 bit precision. One image contains 65536 pixels x 1.5 bytes/pixel = 98304 bytes of data, however, data is stored as 16 bit values in the SOC data files. For normal science observations the Ralph electronics use the value of the frame rate to minimize smearing by compensating for the spacecraft motion and scan rate relative to the target. Reset levels are stored temporarily by the electronics and subtracted from read levels (subtracted mode). The difference in read and reset levels is transferred to the spacecraft. Alternately, the instrument can be forced to use a fixed frame rate value. Un-subtracted Read-out: There is a voltage offset for each LEISA quadrant to assure the sensor signal will be in the correct range of the A/D. The offset values are set from a table when Ralph is powered on. Un-subtracted mode can be used to evaluate the offset values. In un-subtracted readout mode, reset levels are not subtracted from read levels. Both read and reset are transferred. The number of values in one data frame of the un-subtracted mode (131,072) is 2x twice that of subtracted mode. Read and reset values are interleaved by data line and the number and order of the pixel elements in a line are the same as for subtracted mode readout. The reset of a pixel occurs after the integrated signal is read, so read levels correlate with reset levels recorded in the preceding frame. The spacecraft interface supports only uncompressed packetization of the full LEISA array for un-subtracted readout mode. 8.2.4 Housekeeping Needed in Level 1 Files (for Calibration) Most of the H/K values are used for engineering troubleshooting and not needed for data processing. Housekeeping data that are important to further processing (see the following table) are stored in header keywords. Keyword Description SIDE Instrument hardware side DETECTOR Always LEISA for LEISA data FILTER Always WEDGE for LEISA data LEI_OFFx Value used to set voltage offsets for the four LEISA quadrants. x=1-4 LEI_RATE Time between LEISA readouts (ms) 8.2.5 Science Data and/or Housekeeping Requirements Other important information is determined by the SOC while processing the raw observation data. These values are also stored as keywords in the FITS header. Keyword Description MET510 The MET of the Ralph housekeeping packet that marks the start of an observation, used to determine the observation start time and frame rate TRUE510 Is the 0x510 real or assumed from a gap? SCANTYPE Always LEISA for LEISA data LEI_MODE RAW for un-subtracted readout mode SUBTRACTED for subtracted readout mode STARTMET Actual start time of first integration, in MET (s) EXPTIME LEISA exposure time (s). Same as RALPHEXP. There is zero dead time between frames so the frame rate is exactly 1/EXPTIME SPICE and SPICE Kernels: The SOC maintains an archive of SPICE kernels that describe the position and attitude of the spacecraft as any time in the mission. The kernels are used to calculate many values describe the instrument pointing during each observation, stored as header keywords with the prefix SPC. The names of the SPICE kernels used to process the observation are stored as header keywords with the prefix SPCK. 8.3 Calibrated Data Specifics 8.3.1 Algorithm for Pipeline There are six processing steps applied to the raw LEISA data to produce the calibrated output: 1. Validate raw image file 2. Preprocess un-subtracted mode data 3. Process A/D rollover pixels 4. Convert raw counts to calibrated values 5. Compute pointing data 6. Construct FITS file Validate raw image file The input file is validated to assure the data is ready for further 8.processing. Checks are for valid mission, instrument, mode, and 8.image array size. The values of important keywords are validated 8.and collected in this step. Preprocess un-subtracted mode data If the data readout was in un-subtracted mode, the reset values are subtracted from the read values in this step and the rest of the processing is the same, regardless of readout mode. Process A/D rollover pixels There are two instances where the subtracted raw data value will be off by exactly 4096 counts. The first is when the subtraction of the reset count results in a negative number . This happens because of array noise and read noise and results in small negative numbers being returned as large positive number.s. The second instance is when the subtraction of the reset count results in a number greater than 4095 (12 bits). This happens because the most significant bit of the A/D is not read. A count higher than 4095 is normally considered outside of dynamic range of LEISA, but can be corrected on a case by case bases. If a file identifying rollover pixels for the observation exists (see next paragraph), the identified pixels are corrected for rollover. If no file exists, any subtracted count greater than 3850 is considered to have rolled over, and 4096 is subtracted from the raw count value. This is a good first-cut since the observations will not be designed to return signal counts this high. During initial image analysis by the Ralph team, each observation is analyzed in detail. A file identifying rollover pixels is generated which identifies the pixels that are deemed to need rollover correction. The case where the read count is higher than 4095 can be detected by analyzing surrounding pixels and by watching the target scan through the array. These are also included in the rollover file. Once this file is installed on the SOC, processing of the calibrated data for the observation will automatically use the rollover file instead of the default processing. Convert raw counts to calibrated values There are 8 values that make up the conversion between raw signal count and calibrated signal value. 1. Electronics induced readout signal 2. CCD flat field 3. Calibration offset 4. Calibration gain 5. Integration time 6. Filter width 7. Pixel solid angle 8. Gain correction The electronics induced readout signal is a base signal that does not depend on integration time. It has been derived from studies of the dark frames of flight data and is subtracted from the raw signal count. The CCD flat field is derived from laboratory data and refined with in flight observation data. The flat field changes slightly as the mission progresses so a different flat field can be defined for an individual observation or a range of observations. The actual flat field used in the processing is included in the output FITS file. The calibration offsets and calibration gains are derived from laboratory data and refined with in flight observation data. These values can change as the mission progresses. Different calibration values can be defined for an individual observation or a range of observations. The actual calibration values used in the processing are included in the output FITS file. The integration time is divided into the calculated calibrated counts. The filter width of each pixel is divided into the calculated calibrated counts. The pixel solid angle is divided into the calculated calibrated counts. The gain correction is divided into the calculated calibrated counts. All of these values are derived from laboratory data and refined with in flight calibration observation data. They are updated, as needed an applied to the observation data automatically when re-processed Compute pointing data The pointing for each pixel of each frame is computed using the timing information from the observation, reconstructed ephemeris and attitude files, and knowledge of the optical distortion of the instrument. One array is generated giving the cartesian pointing vector of each pixel in the LEISA array. This is a function only of the optical distortion of the system. A second array is generated giving the rotation quaternion of the instrument boresight into the J2000 reference frame for the middle of each exposure. By rotating the pointing vector of a pixel by the quaternion for the image frame, the J2000 pointing vector of each pixel can be derived Construct FITS files A FITS file is constructed to store all the calibrated image data and related processing data. 8.3.2 Dataflow Block Diagram Figure 8-1 8.3.3 Data Format Calibrated Dataset The calibrated LEISA data are stored in Band Interleaved by Line (BIL) order, exactly as the raw data is stored. The resulting frames from LEISA have the (0,0,0) element of the 3-D array corresponds to the location of wavelength 2.5 microns on the LEISA filter at the minimum X axis location in the image in the first frame. The calibrated data product is a FITS format data file and PDS detached label file. Ancillary data for an observation is placed in the primary header of the FITS data file. The 256X256XN data cube is stored in the primary data unit as an array of floating point numbers. The FITS extensions are outlined below. Outline of the calibrated FITS file: - Primary HDU - Calibrated image data - Primary Header (FITS + pointing + observation keywords) - 256 X 256 X N floating point array - Extension 1 - Center wavelength and filter width for each pixel - 256 X 256 X 2 floating point array - Extension 2 - Cartesian pointing vector for each pixel - 256 X 256 X 3 floating point array - Extension 3 - Flat field correction for each pixel - 256 X 256 X 1 floating point array - Extension 4 - Radiometric gain and offset for each pixel - 256 X 256 X 2 floating point array - Extension 5 - Error estimates for each pixel - 256 X 256 X 1 floating point array - Extension 6 - Data quality flags for each pixel - 256 X 256 X 1 Integer array - Extension 7 - Ephemeris time and quaternion for each frame - 5 X N floating point array - Extension 8 - Binary table of Ralph housekeeping - Ext. Header (keywords + binary table definition) - Ext. Binary Table (115 X S binary table of Ralph housekeeping data) *[N is the number of data image frames in the observation, S is the number of seconds in the observation] For a description of the contents of the FITS extension, see the above section describing the SOC calibration processing. Calibrated Image Data The Image Data Unit of the Level 2 file contains data expressed in physical units useful for scientific interpretation. The instrument pipeline converts the data values of raw instrument counts to radiance units, W/cm2/sr. Data quality flags The data quality flag is set if there is a known problem with the given pixel. Quality Flag Value Description 0 Good pixel 1 Defect in one of the calibration files 2 Flat field out of bounds 4 Known CCD defect 32 Bad pixel not in any of above categories 8.3.4 Extra FITS Extensions (planes) and Their Definitions See above 8.3.5 Scientific Units Radiometric units: W/cm2/sr 8.3.6 Additional FITS and PDS Keywords Added See above 8.3.7 Hardware/OS Development Platform The software for processing the raw and calibrated data files has been developed on the SOC computers, running GNU/Linux/i686 Version 2.6.17-1.2142_FC4. 8.3.8 Language(s) Used The software for processing the raw data files is written in Python. The software for processing the calibrated data files is written in C, with a Perl script wrapper. 8.3.9 Third Party Libraries Required Python SPYCE interface library Python MySQLdb interface library Independent JPEG Group's JPEG software CSPICE processing library CFITSIO processing library 8.3.10 Calibration Files Needed (with Quantities) The instrument software package will include additional datasets needed for calibrating the data. All The calibration software requires additional datasets needed for calibrating the data, as describe in the above sections. The instrument pipeline maintains version control on calibration datasets, as with calibration procedures. Some calibration files are associated with specific observations, some are associated with a range of observations. The estimated number of calibration files needed for the mission is 300, totaling 3 GBs. 8.3.11 Memory Required 500MB 8.3.12 Temporary File System Space Needed 500MB 8.3.13 Predicted Size of Output File(s) Up to 500MB 8.3.14 Predicted Execution time Processing time for raw data files is approximately 3 seconds per image frame. Processing time for calibrated data files is approximately .1 seconds per frame 8.3.15 Contact/Support Person(s) Allen Lunsford Dennis Reuter 301-286-2042 Donald Jennings 301-286-7701 Laddawan Miko 301-286-2166 8.3.16 Maintenance Schedule (Code/Data Updates, Documentation) The LPS is installed by extracting files from an archive. The sub-directory structure for the software package is created during extraction. Symbolic links to external directories may be substituted for default directory references. The shell for execution of the LPS is tcsh/csh. Changes will be made to shell initialization files; new elements are appended. Instructions for configuring the shell environment are given. A guide to installation and setup is included with the LPS package. An initial period of testing and refinements is expected. Pieces of the software are tested separately during development. LPS modules are re-tested upon installation to the SOC. Sample datasets are provided to verify the function of software. Integrated testing of the instrument pipeline under the control of the SOC MDM is performed in accordance with the SOC. The instrument software engineer is available exclusively to the SOC to support the integration of pipeline software. Changes to calibration datasets are made as needed. A facility is provided by the SOC so that software changes are reversible. A LEISA team member will be available to assist SOC operators in responding to unexpected errors in the instrument pipeline. Persons supporting the LEISA Instrument Pipeline software are listed above. The LEISA instrument pipeline is developed at GSFC. The first fully functional version (v0) of the software is tested on the GSFC computer system. In advance of completion of LPS v0, a skeleton software package will be made available to the SOC. The skeleton software is a callable code set which installs and functions as the instrument pipeline but does not perform calibration and book keeping on instrument data. Completion of software Version 0 encompasses the creation of instrument calibration files. Documentation including instructions for installation and setup will be delivered to the SOC with Version 1 of the LPS. Updates of software after Version 1 are performed on an as needed basis. Table 8-1 9. LORRI INSTRUMENT DESCRIPTION 9.1 Overview The LOng Range Reconnaissance Imager (LORRI) is a narrow angle (FOV=0.29degrees), high resolution (IFOV=5 microrad), Ritchey-Chretien telescope with a 20.8 cm diameter primary mirror, a focal length of 263 cm, and a three lens field-flattening assembly. A 1024 x 1024 pixel (optically active region), back-thinned, backside-illuminated CCD detector (model CCD 47-20 from E2V) is located at the telescope focal plane and is operated in standard frame-transfer mode. LORRI does not have any color filters; it provides panchromatic imaging over a wide bandpass extending approximately from 350 nm to 850 nm. The LORRI telescope has a monolithic silicon carbide structure, built by SSG Precision Optronics, Inc., is designed to maintain focus over the entire operating temperature range (-125 C to +40 C) without a focus adjustment mechanism. A detailed description of the design and fabrication of LORRI can be found in the paper by Conard, et al., "Design and fabrication of the New Horizons Long-Range Reconnaissance Imager" in SPIE proceedings 5906-49, 2005. A detailed discussion of the performance of LORRI, as measured during calibration testing before launch, can be found in the paper by Morgan et al., "Calibration of the New Horizons Long-Range Reconnaissance Image" in SPIE proceedings 5606-49, 2005. LORRI is a supplemental instrument on New Horizons and is not needed to meet the baseline scientific objectives of the mission. Nevertheless, LORRI adds significant capabilities to New Horizons, including the highest available spatial resolution (50 m/pixel at the Pluto closest approach distance of 10,000 km) and redundancy for the primary optical imager, MVIC on Ralph. The exposure time for LORRI is adjustable in 1 msec increments from 0 ms to 29,967 msec. However, exposure times will normally be limited to < 150 msec to prevent image smear associated with spacecraft motion during observations. Initially, the shortest useful exposure time was expected to be ~40 msec owing to frame transfer smear associated with the transfer of charge from the active CCD region to the storage region, during which time the active region remains exposed to the image scene because LORRI has no shutter, but an improved frame transfer smear removal algorithm was developed that now permits exposure times as little as 1 msec. The LORRI exposure time can be commanded to a specific value, or LORRI can be operated in "auto-exposure" mode, in which the LORRI flight software sets the exposure time automatically based on the signal level in a previous image. In auto-exposure mode, the algorithm used to set the exposure time depends on several adjustable parameters that are stored in an onboard table. The optimal values for these table parameters vary with the type of scene being observed, which means that new table loads may be required prior to some observations. Although the LORRI auto-exposure mode worked well during ground testing, no decision has yet been made on whether it will be used in-flight during encounter observations. LORRI can also be operated in "rebin" mode, in which case the signal in a 4 x 4 pixel region is summed on-chip to produce an active region that is effectively 256 x 256 pixels covering the entire 0.29 degrees FOV. The main purpose of this mode is to provide high sensitivity acquisition of a Kuiper Belt object (KBO), which requires an exposure time of ~10 sec. Although LORRI rebin mode may never be used for science observations, the LORRI pipeline is still required to calibrate rebinned images. Figure 9-1: Cutaway Views of LORRI 9.2 Raw image Specifics 9.2.1 Data Format The raw image data is organized in a FITS file. The primary header 9.and data unit (HDU) is used to store the reconstructed image from 9.telemetry. Additional data are stored in the extensions of the 9.file. The two tables below contain a description of the layout for 9.the extensions for raw data. As described previously, LORRI operates in two binning modes: 1x1 and 4x4. For the 1x1 binning mode, the raw image dimensions are 1028x1024 where columns 0 through 1023 are the optically active region of the CCD and the remaining columns (1024-1027) are from optically inactive region (dark columns) of the CCD and represent a temperature-specific measurement of the bias value. For the 4x4 binning mode, the raw image dimensions ar 257 x 256 where columns 0 through 255 are optically active and column 256 for the dark column. Table 9-5: Raw FITS file extension layout 9.2.2 Data Sources (High/Low Speed, CCSDS, ITF) The LORRI high-rate data is delivered to the Instrument Interface card over a low-voltage differential signal (LVDS) interface and is then transferred to the SSR through the spacecraft high-speed PCI bus by the C&DH software. The image data is stored directly on the SSR and packets are generated by command to the C&DH software as is described in the table below. The APID from which the image originated is part of the filename, so this mapping may provide some assistance in decoding the filenames retrieved from the SOC. Table 9-6: Low Rate Instrument Telemetry Description Table 9-7: LORRI high-speed telemetry description 9.2.3 Definition of an "Observation" Each LORRI image is an "observation." 9.2.4 Housekeeping Needed in Raw Image Files (for Calibration) No special requirements other than pointing 9.2.5 Raw Science Data and/or Housekeeping Requirements No special requirements 9.3 Calibrated Image Specifics 9.3.1 Algorithms for Pipeline Calibration Process The calibration of LORRI images potentially involves all of the following steps: 1) Bias subtraction 2) Signal linearization 3) Charge transfer inefficiency (CTI) correction 4) Dark subtraction 5) Smear removal 6) Flat-fielding 7) Absolute calibration Ground testing has demonstrated that the linearization, CTI, and dark subtraction steps will not be needed, so they are not described below. Nevertheless, the LORRI pipeline architecture will be maintained to allow these additional steps to be incorporated quickly, if in-flight data suggest they are needed. The LORRI pipeline software consists of a series of IDL routines that implement the above processing steps. In general, the IDL routines have the following naming convention: lorri_function.pro, where "function" refers to the specific task performed by that routine. (The "pro" extension will be omitted below when discussing specific routines.) Each routine typically has several command line arguments and keywords that specify the input and output files and, possibly, parameters for tailoring the routine for particular circumstances. The routines that perform the bias subtraction, the smear removal, and the flat-fielding are described below. No special routines are provided to perform the absolute calibration. Instead, the absolute calibration is performed using keywords provided in the FITS header, as described further in Section 9.3.1.4. 9.3.1.1 Bias Subtraction If an image has an associated "dark" image (i.e., an image taken with the same exposure time but without any illumination), then the debiased image is simply the difference of those two images. This was usually the case during on-ground testing when images taken of a scene were immediately followed by images taken with the scene blocked (i.e., an obstruction was placed in the optical path to block the illumination). However, in-flight images may often be taken without accompanying darks either because of limitations on downlink bandwidth, or because a decision is made to take more target images at the expense of concurrent darks. In either case, the same pipeline routine will be used to debias the image (lorri_debias), but the algorithm employed is different in each case and different reference files are required. If in-flight data indicate that bias images are stable over time, many bias images will be combined (after filtering out clearly discrepant pixels) to produce a "super-bias" image. Then the median value of the inactive region of the image (i.e., the median of a 1024 row by 4 column region) is subtracted from the super-bias image to produce a "delta-bias" image. The IDL procedure that produces the delta-bias image is called lorri_delta_bias, but this routine is not part of the standard LORRI calibration pipeline; rather, it is an ancillary routine used to produce a calibration reference file. The delta-bias image will exhibit the pixel-to-pixel variation in the bias and will oscillate about zero. The bias subtraction for any new image is then a two-step process: 1) The median signal level in the inactive region of the image is subtracted from each pixel's value to remove the overall bias level, and 2) The delta-bias image is subtracted from the image created in the previous step to remove the pixel-to-pixel variation and produce the final, debiased image. Ground calibration testing showed that the overall bias level in step (1) above depends on the signal level in the last few columns of the active region of the CCD. The effect is produced by amplifier undershoot, which means that the bias level recorded by the pixels in the inactive region is smaller than the actual bias level. The magnitude of the effect depends on the signal level in the active region and on the column number in the inactive region and can be as large as ~12 DN. Thus, prior to computing the median signal in the inactive region (step 1 above), the intensities of all the pixels must be corrected for amplifier undershoot. This correction step is incorporated into the lorri_debias procedure. If the in-flight bias images vary significantly in time, separate bias images (i.e., 0 ms exposures) must be taken for each science image obtained. In this case, the bias subtraction proceeds exactly as performed during ground calibration testing, with the bias removal achieved by simple subtraction of the bias image from the science image. There are several drawbacks to this approach: (1) more images must be taken, which affects the data volume that must be stored on the on-board solid-state recorder, (2) more data must be downlinked, which may not be possible because of limited downlink bandwidth and/or the cost associated with the extra Deep Space Network (DSN) support required, (3) the signal-to-noise ratio (SNR) may be degraded because the bias subtraction no longer involves a high SNR reference file, and (4) fewer science images can be obtained because they have been displaced in the observing timeline by extra bias images. 9.3.1.2 Smear Removal LORRI does not have a shutter, so the target being observed illuminates the active region of the CCD whenever LORRI is pointed at the scene. In particular, the CCD continues to record the scene as the charge is transferred from the active portion to the storage area, and this results in a smearing of the observed scene. Fortunately, this smear can be removed to high accuracy using the correction algorithm described below. When bright objects are observed, the readout smear makes the raw image difficult to use for analysis purposes. In the image of Jupiter below, the raw image is on the left and the calibrated image with readout smear (aka frame transfer smear) removed is on the right. Figure 9-8: Demonstration of Smear Removal The need for the readout smear removal arises from the operation of the frame transfer CCD used in LORRI, where first the image zone is flushed, then an exposure is taken, and finally the image is transferred into the storage zone. Hence a pixel of the raw image is exposed to the scene radiance from the corresponding geometrical element of the scene, but it is also exposed to the radiances of all the scene elements in the same image column during the image transfers. Thus the raw image is the superposition of the scene radiance and the signal acquired during frame transfers, which is called readout smear. The readout smear is removed as follows. Equation 9-1 The value for T_favg is dependent on the desired exposure time and has been determined empirically using in-flight data. The following table provides the appropriate values at different exposure times. Table 9-9 Value for T_favg from T_exp It should be noted that when the raw data is saturated, the resulting readout smear correction will be inaccurate. The algorithm relies on an accurate accumulation of charge in all rows of each column and if the raw data is clipped for lack of dynamic range to capture that integrated signal, the effect of readout smear cannot be completely and properly removed. This correction algorithm has been implemented in the IDL routine lorri_desmear. 9.3.1.3 Flat-Fielding Flat-fielding refers to the process of removing the pixel-to-pixel sensitivity variations in the image. An exposure obtained by illuminating the LORRI aperture uniformly with light is called a "flat-field" image. During ground calibration testing, flat-fields were obtained by using an "integrating sphere to provide uniform illumination. The light source was a xenon arc lamp with a spectrum similar to that of the sun. The absolute intensity of the input illumination was measured using a calibrated photodiode. For the panchromatic case, which is the one most relevant for flat-fielding LORRI images, the light from the xenon lamp was unfiltered. Flat-field images were also obtained by passing the light through bandpass filters centered at five different wavelengths spanning the range over which LORRI is sensitive, prior to injection into the reference sphere, in order to estimate the sensitivity of the flat-fields to the spectral distribution of the source. The spatial patterns in the flat-field images change fairly dramatically with wavelength. However, the variation in panchromatic flat-fields caused by differences in the spectral distribution of the illumination source should be much less significant. Indeed, panchromatic flat-field images produced using a tungsten lamp were virtually indistinguishable from those produced by the xenon lamp. Flat-fields were obtained at four different telescope temperatures (at standard laboratory room temperature, and at the lowest, nominal, and highest temperatures predicted for in-flight conditions), but no significant temperature variations in the flat-field images were detected. The flat-field reference file used in the LORRI pipeline was produced by averaging 100 flat-field images taken at room temperature using the xenon arc lamp as the light source, debiasing and desmearing the average image as described earlier, and normalizing the intensities in the active region to a median value of 1. If "S" (units are DN) is an image of a target that has already been desmeared and debiased, and if "FF" is the reference flat-field image, then the flat-fielded (i.e., photometrically-corrected) target image ("C"; units are DN) is given by: C = S/FF The flat-fielding correction is implemented in the LORRI pipeline by the routine lorri_flatten. If in-flight measurements indicate that the LORRI flat-field characteristics are different than those measured during ground calibration tests, new reference flat-field images must be obtained. Although LORRI has two internal reference lamps (sometimes referred to as "cal lamps"), the illumination pattern is highly non-uniform and, thus, not very suitable as a secondary flat-field standard. Various test measurements will be performed during the early portion of the mission to determine if scattered sunlight can serve as a suitable secondary flat-field standard. If there is a Jupiter encounter, smeared images of Jupiter might also prove to be useful as a secondary flat-field standard. In any case, there will be an attempt to monitor the flat-field characteristics of LORRI over time, and the reference flat-field image used by the LORRI pipeline will be updated as necessary to maintain an accuracy better than 1% in the correction of the pixel-to-pixel sensitivity variation, except possibly near the center of the field where image ghosts may compromise the quality of the reference flat-field (see further discussion below). During ground calibration tests, intensity artifacts caused by optical ghosts were observed near the center (roughly covering a 200 x 200 pixel region) of the flat-field images. Ray tracing of the optical system indicates that the intensity of the ghost image should be less than ~1% of the intensity produced by the direct illumination, but measurements indicated that ghost intensities have an amplitude of ~5-7% of the direct intensity for panchromatic illumination. The ghost intensity is scene-dependent with most (~80%) of the ghost signal arising from regions outside the nominal field-of-view of LORRI. There is a suspicion that at least some of the ghost signal is an artifact of the test conditions, and the reference flat-fields currently used by the pipeline do not include the ghost signal produced by the out-of-field light. Any flat-field data taken in-flight will be carefully scrutinized to search for any effects attributable to optical ghosts. Depending on those results, further modifications to the reference flat-fields may be required. There is also the possibility that different flat-field reference images may be required depending on the scene being imaged (i.e., a ghost subtraction step may be required prior to application of the flat-field correction under some circumstances). 9.3.1.4 Absolute Calibration (Conversion from corrected DN to physical units) The calibration software pipeline does not perform the conversion from DN to physical units because that conversion requires knowledge of the spectral distribution(i.e. color) of the target. Instead, various LORRI FITS header keywords ("photometry" keywords) are provided that allow users to convert from DN to physical units depending on the spectral type and spatial distribution (diffuse vs. point source) of the target. Photometry keywords are provided for targets having spectral distributions similar to Pluto, Charon, Pholus, Jupiter, and the Sun. The units adopted for the radiance (aka "intensity") of diffuse targets are ergs/cm2/s/sr/Angstrom. The units adopted for the irradiance (aka "flux") of point (i.e., unresolved) targets are ergs/cm2/s/Angstrom. Tables providing the values for the photometry keywords at the time of launch are given below. The latest (i.e., current) values of the photometry keywords are provided in the header of the calibrated image FITS file for the image being analyzed. The absolute calibration is achieved by specifying a keyword (RPLUTO) in the header of the calibrated image file that allows the user to convert a count rate ("C/TEXP" in DN/s/pixel, where "C" is the flat-fielded signal in a pixel and "TEXP" is the exposure time) for a resolved source into a radiance value ("I" in ergs/cm2/s/sr/Angstrom) at LORRI's pivot wavelength (specified by the FITS keyword PIVOT; see below), assuming that the spectrum of the target is identical to the globally-averaged spectrum of Pluto. The relevant formula is: I = C/TEXP/RPLUTO Similarly, the keyword RSOLAR allows the conversion of the count rate for a resolved source into a radiance value at the pivot wavelength assuming that the target has a solar-like spectral distribution: I = C/TEXP/ RSOLAR Finally, the keyword RPHOLUS allows the conversion of the count rate for a resolved source into a radiance value at the pivot wavelength assuming that the target has a spectral distribution identical to that of the centaur object 5145 Pholus, which may be a good analog for the reddest regions on Pluto: I = C/TEXP/RPHOLUS The current best estimates for these sensitivity keywords, based on ground calibration tests, are provided in the table below. In-flight calibration observations of photometric standard stars will be used to verify these values and to monitor them over time. Keyword Value [(DN/s/pixel)/(ergs/cm^2/s/sr/Angstrom)] RSOLAR 2.664 x 10^5 RPLUTO 2.575 x 10^5 RCHARON 2.630 x 10^5 RJUPITER 2.347 x 10^5 RPHOLUS 3.243 x 10^3 If users need conversions for other spectral distributions, they must derive those themselves using the LORRI spectral response function provided in the paper describing LORRI's in-flight calibration results. The pivot wavelength (PIVOT) is given by Equation 9-2, where "P" is the LORRI system quantum efficiency (i.e., fraction of photons detected) at wavelength " lambda". The current best estimate for the LORRI pivot wavelength is 6076 Angstroms. For unresolved sources (e.g., stars), the absolutely calibrated flux (also called "irradiance") at the pivot wavelength can be determined using keywords that are defined analogously to the photometry keywords discussed above for resolved sources. In the case of a source having a spectral distribution identical to that of a globally-averaged Pluto spectrum, the observed count rate integrated over the LORRI PSF ("CINT/TEXP" in DN/s, where CINT is the total number of flat-field corrected counts integrated over the image and "TEXP" is the exposure time) can be related to the flux ("F" in ergs/cm2/s/Angstrom) by: F = CINT/TEXP/PPLUTO Similarly, the flux at the pivot wavelength for a target having the same spectral distribution as the sun is given by: F = CINT/TEXP/PSOLAR And the flux at the pivot wavelength for a target having the same spectral distribution as 5145 Pholus is given by: F = CINT/TEXP/PPHOLUS The current best estimates for these sensitivity keywords, based on ground calibration tests, are provided in the table below. In-flight calibration observations of photometric standard stars will be used to verify these values and to monitor them over time. Keyword Value [(DN/s)/(ergs/cm2/s/Angstrom)] PSOLAR 1.066 x 10^16 PPLUTO 1.030 x 10^16 PCHARON 1.052 x 10^16 PJUPITER 9.386 x 10^16 PPHOLUS 1.297 x 10^16 Synthetic photometry techniques can be used to convert the fluxes derived in the manner described above to fluxes at other wavelengths, and then into standard UBVRI magnitudes in the Landolt (1992) photometric system, which is essentially identical to the Johnson UBV system combined with the Kron-Cousins RI system. The results described in the LORRI calibration paper can be used to derive fluxes for targets whose spectral distributions do not match the three cases discussed above. We provide below some examples showing how to convert from engineering units to physical units, for both diffuse and point targets. Consider a diffuse target whose spectrum is similar to that of Pluto. You should then use the RPLUTO photometry keyword in the header of the calibrated image file to convert a count rate ("C/TEXP" in DN/s/pixel, where "C" is the flat-fielded signal in a pixel and "TEXP" is the exposure time) into a radiance value ("I" in ergs/cm2/s/sr/Angstrom) at LORRI's "pivot" wavelength (specified by the FITS keyword PIVOT for the formal definition of the pivot wavelength): I = C/TEXP/RPLUTO Similarly, the photometry keywords RSOLAR, RCHARON, RJUPITER, and RPHOLUS should be used to convert count rates into radiance values at the pivot wavelength assuming that the target has, respectively, solar-like, Charon-like, Jupiter-like, or Pholus-like spectral distributions. For LORRI, the pivot wavelength is 6076.2 Angstroms, and we don't expect this to change, at least not significantly. Since the solar flux (F_solar) at a heliocentric distance of 1 AU at the pivot wavelength is 176 erg/cm2/s/Angstrom, the value for the radiance can be converted to I/F (where pi*F = F_solar) using: I/F = pi * I * r^2 / F_solar where "r" is the target's heliocentric distance in AU. For unresolved targets (e.g., stars), the absolutely calibrated flux (also called the "irradiance") at the pivot wavelength can be determined using keywords that are defined analogously to the photometry keywords discussed above for resolved targets. In the case of a target having a spectral distribution identical to that of a globally-averaged Pluto spectrum, the observed count rate integrated over the LORRI PSF ("CINT/TEXP" in DN/s, where CINT is the total number of flat-field corrected counts integrated over the image and "TEXP" is the exposure time) can be related to the flux ("F" in ergs/cm2/s/Angstroms; not to be confused with "F" in I/F) by: F = CINT/TEXP/PPLUTO When observing point targets, it is more common to convert the absolute flux to a magnitude in a standard photometric system. The following equation can be used to transform a measured value of the irradiance (aka "flux") of an unresolved target to a magnitude in the standard V band: V = -2.5 log S + PHOTZPT + CC + BC where "V" is the visual magnitude in the Johnson photometric system, PHOTZPT is the "stellar photometry keyword", which is the "zero point" of the LORRI instrumental magnitude system, "S" is the integrated net signal rate from the target in DN/s, "CC" is the color correction (i.e., correction for the spectral distribution of the target), and "BC" is the aperture correction (in case the flux is not integrated over the entire stellar image; a careful analysis of the flux versus aperture size for a bright star in the field can then be used to determine the value of BC for the aperture selected for the photometry). In-flight photometry of stars in the open galactic cluster M7 yield the following: PHOTZPT = 18.94 Table 9-10: Color correction coefficient for various targets The following reference flux information is provided for convenience and was gathered from several sources. The UBV are in the Johnson system, RI are in the Landolt-Kron-Cousins system, and JHK_sK are in the UKIRT system. The fluxes for Vega are from the model STScI absolutely-calibrated spectrum. At near-IR wavelengths, the model underestimates the actual Vega flux by about 5-6% owing to the excess flux from the Vega dust disk. Note also that Vega has U=B=V=0.03 (i.e., not 0). Table 9-11 Fluxes for Vega 9.3.1.5 Pointing Information Pointing information for the LORRI boresight (center of the LORRI field-of-view, which is pixel [511,511]) is included in the FITS header in both the raw and the calibrated image files. An example of this information follows: SPCBLRA = 233.4199004768138 / [degrees] Boresight RA, EME J2000 SPCBLDEC= -17.96897170490819 / [degrees] Boresight DEC, EME J2000 SPCEMEN = 283.935414259362 / [degrees] EME J2k North Clk Angle, CW from UP 9.3.1.6 Conversion of instrument housekeeping items to engineering units The LORRI-specific housekeeping items reported in the raw FITS file are in units of counts or DN. To make these values more useful for data analysis, they have been converted to engineering units (volts, amps, degrees Celsius) and reported at the tail end of the header of the primary HDU of the calibrated FITS file. Because the contents of the raw header are duplicated in the calibrated file, a different set of tag names are used for the values that have been converted to engineering units. The new tags are reported after the comment that reads "LORRI Level 2 Calibrated telemetry items". 9.3.2 Instrument Characterization There are several characteristics of the instrument that are related to the radiometric calibration of LORRI that will be useful when analyzing the calibrated image data. They are the quantum efficiency and spectral responsivity, each as a function of wavelength. There are tables for each of these in the calibration directory for the PDS archive, but a graph for each is reproduced in the figures below. Figure 9-2: LORRI Spectral Response vs Wavelength Figure 9-3: LORRI Quantum Efficiency vs Wavelength After the data have been calibrated, additional processing steps are likely to be required. Obvious examples of this are ghost removal and stray light processing. At present, there have been no algorithms developed for public release because they are highly scene dependent. Individual images must be analyzed to understand the structure of the effects to determine an appropriate method for its removal. In the example below, a cutout from a calibrated image is presented to illustrate the effect of stray light from Jupiter's disk, which is just out of the field of view. The circular structure is an example of the ghost pattern. The image on the right demonstrates the processed version of that image. The gradient from the stray light has been removed, as well as the majority of the effects of the ghost. Figure 9-4 Example of Special Processing of Calibrated Data 9.3.4 Dataflow Block Diagram Figure 9-5 9.3.5 Data Format The calibrated image data is organized in a FITS file. The primary header and data unit (HDU) is used to store the calibrated image that results from the calibration pipeline. The first extension is the error estimate image, followed by the second extension containing the data quality image. The table below contains a description of the layout for the extensions for calibrated data. For 1x1 binning mode, the calibrated image dimensions are 1024x1024, and for 4x4 mode, the dimensions are 256x256 pixels. In both situations, these pixels correspond to the optically active pixels from the raw image mentioned previously. Table 9-12 Calibrated FITS file extension layout LORRI calibrated FITS files have 3 extensions. The debiased, desmeared LORRI image is written into the primary HDU as a 2-dimensional, 32-bit real image. The unit for each data value is photometrically-corrected DN. The estimated errors in these corrected DN values are stored as a 2-dimensional, 16-bit real image in the first extension. A data quality image is stored in the second extension as a 2-dimensional, 16-bit integer image. The error in the photometrically-corrected signal is estimated from Equation 9-3, where "sigma" is the 1-sigma error in the corrected signal for a particular pixel (DN), "Pmeas" is the observed signal in that pixel (DN, after bias subtraction but before smear removal), "g" is the electronics gain (22 e/DN), "RN" is the electronics noise (1.3 DN), "f" is the estimated error in the reference flat-field image (0.005), and "FF" is the value of the reference flat-field image at the relevant pixel. The above formula neglects any noise contributed by the bias and smear removal steps, but those errors are generally expected to be small compared to the other sources of error. The data quality image is used to flag pixels that have known artifacts and may need special consideration when performing scientific analysis. The pixel value in the quality flag image represents the sum of all quality flags present for that pixel. This pixel value can also be described as the result of the bitwise 'OR' of each quality flag value. The list of data quality values and their descriptions are listed in Table 9-13. Table 13 Quality flag value descriptions 9.3.7 Scientific Units Following the convention adopted by the New Horizons Principal Investigator, the unit used for calibrated data product are "photometrically-corrected DN". The procedure given above must be completed to obtain absolutely calibrated data products. The units adopted for the radiance ( aka "intensity") of diffuse targets are ergs/cm2/s/sr/Angstroms. The units adopted for irradiance (aka "flux") of point (i.e. unresolved) targets are ergs/cm2/s/Angstroms. Wavelengths are quoted in angstrom units. 9.3.8 Additional FITS and PDS Keywords Added Listed below are the keywords and sample values for those keywords that have been added to the FITS header and are stored with the primary HDU of the output calibrated image FITS file. COMMENT ********************************************************* COMMENT *** LORRI Level 2 software name and version info *** COMMENT ********************************************************* L2_SWNAM= 'lorri_level2_pipeline' /Level 2 calibration software L2_SWVER= 'untagged' /software version tag COMMENT ********************************************************* COMMENT *** LORRI Level 2 software logic flow control flags *** COMMENT ********************************************************* IMGSUBTR= 'OMIT ' / image subtraction step BIASCORR= 'PERFORM ' / bias subtraction step SLINCORR= 'OMIT ' / signal linearization step CTICORR = 'OMIT ' / charge transfer inefficiency step DARKCORR= 'OMIT ' / dark subtraction step SMEARCOR= 'PERFORM ' / smear removal step FLATCORR= 'PERFORM ' / flat-fielding step GEOMCORR= 'OMIT ' / geometric correction step ABSCCORR= 'PERFORM ' / absolute calibration step COMPERR = 'PERFORM ' / compute error estimate COMPQUAL= 'PERFORM ' / compute quality flags COMMENT ********************************************************* COMMENT *** LORRI Level 2 Reference Filename *** COMMENT ********************************************************* REFDEBIA= 'sap_006_combined_100img_1x1.fit' / debias image filename REFFLAT = 'cflat_grnd_SFA_20050309_v2.fit' / flat field image filename REFDEAD = 'dead_ground_1x1_synthetic.fit' / dead pixel image filename REFHOT = 'hot_ground_1x1_synthetic.fit' / hot pixel image filename REFSUBIM= ' ' / subtraction image filename COMMENT ********************************************************* COMMENT *** LORRI Level 2 Absolute Calibration Parameters *** COMMENT ********************************************************* PIVOT = 6076.20019531 / LORRI pivot wavelength. units=angstroms RSOLAR = 266400.000000 / Conv to radiance for solar source RPLUTO = 257500.000000 / Conv to radiance for pluto source RPHOLUS = 324300.000000 / Conv to radiance for 5145 pholus source RCHARON = 263000.000000 / Conv to radiance for charon source RJUPITER= 234700.000000 / Conv to radiance for jupiter source PPLUTO = 1.03000005170E+16 / Conv to irradiance for pluto source PSOLAR = 1.06600003807E+16 / Conv to irradiance for solar source PPHOLUS = 1.29700002225E+16 / Conv to irradiance for 5145 pholus source PCHARON = 1.05199994793E+16 / Conv to irradiance for charon source PJUPITER= 9.38600033786E+15 / Conv to irradiance for jupiter source PHOTZPT = 18.9400000000 / Zero point for visual magnitude, V COMMENT ********************************************************* COMMENT *** LORRI Level 2 Calibrated telemetry items *** COMMENT ********************************************************* EPU_P5VO= 5.04305504857 / EPU +5 voltage. units=Volts EPU_P5CU= 0.143143000000 / EPU +5 current. units=Amps FPU_P15V= 15.0005851594 / FPU +15 voltage. units=Volts FPU_P15C= 0.0493827000000 / FPU +15 current. units=Amps FPU_P6_V= 6.05666080780 / FPU +6 voltage. units=Volts FPU_P6_C= 0.152152000000 / FPU +6 current. units=Amps FPU_HTRC= 0.00000000000 / FPU heater current. units=Amps EPU_25PV= 2.50943456804 / EPU +2.5 voltage. units=Volts RINGTEMP= -66.8836898878 / Intermediate ring temp. units=celsius MFOOTTMP= -61.8964797242 / Mounting foot-top temp. units=celsius M2MNTTMP= -66.8836898878 / M2 mirror mount temp. units=celsius RADTEMP = -88.9564863168 / Radiator temp. units=celsius BAFATEMP= -62.9653774259 / Baffle-aft temp. units=celsius BAFFTEMP= -70.8007466057 / Baffle-forward temp. units=celsius M1SUPTMP= -67.2398321861 / M1 mirror support temp. units=celsius M1MIRTMP= -66.5275372052 / M1 mirror temp. units=celsius CCDTEMP = -79.5485000000 / CCD temperature. units=celsius M1VFTEMP= -66.0128183000 / M1 V/F temperature. units=celsius M2VFTEMP= -66.3287025000 / M2 V/F temperature. units=celsius FPUBTEMP= 29.5499120000 / FPU board V/F temp. units=celsius STEMPCVR= 'ENABLE ' / Temperature conversion enable SCLMP2PE= 'OFF ' / Cal lamp 2 power enable SCLMP1PE= 'OFF ' / Cal lamp 2 power enable SSOURCE = 'CCD ' / Image source SFORMAT = '1X1 ' / Image format SEXPMODE= 'MANUAL ' / Exposure mode PDUNAME = 'Level 2 LORRI image' / 9.3.8.1 Reading FITS file contents using IDL The main method for accessing the various extensions and headers from the FITS file within IDL rely on a third-party library known as the Goddard Astron library. From within IDL, one can load the primary HDU from a fits file using the following command: IDL> calimg=readfits('lor_0035015237_0x630_sci_1.fit', hdr ) IDL> help, calimg CALIMG FLOAT = Array[1024, 1024] The return value of this function ("calimg") is a two dimensional array containing the image data from the primary HDU and its type depends on the data that is read from the file. In the case of raw data, it will be a 16-bit integer array and for calibrated data, it will be a 32-bit floating-point array. The first argument in the call to readfits() is the name of the FITS file to be read. The second argument is an ASCII string variable that will contain the FITS header for the primary HDU upon completion of the function. The same function may be used in order to read any of the extensions listed in the files. For example, to read the data quality image from the calibrated FITS file, one would use a statement such as: IDL> quality=readfits('lor_0035015237_0x630_sci_1.fit', hdr2, exten_no=2) IDL> help, quality QUALITY UINT = Array[1024, 1024] In this example, the ASCII string variable "hdr2" contains the FITS header associated only with the second extension and has no portion of the header from the primary HDU. 9.3.9 Hardware/OS Development Platform The pipeline software was developed in a variety of environments with the commonality of unix-style operating systems. There are no dependencies on the endian properties of the environment. 9.3.10 Language(s) Used IDL 9.3.11 Third Party Libraries Required There are two third party IDL libraries that are needed by the calibration pipeline software: 1) Goddard Astron library, which contains routines needed to read and write FITS files, the format used by the raw data files. Because this library is provided by the SOC for use by many instruments, we will not be delivering this library, but will rely on the version provided to us. 2) IDLUSR, a collection of useful IDL routines made available for public release at APL. Information about this library can be found at http://fermi.jhuapl.edu/s1r/idl/idl.html 9.3.12 Calibration Files Needed (with Quantities) There are currently five categories of reference files needed to perform the calibration process. The reference image categories are the delta-bias, flat-field, dead pixel, hot pixel and desmear e-matrix. Because the LORRI instrument can produce images in either 1024 x 1024 mode or 256 x 256 mode, there are two varieties of each of these images. The filenames associated with these images will be obvious by inspection, although no formal file naming convention has been adopted. There are two ASCII description files in the calibration directory that don't qualify as calibration files but are related to the operation of the pipeline. The first is a configuration file that details all of the configuration parameters for the pipeline ("default_config.txt"). The other file is a description of the housekeeping items that are stored in the first 34 pixels (51 bytes) of the raw image data ("binary_lorri_image_hdr.txt"). These values can be used to validate the FITS header tags that were produced by associating the high-speed image data with the low-speed telemetry values. The values in the first 34 pixels are guaranteed to be correctly associated with a particular image (provided the were not compressed in a lossy fashion) because the LORRI ASE put them in place prior the transfer of the image data to the SSR. As such, they represent a valuable check of the telemetry processing performed on the ground after receipt. The following is a table of the types of files in the calibration directory: Table 9-14 9.3.13 Memory Required ~ 100 MiB 9.3.14 Temporary File System Space Needed None 9.3.15 Predicted Size of Output File(s) Table 9-15 9.3.16 Predicted Execution time Less than 30 seconds per image. 9.3.17 Contact/Support Person(s) Raw data support: Howard Taylor, John Hayes, and Hal Weaver Calibrated data support: Howard Taylor and Hal Weaver 9.3.18 Maintenance Schedule (Code/Data Updates, Documentation) As in-flight calibration data are collected and analyzed, certain aspects of the calibration pipeline will require updates, either in the form of updated reference files, or updated code for bug fixes or future improvements. 9.4 References A. F. Cheng, H. A. Weaver, S. J. Conard, M. F. Morgan, O. Barnouin-Jha, J. D. Boldt, K. A. Cooper, E. H. Darlington, M. P. Grey, J. R. Hayes, K. E. Kosakowski, T. Magee1 E. Rossano, D. Sampath, C. Schlemm, H. W. Taylor, "LONG-RANGE RECONNAISSANCE IMAGER ON NEW HORIZONS", in Space Science Review, in press(2007) S. Conard, F. Azad, J. Boldt, A. Cheng, K. Cooper, E. Darlington, M. Grey, J. Hayes, P. Hogue, K. Kosakowski, T. Magee, M. Morgan, E. Rossano, D. Sampath, C. Schlemm, and H. Weaver, "Design and fabrication of the New Horizons Long-Range Reconnaissance Imager," in Astrobiology and Planetary Missions, G. R. Gladstone, ed., Proc. SPIE 5906, 2005. F. Morgan, S.J. Conard, H.A. Weaver, O. Barnouin-Jha, A.F. Cheng, H.W. Taylor, K.A. Cooper, R.H. Barkhouser, R. Boucarut, E.H. Darlington, M.P. Grey, I. Kuznetsov, T.J. Madison, M.A. Quijada, D.J. Sahnow, and J.M. Stock, "Calibration of the New Horizons Long-Range Reconnaissance Imager," in Astrobiology and Planetary Missions, G. R. Gladstone, ed., Proc SPIE 5906, 2005. 10. MVIC INSTRUMENT DESCRIPTION: 10.1 Overview The Ralph instrument consists of two sets of focal planes: MVIC a visible, near-IR imager and LEISA, a short-wave IR spectral imager. This document only relates to the MVIC (Multispectral Visible Imaging Camera) part of the Ralph instrument. The LEISA pipeline is described in a different document (New Horizons SOC to Instrument Pipeline ICD LEISA Edition). There are 7 separate CCD arrays in the MVIC focal plane. The MVIC telemetry is communicated via a low-speed interface and the imaging data uses a high-speed interface. Figure 10-1 shows a model of Ralph in the spacecraft coordinate system. The MVIC detector package is the light blue box on the +Y face of the instrument. Data recorded on New Horizons is sent to the ground via the Deep Space Network. From there the data is sent to the Mission Operations Center (MOC) at the Applied Physics Laboratory (APL). The Science Operations Center (SOC) retrieves new data from the MOC daily. The data is in a raw form (packetized). The Level 1 and 2 software pipelines convert the data from these raw packets into FITS (Flexible Image Transport) files with scientifically useful and calibrated data. The Level 1 processing sorts the packets into images (in the case of MVIC) with useful header keywords. These keywords include the mode or filter of the observation, timing information and basic pointing information of the instrument boresight. The Level 1 processing also adds relevant housekeeping telemetry (temperatures, voltages, etc.) in a binary table as an extension to the FITS file. The Level 2 processing performs the basic scientific calibration. Before we get into a description of the MVIC calibration, we will describe the image formats for each of the CCD arrays that comprise MVIC. There are 8 detectors in the Ralph instrument. Seven of those are part of MVIC (the yellow and blue ones). The boresight information in the figure is from ground-testing and will change in the future. Figure 10-2 The "Pan Frame" array (yellow) has 5024x128 pixels. The first and last 12 pixels in each row are not optically active. When observing with the pan frame array, multiple images are recorded in each observation. Correspondingly, we store these images together in one FITS file. The first three letters of these files are "mpf" standing for MVIC pan frame. The images are stored as an image cube (3-dimensions). The first dimension is the column number, the second dimension is the row number and the third dimension is the image number within that observation. The remaining 6 arrays are operated in a time-delay integration mode (TDI). These arrays have 5024x32 pixels. The first and last 12 pixels in each row are not optically active. To take an observation with the TDI arrays, we scan the spacecraft and (typically) clock the charge through each of the 32 rows at a rate that matches the spacecraft's scan rate. Using this method we can build up arbitrarily long images in the row direction. For the "Pan 1" and "Pan 2" (panchromatic -- unfilterred) detectors, the resulting FITS files are standard 2-dimensional images The first three letters of these files are either "mp1" or "mp2" corresponding to MVIC pan 1 or MVIC pan 2. The color arrays (NIR, CH4, Red and Blue) are operated together and they use a time-delay integration to build up images. The data for each filter is stored in separate FITS files. In the table below the variable "Ni" stands for the number of images in a pan frame observation. When we command a pan frame observation, we always take multiple images. The "Nr" in the table is the number or rows in an observations. This is determined by the length of time that we are recording data and the rate that we clock the rows in TDI mode. Table 10.2 Observation Modes and their filename prefixes and data dimensions The Level 2 FITS file has a primary data unit which contains the bias-subtracted, flattened image (or image cube in the case of pan frame) plus 4 extensions. Extension 1 is the bias-subtracted, flattened, distortion-removed image (or image cube). Extension 2 is an array with the per pixel error of the bias-subtracted, flattened, distortion-removed image (or image cube). Extension 3 is an array with a data quality flag for each pixel of the bias-subtracted, flattened, distortion-removed image (or image cube). Table 10.2 Pan Frame Image Data Format Table 10.3 TDI Image Data Format 10.2 Level 1 Specifics 10.2.1 Data Format The Level 1 MVIC files will be FITS files. Details of the dimensions 10.of the FITS images and header keywords are given in the following 10.sections. 10.2.2 Data Sources (High/Low Speed, CCSDS, ITF) MVIC telemetry data are recorded using the low-speed interface and images are recorded with the high-speed interface. There are four Ralph-MVIC data formats: A. MVIC Panchromatic TDI (Time-Delay Integration) Format B. MVIC Panchromatic TDI Binning Format C. MVIC Color TDI Format D. MVIC Frame Format Note the distinction between "format" and "mode" is a hold over from the spacecraft to Ralph ICD where the LEISA component of Ralph has multiple modes for one format. The MVIC Panchromatic TDI format has two modes (either "Pan 1" or "Pan 2" indicating which detector was used). The MVIC Panchromatic TDI binning format is obsolete and is not being supported at this time. The remaining two formats each have only one mode. Table 10 lists the MVIC ApIDs and their corresponding data types. For each data type there are two ApIDs, one for each C&DH (Command and Data Handling) system on the spacecraft. Table 10-4: ApID and Data Type for MVIC Data *Note that the Panchromatic TDI Binned data is 3x1 binning in the cross-track direction with 2 out of 3 along-track lines discarded. Note there is not a different ApID for PAN1 and PAN2 observations. This information is stored in the low-speed housekeeping data (keyword MODE). This information is inserted into the FITS header of the Level 1 FITS file. A value of 3 means that the observation was taken with the "Pan 1" array and a value of 4 indicates the "Pan 2" array was used. For more information, see the Ralph Users Manual. The Pan Frame detector has 5024x128 pixels. The first and last 12 pixels in each row are not optically active. When observing with the pan frame array, multiple images are recorded in each observation. Correspondingly, we store these images together in one FITS file. The first three letters of these files are "mpf" standing for MVIC pan frame. The images are stored as an image cube (3-dimensions). The first dimension is the column number, the second dimension is the row number and the third dimension is the image number within that observation. The remaining 6 arrays are operated in a time-delay integration mode (TDI). These arrays have 5024x32 pixels. The first and last 12 pixels in each row are not o