Experimenter to Planetary Science Archive Interface Control Document RO-MIR-IF-0001 Issue 5.0 31 August 2018 MIRO Experimenter to Planetary Science Archive Interface Control Document Prepared by: Mark Hofstadter Seungwon Lee Lei Pan Paul Springer Approved by ______________________________________ _______________________ Mark Hofstadter Date MIRO Principal Investigator Table of Contents LIST OF TABLES x CHANGE LOG xii 1 Introduction 1-1 1.1 Purpose and Scope 1-1 1.2 Archiving Authorities 1-1 1.3 Contents 1-1 1.4 Intended Readership 1-1 1.5 Applicable Documents 1-2 1.6 Relationships to Other Interfaces 1-2 1.7 Acronyms and Abbreviations 1-2 1.8 Contact Names and Addresses 1-3 2 Overview of Process and Product Generation 2-1 2.1 MIRO Overview and Objectives 2-1 2.2 Instrument Description Summary 2-1 2.3 Data Products Description 2-2 2.3.1 Introduction 2-2 2.3.2 Major Data Modes 2-3 2.3.3 Ground Test Data 2-6 2.3.4 Calibration 2-7 2.3.4.1 Radiometric Calibration 2-7 2.3.4.2 Frequency Calibration 2-11 2.3.5 Geometry Information 2-11 2.3.6 Level 4 Products 2-12 2.3.7 Operational Scenarios and Mission Phases 2-12 2.3.8 Data Flows 2-13 3 Archive Format and Content 3-1 3.1 Format 3-1 3.1.1 Volume Format 3-1 3.1.2 Data Set Naming 3-1 3.1.3 File Name Formats 3-1 3.2 Standards Used in Data Product Generation 3-2 3.2.1 PDS Standards 3-2 3.2.2 Time Standards 3-2 3.3 Data Validation and Flagging 3-3 3.4 Content 3-3 3.4.1 Volume Set 3-3 3.4.2 Data Set 3-3 3.4.3 Directories 3-6 3.4.3.1 Root Directory 3-6 3.4.3.2 Catalog Directory 3-6 3.4.3.3 Data Directory 3-7 3.4.3.4 Index Directory 3-7 3.4.3.5 Label Directory 3-8 3.4.3.6 Document Directory 3-8 3.4.4 Data and Label Files 3-8 4 Detailed Interface Specifications 4-1 4.1 Data Product Identification 4-1 4.2 PDS Label Structure, Definition and Format 4-1 4.3 Overview of Detectors 4-3 4.3.1 Spectrometer Data 4-3 4.3.2 Radiometer (Continuum) Data 4-3 4.3.3 Engineering Data 4-3 4.4 Data Format Description 4-3 APPENDICES A. VOLDESC.CAT A-1 B. STRUCTURE FILES B-1 B.1. Spectrometer Level 2 Data (see Section 4.3.1) B-1 B.2. Spectrometer Level 3 Data Generated by MIRO Pipeline Version 1.0 and 1.1 (see section 4.3.1) B-3 B.3. Spectrometer Level 3 Data Data Generated by MIRO Pipeline Version 2.0 (see section 4.3.1) B-7 B.4. Spectrometer Level 3 Data Data Generated by MIRO Pipeline Version 3.0 (see section 4.3.1) B-11 B.5. Spectrometer Level 4 Version 1.0 Data B-15 B.6. Geometry for Spectrometer Level 3 Data Generated by MIRO Pipeline Versions 1.0 and 1.1 (see section 4.3.1) B-23 B.7. Geometry for Spectrometer Level 3 Data Generated by MIRO Pipeline Version 2.0 (see section 4.3.1) B-27 B.8. Geometry for Spectrometer Level 3 Data Version 3.0 and Level 4 Version 1.0 B-35 B.9. Continuum Level 2 Data Generated by MIRO Pipeline Versions 1.0 and 1.1 (see section 4.3.2) B-43 B.10. Continuum Level 3 Data Generated by MIRO Pipeline Versions 1.0 and 1.1 (see section 4.3.2) B-46 B.11. Continuum Level 3 Data Generated by MIRO Pipeline Version 2.0 (see section 4.3.2) B-49 B.12. Continuum Level 3 Data Generated by MIRO Pipeline Version 3.0 (see section 4.3.2) B-53 B.13. Geometry for Continuum Level 3 Data Generated by MIRO Pipeline Version 1.0 and 1.X (see section 4.3.2) B-58 B.14. Geometry for Continuum Level 3 Data Generated by MIRO Pipeline Version 2.0 (see section 4.3.2) B-58 B.15. Geometry for Continuum Level 3 Data Generated by MIRO Pipeline Version 3.0 (see section 4.3.2) B-58 B.16. Housekeeping Data (See Section 4.3.3) B-58 C. AVAILABLE SOFTWARE TO READ PDS FILES C-1 List of Tables Table 1.1: Primary Contacts for Archiving Purposes. 1-3 Table 2.1: MIRO Instrument Performance Characteristics. 2-2 Table 2.2: Allocation of Time in a Calibration Sequence. 2-7 Table 3.1: Data Sets Delivered. 3-4 Table 3.2: Files in Root Directory. 3-6 Table 3.3: Files in Catalog Directory. 3-7 Table 3.4: Files in Index Directory. 3-8 Table 3.5: Files in Document Directory. 3-8 Change Log Date Sections Changed Description 2003-Aug-12 All Initial version 2003-Sep-09 Many Edits as a result of PDS telecon 2003-Nov 2, 4 2003-Dec Changed doc.ID from UoB-IF- 1234 to MIR-IF-0001. Updated keywords in 4.2 & App.1 per latest Archive Plan. 2004-Jul 3-6 Corrections after review of sample label 2005-Dec All Revisions for delivery of ground testing archive 2006-Nov All Revisions after PDS Internal Review and for delivery of Level 3 data. 2006-Dec-11 Section 3.4.2l Updated delivery dates and added items per email from Maud Barthelemy 2007-May-09 Sections 3.2.3 and 4.2 Added documentation of coordinate systems used. Section 5 Changed VOLUME keywords per revised Archive Conventions 2007-Oct-22 1.5, 2.3.4, 3.2.2, 3.4, 4.2, 6, 5 Added explanation of Times in data files, changes to labels and delivery contents, and revised structure files. 2008-Sep-02 Section 4.4 added, Section 7 revised PDS review requested more documentation of data formats. 2009-May-15 6.4 Removed GMT field from Level-3 continuum files 2010-Nov-04 2.3.3.4 Added reference to Geometry files. 3.4.2 Updated list of archive datasets 4.4 Updated for change to Level-3 files (UTC added) 6.2, 6.4 Added UTC field to Level-3 data files 2012-Apr-30 6.1, 6.2, 6.3, 6.4, 6.5 Corrected MIRPOS codes (2=warm, 3=cold) 6.5 Fixed typo in LDFRQ description. 2015-May-19 Many Delivery of first post-hibernation data, utilizing Pipeline data generation software version 1.0. 2017-Aug-31 Many Changes associated with 2017 August data delivery to PSA/PDS. Updates to discussions of the calibration algorithm, file content and format, contact information, and clarifications/additional information in all sections. 2017-Sep-20 Appendix B Version 4.2 corrects the descriptions of local_solha, scene_eff_solha, and the flag value used for undefined geometric quantities. 2017-Oct-05 3.4.2, Appendix B Version 4.3 adds reference to Level 4 files for PRL through ESC3. 2018-Jan-31 Many Update name of CTS Frequency Calibration document. Clarify discussion of geometry records. Update link to READPDS in Appendix C. Table 3.1 now identifies a data gap in Version 1.0 files. Section 2.3.4 provides additional calibration details. Updates/corrections to Appendices B and C. 2018-May-31 2.3.4, Appendix B Version 4.5 primarily adds information associated with the Level 3 Version 3 deliveries to the archive, which have an improved calibration. Several other minor updates made. 2018-May-31 Appendix B, various others Version 4.6 added description of Level 4 "Folded Spectra" product. 2018-June-15 Appendix B, various others Version 4.7 deletes Appendix B and references to it (making the delivered structure files the sole reference for data formats). 2018-Aug-31 2.3.6, various others Version 5.0 adds discussion of Level 4 V2 products and makes minor corrections and clarifications in various parts of the document. Introduction Purpose and Scope The purpose of this EAICD (Experimenter to Planetary Science Archive Interface Control Document) is twofold. First, it provides users of the MIRO instrument with detailed description of the products and a description of how they were generated, including data sources and destinations. Secondly, the EAICD describes the interface to the Planetary Science Archive (PSA) of ESA and is the official document between each experimenter team and the PSA. Archiving Authorities The Planetary Data System (PDS) Standard is used as archiving standard by NASA for U.S. planetary missions, implemented by PDS ESA for European planetary missions, implemented by the Research and Scientific Support Department (RSSD) of ESA ESA?s online Planetary Science Archive (PSA) was implemented to support and ease data ingestion to offer additional services to the scientific user community and science operations teams as e.g. search queries that allow searches across instruments, missions and scientific disciplines several data delivery options as direct download of data products, linked files and data sets ftp download of data products, linked files and data sets The PSA aims for online ingestion of logical archive volumes and will offer the creation of physical archive volumes on request. Contents This document describes the data flow of the MIRO instrument on Rosetta from the spacecraft (s/c) until the insertion into the PSA. It includes information on how data were processed, formatted, labelled and uniquely identified. The document discusses general naming schemes for data volumes, data sets, data and label files. Standards used to generate the product are explained. The design of the data set structure and the data product is given. Intended Readership The MIRO science, software development and engineering team, the staff of the Planetary Science Archive design team, and any potential user of MIRO data. Applicable Documents AD1 Rosetta Archive Generation, Validation and Transfer Plan, 10 January 2006, RO-EST-PL-5011, Issue 2, Revision 3 AD2 Planetary Data System Standards Reference, 1 August 2003, Version 3.6, JPL D-7669, Part 2 AD3 Planetary Data System Data Dictionary Document, 28 August 2002, Revision E, JPL D-7116 AD4 MIRO Users Manual, RO-MIR-PR-0030, issue 7.0 AD5 Acton, C.H.; "Ancillary Data Services of NASA's Navigation and Ancillary Information Facility;" Planetary and Space Science, Vol. 44, No. 1, pp. 65-70, 1996. AD6 Backus, C. and Gulkis, S., "CTS: Frequency Response as a Function of Temperature", included in MIRO PDS/PSA document directory as CTS_FREQUENCY_CALIBRATION.PDF AD7 Rosetta Time Handling, 28 February 2006, RO-EST-TN-3165, Issue 1, Revision 1 Relationships to Other Interfaces The controlling document of the interfaces discussed here is AD1. For further details on the MIRO instrument and its usage, see AD4. Acronyms and Abbreviations bps bits per second CCSDS Consultative Committee for Space Data Systems CODMAC Committee on Data Management and Computation CTS Chirp Transform Spectrometer DBMS Database Management System DDS Data Distribution System (Darmstadt, Germany) DVD Digital Video Disk ESA European Space Agency GHz GigaHertz (109 Hz) HSK Housekeeping IFP Intermediate Frequency Processor JPL Jet Propulsion Laboratory (Pasadena, CA) kHz kiloHertz (103 Hz) LO Local Oscillator MHz MegaHertz (106 Hz) MM millimeter MIRO Microwave Instrument for the Rosetta Orbiter NAIF Navigation and Ancillary Information Facility NaN ?Not a Number,? digital representation of an undefined or unrepresentable value NASA National Aeronautics and Space Agency (USA) OBT On-Board Time PDS Planetary Data System PSA Planetary Science Archive rms root mean square SUBMM submillimeter TDB Barycentric Dynamical Time USO Ultra Stable Oscillator UTC Coordinated Universal Time Contact Names and Addresses Table 1.1: Primary Contacts for Archiving Purposes. Last Name First Name Institution Phone Email Hofstadter Mark JPL 818-354-6160 mark.hofstadter@jpl.nasa.gov Pan Lei JPL 818 393-0477 lei.pan@jpl.nasa.gov von Allmen Paul JPL 818 393-7520 paul.a.vonallmen@jpl.nasa.gov This page is intentionally left blank Overview of Process and Product Generation MIRO Overview and Objectives The MIRO investigation addresses the nature of the cometary nucleus, outgassing from the nucleus, and development of the coma (both gas and dust) as strongly interrelated aspects of cometary physics. MIRO also performed searches for outgassing activity on asteroids. MIRO is configured both as a continuum and a very high spectral resolution line receiver. Center-band operating frequencies are near 190 GHz (1.6 mm) and 562 GHz (0.5 mm). Spatial resolution of the instrument at 562 GHz is approximately 50 m at a distance of 20 km from the nucleus; spectral resolution is sufficient to observe individual, thermally broadened, line shapes at all temperatures down to 10 K or less. MIRO continuum channels at millimeter and submillimeter wavelengths sense the subsurface temperature of the nucleus to depths of several centimeters or more. Model studies relate these measurements to electrical and thermal properties of the nucleus and address issues connected to the sublimation of ices, ice and dust mantle thickness, and the formation of gas and dust jets. The global nature of these measurements will allow in situ lander data to be extrapolated globally, while the long duration of the mission allows us to follow the time variability of surface temperatures and gas production. MIRO is highly complementary to the IR mapping instrument on the orbiter (VIRTIS), having similar spatial resolution but greater depth penetration. MIRO?s spectrometer only operates at submillimeter wavelengths, and observes four key volatile species simultaneously: H2O, CO, CH3OH, and NH3. The three main oxygen isotopologues of water (H216O, H217O, and H218O) are all measured, as are three different CH3OH transitions. The primary retrieved products are abundance and velocity of each species, along with their spatial and temporal variability. This information is used to infer coma structure and processes, including the nature of the nucleus/coma interface. Instrument Description Summary The MIRO instrument provides both very sensitive continuum capability for temperature determination and extremely high-resolution spectroscopy for observation of molecular species. The instrument consists of two heterodyne radiometers, one at millimeter wavelengths (1.6 mm) and one at submillimeter wavelengths (0.5 mm). The millimeter and the submillimeter radiometers have continuum bandwidths of 0.5 GHz and 1.0 GHz respectively. The submillimeter receiver has a total spectroscopic bandwidth of 180 MHz and a spectral resolution of 44 kHz. In the spectroscopic mode, 4,096 channels, each having a bandwidth of 44 kHz, are observed simultaneously. The performance parameters that govern the MIRO instrument design include system sensitivity, spatial resolution, radiometric accuracy (both absolute and relative), beam pattern and pointing accuracy, together with the mass, power, volume envelope, and environmental conditions available within the spacecraft. The MIRO instrument performance characteristics are summarised in Table 2-1. More detailed information can be found in the MIRO User Manual (AD4). Table 2.1: MIRO Instrument Performance Characteristics. Equipment Property Millimeter-Wave Submillimeter-Wave Telescope Primary Diameter Primary F/D Sidelobes Spatial Resolution Footprint size (at 10 km) 30 cm 1 ?30 dB 23.8 arcmin ~75 m 30 cm 1 ?30 dB 7.5 arcmin ~25 m Spectral Performance Sky Frequency Band (Double Sideband) 1st IF Bandwidth 1st IF Frequency Range Radiometer Bandwidth Spectral Resolution Allocated Spectral Range per line Accuracy 186.7?189.7 GHz 550 MHz 1?1.5 GHz 0.5 GHz n/a n/a n/a 547.6?579.2 GHz 11 GHz 5.5-16.5 GHz 1.0 GHz 44 kHz nominally ~20 MHz 10 kHz Spectrometer Center Frequency/Bandwidth Number of channels n/a n/a 1350/180 MHz 4096 Radiometric Performance DSB Receiver Noise Temperature SSB Spectroscopic Sensitivity (300 KHz, 2 min) : relative absolute Continuum Sensitivity (1 sec): relative absolute ~2000 K n/a n/a 1 K rms 3 K rms ~4000 K 2 K rms 3 K rms 1 K rms 3 K rms Data Rates Continuum Mode Spectroscopic Mode On-board Storage <1 kbps 2.5 kbps 0.2 GB (one day?s data volume, Mode 3, 100% duty cycle) Data Products Description Introduction The MIRO instrument has 6 major modes of operation and data-taking that reflect operational combinations of its two continuum radiometers and the spectrometer: engineering mode, millimeter continuum mode, submillimeter continuum mode, dual continuum mode, CTS/submillimeter continuum mode, and CTS/dual continuum mode. In addition, a special mode has been designed for planetary and asteroid flybys. A number of data compression options are obtained in each mode by varying the data-taking rate (integration time per sample) and/or spectral resolution of the radiometers and spectrometer. The specific parameters for each mode are described in more detail in the MIRO User Manual (AD4), but are summarized below. All engineering (housekeeping) data, continuum and spectroscopic data (both raw and calibrated) are table files, consisting of time sequences of measured data. The detailed structure of these files is defined by the Structure Files (also referred to as format files or .FMT files) provided in each data delivery to the PSA/PDS. Major Data Modes Engineering Mode In engineering mode the MIRO software is collecting engineering data from 56 internal sensors. The sampling of these sensors is at a 5 Hz rate, but only the latest sample available at the time a telemetry packet is generated is returned by the instrument. All engineering measurements are 12-bit A/D converted values. The engineering mode telemetry is sent to the spacecraft in the form of a housekeeping telemetry packet. One engineering telemetry packet is typically generated every 11 seconds. This is the maximum rate possible, which results in the most stable calibration of the instrument. Early and late in the mission, slower engineering telemetry rates were used to minimize data volume. The frequency of housekeeping packet generation is controlled by the Housekeeping Cycle Skip telecommand; see the Users Manual (AD4) for details. Millimeter Continuum Mode In millimeter continuum mode continuum data are collected from the millimeter radiometer at a 20 Hz rate. All continuum data consist of 16-bit values. The millimeter continuum data are returned in science telemetry packets usually containing 200 continuum samples. At the nominal collection rate of 20 Hz, a millimeter packet is generated every 10 seconds. A ?summing value? parameter can cause the MIRO software to sum either 1, 2, 5, 10 or 20 separate radiometer measurements prior to putting them into the telemetry packet. This feature can reduce the data rate to as little as one millimeter continuum packet every 200 seconds, with each of the 200 samples representing a 1-second integration. The continuum data records contain a parameter called ?ND? which provides the number of continuum data samples in the packet. While the vast majority of the time there are 200 continuum samples in a packet, increasing the continuum summing parameter can result in occasional smaller packets, as can operating in modes that do not collect spectrometer data (i.e., Millimeter Continuum, Submillimeter Continuum, and Dual Continuum modes). These smaller packets occur just before a calibration sequence, and are the result of the timing and prioritization of various tasks the instrument must execute. Packets containing less than 200 continuum samples are valid and contain good data. Submillimeter Continuum Mode Sub-millimeter continuum mode is identical to the millimeter continuum mode in data collection and packing except that a different set of electronics is powered on. Millimeter and submillimeter continuum data are contained in separate science telemetry packets, identified by a field in the source data header. Dual Continuum Mode In dual continuum mode, the millimeter and submillimeter continuum receivers are collected simultaneously. When running in dual continuum mode, the summing value parameter mentioned earlier is applied to both sets of data, causing equal amounts of millimeter and submillimeter data to be generated. CTS / Submillimeter Continuum Mode This mode adds the collection of chirp transform spectrometer (CTS) data to Submillimeter Continuum Mode. Normal CTS science data is ?frequency switched? (described below), with one spectrum generated from 30 seconds of data. Selectable integration times are 30, 60, 90, and 120 seconds. Frequency switching is a calibration technique that removes frequency-dependent artifacts from the data. To utilize it, the CTS is programmed to collect data in paired, 5-second periods. An internal LO frequency generator is switched every 5 seconds, introducing a small shift in frequency between the two 5-second periods in a pair. These paired LO observations are repeated 3, 6, 9, or 12 times to produce 30-, 60, 90, or 120-second spectra. Each LO setting is integrated separately over the requested interval, and then the data from the two LO frequencies are subtracted from each other to provide a single 4096-element difference spectrum. Thus, a 30-second frequency-switched spectrum consists of a 15 second integration with the LO set to +5 MHz (LO=1) subtracted from a 15-second integration with the LO set to -5 MHz (LO=0). This subtraction results in 1) many instrument-caused frequency-dependent features being eliminated from the spectrum, and 2) the spectral line generated by the comet appearing twice in the spectrum; once as a positive feature and 10 MHz away as a negative feature. (A future MIRO data product will ?fold? the frequency-switched data so that the two comet spectral features are combined, reducing the noise level.) Frequency switching is only performed when the CTS is powered on. CTS data collection and the LO frequency switching is coordinated with the collection of continuum data. Exactly 100 continuum samples are taken during each LO setting, and only one LO setting is in place during each CTS 5-second internal integration period. This assures that it is known at which LO frequency each of the measurements are made. To reduce instrument data volume, the 4,096 spectrometer channels can be smoothed on board, resulting in telemetered spectra with 4096, 2046, 1364, or 1022 channels. See the Users Manual (AD4) Section 6.1.2.1.4.2 for details of the smoothing function applied. Regardless of whether smoothing is applied, all uncalibrated spectra in the archive have 4096 channels, with a bad data flag padding the higher channel numbers when smoothing is applied. Levels 2 and 3, Version 1 data products use 0.0 or -999.99 as the bad data flag. Level 3 Version 3 uses -999.0. Level 3 Version 2 Sspectra for the PRL and ESC1 mission phases use NaN as the bad data flag, though NaN is not allowed by PDS standards. The CTS channel-to-frequency conversion is temperature dependent (see the Users Manual [AD4] Section 9.4). While the CTS is temperature-controlled, for operational reasons temperature settings were occasionally changed. Temperatures also varied during instrument warm-up. To preserve the native resolution of the spectrometer (approximately 44 kHz) while also making it convenient to compare spectra taken at different temperatures, when reporting calibrated spectra on an absolute frequency scale, two spectra are provided in files generated by MIRO Pipeline software Versions 2.0 and 3.0 (used in Level 3 Version 2 and 3 data sets). One contains the intrinsic 4096 channels, with channel frequencies indicated for the current temperature of the CTS. The second calibrated spectrum is interpolated to a common frequency grid containing 4152 channels. This second spectrum allows all spectra collected during the course of the mission to be provided in a common format. Each calibrated spectrum on the common frequency grid will have either the lowest, highest, or both the lowest and highest channel numbers reported as NaN (Version 2) or -999.0 (Version 3), indicating those frequencies were not sampled at the current CTS temperature. (In Version 1.0 and 1.1 calibrated CTS data sets, only one calibrated spectrum is provided, it contains 4,250 channels, and 0.0 or -999 is used as the fill value.) A mask is applied to the CTS data and only 12 bits of each resulting measurement are returned to Earth. CTS / Dual Continuum Mode This is the same as the CTS / SMM continuum mode except that the millimeter data are also collected. Asteroid Mode This special data-taking mode has been implemented for the asteroid and planetary encounters to enable MIRO to follow the rapid and large Doppler shift of spectral lines that may be visible. The primary characteristic of this mode is that LO frequency switching is turned off. The LO is set to either +5 MHz or - 5 MHz from the nominal frequency prior to the encounter. At a specified time (typically closest approach to the target so the Doppler shift is zero), the LO frequency is switched to the opposite value. Each spectrum of Asteroid Mode CTS data consists of a single 5-second integration with all 32 bits returned for each of the 4,096 channels. Continuum data are collected at 20 Hz during Asteroid Mode. This mode was not used during comet observations. During other mission phases, one must look at the ?asteroidmode? parameter in the CTS data records to determine if the instrument was in this mode. SOME NOTES ON FREQUENCY SWITCHING AND THE TIMING OF CONTINUUM DATA When in a CTS data collection mode, frequency switching (described earlier in this section) is applied. This has several impacts on the continuum data. Instrument performance is slightly different at the two LO settings, requiring the data with LO=0 to be calibrated independently from data with LO=1. This is possible because calibration data are collected separately at each LO setting when frequency switching is active (Section 2.3.4). Consideration of the Power and Summation parameters in the continuum data records allows one to determine the LO state for every continuum sample in a packet except when in Asteroid Mode (discussed below). Power is used to identify when the instrument is in a CTS mode and frequency switching is active. The Summation parameter indicates if each reported continuum measurement integrates for 0.05, 0.1, 0.25, 0.5, or 1 second, indicating that there are 100, 50, 20, 10, or 5 samples during each 5-second LO setting. Because LO switching is synchronized with data collection, the first continuum samples (except when in Asteroid Mode) are collected with LO=0 (the -5 MHz setting), which then alternates with LO=1 (+5 MHz) every 5 seconds. The time required to perform a frequency switch is slightly variable (by about 5 msec). To guarantee accurate timing of every sample in a continuum packet (primarily needed for geometry calculations during rapid slews or flybys), several time stamps are provided. Time is always the start of integration of the first sample in a packet, which is collected with LO=0. The time interval between samples at a given LO setting is determined by the Summation parameter. Time1 marks the beginning of the first integration with LO=1 when Summation=0 or 1 (the 101st or 50th sample in the packet, respectively). For all other Summation values, Time1 is zero. Time2 marks the beginning of the second LO=0 integration when Summation=1(the 101st sample). For all other Summation values, Time2 is zero. Time3 marks the beginning of the second LO=1 integration when Summation=1 (the 151st sample). For all other Summation values, Time3 is zero. For Summation=2 or 3, the integration time of each sample is large enough (0.5 and 1 second, respectively), that the timing uncertainty during a 200-sample packet due to LO switching is negligible. In these modes, only Time along with the nominal integration period is needed to time-tag each continuum sample. Asteroid Mode was not used at the comet, but for completeness we point out that because LO-switching is suppressed in this mode, timing uncertainties in a continuum packet are negligible in spite of the Summation parameter being 0. In this mode, the Time and Summation parameters are sufficient to determine the time of each sample. Note also that a user must look in the CTS data packets at the ?Asteroidmode? parameter to determine whether or not the instrument is in this mode. Ground Test Data Ground tests in a thermal-vacuum chamber were carried out at JPL from 15 May to 29 June 2001 and were intended to determine characteristics of the MIRO instrument in vacuum conditions and as a function of temperature. The emphasis was on deriving parameters that cannot be obtained under ambient conditions, such as the noise figures of various electronic components, the frequency response of the instrument and the linearity of the response, and the stability of several features. The data obtained from these tests and the accompanying log files are delivered as the first MIRO archive dataset. Calibration Radiometric Calibration See the MIRO Users Manual (AD4) Section 9.5 for a detailed discussion of MIRO radiometric calibration and Section 9.7. for a discussion of error bars. The MIRO instrument is calibrated on a periodic basis and immediately following every mode change. An automatic calibration will take place every 1895 seconds (approximately 32 minutes), if not interrupted by a mode change command, which triggers a calibration immediately. The normal interval of 1895 seconds allows 95 seconds for the calibration and 1800 seconds (30 minutes) for the data collection period. The 1800 seconds allows for complete CTS integration periods of 30, 60, 90, and 120 seconds (60, 30, 20, and 15 output spectra respectively). The 95 seconds of calibration data are distributed as follows: Table 2.2: Allocation of Time in a Calibration Sequence. Time (seconds) Activity 5 3 mirror movements/no data collection 30 Warm load position- CTS + continuum + engineering 30 Cold load position - CTS + continuum + engineering 30 Sky position - CTS + continuum + engineering These calibration data are included in the level-2 data files as part of the time sequence. They are flagged by a Calibration field in the header columns (see the Structure files, also referred to as format files or .FMT files, associated with each data delivery). Each calibration sequence includes collecting 30 seconds of data on an internal warm calibration target, 30 seconds on an internal cold calibration target, and 30 seconds on the sky wherever the instrument boresight is pointed during the sequence. The continuum Summation parameter and, if in a spectroscopic mode, the CTS Smoothing parameter, are applied during calibration observations. If the CTS is on, frequency switching is performed during calibration observations but?unlike during normal science observations?the two LO settings are not subtracted from each other. Instead one spectrum is returned for each of the two LO settings (each representing a 15-second integration). Thus there are two spectra returned on the warm load, two from the cold load, and two on the sky. The number of continuum packets returned on each calibration target depend on the Summation parameter: three are returned for Summation=0, two for Summation=1, and one for Summation=2 or 3. These data are used to determine the relation between instrument counts and absolute flux units. Flux is reported as a Rayleigh-Jeans brightness temperature in Kelvin. The brightness temperature is related to, but not the same as, the physical temperature of the target (see AD4 Section 9.2 for a detailed discussion). The instrument response is assumed to be linear, so that the relation between recorded counts, D, and brightness temperature, T, is D=g*T+I (Eq. 1) where g is the gain in counts per Kelvin, and I is the intercept in counts. Since the physical and hence the brightness temperatures of the warm and cold calibration targets are known (see below), and using the subscript H to refer to the warm load and C to the cold, a calibration sequence can be used to determine the gain and intercept by these equations: g=(D_H-D_C)/(T_H-T_C) (Eq. 2) and I=T??*(1-g) (Eq. 3) where T?? is ?(T?_H-T_C)/2.0. We will refer to the gain and offset collectively as the calibration parameters. The above equations are used to determine the calibration parameters at the time of each calibration cycle. These will be referred to as the ?instantaneous calibration parameters.? For continuum data, the ?D? in Equations 1 and 2 is the average number of counts reported while observing each load (and hence may average multiple packets depending on the Summation value), with LO0 and LO1 data averaged separately. For spectroscopic data, the two LO settings are averaged together, but calibration parameters for each channel are calculated independently. Instantaneous gains are not calculated when there is no engineering data from within ?1 minute of the calibration cycle. In addition, instantaneous continuum gains whose values deviate significantly from expected values are not used (mm gains < 4.0 counts/K or > 6.0 counts/K; submm gains < 1.45 counts/K or > 1.7 counts/K). As discussed in AD4, Section 9.5, smoothing the calibration parameters over time results in better science data. Later in this section we describe how the instantaneous calibration parameters are applied to the data. The Brightness Temperature of the Calibration Targets The mounting of the cold calibration target extends outside the spacecraft and radiates to cold space, keeping its temperature below that of the general spacecraft interior. The internally mounted hot load is heated to keep it above the ambient temperature of the spacecraft. These temperatures are not controlled, but are measured. Each calibration target has two thermistors within it, and their temperatures are reported in the Engineering data. At the time of a calibration sequence, the physical temperature of each calibration target is taken to be the average of the reading of its two thermistors in the engineering packet collected at the time closest to the beginning of the calibration sequence. This physical temperature is not the brightness temperature to be used in the calibration equations, however. As discussed in AD4 Section 9.2, a correction needs to be applied T_B=h?/k*1/(e^(h?/(kT_Phys ))-1) (Eq. 4) where TB is the brightness temperature, TPhys is the physical temperature of that load, h is the Planck constant, k is Boltzmann?s constant, and ? is the frequency of observation. A further correction is required for the millimeter-wavelength cold load brightness temperature. This is due to a design error which allows the millimeter receiver to pick up some signal from the mounting structure of the cold target. The final cold-load brightness temperature to use in Eq.?s 2 and 3 is T_C=T_B*w+T_OB*(1-w) where w is 0.91 in Version 1 products, 0.925 in Version 2 and 3 products,TB in the above equation is the cold-load brightness temperature from Eq. 4 and TOB is the brightness temperature of the optical bench from Eq. 4. The physical temperature of the optical bench is found in the engineering data along with the calibration target physical temperatures. Continuum Radiometric Calibration Software Version 1 In Versions 1.0 and 1.1 of the continuum calibration software (used in generating Level 3 Version 1.0 products in the PSA/PDS), gain and offset as a function of time are found from a linear interpolation between the instantaneous calibration immediately preceding a science observation and the instantaneous calibration immediately after. When in a spectroscopic mode (which means frequency-switching is applied), the calibration parameters for LO0 and LO1 are calculated separately and applied separately to the LO0 and LO1 continuum science data. For simplicity, even when not in a spectroscopic mode, the continuum gains are calculated and applied as if frequency switching were active. This calibration algorithm adds noise to the calibrated data because the instrument gain is more stable than the measurement precision of an instantaneous gain calculation. For this reason, Version 1.0 of Level 3 products should not be used when higher version numbers are available. Note that observations of empty sky calibrated with Version 1 software typically show a brightness of about 1.6 K in the submm, and -0.5 K in the mm channel. Expected values of the cosmic microwave background are about 0 and 0.5 K in the submm and mm, respectively, so the MIRO values are in error. These offsets are discussed in AD4 Section 9.5.2. Level 3 Version 3 files contain a correction. Software Version 2 For calibrated continuum science data generated by MIRO Pipeline software Version 2.0 (used in generating Level 3 Version 2.0 products for the PRL and ESC1 mission phases), gain as a function of time is calculated in a two-step process. First, the instantaneous gains are box-car smoothed: at the time of a calibration, 16 instantaneous gains before and 16 after are averaged with the instantaneous gain at that time. This results in an averaged-gain at the time of each calibration cycle. Second, for the start time of each continuum science packet, a least-squares quadratic fit is made to the 16 box-car-smoothed gains nearest the start time (nominally 8 before and 8 after) to determine a gain versus time curve, and that gain curve is used in the calibration equation. A single gain value is used for all continuum science values in a packet. A similar process is used to determine the intercept as a function of time, except that 1) The instantaneous intercepts are not box-car smoothed, 2) The box-car smoothed gain values are used in Eq. 3 to generate a new instantaneous intercept, and 3) A linear fit to the 4 nearest instantaneous intercepts is used (nominally 2 before and 2 after the science data). When in a spectroscopic mode (which means frequency-switching is applied), the calibration parameters for LO0 and LO1 are calculated separately and applied separately to the LO0 and LO1 continuum science data. For simplicity, even when not in a spectroscopic mode, the continuum gains are calculated and applied as if frequency switching were active. Note that the Version 2 algorithm contains the same non-linearity errors when observing the sky or targets below 150 K as described in the Version 1 algorithm. Software Version 3 Level 3 Version 3 data products in the PDS/PSA were generated by MIRO software version 3.0. The continuum radiometric calibration is the same as in Version 2 software, with an additive correction applied to all temperatures to account for some systematic errors. As described in detail in AD4 Section 9.5.2, the correction is linear with temperature, is zero near the temperature of the hot and cold calibration targets, and has maximum amplitude when looking at blank sky. The maximum amplitude is chosen to make the blank sky temperature match the expected cosmic microwave background brightness temperature (0.001 K at submm wavelengths, 0.33 K in the millimeter). CTS Radiometric Calibration Software Version 1 The following description applies to calibrated CTS science data generated by MIRO Pipeline software Versions 1.0 and 1.1 (used in generating Level 3 Version 1.0 data sets in the PSA/PDS). Each of the 4096 CTS channels is calibrated independently. For reasons discussed in AD4 Section 9.5, the Intercept in Eq. 1 is taken to be zero. Because of frequency switching, a science spectrum incorporates data at both LO settings, so for CTS purposes the instantaneous gain is taken to be the average gain of the two LO settings. The CTS gain at the time of a science observation is taken to be one-half of the gain determined by a linear interpolation in time between the instantaneous gains calculated just prior to and after the science observation. The factor of 1/2 (effectively increasing the reported brightness temperature) is due to the fact that MIRO is a double-sideband system. (MIRO measures the total power received in two frequency bands, an upper and a lower sideband. When looking at a continuum target, such as the comet nucleus or a calibration target, roughly equal amounts of power are received in each sideband. A spectral line, however, only exists in one of the sidebands, so the effective CTS gain for that line is only half of what it is for the calibration signal which exists in both sidebands.) In a perfect instrument, frequency-switched spectra would have a baseline brightness (where there is no spectral feature from the comet) of 0 K. Due to a systematic offset between the calibration parameters at each LO setting, differenced spectroscopic data typically have a small positive bias, with the baseline being at about 2 K. This baseline also can vary in some bands (bands are defined in AD4 Section 9.3) due to an amplifier instability. We leave these non-zero baselines in the Version 1 calibrated data because they may be of interest for future engineering uses. For science purposes, however, it is appropriate to estimate the baseline around each spectral feature in a band and subtract that value from all channels in the band, giving it a zero baseline. (This baseline removal is performed in Version 3 data products.) As was discussed for continuum calibration, the instrument gain is actually more stable than the precision of an instantaneous calibration measurement. The linear interpolation in gain just described therefore adds noise to the calibrated data. Later software versions improve upon this by averaging the CTS gains over time. For this reason, Version 1.0 of Level 3 products should not be used when a higher version is available. Software Version 2 The following description applies to calibrated CTS science data generated by MIRO Pipeline software Version 2.0 (used in generating Level 3 Version 2.0 data sets in the PSA/PDS). As described for Version 1, each of the 4096 CTS channels is calibrated independently, the Intercept in Eq. 1 is taken to be zero, and the instantaneous gain is taken to be the average gain of the two LO settings. The gain at the time of a science observation is taken to be one-half of the average gain of the nearest 32 instantaneous gains. (See the description of Version 1 for an explanation of the factor of 1/2.) As explained above, differenced spectroscopic data typically have a small positive bias. For science use it is appropriate to estimate the baseline around each spectral feature in a band and subtract that value from all channels in the band, giving it a zero baseline. This baseline is removed in Version 3 products. Software Version 3 Level 3 Version 3 data sets in the PDS/PSA archive were generated using software version 3. This software starts with Version 2 spectra, and subtracts a linear baseline from around each spectra line, as described in AD4 Section 9.5.2. To determine the baseline, spectral channels on each side of a spectral feature are averaged together, and a line is fit to the resulting two points. The placement of the ammonia line in MIRO's spectrometer, however, does not allow a spectral baseline to be estimated on one side of the line, so in this case the baseline is extrapolated from two sets of points on one side of the line. Frequency Calibration The frequency calibration of the CTS is a complex subject, described in AD6, and summarized in AD4, Section 9.4. The Receiver Frequency of the radiation entering the instrument (in the range 547.5 ? 580.5 GHz, see Table 2-1) is translated by a series of mixers in the IFP to the frequency range of the CTS, centered at 1350 MHz. The relationship between IFP output frequency and channel number is a function of the CTS temperature. In the calibrated data (as well as in the engineering data) the SPECT_T1 field gives this temperature. In Version 2 and 3 data sets (generated by software versions 2 and 3, respectivcely) of the calibrated CTS data, spectra are interpolated to a common frequency grid, allowing spectra to be directly compared regardless of the value of SPECT_T1. (In Versions 1.0 and 1.1 of the calibration, all spectra were interpolated to a frequency grid corresponding to a 67.9 C temperature, and SPECT_T1 within the calibrated data file was reported as 67.9 C, regardless of what the true temperature was at that time.) Geometry Information CODMAC Level 3 Version 2 and 3 files, as well as all Level 4 files, are generated by MIRO Pipeline software Versions 2.0 and 3.0. These products contain geometry information calculated using a shape model for the nucleus and SPICE library routines. See the Users Manual [AD4] Section 9.6 for additional information on the geometry calculations. (In the older Version 1.0 and 1.1 files, geometry information?when provided?is of limited use, as it assumes the comet nucleus is a tri-axial ellipsoid.) Level 4 Products At the time of this writing (August 2018), there are Version 1 and Version 2 data products designated CODMAC Level 4. (Note, however, that MIRO Level 3 products do contain some resampled fields which by themselves would bit the CODMAC definition of Level 4.) Level 4 Version 1 data sets provide CTS "folded" spectra and, in accompanying files, geometry information for those spectra. See the MIRO Users Manual Section 9.8 for a description of folded spectra. They are calibrated, irreversibly resampled, and provided on a comet-centric velocity as opposed to frequency grid. Level 4 Version 2 data sets contain copies of the Version 1 folded spectrometer and geometry information, and also contain mm and submm continuum data averaged into 1-second bins. These 1-second averages are typically much more useful to science users than the native 50 msec resolution. Geometry information for the 1-second data is included within the continuum data files, instead of being stored in separate files as it is for the folded spectra. To create the 1-second continuum data products, the entire mission starting on 4 August 2014 (DOY 216) and ending on 30 September 2016 (DOY 274) was divided into 1-second time bins, with bin centers on integer seconds (i.e. the first 3 time bins are centered on 00:00:00, 00:00:01, and 00:00:02 of 4 August 2014). All continuum data from the instrument is then averaged into these bins. A typical 1-second bin averages 19 or 20 continuum measurements (corresponding to the native 50-msec sampling of the instrument), but there are often fewer samples because of the instrument mode or other activities. When the instrument was not collecting continuum data, 1-second data records are still generated, but they contain the -999.0 flag value in the temperature field and report 0 samples in the average. Note that the time fields associated with a 1-second data record mark the midpoint of the averaging period. (In all other data products, the primary time stamp marks the beginning of the integration time.) Operational Scenarios and Mission Phases MIRO collected scientific and calibration data prior to entering deep space hibernation in June 2011, which was described in previous archive deliveries. For most of its time at comet 67P, the MIRO instrument operated in CTS/Dual Continuum mode (see Section 2.3.2), with frequency-switching turned on, continuum science data collected at 20 Hz, and CTS spectra generated every 30 seconds. The spacecraft was power-limited in the earliest and latest mission phases at the comet. Stringent data-volume limitations also existed near solar conjunction (February 2015) and at the end of mission. At those times, other instrument modes and longer integration times were frequently applied. If the instrument is in a continuum-only mode, frequency switching is turned off. Numerous observing sequences were utilized during the mission to optimize science return given various operational constraints. It is beyond the scope of this document to describe the sequences in detail, but in general they include long stares at the center of the nucleus, scans over portions of the nucleus (which may or may not include parts of the surrounding coma), scans over the entire nucleus (which may or may not include significant parts of the coma), stares or scans in parts of the coma, and scans over large regions of the coma. Information on observing sequences commanded are available in the DOCUMENTS directory of data sets (see Section 3.4.3.6). The instrument mode called ?Asteroid Mode? was used only for asteroid and planetary encounters, and was not use during the mission phases at the comet. Archived MIRO data on comet 67P are divided into time-based mission phases as indicated below. The time ranges specified are defined by the Rosetta Project. In some cases, due to MIRO not being turned on or missing data packets, MIRO data may not be available in the entire time range specified. PRL: Pre-landing covers the time period from 2014 January 21 (science-instrument recommissioning after hibernation exit) through 2014 November 18 (the end of lander operations). MIRO post-hibernation re-commissioning and check-out occurred on 27-29 April 2014. Early in the PRL phase the spacecraft was power-constrained, requiring MIRO to be switched off much of the time. By August 2014, however, power levels were such that MIRO could be on continuously. ESC1: Escort 1, the first period of operations after landing operations, covers 2014 November 19 through 2015 March 10. Solar conjunction occurred in February 2015, drastically reducing data rates within ~1 month of that time. ESC2: Escort 2 covers 2015 March 11 through 2015 June 30. The spacecraft entered a safemode (switching MIRO off for several days) in late March 2015. ESC3: Escort 3 covers 2015 July 01 through 2015 October 20. ESC4: Escort 4 is the last phase of the primary mission, covering 2015 October 21 through 2015 December 31. EXT1: Extended 1, the first block of time in the extended mission covers 2016 January 01 through 2016 April 05. EXT2: Extended 2 covers 2016 April 06 through 2016 June 28. The spacecraft entered a safemode (switching MIRO off for several days) in May 2016. EXT3: Extended 3 covers 2016 June 29 through 2016 September 30 (end of mission). Data Flows The MIRO telemetry packets coming from the spacecraft were retrieved from the ESOC Data Distribution System (DDS) at Darmstadt by PI-controlled workstations located at the Jet Propulsion Laboratory in Pasadena, CA, under the direct responsibility of the PI. The telemetry records are written in their original SFDU formats for permanent safekeeping in the MIRO archival system at JPL. These telemetry records will be kept in the MIRO project but are not considered part of the formal science archive. The Data Archive has the following characteristics: The MIRO Data Archive system is located at JPL in Pasadena, CA. The Data Archive system has the capability to store and maintain all the data coming from ESA (instrument/science data, housekeeping data, auxiliary data, navigation data, command logs) in their original format (SFDU format where applicable). The Data Archive system is capable of transferring data to the MIRO data base management system (DBMS) for further processing. The Data Archive system has the capability to store and maintain all the data in PSA/PDS format that will be delivered to ESA?s Planetary Science Archive and NASA?s Small Bodies Node of the PDS. All data (science, housekeeping and auxiliary) in the MIRO Raw Data Archive at JPL are capable of being ingested into the MIRO DBMS. This DBMS is the means of access to the data for team members doing science analysis of these data. Delivery of data to the Rosetta Mission Archive of the PSA of ESA and the Small Bodies Node of the PDS of NASA is done by extracting data from the MIRO DBMS into file formats defined by this document and generating PDS labels for these files. These files are placed in directory trees in the MIRO Data Archive, along with all associated documentation and index tables. Compressed copies of these directories are delivered to the PSA and PDS for external archival and will also remain online in the MIRO Data Archive. The MIRO team will support the peer reviews of MIRO-related data that are conducted by the ESA-PDS archiving team and will correct or otherwise appropriately resolve any liens identified by the peer review(s). The Small Bodies Node of PDS will work jointly with the archiving scientists at ESA to prepare the complete ROSETTA archive within ESA consistent with all PDS standards (see AD2). The ROSETTA archive resides both at ESA and with NASA's PDS. PDS and the ESA archiving scientists will carry out the peer review of all data to ensure that outside users can make good scientific use of the data from the archive. The final archive will be maintained electronically both by the PDS Small Bodies Node and by ESA. ESA will prepare CD ROM (or successor media such as DVD ROM) copies of the archive for distribution both through ESA and through PDS. As refinements to the primary MIRO data products are made (e.g. improved calibration algorithms, higher-level data products), they will be stored in the MIRO database at JPL and, when appropriate, delivered to the PSA/PDS public archives. Members of the MIRO science team will also produce at their home institutions data products as needed to perform their science tasks. This page is intentionally left blank Archive Format and Content Format This section describes the format of the MIRO Instrument Team Archive volumes. Data in the archive will be formatted in accordance with Planetary Data System specifications (AD2). Volume Format This document will not be concerned with any particular media formats such as DVD?s because data will be delivered electronically. When applicable, media formats will be determined by the PDS. Also, for present purposes, datasets will be regarded as equivalent to volumes. Data Set Naming The informal Dataset Names used in this document are formed by appending the mission phase descriptor to the instrument name. Examples are: MIRO_THERMALVAC MIRO_PRL MIRO_EXT3 The formal PDS values for DATA_SET_NAME and DATA_SET_ID are formed according to the rules defined in AD1: DATA_SET_NAME = ?ROSETTA-ORBITER MIRO ?. DATA_SET_ID = ?RO--MIRO----?. Examples of DATA_SET_NAME are: ?ROSETTA-ORBITER CAL MIRO 2 GRND THERMAL-VAC V1.0? ?ROSETTA-ORBITER EARTH MIRO 2 EAR1 Earth-1 V1.0? Examples of DATA_SET_ID are: "RO-E-MIRO-2-EAR1-EARTH1-V1.0" ?RO-C-MIRO-3-ESC4-67P-V1.0? See AD1 for allowed values for these items. File Name Formats The following scheme will be used for names of files containing data products: MIRO___begindate. The level field is the CODMAC processing level. Valid names for the detector field include: MM SUBMM CTS HSK MMCAL SUBMMCAL CTSCAL CTSFOLDED MM1S SUBMM1S File extensions can be at least: DAT binary data TXT acsii data, lines of variable length, delimited typically with LBL ascii detached label file DOC text description where necessary In data file names, begindate format will be yyyyddd, where ddd is 1-based Julian day, i.e. Jan 1 is day 1. Standards Used in Data Product Generation PDS Standards The MIRO Data Products comply with the Planetary Data System standards for file formats and labels, as specified in the PDS Standards Reference (AD2). Time Standards The MIRO Data Products are intended to comply with the CCSDS Time Code Format Standard (CCSDS 301.0-B-2). The On-Board Time (OBT) of the Rosetta spacecraft is used in the PDS keywords SPACECRAFT_CLOCK_START_COUNT and SPACECRAFT_CLOCK_STOP_COUNT. The format of this time (as defined in RO-EST-TN-3165, Rosetta Time Handling) is: ?i/mmmmmmmm[.nnnnn]? where: i = integer signifying which zero point is in use. (Currently, all OBTs have i=1, signifying that the zero point is at 2003-01-01T00:00:00 UTC. This integer will change if the clock is ever reset, which is not planned but may happen as a result of unforeseen circumstances.) mmmmmmmm = integer seconds since the zero point. nnnnn = (optional) fractional seconds in units of 1/65536 sec. Therefore, the floating-point time since the zero point represented by a given OBT is: time = mmmmmmmm + nnnnn/65536. The OBT is not used internally in any MIRO data files. Instead, table entries are marked by Sun Modified Julian Time (SMJT) or ?unix time?, which is elapsed seconds since 1970-01-01T00:00:00 UTC. This takes leap seconds into account and is therefore in the UTC system. The conversion from SMJT to Ephemeris Time (ET2000), which is the standard TDB time system used by NAIF, is given by: ET2000 = SMJT ? 946727958.816 + LEAPSECS + O(0.0017) where the last term represents a sinusoidal correction for the Earth?s motion that never exceeds 0.0017 seconds, and LEAPSECS is the number of leap seconds that have been added between 1970 and the relevant date. A Fortran-77 program, named UTCCON, that converts between SMJT and the ISO standard UTC representation, is provided in the DOCUMENT directory. See AD7 for further discussion and conversions to other time systems. Data Validation and Flagging Data validation on MIRO archive products consists of basic checks on the completeness of Continuum packets and CTS spectra, automated flagging of some suspect data, and statistical analyses of the entire data set to confirm expected behaviors. CODMAC Level 2 archive products (de-commutated telemetry values) only have completeness checks performed on them. Version 2.0 and 3.0 CODMAC Level 3 (radiometrically calibrated) and Level 4 (higher-order) data products have warning flags set to indicate when the applied calibration is suspect and, for CTS data, if the frequency stability of the instrument is in question. In addition, products were tested in aggregate to check that noise statistics were approximately Gaussian (e.g. by making histograms of calibrated radiances looking at empty space), and that relatively few non-physical radiances were being generated (e.g. continuum brightness temperatures less than -5 K or above 400 K). Content This section describes the directories and contents of the MIRO Data Product volumes, including the file names, file contents, file types, and organization responsible for providing the files. The data described herein appear on each volume of the MIRO Data Product volume series. Volume Set Since the Rosetta Project plans for electronic delivery and there is no need to bundle several datasets into one volume set, as a general rule, a volume shall be a dataset. Data Set The following table shows data set name (informal), DATA_SET_ID, delivery date, and data types contained, for each volume of the MIRO Data Product volume series, as of August 2018. The naming follows section 3.1.2. Many data sets have been delivered with two or more version numbers (e.g. MIRO Level 3 Commissioning data has V1.0 and V1.1). In general, the highest version number is the best data set to use. Table 3.1: Data Sets Delivered. Dataset Name and DATA_SET_ID Delivery Date Description DIRECTORIES This section describes the contents of each directory in a Data Product dataset. Root Directory The following table lists the files located in the root directory. Table 3.2: Files in Root Directory. File Name File Contents AAREADME.TXT Introductory information about the contents and format of the volume. CATALOG Directory containing catalog files: mission, instrument, and dataset Descriptions which are duplicated in the PDS higher-level catalog. DATA Root directory for each data type present in this volume: Science (Spectroscopic and Continuum) Engineering, and Geometry. DOCUMENT Directory containing basic documentation. ERRATA.TXT Cumulative listing of comments and corrections. INDEX Directory containing index tables for the data files in this volume. LABEL Directory containing structure files references by PDS labels. VOLDESC.CAT Description of the contents of this volume in a PDS-labelled format. Appendix A contains a listing of the VOLDESC.CAT file for one of the datasets listed in 3.4.2. All others are similar. Catalog Directory This directory contains files providing a top-level description of the Rosetta mission and spacecraft, the MIRO instrument and its team, and its data products. The following table describes the files in the Catalog Directory. Table 3.3: Files in Catalog Directory. File Name File Contents CATINFO.TXT A description of the contents of this directory. DATASET.CAT PDS data set catalog information about the MIRO Data Product data sets. INSTHOST.CAT PDS instrument catalog information about the Rosetta Spacecraft. INST.CAT PDS instrument catalog information about the MIRO instrument. MISSION.CAT PDS mission catalog information about the Rosetta. PERSONNEL.CAT PDS personnel catalog information about the MIRO Team members responsible for generating the data products. REF.CAT PDS references mentioned in other files. SOFTWARE.CAT PDS software information about provided software (if any). TARGET.CAT PDS catalog information about the target bodies observed by Rosetta. Data Directory This directory contains two or three sub-directories. For CODMAC Level 2 data sets, the directories are Spectroscopic, Continuum, and Engineering, which each contain all the data files for the corresponding data type in the data set. For CODMAC Level 3 data sets, the three directories are Spectroscopic, Continuum, and Geometry, except for Version 1.0 of the ESC4, EXT1, EXT2, and EXT3 Level 3 data sets. For those data sets, no Geometry files are provided. The CODMAC Level 4 Version 1.0 DATA directory contains only Spectroscopic and Geometry subdirectories. The Level 4 Version 2.0 DATA directory contains Spectroscopic and Continuum subdirectories. Continuum Data Directory This directory contains science data files containing Continuum (MM and SMM) data, and their detached labels. The Level 4 Version 2 Continuum directory also contains geometry files for the continuum data (unlike most other data sets, in which a separate Geometry directory is used). Engineering Data Directory This directory (only in CODMAC Level 2 data sets) contains files containing Engineering data, and their detached labels. Spectroscopic Data Directory This directory contains science data files containing Spectroscopic (CTS) data, and their detached labels. Geometry Data Directory This directory (when present) contains geometry data files containing geometry data and their detached labels. Separate geometry files are generated for the Level 3 millimeter continuum, submillimeter continuum, and spectroscopic data. Separate geometry files are also generated for Level 4 continuum data products. (Note that Level 4 spectroscopic data do not have separate geometry files.) When geometry files are in a Data Set, there is a one-to-one correspondence between science data files and geometry data files, and all time tags within science data records are found in geometry data records. (Note that continuum data records contain more than one time tag---see Section 2.3.2---and geometry calculations are provided for each time tag, so there are more continuum geometry recrods than continuum data records.) Index Directory This directory contains index files providing summary information for all the data products in this data set. The following table describes the files in the Index Directory. Table 3.4: Files in Index Directory. File Name File Contents INDEX.LBL A volume index file. INDXINFO.TXT A description of the contents of this directory. INDEX.TAB Index table file for all data products. Label Directory This directory includes files referenced by data files on this volume, e.g. FMT files containing header descriptions. Document Directory This directory contains various files documenting the contents of this data set. Software is also in this directory. The following table describes the files in the Document Directory. Table 3.5: Files in Document Directory. File Name File Contents DOCINFO.TXT A description of the contents of this directory. MIRO_READ_DATA.ASC Either a python program capable of reading all MIRO data products generated through its release data, or (in some older data sets) a Fortran-77 program to list selected parts of certain MIRO data files. These programs are provided as additional documentation for the structure files. UTCCON.ASC A Fortran-77 program that converts between the time system used in the data files and standard UTC notation. RO_MIR_IF_001.PDF This interface document. USER_MANUAL.PDF The MIRO User Manual [AD4]. Other documents, as available, include log files (typically with names including MIRO_LOG or SCAN_LOG), ascii versions of documents, and corresponding LBL files. Data and Label Files Science and Engineering data files are placed in the appropriate subdirectories of the data directory (3.4.3.4), together with their detached labels. Other data files shall be placed in their appropriate directories, all with detached PDS labels. Detailed Interface Specifications In this chapter, detailed information about the archive design at instrument and detector level is given. Data Product Identification The basic MIRO data product is a binary file containing scientific or ancillary data in table format, and an associated detached label file in PDS format describing the data. The file naming convention for these files is given in 3.1.3. A data file contains a continuous stream of data for one of the MIRO instruments (CTS, MM radiometer, or SUBMM radiometer), for Engineering data, or for geometry information (see section 2.3). Note that the instrument Data Mode (section 2.3.2) could change several times a day (though typically it did not), and some data products are generated in multiple modes. For this reason, Data Mode is not stored in the meta-data of a product, but it is stored in each row of a data table (each row corresponding to a small time range). The length of a data file is arbitrary, being defined by the process of obtaining the data from the database, but it shall not exceed an observing time of one week. PDS Label Structure, Definition and Format The following keywords are used in the PDS labels for MIRO data products (with the values given when these will be invariant): PDS_VERSION_ID = PDS3 LABEL_REVISION_NOTE RECORD_TYPE = FIXED_LENGTH RECORD_BYTES FILE_RECORDS ^TABLE DATA_SET_NAME DATA_SET_ID MISSION_NAME = ?INTERNATIONAL ROSETTA MISSION? MISSION_ID = ROSETTA INSTRUMENT_HOST_NAME = ?ROSETTA ORBITER? INSTRUMENT_HOST_ID = RO INSTRUMENT_NAME = ?MICROWAVE INSTRUMENT FOR THE ROSETTA ORBITER? INSTRUMENT_ID = MIRO INSTRUMENT_TYPE = {?RADIOMETER?,?SPECTROMETER?} ^INSTRUMENT_DESCRIPTION = ?RO-MIR-IF-0001_16.TXT? TARGET_NAME TARGET_TYPE MISSION_PHASE_NAME ORBIT_NUMBER SPACECRAFT_CLOCK_START_COUNT SPACECRAFT_CLOCK_STOP_COUNT START_TIME STOP_TIME SC_SUN_POSITION_VECTOR SC_TARGET_POSITION_VECTOR SC_TARGET_VELOCITY_VECTOR SUB_SPACECRAFT_LATITUDE SUB_SPACECRAFT_LONGITUDE SPACECRAFT_ALTITUDE NOTE = ? The values of the keywords SC_SUN_POSITION_VECTOR, SC_TARGET_POSITION_VECTOR and SC_TARGET_VELOCITY_VECTOR are related to the ECLIPJ2000 reference frame. The values of SUB_SPACECRAFT_LATITUDE and SUB_SPACECRAFT_LONGITUDE are northern latitude and eastern longitude in the standard planetocentric IAU frame. All values are computed for the time 20xx-xx-xxTxx:xx:xx.xxx, the midpoint of the observations. Distances are given in , velocities in , angles in .? PRODUCT_CREATION_TIME PRODUCT_ID PRODUCT_TYPE PROCESSING_LEVEL_ID PRODUCER_FULL_NAME = {?Dr. Samuel Gulkis?, ?Dr. Mark Hofstadter?} PRODUCER_INSTITUTION_NAME = ?JET PROPULSION LABORATORY? PRODUCER_ID = JPL DATA_QUALITY_ID DATA_QUALITY_DESC = ?1 = nominal, 2 = problematical? OBJECT = TABLE INTERCHANGE_FORMAT = BINARY COLUMNS ROWS ROW_BYTES ^STRUCTURE = ?xxxx.FMT? END_ OBJECT = TABLE END The FMT file pointed to by the ^STRUCTURE keyword will contain the detailed specification of the contents of the data. The file pointed to by the ^INSTRUMENT_DESCRIPTION resides in the Document directory (3.4.3.7). No mission-specific keywords will be used. All keywords are defined in the PDS data dictionary (AD3 or online at https://pds.jpl.nasa.gov/tools/dd-search/). The coordinate system used for the geometric items in the label (SC...VECTOR) is the J2000 system, which is an inertial Cartesian frame based on the Earth mean equator of Epoch J2000. Overview of Detectors Spectrometer Data The contents of the spectrometer (CTS) level-2, -3, and -4 data products are fully defined by their structure files. For further details, see MIRO User Manual (AD4) Sections 7.1.3 and 9.8. Radiometer (Continuum) Data The contents of the mm and submm radiometer level-2, -3, and -4 data products are fully defined by their structure files. For further details, see MIRO User Manual (AD4) 7.1.4 and 7.1.5. Engineering Data The contents of the Engineering data products are fully defined by the structure file ENG_LEVEL_2_FORMAT.FMT. For further details, see MIRO User Manual (AD4) 7.1.2. Data Format Description The contents of the MIRO data files are fully defined by the *.FMT files in the LABEL directories of the archives. Here, a brief explanation is provided of the science-data portion of CTS and Continuum files. Additional information is in section 2.3. (The Engineering files are not discussed further as they are not likely to be of interest to the general user.) It is important to understand that the Data column(s) and, in CTS files, frequency columns, of the data products contain large data arrays. In the CTS files, the arrays contain a complete spectrum, whereas in the Continuum files this is a packet of data in time order. The name of the Data column is simply D in Level 2 files. It is also D in Level 3 continuum files. In Level 3 CTS files, the data arrays are D in Versions 1.0 and 1.1. In Version 2.0 and 3.0 Level 3 CTS files, the data arrays are D and D_INTERPFREQ, with corresponding arrays of frequency information in IFREQ and IFREQ_INTERP. Data arrays are also present in the Level 4 Version 1.0 and 2.0 files (see the Level 4 Structure Files). A very important item is the Cal flag in the data products. When this flag is 0, then the spectrum is for a calibration sequence. When the flag is 1, then the data are science observations. It is unfortunate that Cal=0 means calibration, but this is a historical accident which, if changed, risks greater confusion. Note that calibration data is only included in CODMAC Level 2 files. Level 3 and 4 files only contain science data (Cal=1). See Appendix B for a description of an IDL-based tool to read MIRO data that is provided by PDS, called READPDS. A python program called MIRO_READ_DATA is present in the DOCUMENT directory within some MIRO data sets. It is not maintained, but serves as an example of how to read all data formats used by MIRO. See Appendix B for details. Frequency calibration: the total bandwidth of MIRO?s CTS is 180 MHz, with the frequency increasing inversely with the bin (channel) number of the spectra, in an approximately linear fashion. The exact dependence is dependent on the temperature, which is why the number of bins are increased from 4096 for the raw data to 4152 for the calibrated data generated by Pipeline software Versions 2.0 and 3.0 (4250 in earlier software versions). This is described in Section 9.4 of the Users Manual (AD4), which is included in the DOCUMENT directory of the archive. This also describes how the true frequencies of the lines observed (which span 33 GHz, far more than the nominal bandwidth) are mapped into the CTS spectrum. Discontinuities between the eight regions of the different mappings appear as smooth transitions, because of the design of the CTS. Data in the transition regions between these bands are not usable. Appendix A, VOLDESC.CAT What follows is a sample VOLDESC.CAT file for a Level 3 Version 3 product. All other VOLDESC.CAT files are similar. PDS_VERSION_ID = PDS3 RECORD_TYPE? ? = "STREAM" LABEL_REVISION_NOTE = "2018-05-11 L. Pan updated calibration algorithm and documentation" OBJECT ? ? = VOLUME ? DATA_SET_ID? = "RO-C-MIRO-3-ESC2-67P-V3.0" ? VOLUME_SERIES_NAME = "ROSETTA SCIENCE ARCHIVE" ? VOLUME_SET_NAME = "ROSETTA: MIRO DATA" ? VOLUME_SET_ID = "USA_NASA_JPL_ROMIR_1036" ? VOLUME_NAME = "CALIBRATED MIRO DATA FOR THE ROSETTA COMET ESCORT 4? PHASE" ? VOLUME_ID = "ROMIR_1036" ? VOLUME_VERSION_ID = "VERSION 3" ? VOLUMES = 1 ? VOLUME_FORMAT = "ISO-9660" ? MEDIUM_TYPE = "ELECTRONIC" ? PUBLICATION_DATE = 2018-05-11 ? DESCRIPTION = "This volume contains the data from the Microwave Instrument for the Rosetta Orbiter (MIRO). It contains calibrated data obtained during the ROSETTA COMET ESCORT 2 Mission phase." ? OBJECT ? ? = DATA_PRODUCER ? ? INSTITUTION_NAME = "JET PROPULSION LABORATORY" ? ? FACILITY_NAME = "MIRO DATA PROCESSING TEAM" ? ? FULL_NAME = "MARK HOFSTADTER" ? ? ADDRESS_TEXT = "JET PROPULSION LABORATORY \N ? ? 4800 OAK GROVE DRIVE \n ? ? ? MAILSTOP 183-301 \n ? ? ? PASADENA, CA 91109 \n ? ? ? USA" ? END_OBJECT = DATA_PRODUCER ? OBJECT ? ? = CATALOG ? ? ^MISSION_CATALOG = "MISSION.CAT" ? ? ^INSTRUMENT_HOST_CATALOG = "INSTHOST.CAT" ? ? ^INSTRUMENT_CATALOG = "INST.CAT" ? ? ^DATA_SET_CATALOG = "DATASET.CAT" ? ? ^REFERENCE_CATALOG = "REF.CAT" ? ? ^PERSONNEL_CATALOG = "PERSONNEL.CAT" ? ? ^SOFTWARE_CATALOG = "SOFTWARE.CAT" ? ? ^TARGET_CATALOG = "TARGET.CAT" ? END_OBJECT = CATALOG END_OBJECT = VOLUME END Appendix B, Available Software to Read PDS Files AVAILABLE SOFTWARE TO READ PDS FILES The MIRO data files can be read by PDS-supported software such as NASAVIEW. The PDS also provides an IDL routine, called READPDS, for reading archived data. It can be found on-line at: https://pds-smallbodies.astro.umd.edu/tools/tools_readPDS.shtml The MIRO team has provided, in the DOCUMENT directory, a python program named MIRO_READ_DATA. In Level 3 Version 3 data sets, it is a python program that can read all MIRO data formats for comet data as of the release date of the data set. The program serves as an example for understanding the data structure of MIRO products and for developing data processing software. Limitations of the python version of MIRO_READ_DATA to be aware of are: * MIRO_READ_DATA does not read archived data for mission phases prior to encountering the comet (i.e. prior to PRL). * The last three fields in MIRO Level 2, Version 1.0 Engineering data are "bit column" fields. This means each is a sequence of 1 to 16 bits, each bit being an independent data field, but the bits are stored as a 2-byte integer. MIRO_READ_DATA does not separate out the separate bit column values, and reports each bit column field as a single integer. * As described in the format files associated with each data delivery, MIRO products use several different formats. MIRO_READ_DATA uses the name of the input file to identify the correct format to use, so the naming conventions used in the PSA/PDS archive must be maintained when using MIRO_READ_DATA. Note that MIRO_READ_DATA expects filenames to be in all capital letters. In older data sets, a FORTRAN program may be found. The FORTRAN version only works on Version 1 of Level 1, 2, and 3 data sets. The python version should be used whenever possible. This page is intentionally left blank