============================================================================ ============================================================================ SOHO 1996 ============================================================================ ============================================================================ Reviewers: K. Battams, L. Feaga General ------- - Need specific provenance for comets in this archive. - Strongly consider which data products are archived at PDS. Is a fully-calibrated level 1.0 image required here? All current publications use level 0.5 data. These data are well understood by the scientific community and are already available in an archive. M. Knight's work is the intermediate product for generating light curves. Ultimately, archive the light curves. This is the most crucial data. - Recommend taking level 1.0 data product and write a simple routine to rescale for our work, then we don't need all the calibraiton products, etc. - For absolute completeness of the SOHO comet archive at PDS, consider archiving: o Comets discovered by SOHO SWAM instrument o Data from observations of comets by the SOHO UBCS instrument - Smaller images, 512x256, give a comet position is not on the image, e.g. data/1996a/1996a2_c3_****.fit. These images should be identified. This may occur on any image that does not have one dimension that is 1024. May need to manually look at these images to see if position is outside the frame. Affects comet post files, not astrometry files. After 1997-1998, this happens infrequently since nearly all data are full-frame. See note for data/1996a2/ in L. Feaga's list appended below. - Verify documentation correctly describes what is on the volume. - Implement changes and corrections noted by K. Battams' (appended below). Most of these items are duplicated in this section and the following sections. - Make corrections noted by L. Feaga (appended below). List includes typos and inconsistencies in voldesc.cat, aareadme.txt, /document, /calib, /index, and /data. - Make corrections noted by A. Raugh (appended below, from "problems.acr"). Images ------ - Problems remain with the LASCO images in the PDS SOHO archive. As they stand, they are not suitable for release/archiving. M. Knight and K. Battams are still working on the processing issues (scaling for dynamic range is not correct). C2 pretty good. C3 needs work because comet was "erased" in nearly all of the of these images. Labels ------ - Problem with image start and stop times: o LASCO is accurate to only to one tenth of a second. o Absolute times (SOHO clock) are accurate to better than 15 seconds. Consider removing stop time? See Chapters 7 & 8 of the LASCO C3 calibration paper which discusses this topic in detail. /document/comet_pos/sub-directory --------------------------------- pos_1996.tab - Missing constants are incorrect. Document claims they are of the form -9999.99 but in table, just ----'s. - Three different image times are given; reconsider the inclusion of these time formats in favor of a single corrected start time - Add the corrected exposure time to this file - Missing "T" in ISO-formatted dates pos_1996.txt - Add note that post Aug 2004, the astrometry files contain extra information in the first row (SOHO#, Camera, Discoverer) - Clarify that the column "know" differentiates comet positions (know=0) from star positions (know=1) - Remove the comment "For images which do not have an identified comet, these values are replaced to -9999.99 of the same width as the corresponding field."? The astrometry file does not have entries with this value. /document/comet_elements.txt ---------------------------- - Remove inconsistencies in spacing for the name of comet discoverer. References ---------- - Review and add to list ""Calibration of the SOHO/LASCO C3 White Light Coronagraph", Morrill et al, Solar Physics (2006) 233: 331-372. calib/filter/*.tab ------------------ - Some values are negative, do not change to 0; note this in description/ index.lbl --------- - Add column descriptions. - Describe how SOLAR_NORTH_POLE_CLOCK_ANGLE is measured in an image. ______________________________________________________________________________ ______________________________________________________________________________ Karl Battams' List of Corrections --------------------------------- Images (Processing) ------------------- - Problems remain with the LASCO images in the PDS SOHO archive. As they stand, they are not suitable for release/archiving. There is ongoing correspondence between myself and Matthew Knight to resolve the processing issue(s), so unless requested (...??), they will not be discussed further at this point (with the exception of a brief mention in the summary). Labels ------ - Image "Start" and "Stop" time: Within the .lbl files for each image (also the "keyword_glossary" file), the beginning and end time of each individual exposure is given. (The "STOP_TIME" value is calculated by "adding the exposure duration value to the start time as extracted from the FITS file". Additionally, the exposure duration is corrected using the exposure correction factor.) The "start","stop" and "exposure duration" values are formatted to 1000ths of a second. It must be noted in the documentation that after all appropriate corrections: 1. LASCO exposure durations are accurate to better than 1/10th second 2. Absolute times (SOHO clock) are accurate to better than 15 seconds - The current scheme implies (to me, at least) a precision in time recording that does not exist within any SOHO data product! - Consider removing "STOP_TIME" value in data label files (and associated documentation). - Chapters 7 & 8 of the LASCO C3 Calibration paper (see below) discuss this in detail. Documentation ------------- /document/comet_pos/ sub-directory pos_1996.tab - Missing constants are incorrect. The documentation claims they are of the form -9999.99 (or variations thereof), but in the actual pos_1996.tab file, they are just ----'s. - Three different image times are given: 1. Corrected (calibrated) start time 2. Mid-time (corrected UTC time of s/c at midpoint of image) 3. "Old" time (uncorrected UTC time of s/c at midpoint of image) - Bearing in mind the aforementioned uncertainties, reconsider the inclusion of these three time formats in favor of a single (corrected) start time _ Add the corrected exposure time to this file pos_expl.txt - Note that post Aug 2004, the astrometry files contain extra information in the first row (SOHO#, Camera, Discoverer) _ Maybe clarify that the column "know" differentiates comet positions (know = 0) from star positions (know = 1) - Regarding the statement "For images which do not have an identified comet, these values are replaced by -999.999 of the same width as the corresponding field." -- there are no such lines in the astrometry files as they exist now. Only images with an identifiable comet were used for SOHO astrometry measurements. If this "missing" data is to be subsequently added to the astrometry files, it will need to be made very clear that the new measurements were performed "after the fact", on a different machine/OS, with a different operator (SOHO measurements can be subjective...), and potentially with a different version of software. /document/comet_elements.txt - Inconsistency in spacing for the name of comet "discoverer". Example: "X.Leprette" and "X. Leprette". This could cause issues for someone who has written a whitespace-dependent program of some kind... References ---------- _ Review (and add to list) "Calibration of the SOHO/LASCO C3 White Light Coronagraph", Morrill et al, Solar Physics (2006) 233: 331-372 Bottom line ----------- - The processed images continue to be a problem and are still not suitable for the PDS archive. Again, I will restate my previous recommendation that the use of a 'Level-1 variant' data product be reconsidered in favor of the well- established and widely available level-0.5 data (upon which all current SOHO comet data and papers are based). Additionally/alternatively, consider the use of the easily available NRL LASCO Level-1data product, which would only required a conversion to the desired units. - Aside from the images, and the few notes described earlier, the SOHO data archive appears to be in very good shape. Developers should keep in mind the possible future addition of (the currently incomplete/unpublished) SOHO comet photometry data. For absolute completeness of the SOHO comet dataset, additional consideration should be given to including: 1. comets discovered by the SOHO SWAN instrument, and 2. data from observations of comets by the SOHO UVCS instrument ______________________________________________________________________________ ______________________________________________________________________________ Lori Feaga's List of Corrections -------------------------------- voldesc.cat - MISSION_CATALOG entry not correct, the file name was shortened. aareadme.txt - Missing comet_elements.txt and eventual label should data directories be listed separately? - Change Puneet as contact person document/docinfo.txt - Be consistent in "PNG", "png", "pdf" as all caps or all lower case document/capture_dates.txt - Missing a lot of files that I believe are downloaded from LASCO: o document/spacecraft_info/time_and_rotation_axis.txt o document/calib_docs/filter/ 13 plots o document/calib_docs/quantum_eff/ 2 plots o document/sun_position/sc_body_axis_avg_roll.tab o document/sun_position/sunpos.txt o document/comet_pos/soho*.txt (26 files) o document/telemetry_stats/tm_plot_1996_* (12 files) o calib/filter/*.tab (11 files) o calib/quantum_eff/*.tab (2 files) o calib/vignetting/*.fit (3 files) document/comet_elements.txt - M3 and M2 are "out of order", make sure the rows have the correct data. YES, Karl said in perihelion order document/keyword_glossary.txt - Compared with an actual data label, the following keywords are not defined here, should they be included? o PDS_VERSION_ID o RECORD_TYPE o RECORD_BYTES o FILE_RECORDS document/inst_docs/handbook.txt - Spelling error from old liens: near the end of the file, "accessable" should be "accessible" document/inst_docs/temperature/*.png files - Not of greatest quality document/comet_pos/astrometry.txt - "pos_1996.*" should just be "pos_1996.tab" document/comet_pos/pos_1996.tab and .lbl - Missing data values are "-----" in the .tab and are specified as "-999.99" in .lbl - Space between date and time, should be one single column w/ "T" - Impose format for values in table especially sun_comet_dist and spacecraft_comet_dist - Label values for RECORD_ BYTES, FILE_RECORDS, ROW_BYTES, ROWS, BYTES, FORMAT, MINIMUM, MAXIMUM, START_BYTE need to be checked/corrected document/comet_pos/soho*.txt - All 26 files need labels document/sun_position - Some bad quality images: o pes_a.png o bs.png o 960101sc_roll.png calib/filter/*.tab - Small negative numbers (-1<#<0) in the tables for transmittance values, but in the label, there is a note that these run from 0-100, so note there is noise index/index.tab - Many columns do not have correct description and fix all BYTES, START_BYTE, FORMAT, etc. data/1996*/*.lbl - Check FILE_RECORDS value data/1996a2/ - All data are 512x256, comet is listed as being in a y pixel out of the appropriate range in the pos_1996.tab ______________________________________________________________________________ ______________________________________________________________________________ Anne Raugh's List of Corrections (from the problems.acr file) ------------------------------------------------------------- 30 March 2006 ============= Known problems with SOHO 1996 update (PDS Standards) GENERAL o wrong DSID throughout aareadme.txt o Refers to this as separate data set. o Sentence describing the DATA/ directory makes no sense o Does not mention document/keyword_glossary voldesc.cat o Does not list target catalog files (these are missing) calib/vignetting o All files missing labels catalog/ o Missing all TARGET files catalog/dataset.cat o Syntax errors o Missing DATA_SET_MISSION catalog/lasco.cat o Syntax errors catalog/reference.cat o Syntax errors o Incomplete REFERENCE_TEXT for BIESECKERETAL2002 o Incorrect REFERENCE_TEXT for FLECKETAL1995 document/ o Unlabelled files: comet_elements.col, comet_elements.txt document/docinfo.txt o Information in here does not match file list in the actual directory. document/capture_dates.lbl o RECORD_TYPE, RECORD_BYTES and FILE_RECORDS are incorrect document/keyword_glossary.lbl o Invalid RECORD_TYPE o Incorrect RECORD_BYTES document/calib_docs/exp_fac/expfac.lbl o RECORD_BYTES, FILE_RECORDS, RECORD_TYPE incorrect document/* o SAME ERRORS AS ABOVE FOUND IN ESSENTIALLY ALL LABELS IN THE DOCUMENT/ TREE index/index.lbl o ROWS, ROW_BYTES, RECORD_BYTES, FILE_RECORDS all incorrect. o index.tab is improperly formatted. o Column definitions don't work ============================================================================ ============================================================================ SOHO 2000 ============================================================================ ============================================================================ o Some of the LASCO calibration files used were out of date, or will be shortly. These are now stable, but the calibration will need to be rerun. There is an IDL library (SolarSoft) available. o Add a reference to the SolarSoft IDL library with a descriptive note to the data set description. o Add a table with orbital elements and family membership (if that makes sense) for the comets. o Consider whether this should be one complete data set rather than one for each year. (This data set will require a special interface to select data by family.) o Consider adding Doug Beisecker's published photometry to this data set. o Include SOHO atrometry files. o Consider who to credit in the label and how for the people involved in discovering the comet in the image. o Consider what to do about "X" comets - uncertain detections (fewer than five reasonably accurate positions) for which only minimal coordinates are available. o Make corrections indicated in K. Battams presentation "Documentation" slides (appended below). o Separate vignetting functions for the different filters are not necessary. One full-resolution function for each band is sufficient. o Make corrections included in Lori's presentation (appended below). o Make sure positions in the astrometry table do correspond to comet positions for faint comets. o Consider adding a family subdirectory under the "data/" directory. o Correct typos noted by L. Feaga (on paper in Anne's office). o Make corrections noted by T. Farnham (appended below). o Make corrections noted by S. Adams (appended below). ______________________________________________________________________________ ______________________________________________________________________________ Karl Battam's Documentation Corrections --------------------------------------- In the catalog/dataset.cat file: "Data Set Overview" - "Nearly all of the SOHO discovered..." should read "Approximately 95% of the SOHO-discovered..." - Mention that synoptic programs are full-resolution (1024x1024). - Note increased LASCO telemetry during 'keyholes'. - Note reduced FOV during the early part of the mission. "Processing" - Level-0.5 and/or Level-1 processing should perhaps be explained with reference to SolarSoft (software library). - Clarify the difference processing levels. [Details are on the presentation slides. See PDS_SOHOComet.pdf -AR] "Calib" - Note the ZP's might change (calibration papers from NRL/Marseille due for publication) "Processing" (cont...) - "vignetting corrections have remained constant throughout the mission" o rephrase o not true for c3 (post-interrupt) ______________________________________________________________________________ ______________________________________________________________________________ Lori Feaga's List of Corrections -------------------------------- SOHO/calib/filter/*.lbl - In the DESCRIPTION of transmittance (column 2), range should be 0-100. - c2_ha.*, c3_ha.* and c3_i75.* units for wavelength are inconsistent from label to table file SOHO/catalog/dataset.cat - In "Processing" is this: "vignetting corrections have remained constant through the mission" yet there are versions 1 and 2 for some of the filters and some filters have multiple files (different filenames). - In "Data": "There images are listed in the reduction.log file found in the documents directory." but there is no such file. SOHO/document/capture_dates.txt - There is an asterisk after the listing for handbook.txt. Why? - For the list of files in document/calib_docs/exp_fac, only 24 of 192 files were actually available. SOHO/document/calib_docs/expfac.txt - In the description of the exposure factor plots, several colars are mentioned to distinguish the method of calculating the exposure factor, but I only saw red and green points. [Note: Using color to encode important information is strongly discouraged. There are quite a few color-blind astronomers out there. -AR.] SOHO/document/calib_docs/filter - Missing *.tab files as described in filter.txt - Files reside in /calib/filter/, needs a pointer to that directory. SOHO/document/calib_docs/filter/c2_all.png - Plot of filter transmission has 2 orange filter curves with no explanation. SOHO/document/calib_docs/filter/c3_all.png - Doesn't include the transmission curve of the polatization I-75 filter. SOHO/document/calib_docs/quantum_eff/qe.txt - ASCII tables are in /calib/quantum_eff/ directory, not here. - c3_qe.ps plot is also available, but not listed. SOHO/document/calib_docs/vignetting/vign.txt - Vignetting images are found in the /calib/vignetting directory, a pointer to this directory should be added. - Missing file: c3_cl_vig_951201_v03.fit SOHO/document/comet_pos/pos_expl.txt - "values are replaced by dashes of the same width as the corresponding field" isn't true - they're replaced by -999.999 of same width. SOHO/index/index.lbl - Units need to be added to columns 5, 6, 7 and 11. Liens from previous reviews still unresolved: /calib/filter o Should there be a unit mentioned for transmittance, i.e., percent? o Pointer still needed to the PNG files in the label files. /calib/quantum_eff o Should there be a unit for quantum efficiency? o Pointer still needed to the PNG and PS files in the label files /document/handbook o Spelling error remains, "accessable" Typos and grammatical errors supplied on paper. ______________________________________________________________________________ ______________________________________________________________________________ Tony Farnham's Corrections -------------------------- In AAREADME.TXT Contact information for Puneet should be changed. In EXPFAC.TXT file: Launches right into an explanation about what is done. Should have a sentence (or reference to another file) that describes why it is being done. In POS_EXPL.TXT States that if the comet is not identified in the frame, the phase angle etc are replaced by dashes. In fact they are replaced by the missing data value ______________________________________________________________________________ ______________________________________________________________________________ Steve Adams' Corrections ------------------------ 1. root directory A. dataset.html 1. delete this file on final archive B. aareadme.txt 1. Directory tree data directory missnamed. C. voldesc.cat 1. Files listed in CATALOG object do not match what is on the volume. 2. calib directory A. Nothing (documentation) on the volume that explains how to conduct calibration with this data. Fifty or a hundred years from now, long after we have retired, how will anyone understand what is here and how to utilize it? 3. catalog directory A. catinfo.txt 1. Files referenced in this document and what is on the volume don't match each other. B. dataset.cat 1. Delete comment after DATA_SET_RELEASE_DATE, not needed and makes ingestion fail 2. CITATION_DESC = Move opening quote to the same line as the keyword. 3. missing required object, (i.e., DATA_SET_MISSION) C. lasco.cat 1. no comments D. reference.cat 1. List all authors in citation a. BIESECKERETAL2002 b. BRUECKNERETAL1995 E. soho.cat 1. no comments F. solar_and_heliospheric_obser.cat 1. no comments 4. document sub-directory A. docinfo.txt 1. /document/calib/ directory and its sub-directories listed are not on the volume. See 2.1. 2. /document/comet_pos/ directory listed is missing on this volume. 5. index sub-directory A. index.tab 1. index.tab refers to a data directory. There is no data directory on this volume. Perhaps your "sample" directory should be named "data"? 6. index sub-directory A. no comments 7. sample sub-directory (data sub-directory?) A. no comments