Reduction of SQIID Data Using the IRAF upsqiid Package K. Michael Merrill, NOAO/KPNO 16 February 2001 ------------------------------------------------------------------------ Return to SQIID Web Page ------------------------------------------------------------------------ * I. Overview <#over> * II. Data Sets <#data> * III. DARK, FLAT, and SKY Frames <#dark> * IV. Processing of Individual Images <#proc> * V. Building a Database from a Dataset <#buil> * VI. Specifying the Relative Spatial Offsets <#spec> * VII. Creating the Final Images <#crea> * VIII. Registration summary <#regi> * IX. Handling larger images <#hand> o Appendix A: Some Useful Tools <#appa> o Appendix B: UPSQIID package <#appb> o Appendix C: SQIID Specific Addenda <#appc> NB: The UPSQIID package is not an officially released and supported IRAF package; all queries should be directed to KMM at NOAO (merrill@noao.edu; 520-318-8319). Although specifically targeted for SQIID, the routines are suitable for other image data as well. See APPENDIX B for more information. ------------------------------------------------------------------------ I. Overview The NOAO SQIID Infrared Camera produces simultaneous images of the same field in the J, H, K, and narrowband L passbands, using individual 512X512 quadrants of ALADDIN InSb arrays. The observations are generally background (photon statistics) limited. Typical observing programs include: * taking a few (2-5) exposures on the same target with small offsets (to counter ghosts and bad pixels and improve spatial sampling of the images); * taking many exposures of the same target with a dither pattern of offsets (to build up long exposures); * spatial mosaics of dithered pairs of images covering larger regions with limited overlap between images (to build up large images). These three kinds of observations are distinguished because they require somewhat different data reduction strategies. This document describes a set of IRAF programs (created by Michael Merrill at NOAO) designed to facilitate the reduction of SQIID datasets. These programs reside in the IRAF package "upsqiid". They have a number of imperfections in the user interface (especially a large number of irrelevant parameters) and currently do not have help files. We will define a "dataset" to be the set of direct observations of a given field. These may be dithered observations of a single target or a mosaic of a larger region. In the extreme case, the dataset may contain only a single exposure (and only the first two steps listed below would be required). The basic path for the reduction of SQIID dataset can be described as follows: * Create DARK, FLAT, and SKY frames for each passband. * Process each individual source frame to remove instrumental dark current, sky background, and pixel to pixel sensitivity variations. * Combine all images of a given target into a "database". * Interactively define the relative spatial offsets between each image in the database for one of the channels. * Combine the images in the database into a single image suitable for analysis, using bad-pixel masks to exclude bad pixels. Users are cautioned that IR image datasets often present a greater data reduction challenge than optical CCD images both due to the superior performance of optical CCD detectors (lower dark current, readout noise, and pixel to pixel sensitivity variations) and especially due to the extreme background limited nature of most IR observations. The results at each step in the process should be carefully examined and problems understood before proceeding. Many problems can be solved by the exclusion of bad images from the datasets. ------------------------------------------------------------------------ II. Data Sets a) Image names and channel ID The SQIID system generates FITS images with the following naming convention: "base_name"//XXX//"channel_id"//.fits (e.g., n1_081j.fits) where "base_name" ("n1_") was supplied by you at the time of observation, "channel_id" is j,h,k or l, and the number "XXX" is sequentially numbered (it was automatically incremented for each exposure until a new "base_name" was declared, typically at each new object). See APPENDIX A.2 for further information on image naming conventions and data restoration. To facilitate the handling of the large numbers of image sets typically obtained during a SQIID observing run, the SQIID software is designed to operate on IRAF "@list" files (inside IRAF, one instructs the system to read data from the file "list" by prepending "@" to it; "@list" tells the task to read list to get the image names.) The task 'chlist' is provided to create list files. (See APPENDIX A.3 for further discussion of the "chlist" tasks.) To use 'chlist' the image names must end in an integer. For example, chlist raw001 20 > jlist will create a list file name "jlist" (note that the output is to the standard output) with 20 image names starting with "raw001, raw005, ..., raw077". The increment between names is the hidden parameter 'delta' which defaults to 1. Directory pathnames can be prepended to the baseid. To facilitate data processing of image lists, the RECHANNEL, SQSECTIONS, SQPARSE tasks have been provided to assist in producing names of alternative channels from a given channel image name: RECHANNEL produces a list of images from an specific channels from a list of another channel. RECHANNEL: image = Image list channel = Image color: |j|h|k|l| (section = "") Image subsection [xmin:xmax,ymin:ymax] SQSECTIONS functions like the IRAF standard SECTIONS task, allowing one to count images resulting from a list or image template, output assorted subsets. "ch_id" allows one to produce a specific channel output from any channel input. SQSECTIONS: images = Image template (option = "root") Option (nolist, fullname, root, section) (nimages = ) Number of images in template (ch_id = "") Returned channel_id (jhkl) SQPARSE: image = Image name (root = "") Returned filename root sans channel (chroot = "") Returned filename root with channel (ch_id = "") Returned channel_id (jhkl) (alt_ch = "") Returned alternative channel(s) (extn = "") Returned image extension (section = "") Returned image section (verbose = no) Print results b) Orientation Since the individual channels of SQIID illumine a single quadrant of a four quadrant array and the best quadrant of each device is selected, the relative orientation on the sky is channel specific. Until the image saver task itself is modified to properly orient the channels (North up and East to the left) in the saved images, this task needs to be performed after the images are saved to disk. * [NOTE: As of September, 2000, the saver task correctly handles the orientation issue during the saver process. Images saved to disk (and automatically displayed) are properly oriented with North up and East to the left.]* The SQIID software can handle this several different ways. However, it is important to keep track of the orientation issue as you process data. The channels are oriented as follows: J H K L W N S E S N W E E W N S E S N W One can process all the data, producing sky frames, darks, flats, and masks, for each channel "as is". One can then use the orient parameter to select within the basic processing tasks (USQPROC, PATPROC, MOVPROC) to determine the final orientation: * ("orient = yes", "orient+") generates output images with all channels properly oreinted on the sky (North up and East to the left) from "as is" constituents * ("orient = no ", "orient-") leaves the orientation of the output images unchanged Alternatively, one can orient the raw data prior to processing and sky oriented consituents throughout the processing. The CHORIENT task allows one to orient images one at a time, using the image name template from one channel to optionally orient all channels. The image header keyword "ORIENT" is added to signal that the orientation process has occured. One can either orient in place, without changing the name of the image, or, using the "newid" string to append an identifier between the image root and the channel id. CHORIENT: image = Image name (channels = ".") Orient these channels (eg, "j", "jhk"; "." == named image channel only (verbose = no) Print results (newid = "") ID append to existing name? Starting with image = n1_001j.fits: * channels = "." orients n1_001j * channels = "j" orients n1_001j * channels = "k" orients n1_001k and leaves n1_001j alone * channels = "jhk" orients n1_001j, n1_001h, n1_001k By convention, the channel ID always remains at the end. * newid = "" leaves the name alone as n1_001j.fits * newid = "_ok" produces n1_001_okj.fits Note that images which have already been oriented (which have retained the image keyword ORIENT = yes). ------------------------------------------------------------------------ III. DARK, FLAT, and SKY Frames Proper data reduction requires accurate solutions for the small additive effects of internal illumination and charge generation (DARK frames), the large additive effects of sky illumination (SKY frames) and the multipicative effects of position dependent pixel sensitivity (FLATFIELD frames). The creation of DARK, FLAT, and SKY calibration frames are the first step in the data reduction process. The DARK frames are simple to obtain and process but the FLAT and SKY frames are more difficult to create and are crucial to the quality of the final images. Compared to optical band CCD observations, infrared observations are extremely background limited. Furthermore, the background in the near-infrared is variable at many temporal and spatial scales. System SKY frames are established from observations with the same integration time and near in time and place to the source frames. This is done by interspersing frames of off-source (ideally) blank sky fields in the sequence of source observations. Many of these sky frames, each at different position, are then median filtered to remove celestial sources to generate a SKY frame, which is measure of the temporally stable illumination on the array, including dark current and fixed pattern noise. System FLATFIELD frames are established using the sky as a uniform external illumination source. The background in the IR results in sky levels with S/N of better than 100 in a single 3 minute exposure. By taking the median of a reasonable number of blank fields (or target fields without large objects) obtained at different times during the night and at different locations on the sky, a satisfactory measure of the system flatfield can be derived. a) Dark Frames Generally, DARK frames (taken with the internal cold dark slide in place) are stable over a night (at about 1 electron/sec) and are probably stable over an entire observing run. Changes in dark current can accompany changes in the ALADDIN array temperatue. Since the SQIID dark current has both a base level and a time-dependent component, a dark frame must be created for each exposure time (that you intend to use to determine flatfields or for dark subtraction) during the observing run (it is assumed here that the observer has obtained such dark exposures). DARK frames are created with the task "SQIID". For example, usqdark @ldarks ldark will convert the list "ldarks" of L filter dark images (see chlist above) into the DARK frame "ldark". The task "usqdark" has a number of parameters which should be set as indicated in the "lpar" listing below. This format will be used henceforth to document the parameter default settings and the parameters of interest to the user (which are indicated with an * in the leftmost column). Parameters which feed directly into the IMCOMBINE task set apart. USQDARK: * input = Input raw dark images * output = Output IMCOMBINED dark image * (stat_calc = yes) Calculate statistics? * (statsec = "[300:700,400:600]") Image section for calculating statistics * (logfile = "STDOUT") Log file name * (verbose = yes) Verbose output? # IMCOMBINE parameters ========================================================= (common = "none") Pre-combine common offset: |none|median|mode| * (reject_opt = "minmax") Type of rejection operation * (comb_opt = "average") Type of combine operation: |average|median| (lthreshold = INDEF) Lower threshold for exclusion in statistics (hthreshold = INDEF) Upper threshold for exclusion in statistics (blank = 0.) Value of output pixel when all are rejected (expname = "") Image header exposure time keyword * (nlow = 1) minmax: Number of low pixels to reject * (nhigh = 1) minmax: Number of high pixels to reject (nkeep = 0) Min to keep (pos) or max to reject (neg) (lsigma = 3.) Lower sigma clipping factor (hsigma = 3.) Upper sigma clipping factor (pclip = -0.5) pclip: Percentile clipping parameter # IMCOMBINE parameters ========================================================= See APPENDIX A.4 for a discussion the input image parameter. The 'statsec' default should generally be satisfactory. The 'reject_opt' should probably be "minmaxreject". The 'comb_opt' should probably be "average" if the number of dark exposures is less than or equal to 5 and "median" if greater than 5. One does not normally threshold DARKS or bring the data to a common zero-point offset (thus, 'common = "none"'). If you want to keep a record of the USQDARK operations set 'logfile' to a filename (else set to ""). Setting "verbose = yes" increases the amount of information reported (including statistics). Setting "stat_calc = yes", assures that complete statistics will be reported. USQDARK calls "images.immatch.imcombine". See APPENDIX A.6 for more information on IMCOMBINE. b) FLAT Frames The pixel to pixel sensitivity (i.e., the flatfield) is also generally stable over the night and perhaps several nights. Unlike optical CCD observations, images of the interior of the dome or the twilight sky are not practical for the flatfielding of infrared images. Single filter infrared observations of faint sources were typically limited by variations in the background during chopping but the use of the single detector meant that "flatfielding" was not required. With the field of view of infrared array detectors now larger than the sources of interest, the sky background may be relatively easier to compensate for (see below) but since infrared sources are often much fainter than the sky background very precise flat field calibration is required. The background in the IR results in sky levels with S/N of better than 100 in single exposures (order 30 sec at J, H, K and order 1 sec at L band). By taking the median of a reasonable number of blank fields (or target fields without large objects) obtained at different times during the night and at different locations on the sky, a satisfactory measure of the detector flatfield can be derived. A list of suitable observations, carefully edited to exclude fields with sources larger than the dithering range, should be prepared. FLAT frames are created with the task "usqflat". For example, usqflat @limages lflat ldark will create the FLAT frame "lflat" using the list "limages" of L filter images and the DARK frame "ldark". The flat frames should be examined and compared to those created on other nights. A common problem is the presence of a large source (e.g. a galaxy) in the center of many observations. This can be avoided by the observation of sky frames, the combination of distinctly different observing programs during the same night, and the use of sufficiently large dithering motions. Post observation options include the use of carefully selected observations which avoid overlap of the stronger sources. Since direct illumination of the array is possible (remember that IR secondary mirrors are usually undersized), observations near bright sources (such as the Moon), which have atypical illumination, should not be used to determine global flatfields. Observations of illuminated targets inside the dome and during twilight will also have illumination atypical of nighttime observations, and should be used with caution. There are several problems with the twilight sky: * It is changing rapidly - 15 minutes into astronomical twilight, it is doubling every five minutes. * You need to be cautious about observing it when it's too bright (and you might not know until it's too late!). * It has a different character from night-sky. Operationally, we create a FLATFIELD by subtracting a DARK frame (taken with the internal cold dark slide in place) from appropriately selected sky frames, normalize the result by dividing by its the median value within the central region, and median filter the ensemble to produce a FLATFIELD with high S/N. The SQIID system response is stable and is very flat across the arrays, with a odd-even column modulation of +/- a few percent. Consequently, the flatfield for each channel should be stable at the percent level under normal illumination and global flatfields can be constructed which are viable for extended periods of time. The task "usqflat" has a number of parameters which should be set as indicated in the "lpar" listing below: USQFLAT: * input = Input raw sky images * output = Output normalized flat_field image * darkimage = Input dark_count image ("null"==noaction) (ref_flat = "") Reference normalized flat_field images (statsec = "[300:700,400:600]") Image section for calculating statistics * (lo_reset = 0.1) Lower limit for exclusion in output * (hi_reset = 2.) Upper limit for exclusion in output (logfile = "STDOUT") Log file name (verbose = yes) Verbose output? # IMCOMBINE parameters ========================================================= (common = "none") Pre-combine common offset: |none|median|mode| * (prenorm = "median") Pre-IMCOMBINE frame divisor:|none|median|mode| * (imscale = "none") IMCOMBINE scale option: |none|median|mode| (comb_opt = "median") Type of combine operation: |average|median| (reject_opt = "none") Type of pixel rejection operation (mclip = no) Use median, not mean, in clip algorithms (pclip = -0.5) pclip: Percentile clipping parameter (lthreshold = INDEF) Lower threshold for exclusion in statistics (hthreshold = INDEF) Upper threshold for exclusion in statistics (blank = 0.1) Value if there are no pixels (weight = "none") Image weights (expname = "") Image header exposure time keyword (nlow = 1) minmax: Number of low pixels to reject (nhigh = 1) minmax: Number of high pixels to reject (nkeep = 0) Min to keep (pos) or max to reject (neg) (lsigma = 3.) Lower sigma clipping factor (hsigma = 3.) Upper sigma clipping factor (rdnoise = "0.") ccdclip: CCD readout noise (electrons) (gain = "1.") ccdclip: CCD gain (electrons/DN) (snoise = "0.") ccdclip: Sensitivity noise (fraction) (sigscale = 0.1) Tolerance for sigma clipping scaling correction (grow = 0) Radius (pixels) for 1D neighbor rejection # IMCOMBINE parameters ========================================================= See APPENDIX A.4 for a discussion the input image parameter. Since 'prenorm' and 'imscale' are complimentary parameters which determine how the image scaling is accomplished, one of them should be set to "none". The 'prenorm' parameter controls scaling operations before submission of the data to the IRAF 'imcombine' task. The IRAF 'imstatistics' task is used to determine the selected statisitic for each image within the image section selected by 'statsec' which is used to scale the image by division. The 'imscale' parameter controls the scaling operations after submission of the data to the IRAF 'imcombine' task. Imcombine samples the data within 'statsec' for the 'imscale' selected statistic, finds the average statistic for all the images and scales each image to the average value prior to combination. Setting 'reject_opt = "minmax"' might be an appropriate choice. The data which survive the pixel rejection and threshold algorithms are then combined according to 'comb_opt'. The 'usqflat' task then divides the result by the average statistic to produce the FLAT. The 'statsec' default should generally be satisfactory. The 'reject_opt' should probably be "minmaxreject". The 'comb_opt' should be "median". One does not normally threshold FLATS or bring the data to a common zero-point offset (thus, 'common = "none"'). The parameters 'lo_reset' and 'hi_reset' are used to range the FLAT to be between two values. Data below 'lo_reset' are set to lo_reset; data above 'hi_reset' are set to hi_reset. If you want to keep a record of the USQFLAT operations set 'logfile' to a filename (else set to ""). Setting 'verbose = yes' increases the amount of information reported (including statistics). USQFLAT calls "images.immatch.imcombine". See APPENDIX A.6 for more information on IMCOMBINE. c) SKY Frames The background illumination in the near infrared is far from stable with time or location. In the near IR (especially at J) moonlight (direct or reflected off clouds) is a factor, and the night sky emission (especially at H, K, and L) is a function of temperature and humidity. Although the 10 to 30 percent variations in background caused by these factors do not strongly limit the S/N of observations (except at K and L for large changes in temperature), they greatly complicate both the creation of mosaics of large regions and accurate surface photometry of objects with extents comparable to the SQIID's field of view. For such observing programs, it is best to obtain a sufficient number of exposures (and intermixed sky exposures if necessary) to create a sky frame for that specific observation. For programs with single or a few observations of many targets, a sky calibration based on the average over the night combined with the fitting of a surface to the final image is the best that can usually be accomplished. It is useful to remember that variable airglow can cause the sky background at H can vary by a factor of 2 and J can vary by 40% on hour time scales. After examinination of the individual images a list of suitable images should be prepared as described above. The selection of suitable images may require that the individual images be dark subtracted and flatfielded prior to examination on the image display (see section IV). However, raw images must be used as input to the "usqsky" task. SKY frames are created with the task "usqsky". For example, usqsky @limages lsky will create the SKY frame "lsky" using the list "limages" of L filter images. The task "usqsky" has a number of parameters which should be set as indicated in the "lpar" listing below: USQSKY: * input = Input raw sky images * output = Output sky image (darkimage = "null") Input dark_count image ("null"==noaction) (statsec = "[300:700,400:600]") Image section for calculating statistics * (logfile = "STDOUT") Log file name * (verbose = yes) Verbose output? # IMCOMBINE parameters ========================================================= (comb_opt = "median") Type of combine operation: |average|median| (norm_opt = "zero") Type of pre-combine operation: |zero|scale|none (norm_stat = "median") Pre-combine common statistic: |none|mean|median (reject_opt = "none") Type of pixel rejection operation (mclip = no) Use median, not mean, in clip algorithms (pclip = -0.5) pclip: Percentile clipping parameter * (lthreshold = INDEF) Lower threshold for exclusion in statistics * (hthreshold = INDEF) Upper threshold for exclusion in statistics (blank = 0.1) Value if there are no pixels (weight = "none") Image weights (expname = "") Image header exposure time keyword (nlow = 1) minmax: Number of low pixels to reject (nhigh = 1) minmax: Number of high pixels to reject (nkeep = 0) Min to keep (pos) or max to reject (neg) (lsigma = 3.) Lower sigma clipping factor (hsigma = 3.) Upper sigma clipping factor (rdnoise = "0.") ccdclip: CCD readout noise (electrons) (gain = "1.") ccdclip: CCD gain (electrons/DN) (snoise = "0.") ccdclip: Sensitivity noise (fraction) (sigscale = 0.1) Tolerance for sigma clipping scaling correction (grow = 0) Radius (pixels) for 1D neighbor rejection # IMCOMBINE parameters ========================================================= See APPENDIX A.4 for a discussion the input image parameter. Setting 'darkimage = "null"' avoids subtraction of the DARK prior to image combination. You can enable dark subtraction by setting 'darkimage' to the appropriate DARK (DARK will be added to the result after image combination). The data are offset to the average of the statistic selected by 'common' prior to image combination. Setting 'reject_opt = minmax' might be an appropriate choice. The data which survive the pixel rejection and threshold algorithms are then combined according to 'comb_opt'. The parameters "lthreshold" and hthreshold" should be set to include the range of all good pixels in all images. For example, the range 0 to 20000 would certainly be safe. Setting these top closer tolerences may improve the precision of the common median determination. If you want to keep a record of the USQSKY operations set "logfile" to a filename (else set to ""). Setting "verbose = yes" increases the amount of information reported (including statistics). USQSKY calls "images.immatch.imcombine". See APPENDIX A.6 for more information on IMCOMBINE. ------------------------------------------------------------------------ IV. Processing of Individual Images a) UPSQPROC: processing with discrete sky frames Each image (except for dark exposures and blank sky frames) needs to be processed prior to combining the collective datasets into dithered images or mosaics. It is most efficient to create a list of images for each dataset (see "chlist" above). The processing applied is: * Subtract BLANK frame (DARK or SKY, this removes bias and dark current) * Normalize (divide) by the FLAT frame This processing is done with the task "upsqproc" (basic process). For example, upsqproc @limages %raw%p% lflat lsky will process each of the images in the list "limages". The DARK subtraction is implicit in the SKY removal given the sequence of steps listed above. The expression "%raw%p%" assumes that all of the file names in the list "limages" start with the string "raw" (directory information is skipped so that only that part of the pathname after the final "/" survives the substitution process). The output files will have file names starting with the string "p" in place of "raw". The list "limages" then may be edited with a text editor or a new list created with "chlist" for use in the next stages of the reduction. The task "upsqproc" has a number of parameters which should be set as indicated in the "lpar" listing below: UPSQPROC: * input = Input raw images * output = Output image descriptor: @list||.ext||%in%out% * flatimage = Input flat field image name * blankimage = Input blank field image name scaleimage = Input scale image name (opt_norm = "none") Normalization: |none|mean|median|mode| (sec_norm = "[300:700,400:600]") Image section for calculating norm (result_norm = 0.) Resultant value after normalization (int_time = 5.) Integration_time (delay_time = 3.) Delay_time (rescale = no) Rescale scaleimage to make sky? (fixpix = no) Run FIXPIX on data? (badpix = "badpix") badpix file in FIXPIX format * (setpix = no) Run SETPIX on data? * (maskimage = "badmask") untransposed bad pixel image mask * (value = 0.) pixel value for masked pixels * (orient = no ) Orient image with N up and E left (lowerlim = INDEF) Lower limit for exclusion in stats (upperlim = INDEF) Upper limit for exclusion in stats * (verbose = yes) Verbose output? * (logfile = "STDOUT") logfile name See APPENDIX A.4 for a discussion the input image parameter and APPENDIX A.5 for a discussion of the UPSQPROC output image parameter. The parameter 'blankimage' should be set to the SKY frame except for very short exposures (e.g. standard star observations) where the appropriate DARK frame may be used. The parameter 'scaleimage' is not presently relevant. For SQIID data, the parameter 'orient' could be set to "yes". This will orient the images to the usual North up, East left orientation. Normally, the bad pixel corrections controlled by 'fixpix' and 'setpix' are deferred until the images are actually combined. However, if you wish to mask bad pixels at this stage set 'setpix = yes' and 'maskimage' to an mask image appropriate to the channel selected and the orientation of the raw image. Masked pixels are set to 'value'. (see APPENDIX A.7 for how to make a pixel mask.) The processing order is: * 1) subtract 'blankimage' * 2) divide by 'flatimage' * 3) perform FIXPIX and SETPIX as required * 4) orient image as required If you want to keep a record of the UPSQPROC operations set "logfile" to a filename (else set to ""). Setting "verbose = yes" increases the amount of information reported (including statistics). It is useful to remember that UPSQPROC produces 4-byte real images. b) MOVPROC: processing data with moving median sky frames MOVPROC list processes SQIID raw image data using sky frames generated for each processed image. For a given image the running (moving) median from a subset of frames within selected list distance of each frame (excluding the given frame) is processed to produce a SKY frame. This mode is most appropriate for "quick look" type processing of data where the stability of the sky is either unknown or is known to be varying on a time scale shorter than the full data set (so that more traditional approach of grouping skys over long periods is inadequate). The number of frames which enter into the sky can be selected. To facilitate list processing, the data processing can begin and end at selected element in the list. Data which are off source can be optionally included in the running sky frame generation but withheld from full processing into final images. MOVPROC makes no distinction between on and off source frames when determining membership in the sky lists. MOVPROC: * input = Input raw images * output = Output image descriptor: @list||.ext||%in%out% * flatimage = Input flat field image name * (blankimage = "") Input blank field image name: "compute"==compute (fixpix = no) Run FIXPIX on data? (badpix = "badpix") badpix file in FIXPIX format (setpix = no) Run SETPIX on data? (maskimage = "badmask") untransposed bad pixel image mask (bvalue = 0.) Value if there are no pixels * (orient = no) Orient image with N up and E left * (verbose = yes) Verbose output? * (logfile = "STDOUT") logfile name * (include = 0) Number of included images in blankimage subset * (improc = 0) Number of images to process in group * (imskip = 0) Number of images to skip between process * (first_proc = 1) List number of first image to be processed * (last_proc = 1000) List number of last image to be processed (darkimage = "null") Input dark_count image ("null"==noaction) # IMCOMBINE parameters ========================================================= (norm_opt = "zero") Type of combine operation: |zero|scale|none| (norm_stat = "median") Pre-combine common offset: |none|mean|median|mo (comb_opt = "median") Type of combine operation: |average|median| (reject_opt = "none") Type of pixel rejection operation (statsec = "[300:700,400:600]") Image section for calculating statistics (lthreshold = INDEF) Lower threshold for exclusion in statistics (hthreshold = INDEF) Upper threshold for exclusion in statistics (mclip = no) Use median, not mean, in clip algorithms (pclip = -0.5) pclip: Percentile clipping parameter (weight = "none") Image weights (expname = "") Image header exposure time keyword (nlow = 1) minmax: Number of low pixels to reject (nhigh = 1) minmax: Number of high pixels to reject (nkeep = 0) Min to keep (pos) or max to reject (neg) (lsigma = 3.) Lower sigma clipping factor (hsigma = 3.) Upper sigma clipping factor (rdnoise = "0.") ccdclip: CCD readout noise (electrons) (gain = "1.") ccdclip: CCD gain (electrons/DN) (snoise = "0.") ccdclip: Sensitivity noise (fraction) (sigscale = 0.1) Tolerance for sigma clipping scaling correction (grow = 0) Radius (pixels) for 1D neighbor rejection # IMCOMBINE parameters ========================================================= See APPENDIX A.4 for a discussion the input image parameter and APPENDIX A.5 for a discussion of the output image parameter. MOVPROC processes data from list number "first_proc" to list number "last_proc". For each processed data frame from the list it creates a sky frame from the "include" number of images nearest the processed frame. This is equivalent to a running mean/median (selected by "comb_opt") which excludes the image being processed. The task is well suited to handle data with a regular pattern of "on" frames ("improc") interspersed with "off" frames ("imskip"). The parameters "patproc" and "imskip" together determine which of the data are processed into final images. Setting "imskip" to a number other than zero signals the task to avoid processing the "off" frames while still including them in the running sky frame. The preferred setting for IMCOMBINE parameters is: "norm_opt" = "zero" (additive corrections) "norm_stat" = "median" (median statistic) "comb_opt" = "median" (combine operation: median filter) "reject_opt" = "none MOVPROC then would median filter ("comp_opt") the images which have been brought to a common median value ("norm_stat") by adjusting the median value within the "statsec" part of the image to a common value by changing adding/subtracting (norm_option) the appropriate constant prior to combining the images into a composite sky frame. For SQIID data, the parameter 'orient' could be set to "yes". This would orient the images to the usual North up, East left orientation. Normally, the bad pixel corrections controlled by 'fixpix' and 'setpix' are deferred until the images are actually combined. However, if you wish to mask bad pixels at this stage set 'setpix = yes' and 'maskimage' to an mask image appropriate to the channel selected and the orientation of the raw image. Masked pixels are set to 'value'. (see APPENDIX A.7 for how to make a pixel mask.) The processing order is: * 1) generate and subtract 'blankimage' * 2) divide by 'flatimage' * 3) perform FIXPIX and SETPIX as required * 4) orient image as required If you want to keep a record of the MOVPROC operations set "logfile" to a filename (else set to ""). Setting "verbose = yes" increases the amount of information reported (including statistics). It is useful to remember that MOVPROC produces 4-byte real images. For example, a 95 frame sequence of 5 off, 10 on, 5 off, 10 on, .... 5 on (ABBABBABBABBABBABBA where A is a set of five off-target sky frames and B is a set of 5 on-target object frames) could be processed by setting: include=5 first_proc=6 last_proc=90 patproc=10 imskip=5 Any last_proc greater than 90 would achieve the same result. One gets 60 processed images corresponding to list numbers 6-15, 21-30, 36-45, 51-60, 66-75, and 81-90. c) PATPROC: processing data in fixed patterns PATPROC processes raw image data generated for representative data aquisition protocols which intersperse object (+) and sky (-) frame in known fixed repeating patterns. These patterns vary in the amount of time devoted to off target images: all_on: + + + + + + all time on source field pair: +- +-+ - equal time on and off triad: +-+ +-+ +-+ 2/3 on and 1/3 off alt-triad: -++ -++ -++ quad: +--+ +--+ +--+ equal time on and off but less telescope motion alt-quad: -++- -++- -++- than pair The data are normally proceed - sky subtracted, then flat fielded, etc, using the same protocol as UPSQPROC. However, the sky frames are generated from the data (sub)set as part of the data processing. The sky frames can come from all the data, all the off-source frames, or the off-source frame nearest in order time in the data sequence to a given frame. The data at each +|- pattern position can either be single frames or same-size groups of images. When observing conditions are sufficiently stable to permit one to benefit from a composite sky (improved noise and spatial structure rejection), PATPROC can be used to generate composite off-source frames for subtraction from the appropriate on-sky frames. Each group (size="multiple") off-field images is used to generate composite sky frames for each member of the (size = "multiple")0 group. The memeber of a given group of on-source frames is processed using the group sky frame nearest in time in the data sequence. PATPROC: * input = Input raw images * output = Output image descriptor: @list||.ext||%in%out% * flatimage = Input flat field image name * skyimage = Input blank field image name * (order = 1) Pattern # 1:++ 2:+- 3:+-+ 4:+--+ 5:-++- 6:-++ ? * (multiple = 1) # of frames at each +/- pattern state? * (seq_id = "none") Sequential imager id template? (eg, ".00"|"000") (rescale_opt = "none") Rescale blank to object via: |none|mean|median| (sec_norm = "[300:700,400:600]") Image section for calculating statistics (save_blank = no) Save generated blank_frame? (fixpix = no) Run FIXPIX on data? (badpix = "badpix") badpix file in FIXPIX format * (setpix = no) Run SETPIX on data? * (maskimage = "badmask") untransposed bad pixel image mask * (value = 0) pixel value for masked pixels * (orient = no) Orient image with N up and E left (verbose = yes) Verbose output? * (logfile = "STDOUT") logfile name # IMCOMBINE parameters ========================================================= (comb_opt = "median") Type of combine operation: |average|median| (reject_opt = "none") Type of pixel rejection operation (norm_opt = "zero") Type of pre-combine operation: |zero|scale|none (norm_stat = "median") Pre-combine common statistic: |none|mean|median (mclip = no) Use median, not mean, in clip algorithms (pclip = -0.5) pclip: Percentile clipping parameter (lthreshold = INDEF) Lower threshold for exclusion in statistics (hthreshold = INDEF) Upper threshold for exclusion in statistics (weight = "none") Image weights (expname = "") Image header exposure time keyword (nlow = 1) minmax: Number of low pixels to reject (nhigh = 1) minmax: Number of high pixels to reject (nkeep = 0) Min to keep (pos) or max to reject (neg) (lsigma = 3.) Lower sigma clipping factor (hsigma = 3.) Upper sigma clipping factor (rdnoise = "0.") ccdclip: CCD readout noise (electrons) (gain = "1.") ccdclip: CCD gain (electrons/DN) (snoise = "0.") ccdclip: Sensitivity noise (fraction) (sigscale = 0.1) Tolerance for sigma clipping scaling correction (grow = 0) Radius (pixels) for 1D neighbor rejection # IMCOMBINE parameters ========================================================= See APPENDIX A.4 for a discussion the input image parameter and APPENDIX A.5 for a discussion of the output image parameter. The "blankimage" parameter is used to select the type of SKY frame generated: vaid_image_name : use this image as basis for blank image null : don't subtract blank image make_all : make sky from all images make_off : make sky from all off images make_order : use sky from order selected off field nearest neighbor(s) in the data sequence The pattern "order" number identifies the fixed data acquisition pattern: 1: all_on: + + + + 2: pair: +- +- 3: triad: +-+ +-+ 4: quad: +--+ +--+ 5: alt-quad: -++- -++- 6: alt-triad: -++ -++ Each image in the sequence is processed using the sky frame nearest in time. The parameter "multiple" extends the capabilities of PATPROC to cases where the input list to contain more than one image at each +|- pattern position. The "multiple" off-field images are grouped to generate composite sky frames for each member of the "multiple" group. The "seq_id" parameter depicts the sequential imager id template (".00"|"000") for generating descriptive sky names for the case "blankimage" = "make_order". The images are assumed to be named according to the following convention: image_name == imroot//seq_id//"."//imextn seq_id == seq_mark//seq_number For example, for an image name of 982507.012.fits, sed_id = ".000" or ".012" and for an image name of image012.fits, seq_id = "000" or "012". The number of digits and the sequence marker ("" or "." above) are important. The preferred setting for IMCOMBINE parameters is: "norm_opt" = "zero" (additive corrections) "norm_stat" = "median" (median statistic) "comb_opt" = "median" (combine operation: median filter) "reject_opt" = "none PATPROC then would median filter ("comp_opt") the images which have been brought to a common median value ("norm_stat") by adjusting the median value within the "sec_norm" part of the image to a common value by changing adding/subtracting (norm_option) the appropriate constant prior to combining the images into a composite sky frame. For SQIID data, the parameter 'orient' could be set to "yes". This would orient the images to the usual North up, East left orientation. Normally, the bad pixel corrections controlled by 'fixpix' and 'setpix' are deferred until the images are actually combined. However, if you wish to mask bad pixels at this stage set 'setpix = yes' and 'maskimage' to an mask image appropriate to the channel selected and the orientation of the raw image. Masked pixels are set to 'value'. (see APPENDIX A.7 for how to make a pixel mask.) The processing order is: * 1) generate and subtract 'blankimage' * 2) divide by 'flatimage' * 3) perform FIXPIX and SETPIX as required * 4) orient image as required If you want to keep a record of the PATPROC operations set "logfile" to a filename (else set to ""). Setting "verbose = yes" increases the amount of information reported (including statistics). It is useful to remember that PATPROC produces 4-byte real images. For example, a 95 frame sequence of 5 off, 10 on, 5 off, 10 on, .... 5 on (ABBABBABBABBABBABBA where A is a set of five off-target sky frames and B is a set of 5 on-target object frames) could be processed by setting: blankimage="make_order" order=6 multiple=5 One gets 60 processed images corresponding to: list numbers 6-10 processed with composite sky from list numbers 1-5 11-15 16-20 21-25 16-20 26-30 31-35 36-40 31-35 40-45 46-50 51-55 46-50 56-60 61-65 66-70 61-65 71-75 76-80 81-85 76-80 86-90 91-95 An 90-image list would have reused the 76-80 sky to process 86-90. ------------------------------------------------------------------------ V. Building a Database from a Dataset Following the calibrations applied to the individual frames discussed above, each dataset needs to be converted into a "database". This applies even to datasets consisting of a single exposure on a target. As with datasets, each database contains the observations of a single field or target in a single filter passband. The database construct will be used to bring the individual exposures together into the desired final image (dither or mosaic) with appropriate spatial and intensity offset registration. The 'UPSQIID' package contains a set of image registration tools designed to facilitate image registration and combination. The image registration process naturally separates into the following steps: * 1) Determine the relative spatial offsets between images for a selected channel. * 2) Link these relative offsets into a single map which contains the offset of each image relative to the origin of the final image. * 3) Determine the relative intensity offsets between images which overlap. * 4) Combine the images into a single image using bad-pixel masks to eliminate bad pixels. * 5) If possible, geometrically transform images at other channels to the selected channel so that they do not need to be spatially registered separately (this is a feature of SQIID). The 'SQIID' package image registration tools are designed to operate from ordered image lists. The initial data flow through the system is determined to a large extent by the degree of spatial overlap among the images in a given data set. IR observations of extended regions (FOV greater than array FOV) are built up from spatial grids of pointed data sets which overlap on the edges. While the degree of spatial overlap is an uneasy compromise between the competing desire to map efficiently and the need to have objects in common along the edges, these data do not in general share a common overlap region. IR observations of high sensitivity are built up from pointed data sets which are heavily overlapped and share a (generally large) region in common. (The process of producing such data multiply overlapped with relatively small spatial offsets between frames is commonly called "dithering".) Off course, actual data collection may involve a complex hiearchy spatial offsets (e.g., grids of dithered data) and might require cobination of registration techniques. The dataflow for purely WIDE or DEEP observations through the 'UPSQIID' package image registration tools proceeds as follows after generation of the appropriate image list: WIDE DEEP tile spatially ordered image usqmos interactive spatial registration center xyget link registration xylap intensity match zget zget make composite image nircombine nircombine These tools generate a database which links the untouched processed data (generally either stored as mosaic of tiled images using USQMOS, or as lists of images) via a prescription which describes how each image is the be transformed, shifted, intensity offset, and masked to make a composite image. The input images (in the database mosaics or lists) are not themselves modified during this process; rather, each tool in turn inserts the necessary information into the database text files. When required, temporary copies of the images are created, modified, and discarded at task completion. One can edit the prescription as required to meet the special needs of each data set. Tools are provided for merging registration data which share a common image into larger databases. Ultimately, the images are combined according to the database prescription using the powerful IRAF task 'imcombine' (see APPENDIX A.6 and its IRAF help page for a discussion of IMCOMBINE). The user is warned that while the toolkit exists and is filled with (most) of the requisite tools, the high-level tasks which would automate the registration task have not been written - you are still in for a lot of work! The user is advised to read APPENDIX A.9 for guidelines on "keeping track of your reductions". The 'SQIID' task 'usqmos' can be used to gather observations of a single object for a given filter into its own database. Each database consists of a text file and an image containing a mosaic (with no overlap) of the images in the database. Hence the disk storage requirements grow during this stage of the processing. If possible, retain the individual images and calibration frames on the disk since iteration of the processing is often necessary and usually involves the recreation of the FLAT or SKY frame and the reprocessing of the individual images. (See APPENDIX A.9 for assistance in "keeping track of things".) Note: 'usqmos` calls the 'noao.nproto.irmosaic' task which is also part of the 'noao.imred.irred' package loaded by 'UPSQIID'. USQMOS is really just a pre and post processor for IRMOSAIC, initializing the 'UPSQIID' registration database files. See APPENDIX A.11 and the help page for IRMOSAIC for more details. NB: Given the 512X512 image size for SQIID data, this step has become impractical for grids above 4 X 4 so that some other mechanism will be required to operate on larger grids. A dataset of processed images is assembled into a database with the task 'usqmos'. For example, usqmos @limages lmos 3 4 will create a database mosaic able to contain up to 12 input images. The user must specify dimensions of the database mosaic which are large enough to contain all of the images in the input list and which, typically, is approximately square to facilitate display with the image display tool. The task 'usqmos' has a number of parameters which should be set as indicated in the "lpar" listing below: USQMOS: * input = Input images * output = Output image * nxsub = Number of subrasters in x * nysub = Number of subrasters in y (trim_section = "[*,*]") Input image section written to output image * (null_input = "") List of missing input images * (corner = "ll") Starting corner for the mosaic * (direction = "row") Starting direction for the mosaic (raster = no) Raster scan? (median = no) Compute the median of each subraster? (median_secti = "[*,*]") Input image section used to compute median (subtract = no) Substract median from each subraster? (oval = 10000.) Mosaic border pixel values (nimcols = INDEF) The number of columns in the output image (nimrows = INDEF) The number of rows in the output image (opixtype = "") Output pixel type? (tran = no) Apply image transform to mosaiced images? (task_tran = "geotran") transform task: imlintran|geotran (db_tran = "") name of database file output by GEOMAP (co_tran = "") name of coordinate file input to GEOTRAN (geom_tran = "linear") GEOTRAN transformation geometry (max_tran = yes) Offset GEOTRAN to save maximum image (interp_tran = "linear") GEO(IMLIN)TRAN interpolant (bound_tran = "nearest") GEO(IMLIN)TRAN boundary (const_tran = 0.) GEO(IMLIN)TRAN constant boundary extension valu (flux_tran = yes) conserve flux during GEO(IMLIN)TRAN (save_tran = no) Save the intermediate GEO(IMLIN)TRANed images? (save_dbmos = no) Save the IRMOSAIC database file? (logfile = "STDOUT") Log file name (infofile = "") Output information file name The input image list is specified as above (it wants to be an @list). The output file name (without extension) will be the rootname of both the database mosaic image (with the extension .imh) and the database text file (with the extension .dbmos). The images must all be the same size. Setting 'infofile' to other than "" names the output database file accordingly. The 'nxsub' and 'nysub' parameters specify the dimensions of the database mosaic. When there will be unfilled cells in the mosaic, a list of the missing images must be provided in the parameter 'null_input'. For example, if the dataset contains 10 images, a 3 by 4 mosaic could be specified. Then 'null_input' should be "11,12" or "11-12". If their are no null images, set 'null_input = ""' (a " " will cause the task to abort!). The image numbering refers to the postion of the image in the input list. The images in the input list will be tiled into a mosaic with 1-pixel boundaries (value determined by 'ovalue') according to the selected prescription. The parameter "corner", which indicates where to start tiling the images from the input list and the parameter "direction", which indicates the direction (row or column) to proceed with the tiling operation, should be set to the values which describe the way you have ordered your data in the input list. A natural choice might be 'corner' = "ll"' (lowerleft) and 'direction = "row"'. Then the images in the database mosaic would be numbered starting with the lower left image as "1" and proceeding by row. For example, 9 10 11 12 5 6 7 8 1 2 3 4 are the image numbers in the above example. The options to subtract the median and to transform the image should be set to "no" since these steps are done elsewhere. Setting 'median = yes' calculates and reports the image statistics. However, occasionally (for reasons unknown), 'irmosaic' can take an inordinately long time to perform these calculations. The parameters 'task_tran', 'db_tran', 'co_tran', 'geom_tran', 'max_tran', 'interp_tran', 'bound_tran', 'const_tran', 'flux_tran' are used to geometrically transform the images prior to storage in the mosaic. Normally one saves the data as is and defers image transforms to later stages. (See APPENDIX A.12 for further discussion.) Note that if the task aborts for any reason, copies of the raw database file produced by 'irmosaic' (they have "db" prepended to the "output" name) should be deleted before proceeding, since the 'irmosaic' task appends data to any existing database by that name. When assembling the database for the other channels, it is important to realize that the data at each channel must be tiled into its USQMOS mosaic at the same locations. Subsequent 'UPSQIID' registration tasks assume that the mosaics are equivalent accept for channel. Use of the "@list:channel_id" convention assures that such will be the case. ------------------------------------------------------------------------ VI. Specifying the Relative Spatial Offsets To combine a database of images into a single, final image, the relative offsets between the individual images in the database must be determined. IRAF does not yet have a general image registration task and many of the tools which are available (see APPENDIX A.11) are tailored to tiling images sets taken in a fixed grid pattern with minimal overlap (The images are not combined: each successive image covers up its neighbors in the overlap region.) See also the help pages for the tasks 'proto.imalign' and 'proto.imcentroid' for descriptions of system tasks which are useful for multiply-overlapped images which share an object in common. a) Interactive Specification of the Spatial Offsets for grid data The interactive determination of the offsets is accomplished by identifying common sources in the images in the database mosaic and requires an IRAF image display tool (e.g., 'ximtool' or 'saoimage'). Since registration of grids of images with minimal overlap between adjacent images (taken to cover large area) and heavily overlapped images which share a common overlap region (taken to reach faint magnitudes) require different strategies, there are two sets of tools to accomplish this task: * The 'center' and 'xylap' tools are best suited for dataset grids * The 'xyget' task is best suited for multiply-overlapped datasets These tools produce a common database which can be used by subsequent steps in the registration process. In addition, the tool 'mergecom', discussed below, allows you to merge databases from both types of data into a common database. (e.g., to bring the second members of dithered pairs for grid data in registration with the first members of the pair.) Grid data are most easily handled using the 'center' task discussed in APPENDIX A.10. To use this task, the images must be ordered in the USQMOS image so that overlapping images are adjacent to one another. If data are in dithered pairs on an M x N grid, place the first members of each pair in an appropriate order in an M x N grid within the USQMOS image and the second pair in the same order either above it in the same image (to make a M x 2N image) or in a second USQMOS image. Display the USQMOS image "usqmosl" and enter the 'center' task by typing: center usqmosl Using the image cursor (see APPENDIX A.10) successively mark images which appear in adjacent frames: the data must be paired such that each member of the pair corresponds to the position of the selected object in an adjacent frame. The overall order in which the object pairs is selected is unimportant. Avoid very bright stars (which might be saturated) and stars at the edge of the subrasters (whose position can be influenced by the one pixel gap between adjacent sub-rasters. Hopefully, all edge pairs contain at least one object within their common overlap region. The completed file, called "usqmosh.ctr.1" by default, becomes the basis file for the 'xylap' task, which collects and averages the spatial offsets between adjacent frames (calling 'getcenters') and links them into a common map (calling 'linklaps'). For, example, you can access the 'xylap' by typing: xylap usqmosl usqmosl.ctr.1 XYLAP employs many of the conventions of the IRALIGN/IRMATCH1(2)D IRAF tasks discussed in APPENDIX A.11. Task parameters for 'xylap' are as follows: XYLAP: * mosimage = usqmosh MOSAIC image name * basisfile = usqmosh.ctr.1 File produced by CENTER||IRMATCH * (linkfile = "default") File with selected XY linkage paths * (mos_info = "default") Images info file from IRMOSAIC|USQMOS * (nx_sub = INDEF) Number of input images along x direction * (ny_sub = INDEF) Number of input images along y direction * (nxrsub = INDEF) index of x reference subraster * (nyrsub = INDEF) index of y reference subraster * (guess = no) Replace null links with average values? * (new_origin = yes) Move origin to lower left corner? * (trimlimits = "[0:0,0:0]") trim limits on the input subrasters (verbose = yes) Verbose output during LINKLAPS? * (outfile = "") Output file name (tran = no) Request GEOTRAN mosimage before IMCOMBINE? (db_tran = "") name of database file output by GEOMAP (geom_tran = "linear") GEOTRAN transformation geometry (max_tran = yes) Offset GEOTRAN to save maximum image? (interp_tran = "linear") GEOTRAN interpolant (bound_tran = "nearest") GEOTRAN boundary (const_tran = 0.) GEOTRAN constant boundary extension value (flux_tran = yes) Conserve flux upon GEOTRAN? (interp_shift = "linear") IMSHIFT interpolant (nearest,linear,poly3,poly5 (logfile = "STDOUT") Logfile name mosimage name of the 'usqmos' image to be registered basisfile 'basisfile' is a text file listing the coordinates of objects in the input image one object per line in the following format: 1) the x and y coordinates of the object in the first subraster 2) the x and y coordinates of the same object in the second subraster 3) the x and y coordinates of the next object in the first subraster etc. The ouput of the 'center' task, or any other file with these attributes, can be used. link_file = "default" By default ("default") the linkage path for each subraster is computed along two orthogonal paths and the average linkage is used: one path proceeds along the row to the reference subraster row, then along the column; the other path proceeds along the column and then along the row. If 'link_file' is given a linkage filename, that file is used to determine the linkage paths for each subraster. Since arbitrary linkage paths are allowed, one can accommodate problem images which are not linked on sufficient sides. You need to walk each piece to the same final link (one path per line), giving the path position of each image you trespass on the way from each subraster to the number of the reference subraster declared by 'nxrsub' and 'nyrsub'. Here are three sample formats for xylap linkfiles for the 3 x 4 mosaic in the 'usqmos' example above and nxrsub = 2 and nyrsub = 2: Tile order: 9 10 11 12 5 6 7 8 1 2 3 4 file1 file2 file3 1 2 6 1 5 6 1 2 6 2 6 2 6 2 6 3 2 6 3 7 6 3 2 6 4 3 2 6 4 8 7 6 4 8 7 6 5 6 5 6 5 6 6 6 6 7 6 7 6 7 6 8 7 6 8 7 6 8 7 6 9 10 6 9 5 6 9 10 6 10 6 10 6 10 6 11 10 6 11 7 6 11 7 6 12 11 10 6 12 8 7 6 12 8 7 6 File1 proceeds along the row to the reference subraster row, then along the column; file2 procceds along the column, then along the row; file3 contains a variety of linkages to avoid missing links. nx_sub = INDEF, ls ny_sub = INDEF The number of subrasters along a row and a column of the input image. This will defaults to the entire image. If you are only registering part of the image (e.g., the lower half of the dithered M x 2N example above) you need to enter the appropriate numbers here. nxrsub = INDEF, ls nyrsub = INDEF The column and row index of the reference subraster. This will default to the central subraster. guess = no If 'guess = yes', links which are absent from the database are estimated on the basis of the average value for the link along the row or column where the link is missing. Setting 'guess = no' prevents the use of average links when none exists. (By default, orthogonal paths are computed and the average is used; if one path exists and the other doesn't, one does not want the average!). new_origin = yes Setting 'new_origin = yes' tells the task to find the size of the combined image in the database and shift the implied origin of the database to include all of the images by putting the database origin in the lower left corner of the dataset. 'new_origin = no' does not reset the origin from wherever it happens to be. trimlimits = "[1:1,1:1]" The number of columns or rows to trim off each edge of each input subraster before inserting it in the output image in section notation ([left:right,top:bottom]). The default is to trim 1 column or line at each edge. These trims differ with each channel (see the SQIID manual for appropriate choices. The input images are not touched; rather the database is marked to exclude these edge regions from the final image. tran = no Setting 'tran = yes' requests that the task transform the database coordinates according to the 'geotran' conventions and to mark the database so that the images will be geotransformed during the combination process. Setting 'tran = no' leaves things alone. 'db_tran', 'geom_tran', 'max_tran', 'interp_tran', 'bound_tran', 'const_tran', 'flux_tran' are parameters employed by GEOTRAN (see APPENDIX A.12) and should be left at their default values. They are ignored when 'tran = no'. interp_shift = "linear" The type of interpolant used to shift the subrasters. Uses IMSHIFT convention ("linear", "nearest", "poly3", "poly5" and "spline3" are allowed); "linear" is the perferred choice. outfile = "" Outfile is the name of the database file generated. If 'outfile = ""' or 'outfile = "default"', this file is named "mosimage.xylap". As the XY offset data are being collected, XYLAP ('getcenters' task) signals the presence of mis-matched object pairs as follows: #Note: non-adjacent pair:482.06 212.9 514. 218. If you see this message (which generally means that one of a pair of positions is missing), the linkage will be flawed. You need to examine your 'basisfile' at the reported coordinate location, fix the problem (by editing out the bad position or inserting a missing position from a pair), and redoing 'xylap'. During linkage operation XYLAP ('linklaps' task) signals missing links and the action taken: #DBL Note: no data for link | r 2,3 | #DBL Note: using row_ave para_laps 3 for link | r 2,3 | #DBL Note: no data for link | c 2,1 | #DBL Note: using row_ave perp_laps 1 for link | c 2,1 | #DBL Note: no data for link | c 2,2 | #DBL Note: using row_ave perp_laps 2 for link | c 2,2 | #DBL Note: null links not used in pairs Note: XYLAP, which uses the output from the CENTER task is set up to operate on mosaics no larger than 10 in X or Y. (IRAF scripts do not handle arrays well and larger arrays eat up too much space on the stack.) b) Interactive Specification of the Spatial Offsets for multiply-overlapped data Many datasets are either irregularly or multiply-overlapped and cannot be put into a simple grid. The 'xyget' task is designed to accommodate datasets which share an object in common. In many instances, the entire dataset will share a common object and 'xyget' can be run once on the entire dataset. In other cases, subsets of data will share a common object and the dataset can be broken into a series of multiply-overlapped images. Then 'xyget' can be run on each subset and on pairs of images which link each subset and the 'mergecom' task (discussed below) can be used to merge the data into a single database for the entire subset. For example, xyget usqmosl "1-10|1" starting from USQMOS image "usqmosl" xyget @sqlist "1-10|1" starting form SQIID image list "sqlist" will permit the user to interactively indentify a number of point sources in image 1 and then a single source which appears in each of images 1 to 10. Prior to running this task it is advisable to examine the database mosaic image to identify a single point source which is present on all of the images. During the manual registration process one can reject (exclude from further consideration as part of the registration database) specific frames by typing "x" (instead of spacebar) when the offending frame is up for review. Selected XYGET parmeters are discussed below: XYGET: * images = sqmosh MOSAIC image name | @list of images to compare * frame_nums = "1-10|1" Selected frame numbers within MOSAIC|@list getoffset = yes Do you want to get frame offsets? answer = yes Do you want to continue? compute_size = yes Do you want to [re]compute image size? * (mos_info = "default") Images info file from IRMOSAIC|USQMOS * (ref_info = "default") Reference info file from IRMOSAIC|USQMOS (outfile = "") Output info file - default: images.xycom * (trimlimits = "[0:0,0:0]") trim limits on the input subrasters (coord_in = "") Input initial coordinate file (in_shifts = "") Initial shift file between ref and images (bigbox = 11.) Size of coarse search box (boxsize = 7.) Size of final centering box (tran = no) Request GEOTRAN images before IMCOMBINE? (db_tran = "") name of database file output by GEOMAP (geom_tran = "linear") GEOTRAN transformation geometry (max_tran = yes) Offset GEOTRAN to save maximum image? (interp_tran = "linear") GEOTRAN interpolant (bound_tran = "nearest") GEOTRAN boundary (const_tran = 0.) GEOTRAN constant boundary extension value (flux_tran = yes) Conserve flux upon GEOTRAN? * (zscale = no) DISPLAY using zscale? * (z1 = 0.) minimum greylevel to be displayed * (z2 = 1000.) maximum greylevel to be displayed * (verbose = no) Verbose reporting images The name of the USQMOS mosaic image or the image file "@-file" you want to register. An "@-file" is file containing a list of image names, (one per line) - the filename is preceded by "@" to tell IRAF to look for images within the file. NB: When submitting an "@-file" for images stored within an USQMOS image, you need to give the full name of the USQMOS mosiac database in 'mos_info' instead of using the default. frame_nums A specially formatted string which tells which images to register and what image to register them against. The images are identified by their path order in the USQMOS database (MOS_XXX lines, where XXX is the path number). The format for 'frame_nums' consists of up to two to three fields separated by "|". (Note, because of the use of the "|" delimiter, the 'frame_nums' string must be encloded in quotes). The first field indicates the range of images to be registered; the second field indicates the number of the reference image these images are to be referenced to; the third field (when present) gives the name of the 'usqmos' image where the reference image can be found. By default, the images are all from the same database; inclusion of the third field allows you to register images against another image not in the current database. Since the 'SQIID' registration database files assign a single line which begins with "COM_XXX", where XXX is the order of the image in the list these frame_numbers will often be refered to as COM# (see APPENDIX A.13). RANGE SYNTAX The syntax for the range portion of the frame_num string consists of positive integers, '-' (minus), '*', ',' (comma), and whitespace. The commas and whitespace are ignored and may be freely used for clarity. The remainder of the string consists of sequences of three fields. The first field is the beginning of a range, the second is a '-', and the third is the end of the range. The following examples illustrate the range syntax: range result 1-10 1 2 3 4 5 6 7 8 9 10 1,5,9 1 5 9 1-3,5 1 2 3 5 * 1 to the number of 'usqmos' subrasters read from the database Warning: since this number is the original size of the USQMOS mosaic, it may not include all the images in a merged dataset. For example: 'frame_nums = "1-4|2"' registers images 1,2,3,4 against image 2 in the same mosaic 'frame_nums = "1-4|2|mosh2' registers the same images against imagex 2 in the mosaic mosh2 mos_info = "default" Name of the 'usqmos' database file associated with the 'images' parameter. 'mos_info = "default" points to the file "images.dbmos" ref_info = "default" Name of the 'usqmos' database file associated with the indirect image reference field in the 'frame_nums' string noted above. 'ref_info = "default" points to the file "third_field.dbmos" outfile = "" Outfile is the name of the database file generated. If 'outfile = ""' or 'outfile = "default"', this file is named "images.xycom". trimlimits = "[1:1,1:1]" The number of columns or rows to trim off each edge of each input subraster before inserting it in the output image in section notation ([left:right,top:bottom]). The default is to trim 1 column or line at each edge. These trims differ with each channel (see the SQIID manual for appropriate choices. The input images are not touched; rather the database is marked to exclude these edge regions from the final image. tran = no Setting 'tran = yes' requests that the task transform the database coordinates according to the 'geotran' conventions and to mark the database so that the images will be geotransformed during the combination process. Setting 'tran = no' leaves things alone. 'db_tran', 'geom_tran', 'max_tran', 'interp_tran', 'bound_tran', 'const_tran', 'flux_tran' are parameters employed by GEOTRAN (see APPENDIX A.12) and should be left at their default values. They are ignored when 'tran = no'. The parameter 'getoffset' will be prompted for at runtime and set to "yes". The parameter 'trimlimits' excludes masked regions on the edges of the detectors. The values for this parameter have changed over time during the development of the SQIID. Appropriate values are included in the appendix. Since the interactive offsets are determined for the H band images which will be used as the basis of the final transformation, the parameter 'tran' must be set to "no". The scaling of the image display during the interactive phases of the 'xyget' task may be automatic if the parameter 'zscale' is "yes", otherwise the images will be scaled to fill the range between the values of the parameters 'z1' and 'z2'. It is often best to set the parameter 'verbose' to "yes". c) Specification of the J, K, and L Band Offsets Since the relative geometries of the detectors remain constant over an observing run (in fact, between instrument disassemblies), once the H band relative offsets are known the offsets fo images in the other passbands can be automatically determined. The necessary calibrations reside in files as documented in the APPENDIX A.12. The task "xyadopt" will put the necessary information into the database text files for the J, K, and L databases. For example to handle the J channel, xyadopt sqmosj "1-10|1" sqmosh.xycom will obtain the relative offsets from the file "sqmosh.xycom" (which is created by "xyget") to produce the registration file "sqmosj.xycom" appropriate for the J channel image. XYADOPT uses geomap output files appropriate for the observing run and converts assembly instructions of a given channel to those needed to combine another channel. The database is transformed so that the XY locations are OK, and the images are set-up to be geotran`ed into the coordinate system of the target channel prior to combination. Selected XYAPOPT parameters are described below: XYADOPT: * images = IRMOSAIC image name | @list of images to compar * frame_nums = "*|1" Selected image numbers within IRMOSAIC * basis_info = Lap_basis info file from GETCOMBINE|IRCOMBINE answer = yes Do you want to continue? (mos_info = "default") Images info file from IRMOSAIC|MKMOS * (trimlimits = "[0:0,0:0]") trim limits on the input subrasters (outfile = "") Output information file * (tran = yes) Request GEOTRAN images before IMCOMBINE? * (db_tran = "") name of database file output by GEOMAP (geom_tran = "linear") GEOTRAN transformation geometry (max_tran = yes) Offset GEOTRAN to save maximum image? (interp_tran = "linear") GEOTRAN interpolant (bound_tran = "nearest") GEOTRAN boundary (const_tran = 0.) GEOTRAN constant boundary extension value (all_frames = no) Use entire frame_nums range (else nsubraters) (flux_tran = yes) Conserve flux upon GEOTRAN? * (verbose = no) Verbose reporting images The name of the SQMOS mosaic image you want to register. Although an @list indicated a file containing image names (one per line) is accepted, the software has not been fully tested in that mode. frame_nums A specially formatted string which tells which images to register and what image to register them against. Same syntax as discussed in XYGET above. Since the 'sqiid' registration database files assign a single line which begins with "COM_XXX", where XXX is the order of the image in the list these frame_numbers will often be refered to as COM# (see APPENDIX A.13). basis_info Name of the database file produced by either 'xylap' or 'xyget' for the channel you whose registration you wish to adopt trimlimits = "[1:1,1:1]" The number of columns or rows to trim off each edge of each input subraster before inserting it in the output image in section notation ([left:right,top:bottom]). The default is to trim 1 column or line at each edge. These trims differ with each channel (see the SQIID manual for appropriate choices. The input images are not touched; rather the database is marked to exclude these edge regions from the final image. outfile = "" Outfile is the name of the database file generated. If 'outfile = ""' or 'outfile = "default"', this file is named "images.xycom". tran = yes Setting 'tran = yes' requests that the task transform the database coordinates according to the 'geotran' conventions and to mark the database so that the images will be geotransformed during the combination process. Setting 'tran = no' leaves things alone. 'geom_tran', 'max_tran', 'interp_tran', 'bound_tran', 'const_tran', 'flux_tran' are parameters employed by GEOTRAN (see APPENDIX A.12) and should be left at their default values. They are ignored when 'tran = no'. db_tran Name of the GEOMAP database file which transforms the current channel image to that of the adopted channel. See APPENDIX A.12 for a discussion of these files. (Note: since the path to the associated co-ordinate file, ".geoco" is found within the 'db_tran' ".geodb" file, they both must be in the current directory to be found properly.) In the current example, the parameter 'basis_info' should be set to the H band ".xycom" file and the parameter "'rimlimits' must be set for each detector (see the description of XYGET above). The parameter 'tran' must be set to "yes". The transformation coordinates are in the appropriate ".geodb" file as described in APPENDIX A.12. This filename must be specified in the parameter 'db_tran'. (Note: since the path to the associated co-ordinate file, ".geoco" is found within the ".geodb" file, they must be in the current directory to be found properly.) d) Merging registration files into larger databases XYGET/XYLAP and MERGECOM allow you to operate in larger domains. If you assemble the data in smaller mosaics, you can pull them together using XYGET to link the individual mosaics and MERGECOM to put it together into a single prescription. As noted above, XYGET will accept pieces from different mosaics. The format can be summarized as follows: "Range in included image "COM# | reference COM# | mosaic name of reference" the final "|" and field is left out if they are from the same mosaic if the reference USQMOS database is mos_name.dbmos, set 'ref_info = "default"' else set 'ref_info' = name of USQMOS database for reference image Collect the names of the separate registration database files (same channel only, one per line) into a file (e.g., usqmosh.merge) and use 'mergecom' to merge them into a single file (e.g., usqmoshall.xycom) as follows: mergecom @usqmosl.merge usqmoslall.xycom Note: You must feed the information to mergecom in the proper order when you organize the list of filenames: each reference frame must be found within the list of frames which has accummulated from the files earlier in the list. Selected MERGECOM parameters are discussed below: MERGECOM: infofiles = @usqmosl.merge files produced by XYGET|XYLAP|IRCOMBINE mergedfile = usqmoshall.xycom name of resultant composite image compute_size = yes Do you want to [re]compute image size? answer = yes Do you want to continue? (subset = no) Keep only COM_000 and mos_name COM's? (mos_name = " ") mosaic name for saved COM's (renumber = no) renumber secondary referenced frames? (verify = no) Verify alias before proceeding? infofiles List of registration files you wish to merge. Either a comma delimited inline list or an "@list" filename with the files listed one per line. Remember to feed the information to mergecom in the proper order when you organize the list of filenames: Each reference frame must be found within the list of frames which has accummulated from the files earlier in the list. mergedfile Name of resultant composite file. compute_size = yes After all files have been accepted into the database, you will be prompted: Do you want to [re]compute image size? (yes) Answer "yes" if you wish to have the task to find the new size of the combined image in the database and shift the implied origin of the database to include all of the images by putting the database origin in the lower left corner of the dataset. Answer "no" if you want to leave things as they are. subset = no If some of the 'infofiles' include references to reference images outside the dataset which you do not want to include in the 'mergedfile', set 'subset = yes'. The task will then only include those images which have the rootname 'mos_name'. mos_name = " " Root name of the 'usqmos' mosaic image which you wish to include in the 'mergedfile'. Ignored if 'subset = no'. renumber = no Set 'renumber = yes' if you want to automatically renumber COM# after the first file, so they cannot conflict with earlier COM#. Each new file is renumbered to begin where the previous file left off. verify = no MERGECOM makes the reference frame found in the first file the master reference frame for the entire dataset (its the COM_000 image). Each subsequent file is searched for this image. If it cannot find the image, it will search for a match between the reference image in the current file and any image in the accumulated list of images. If 'verify = no' it automatically begins this search. If 'verify = yes', it prompts you: If you answer "no", the task ends; else it continues. When you are satisfied with the result, you can use XYADOPT on the merged file to create merged registration database files for your other channels. e) Matching image intensity For a variety of reasons, including changing sky background, the processed images will generally not perfectly match inside their regions of common overlap. The 'zget' task is used to determine the relative intensity offsets between images you wish to combine into a composite image. If the images share a sizable overlap region which is not dominated by high surface brightness, intensity (z) registration can be deferred to the final stage in the image combination process, the 'nircombine' task itself. However, for the more general case, where images do not share a proper common overlap region, the ZGET task is generally able (albeit slowly!) to get the job done. You must specify the linkage between overlapping images you want to use to accomplish this registration. The task uses the registration database to determine the overlap regions for each image pair. Copies of the images are geotransformed, fractionally shifted, and masked. The statistics (selected by 'common') within the overlap regions of each image pair are determined and the prescription is them followed to accumulate the net offset of each image compared to the first image in the prescription. Currently, the entire overlap region is included in the statistics. Following our example for the mosaic "usqmosl", you can use 'zget' by typing: zget cousqmosl usqmosl.xycom @usqmosl.zget where "cousqmosl" is the name of the composite image, "usqmosl.xycom" is the name of the XY registration database file ouput by XYGET/XYLINK/MERGECOM, and "usqmosl.zget" is the name of a file (discussed below) contain paths for registering each image. Selected ZGET parameters are discussed below: ZGET: * match_name = name of resultant composite image * infofile = "" file produced by GETCOMBINE * image_nums = Selected image numbers|ref_id answer = yes Do you want to continue? compute_size = yes Do you want to [re]compute image size? (outfile = "") Output information file name (gxshift = 0.) global xshift for final image (gyshift = 0.) global yshift for final image (goverlap = "") Global overlap section (overrides all else) (gsize = "") Global image size (overrides all else) (trimlimits = "[2:2,2:2]") added trim limits for input subrasters * (setpix = yes) Run SETPIX on data? * (maskimage = "badmask") bad pixel image mask (fixpix = no) Run FIXPIX on data? (fixfile = "badpix") badpix file in FIXPIX format * (common = "median") Pre-combine common offset: |none|adopt|mean|med * (lowerlim = -1000.) Lower limit for exclusion in IMSTATS * (upperlim = 60000.) Upper limit for exclusion in IMSTATS (interp_shift = "linear") IMSHIFT interpolant (nearest,linear,poly3,poly5 (bound_shift = "nearest") IMSHIFT boundary (constant,nearest,reflect,wrap (const_shift = 0.) IMSHIFT Constant for boundary extension * (svalue = -10000000.) Setpix value (<< floor) (floor = -200.) Rejection floor for imcombine (ceiling = 60000.) Rejection ceiling for imcombine (ovalue = -1000.) Rejected pixel value for new image? (size = no) Compute image size? (save_images = no) Save images which were IMCOMBINED? (verbose = yes) Verbose output? (format = yes) Format output table match_name Name of the resulant composite image infofile 'UPSQID' registration database file corresponding to the image you wish to assemble. THe database files output by XYLAP, XYGET, ZGET, or MERGECOM and any of their transformations via XYADOPT are acceptable. Remember that you can edit COM-line prescription contained in these files when necessary to get your desired result. image_nums 'image_nums' tells the task how to determine the intensity offsets: it tells which images to register and what image to register them against. It can either be a specially formated string or an @list file. The specially formatted string uses the same syntax as discussed in XYGET above. The format can be summarized as follows: "Range in included image "COM# | reference COM# | mosaic name of reference" When a filename is given (with "@" prepended) the file must contain a complete prescription of how to connect each image with its neighbor. The format for each line is as follows: "COM# of included image | COM# of reference" You have considerable freedom of choice in assembling this file. The only requirement on this file is that each image be connected to a neighbor somewhere: Pick a starting point (which only appears to the right of the "|"). Build up paths to each image (one per line). Every image (except the starting point) must appear once and only once on the left side of the "|". Consider the following spatial orientation: 7 8 9 4 5 6 1 2 3 A simple files might be: file1 file2 2|1 2|1 3|2 3|2 4|1 4|1 5|4 7|4 6|5 8|7 7|4 9|8 8|7 5|8 9|8 6|9 or 6|5 or 6|3 where file1 is a straight forward implementation and file2 would avoid an unfortunate overlap between 4 and 5. outfile = "" Outfile is the name of the database file generated. If 'outfile = ""' or 'outfile = "default"', this file is named "match_name.zcom". trimlimits = "[2:2,2:2]" (Not currently used by ZGET) The number of columns or rows to trim off each edge of each input subraster before inserting it in the output image in section notation ([left:right,top:bottom]). The default is to trim 2 column or line at each edge. This compensates for the 2 pixel border trashed (i.e. made up) by the IMSHIFT operations. The input images are not touched; rather the database is marked to exclude these edge regions from the final image. However, the images the trimmed portions of the image will be excluded during the image overlap statistics. Normally, 'trimlimits' are cumulative in the database. This 'trimlimits' is not. It just communicates to the task generating the overlap regions the message that the edges are flawed so don't count them as useful during the overlap computations. common = "median" 'common' selects the statistic you want to use to determine the z offsets between images. It can be either "mean", "median", or "mode". lowerlim = -1000. Lower limit for exclusion in IMSTATS during intensity offset operations. When using 'setpix = yes' you must be sure to exclude the mask regions, which will be set to 'svalue' from the statistics. You might consider choosing a higher value to improve image statistics by restricting the range. However, be sure to keep this limit safely below the lowest real data in your image! upperlim = 60000. Upper limit for exclusion in IMSTATS during intensity offset operations. You might consider choosing a lower value to limit the effects of bright stars and improve image statistics by restricting the range. setpix = yes Since the over-lap region includes the edges of the frame, it is best to mask the images when computing intensity offsets and combining the images into a composite image. 'setpix = yes' will mask the images using the mask 'maskimage' before calculating overlap statistics. ZGET works on image copies. The original images remain untouched. maskimage = "badmask" Name of the channel-specific mask image. See APPENDIX A.7 for a discussion of SQIID image masks. Remember to use the mask appropriate to the current channel. svalue = -10000000. The value given to masked pixels. The large negative number shown is assures that any output pixel which is generated through subsequent image shift and geotransformed from masked input pixels stays recognizably negative compared to good pixels so that such pixels can be thresholded from the final image. When using such a 'svalue', you must remember to exclude this value by choice of lowerlimit from statistics and save such images as real images. To avoid loss of precision, you should use 'imreplace' to replace these values with safe values before you save them using 'wfits' with image scaling. (Saving images via 'wfits' using 'bitpix = -32' option will save the images in IEEE 32bit format without image scaling.) size = no Since one has presumably determined the final image size during the XY registration process, 'size = no' is the preferred choice. If you set 'size = yes', the task we redetermine the image size and edit the prescription accordingly (loosing your registration with other images). If the task cannot find the proper information it will ask you for permission to do it: Do you want to [re]compute image size? (yes) Answer "yes" if you wish to proceed and "no" if you wish to exit. save_images = no Setting 'save_images = yes' preserves the temporary transformed, shifted, and masked images generated by the task for examination of pathological intensity offsets (you could use the 'imexamine' task). Generally, you do not want to save the images and set 'save_images = no'. verbose = yes Set 'verbose = yes' if you want a more complete log of the ZGET operations including in the output file. format = yes Setting 'format = yes' generates output COM-lines aligned into columns so that they are easier to read. To accomplish this, the UNIX 'awk' task is called. If this is not well supported on your system, set 'format = no'. 'interp_shift', 'bound_shift', and 'const_shift' are parameters employed by IMSHIFT and should be left at their indicated default values. 'floor', 'ceiling', and 'ovalue' are parameters which are currently unused by ZGET, but are used in the final combine operations in the task 'nircombine'. If ZGET is crashing, saying things like "rmedian not found", or giving bizarre results, check the 'upperlim' and 'lowerlim' parameters to be sure you aren't "trowing the baby out with the bathwater". In particular, 'lowerlim' must not be INDEF (otherwise all those -1e8's generated by the SETPIX operation and choice of 'svalue' make it in and really mess things up!). The other possibility is that 'lowerlim' is set too high. Although you have subtracted background, H has a way of varying alot. Consequently the residual background can vary several hundred counts from that of the subtracted skyframe. If you have been saving the UPSQPROC output to file you can check there, or you can use IMEXAM on the USQMOS image to see what a better 'lowerlim' would be. If you have residual problems intensity matching in the final image (due to ghosts on one image which are not on the other, the presence of bright stars, or whatever), try different paths to reach the offending images: link from above below or the other side. You just need to be sure that a linkage path to each image exists. Failing this, you can examine the combined image using the "l" and "c" line and column plotting option withim 'imexam' to find the offset error and to edit the ".zcom" files accordingly. (Note: since the intensity offsets within the COM-lines are not cumulative - they are only applied to the image on that line - the editing task is straight forward. See APPENDIX A.13 for a discussion of the registration database file format.) ------------------------------------------------------------------------ VII. Creating the Final Images Following the construction of the database files containing the offset and transformation information, the final images are created with the task 'nircombine'. NIRCOMBINE follows the submitted registration prescription and transforms (GEOTRANSFORM), spatially shifts (IMSHIFT), intensity offsets IMARITH), and masks (USQMASK) copies of the input database images and presents these copies to the IRAF 'imcombine' task along with the integer spatial offsets necessary for assembling the final image. See APPENDIX A.6 for a discussion of 'imcombine'.) In this task all of the transformations are actually applied. Also, the few bad pixels in the detectors can be corrected for either by exclusion from the dithered final image or by interpolation in a mosaic image. For example, nircombine jfinal jmos.xycom will create an IRAF image "jfinal" from the "jmos" database files. Whenever you run 'nircombine' it checks to see if there is a common overlap region for the dataset. If there is no overlap region, it issues a warning: #WARNING: overlap section: [463:253,456:254] is unphysical! The warning is valid, if you intended to use the common overlap region as the basis for determining the offsets ('common' != "adopt"). However, if you have provided an viable alternative with your .zcom file, you can safely ignore this warning. Selected NIRCOMBINE parameters are described below: NIRCOMBINE: * match_name = name of resultant composite image * infofile = "" file produced by XYGET|XYLAP|XYADOPT answer = yes Do you want to continue? compute_size = yes Do you want to [re]compute image size? * (outfile = "") Output information file name * (trimlimits = "[2:2,2:2]") added trim limits for input subrasters * (register = yes) Maintain input image origin and size * (common = "median") Pre-combine common offset: |none|adopt|mean|med * (comb_opt = "median") Type of combine operation: |average|median| * (reject_opt = "none") Type of pixel rejection operation * (lthreshold = -200.) Rejection floor for imcombine and stats * (hthreshold = 65000.) Rejection ceiling for imcombine and stats * (blank = -1000.) Value if there are no pixels * (setpix = yes) Run SETPIX on data? * (maskimage = "badmask") bad pixel image mask * (svalue = -100000000.) Setpix value (<< lthreshold) (fixpix = no) Run FIXPIX on data? (fixfile = "badpix") badpix file in FIXPIX format (to_scale = 0.) Scale to value (0 = noscale) via to_name (to_name = "INT_S") Image header to_name keyword (make_stack = no) Imstack images prior to imcombine? (apply_zero = yes) Apply zeropoint to images prior to imcombine? (invert_zero = no) Invert database zeropoints prior to application (save_images = no) Save images which were IMCOMBINED? (do_combine = yes) IMCOMBINE (else IRMATCH) into final image? * (verbose = yes) Verbose output? (size = no) Compute image size? (interp_shift = "linear") IMSHIFT interpolant (nearest,linear,poly3,poly5 (bound_shift = "nearest") IMSHIFT boundary (constant,nearest,reflect,wrap (const_shift = 0.) IMSHIFT Constant for boundary extension (gxshift = 0.) global xshift for final image (gyshift = 0.) global yshift for final image (goverlap = "") Global overlap section (overrides all else) (gsize = "") Global image size (overrides all else) (weight = "none") Image weights (mclip = no) Use median, not mean, in clip algorithms (pclip = -0.5) pclip: Percentile clipping parameter (nlow = 1) minmax: Number of low pixels to reject (nhigh = 1) minmax: Number of high pixels to reject (lsigma = 3.) Lower sigma clipping factor (hsigma = 3.) Upper sigma clipping factor (sigscale = 0.1) Tolerance for sigma clipping scaling correction (grow = 0) Radius (pixels) for 1D neighbor rejection (expname = "") Image header exposure time keyword (rdnoise = "0.") ccdclip: CCD readout noise (electrons) (gain = "1.") ccdclip: CCD gain (electrons/DN) match_name Name of the resulant composite image infofile 'UPSQIID' registration database file corresponding to the image you wish to assemble. THe database files output by XYLAP, XYGET, ZGET, or MERGECOM and any of their transformations via XYADOPT are acceptable. Remember that you can edit COM-line prescription contained in these files when necessary to get your desired result. outfile = "" Outfile is the name of the database file generated. If 'outfile = ""' or 'outfile = "default"', this file is named "match_name.src". trimlimits = "[2:2,2:2]" The number of columns or rows to trim off each edge of each input subraster before inserting it in the output image in section notation. The default is to trim 2 columns or lines at each edge. This compensates for the 2 pixel border trashed (i.e. made up) by the IMSHIFT operations. The input images are not touched; rather the database is marked to exclude these edge regions from the final image. However, the images the trimmed portions of the image will be excluded during the image overlap statistics. Normally, 'trimlimits' are cumulative in the database. This 'trimlimits' is not. It just communicates to the task generating the overlap regions the message that the edges are flawed so don't count them as useful during the overlap computations. register = yes The new imcombine task trims the edges of the output image at the top and right to the miminal size required. Consequently final image size can very with channelx. Setting 'register = yes' restores images to the full size requested in 'infofile'. Setting 'register = no' leaves the output image of IMCOMBINE alone. common = "median" (mean|median|mode) If your dataset contains images which share a common overlap region, you can use 'common' to select the statistic you want to use to determine the z offsets between images. It can be either "mean", "median", or "mode". If you wish to adopt previously determined intensity offsets (e.g. from ZGET) from 'infofile', set 'common = "adopt". (feeds 'zero' parameter in IMCOMBINE) comb_opt = "median" (average|median) Type of combining aperation performed on the final set of pixels (after offsetting, masking, thresholding, and rejection). The choices are "average" and "median" (only in IRAF 2.10). THe median uses the average of the two central values when the number of pixels is even. reject_opt = "none" (none|minmax|ccdclip|crreject|sigclip|avsigclip|pclip) Type of rejection operation performed on the pixels remaining after offsetting, masking and thresholding. The rejection choices are: none - No rejection minmax - Reject the nlow and nhigh pixels ccdclip - Reject pixels using CCD noise parameters crreject - Reject only positive pixels using CCD noise parameters sigclip - Reject pixels using a sigma clipping algorithm avsigclip - Reject pixels using an averaged sigma clipping algorithm pclip - Reject pixels using sigma based on percentiles If earlier than IRAF2.10, the reject_option should be set to its default ("none"). "minmax" is a conservative choice. See APPENDIX A.6 and the help page for IMCOMBINE for discussion of the alternatives. Depending on the choice of 'reject_opt, the parameters from 'weight' through 'gain' listed above may be better served by choices other than those shown. lthreshold = -1000. Lower limit for inclusion in IMSTATS during intensity offset operations and for inclusion in the final image in IMCOMBINE during threshold operations. When using 'setpix = yes' you must be sure to exclude the mask regions, which will be set to 'svalue' from the statistics. You might consider choosing a higher value to improve image statistics by restricting the range. However, be sure to keep this limit safely below the loweset real data in your image! IMCOMBINE thresholds its input pixels prior to any scaling, rejection, and combining; if INDEF the thresholds are not used. hthreshold = 60000. Upper limit for inclusion in IMSTATS during intensity offset operations and for inclusion in the final image in IMCOMBINE during threshold operations. You might consider choosing a lower value to limit the effects of bright stars and improve image statistics by restricting the range. IMCOMBINE thresholds its input pixels prior to any scaling, rejection, and combining; if INDEF the thresholds are not used. blank = -1000. 'blank' is the value given output pixel locations where no input pixels have survived the threshold and rejection algorithms. setpix = yes Since the over-lap region includes the edges of the frame, it is best to mask the images when computing intensity offsets and combining the images into a composite image. 'setpix = yes' will mask the images using the mask 'maskimage' before calculating overlap statistics. NIRCOMBINE works on image copies. The original images remain untouched. maskimage = "badmask" Name of the channel-specific mask image. See APPENDIX A.7 for a discussion of SQIID image masks. Remember to use the mask appropriate to the current channel. svalue = -100000000. The value given to masked pixels. The large negative number shown is assures that any output pixel which is generated through subsequent image shift and geotransformed from masked input pixels stays recognizably negative compared to good pixels so that such pixels can be thresholded from the final image. When using such a 'svalue', you must remember to exclude this value by choice of lowerlimit from statistics. verbose = yes Set 'verbose = yes' if you want a more complete log of the NIRCOMBINE operations including in the output file. to_scale = 0. Scale to value (0 = noscale) via to_name. This allows one to scale images to a common integration time prior to IMCOMBINE. For example, if to_scale = 10, based on the image header to_name keyword (default is "INT_S", the WILDFIRE image header keyword for integration time in seconds), data with INT_S = 10 are untouched and data with INT_S = 8 are multipled by 10/8=1.25 after masking and spatial shifting but prior to submission to IMCOMBINE. to_name = "INT_S" Image header to_name keyword make_stack = no Imstack images prior to imcombine? Explicitly incorporates creation of an image stack (via IMSTACK) for submission to the IMCOMBINE task to get around 102 image limit. IMCOMBINE implicitly does this internally in IRAF 2.11EXPORT. This allows one to stave this image stack for later review. THe RECOMBINE task can be used to interactively adjust the relative intensity offsets wihin this stack and recombine to see the result. This is more useful in tiled images than in fully overlapped images. apply_zero = yes Apply zeropoint to images prior to imcombine? The apply_zero flag to allows choice of: (1) applying zero-point shifts via imarith prior to imcombine, so that zeropoint adjusted results will be saved via make_stack+ or save_images+. (2) submitting intensity offsets file directly to imcombine, so that zeropoint unadjusted results will be saved via make_stack+ or save_images+. invert_zero = no Invert database zeropoints prior to application. IRAF V2.10EXPORT April 1992 imcombine has an inconsistency in the sign convention on applied zeropoint. The sign of the file based zeropoint assumes the value to be subtracted, whereas the reported number is the value to be added. These have been reconciled to "value to be added" in the IRAF V2.11EXPORT release. zero_invert flag toggles these situations. save_images = no Save images which were IMCOMBINED? Useful fto see what the shifted and masked images look like. do_combine = yes By default, 'do_combine = yes' and the images are combined into a single composite image using all the available pixels. If you would rather emulate the tiling operations of the IRALIGN/IRMATCH1(2)D tasks and have each succesive image cover up the overlap regions with prior images in the prescription, set 'do_combine = no'. 'interp_shift', 'bound_shift', and 'const_shift', are used internally by IMSHIFT and should remain at their indicated default values. The query parameter "computer_size" should be set to "no". The trimlimits should be [2:2,2:2] to exclude the edges which will have been affected by the transformation. Except at the edges (which were excluded earlier), the SQIID detectors have only a few bad pixels and fewer clusters of bad pixels. For dithered datasets, these pixels should be excluded from the assembly of the final images by setting the parameter "setpix" to "yes" and providing the name of the detector's the bad pixel mask (see the Appendix) in the parameter "maskimage". Masked pixels will be set to "svalue" if they are not overlapped by a good pixel. For mosaics and single images, the "setpix" option may also be used. However, the parameter "fixpix" may be set to "yes" instead ("setpix" must be "no") to interpolate over the bad pixels. These used the IRAF "fixpix" algorithm and requires a fixpix format filename in the parameter "fixfile". When using SETPIX, the image mask must either be in the current directory or a valid IRAF pathname to it must be included in the 'maskimage' parameter name, e.g.: mask if in current directory ../mask if one directory above data$mask if the pathname data has been defined and set in you IRAF login If GEOTRAN is requested ('do_tran = yes' in the 'infofile' database), both the ".geodb" and ".geoco" files, which are embedded in the 'infofile' database, must be in the current directory. The parameters 'lthreshold' and 'hthreshold' should be set to include the range of all good pixels in all images. For example, the range -100 to 20000 would certainly be safe for images which have been optimally sky-subtracted. Setting these top closer tolerences may improve the precision of the common median determination. The parameters "lthreshold" and "hthreshold" also limit the range of pixels values which will be included in assembly of the final image. The limits should certainly be set to exclude pixels flagged as bad (i.e., 'svalue' after masking) and pixels with unreasonable values. ------------------------------------------------------------------------ VIII. Registration summary Separately assemble the individual processed frames of the images you wish to combine into a mosaics (one per channel) using USQMOS. When assembling the database for the other channels, it is important to realize that the data at each channel must be tiled into its USQMOS mosaic at the same locations. Subsequent 'SQIID' registration tasks assume that the mosaics are equivalent accept for channel. Use of the "@list:channel_id" convention assures that such will be the case. Defer image masking and image transformation until the final stage within NIRCOMBINE. IF you have dithered frames etc, in addition to an NxM set of overlapped images, you can put them all into a single mosaic if you put the NxM image into the lower left corner of the mosaic. Then, when you use XYLAP, set nx_sub = N set ny_sub = M set nxr_sub and nyr_sub to the position (within the confines of NxM but with the proper co-ordinates from the larger array) Potential grief is minimized if you keep the X dimension of the full array at N. XYGET/XYADOPT and MERGECOM allow you to operate in larger domains, which are not necessarily organized into a simple grid. If you assemble the data in smaller mosaics, you can pull them together using XYGET to link the individual mosaics and MERGECOM to put it together into a single prescription. XYGET accepts pieces from different mosaics. The format is as follows: "Range in included image "COM# | reference COM# | mosaic name of reference" the final "|" and field is left out if they are from the same mosaic if the reference USQMOS database is mos_name.dbmos, set ref_info = "default" else set ref_info = name of USQMOS database for reference image To assemble the datasets for other SQIID channels, assuming you have assembled the data at all wavelengths in the same XY order, you can use XYADOPT to create the prescription for the other colors, once you have a successful prescription for one color. Set the parameters for XYADOPT as follows: set images to the basename of the new color set frame_nums = "Range in COM# of included images | #_of reference image in original color" set mos_info = "default" or "name_of_newcolor_usqmos_database" set basis_info = the ".xycom" file for your complete original color success set trimlimits to value approriate to new color set tran = yes set db_tran = name of appropriate ".geodb" file for transforming new_color to original_color. Both the ".geodb" and ".geoco" files must be in the current directory where you are assembling everything answer "no" whenever you are prompted by XYADOPT/ZGET/NIRCOMBINE "Do you want to resize image?". You want to stick with the origin determined by you first successful channel image. Where images do not share a proper common overlap region, use the ZGET task to determine the intensity offsets. You must specify the linkage between overlapping images you want to use to accomplish this registration. The task uses the registration database to determine the overlap regions for each image pair. Copies of the images are geotransformed, fractionally shifted, and masked. The statistics (selected by 'common') within the overlap regions of each image pair are determined and the prescription is them followed to accumulate the net offset of each image compared to the first image in the prescription. Currently, the entire overlap region is included in the statistics. Set the parameters for ZGET as follows: set match_name to the name of the image you wish to create for the channel set info_file to the name of the complete ".xycom" file for the channel set image_nums to the name of the file containing the linkage paths you want to follow in assembling the image set setpix = yes set maskimage to the bad pixel mask for the channel set common = "median" set lowerlim to a value safely below the lowest data in your images (but not INDEF!) answer "no" whenever you are prompted by XYADOPT/ZGET/NIRCOMBINE "Do you want to resize image?". You want to stick with the origin determined by you first successful channel image. Use NIRCOMBINE to assemble the final image using the XY(Z) regustration database table generated above. set match_name to the name of the image you wish to create for the channel set info_file to the name of the complete ".zcom" file for the channel If your images share a proper common overlap, use the ".xycom" file and set common = "median" set trimlimits = "[2:2,2:2]" to trim off the "rough edges" generated by IMSHIFT set register = yes to preserve image size for all channels set common = "median" if you have a common overlap region set common = "adopt" if you want to use the ".zcom" info from ZGET set setpix = yes set maskimage to the bad pixel mask for the channel set common = "median" set svalue = -1e8 to make sure bad pixels and there progeny are thresholded set lthreshold to a value safely below the lowest data in your images (but not INDEF!) set blank to a number recognizably below your lowest data answer "no" whenever you are prompted by XYADOPT/ZGET/NIRCOMBINE "Do you want to resize image?". You want to stick with the origin determined by you first successful channel image. ------------------------------------------------------------------------ IX. Handling larger images Assembling grids beyond order 6x6 can be a daunting task. The larger the grid, the more likely there will be weak or missing links in the registration process. At some point, one needs to worry about cumulative effects. Furthermore, XYLAP, which uses the output from the CENTER task is set up to operate on mosaics no larger than 10 in X or Y. (IRAF scripts do not handle variable arrays well and larger arrays eat up too much space on the stack.) The techniques summaried above are well suited for minimally-overlapped spatial grids up to 6x6 in size. What is the best way to register larger images? The answer begins with it depends ...! You have a number of choices, since the database generated by XYLAP/XYGET does not really care where the individual frames are. Here are some recipes for linking datasets composed of dithered pairs of a 3x3 grid (arranged as separate 3x3 grids arranged vertically into a 3x6 mosaic), which is extensible to larger situations: 1) The plan ahead option (probably too late for this!). Often, I will include a key piece of an adjacent frame inside my 3x6 (make a 3x7 with the additional pieces in the top row - remember that some of the entries to USQMOS/IRMOSAIC can be "null". Then I can use XYGET followed by MERGECOM to handle the overlapped piece and bring them all to the same universe. You can still do this by creating a new mosaic with the same number of frames in the X direction as your original. Then you can reuse your CENTER about and run XYLAP again with nx_sub and ny_sub set to 3. Then you run XYGET on the additonal piece to get the link to the next 3x3 and use MERGECOM to put it all together. 2) Use the option on XYGET (created for folk, like myself, who don't always plan ahead!) which allows the matched frames to be in separate USQMOS mosaics. Match frame pairs for each USQMOS that way and use MERGECOM to put it together. 3) Make a USQMOS of NIRCOMBINEd frames. Here you need to get crafty, since USQMOS/IRMOSAIC only accepts pieces of the same size. You can use MKPATTERN in the 'noao.artdata' package to make an image big enough to fit any of the pieces and then imcopy each NIRCOMBINE frame to the larger template. You should use IMREPLACE to set all the bad and border pixels to that large negative number the 'setpix' option uses (e.g. -1e8; they have been set to 'blank' by NIRCOMBINE.). Then use USQMOS to make up new mosaics for each channel (each ordered in the same way) and CENTER/XYLAP or XYGET on your favorite channel to link them. Use XYADOPT to set up the other channels. Don't use the 'setpix' option (you took care of that when you did NIRCOMBINE; now you don't have a proper mask image) or the 'tran' options (you've already done that during NIRCOMBINE) along the way in XYGET, XYLAP, XYADOPT, ZGET or NIRCOMBINE. Which way to go is a toss up. Option 1&2 rely on a single overlapping frame for the link, while option 3 let's you use several. However, if the 3x3's aren't put together well (because of weak or missing links), option 3 might lead to contradictions at the edges. Remember to feed the information to MERGECOM in the proper order when you organize the list of filenames: Each reference frame must be found within the list of frames which has accummulated from the files earlier in the list. ------------------------------------------------------------------------ APPENDIX A. Some Useful Tools A.0 Parameter Files As usual "lpar taskname" lists and "epar taskname" edits the parameter file for "taskname". Lines which are shown inside parenthesis are hidden parameters, and are not prompted when you run the task. You can set them using epar or include at the end of tour command line (following any prompted parameters you wish to include). Within the listings shown here some parameters (mode and assorted "list" parameters - parameters which appear on the left of the file listing without prompts on the right) are not shown, since they internal task parameters which cannot be changed by you and only appear because the CL language requires them to appear external to the task. (The "list" type variables which appear in an IRAF script file refer to pointer structures which are used internally to an IRAF script procedure, but because of the nature of the beast must appear externally. They show the temporary values that they had the previous invocation and will update.) Within this writeup, we have marked parameters which are of special interest to the user with an asterisk "*" in the left-hand column. A.1 CLEANUP A lot of the tasks found in this package create intermediate images/files (rescaled images, dark substracted images ...). When everything is all right, these images/files are deleted before the task is exited. Thus you are never really aware that they were ever created. If for some reason a task crashes on you, then you may be left with a few of these 'temporary' images/files in your directory. They are the images/files with names starting with _T. Typing "cleanup" after a task crashes, deletes these images/files (if any). Additional temporary files are generated within the IRAF named directory "tmp" and can be accessed via the "tmp$" alias (e.g. cd tmp$; dir tmp$; head tmp$*). A.2 SQIID image naming conventions The SQIID system generates FITS images with the following naming convention: "base_name"//XXX//"channel_id"//.fits (e.g., n1_081j.fits) where "base_name" ("n1_") was supplied by you at the time of observation, "channel_id" is j,h,k or l, and the number "XXX" is sequentially numbered (it was automatically incremented for each exposure until a new "base_name" was declared, typically at each new object). A.3 List tools CHLIST To facilitate the handling of the large numbers of image sets typically obtained during a SQIID observing run, the UPSQIID software is designed to operate on IRAF "@-list" files (inside IRAF, one instructs the system to read data from the file "list" by prepending "@" to it - "@list" tells the task to read list to get the image names.) The task "chlist" is provided to create list files. (Alternatively the task "nlocal.ranges" can be used.) To use "chlist" the image names must end in an integer. The first useful tool is called CHLIST and its parameter list is shown below. It is used to create list of images of a given color. First_image is the name of the first image name to be in the list. Number is the total number of images in the list. Delta is the increment size. This number is added to the previous image name in the list to create the next image name of the list. CHLIST: first_image = First image in sequentially numbered images number = Number of images in output list (delta = 1) (id_color = "none") Location of color indicator? (end|none|predot|s As indicated, you can include directory pathnames. Set delta = 1 for sequential image names. If you were to give IRAF the following command (with delta = 4): chlist raw001 2 > jlist then jlist would read raw001 raw005. If you were to give IRAF the following command: chlist raw003 5 > klist then klist would read raw003 raw007 raw011 raw015 raw019. If you were to give IRAF the following command (with delta = 1): chlist raw/n1_001j.1 3 > jlist then jlist would read raw/n1_001j.1 raw/n1_001j.2 raw/n1_001j.3. CHLIST has no intrinsic output file; that is why you have to use redirection. If you don't the output is printed on the terminal screen (STDOUT) but no saved. Note that CHLIST acts totally independently of what images you have in your directory. It does not check whether the images exist or not; they just create a list of names. A.4 UPSQIID task input image parameter The input image string parameter for UPSQIID tasks can be the usual "comma delimited command line lists" or "@-file" type lists. One can also use wildcards within image pathnames to operate on all images which match the template (the IRAF "section" command is invoked). For example, assuming the following images exist: raw/n1001 raw/n1002 raw/n1003 and that jlist contains: raw/n1001 raw/n1002 raw/n1003 Then, "raw/n1001,raw/n1002,raw/n1003" "@jlist" "raw/n100?.*.imh" are equivalent j-image inputs. Within the XYGET/XYADOPT tasks, the input to images should be the name of the USQMOS mosaic rather than the list. (Although an alternative path exists for submiting image lists rather than the name of the mosaic, they are not fully debugged. If you submit the list, you need to give the full name of the USQMOS mosaic database rather than "default", because default says "append .dbmos to the image name". A.5 "UPSQPROC" task output image parameter The input and out image string parameters in UPSQPROC can be the usual "comma delimited commanfd line lists" or "@-file" type lists. In addition, the IRAF substitution operator "% % %" can be used (in the same way as imrename works) to generate a new image name based on the old name with string substitution. If the output string begins with a period, the string is appended to the input names. To operate on only one image named in input and output the output name needs a prepended "," to get things to work. Consider the following examples: Let the file inlist contain im1 and the file outlist contain out1 im2 out2 im3 out3 Then the given command lines produce the corresponding outputs as shown: COMMAND LINE RESULTANT ASSIGNMENTS "upsqproc @inlist @outlist" im1 -> out1 im2 -> out2 im3 -> out3 "upsqproc @inlist %im%out%" im1 -> out1 im2 -> out2 im3 -> out3 "upsqproc im1,im2,im3 out1,out2,out3" im1 -> out1 im2 -> out2 im3 -> out3 "upsqproc @inlist .ds" im1 -> im1.ds im2 -> im2.ds im3 -> im3.ds "upsqproc im1 ,out1" im1 -> out1 Note: directory information is skipped over during the "% % %" substitution so that only that part of the pathname after the final "/" survives the substitution process. A.6 IMCOMBINE task Many of the UPSQIID image tasks (usqdark, usqsky, usqflat, nircombine) call the IRAF task "images.imcombine", which contains many parameters (most of which are only useful for the very discerning and sophisticated!). In order to make these parameters available to the UPSQIID package user, I have had to replicate them in the appropriate UPSQIID tasks. See the manual page for "imcombine" for more complete description of the parater choices. The parameters for imcombine have been extensively revised with IRAF2.10EXPORT. Some of key imcombine parameters are listed below: (combine = "average") Type of combine operation (reject = "none") Type of rejection (project = no) Project highest dimension of input images? (outtype = "real") Output image pixel datatype (offsets = "none") Input image offsets (blank = 0.) Value if there are no pixels\n (scale = "none") Image scaling (zero = "none") Image zero point offset (statsec = "") Image section for computing statistics (lthreshold = INDEF) Lower threshold (hthreshold = INDEF) Upper threshold combine = "average" (average|median) Type of combining operation performed on the final set of pixels (after offsetting, masking, thresholding, and rejection). The choices are "average" or "median". The median uses the average of the two central values when the number of pixels is even. reject = "none" (none|minmax|ccdclip|crreject|sigclip|avsigclip|pclip) Type of rejection operation performed on the pixels remaining after offsetting, masking and thresholding. The rejection choices are: none - No rejection minmax - Reject the nlow and nhigh pixels ccdclip - Reject pixels using CCD noise parameters crreject - Reject only positive pixels using CCD noise parameters sigclip - Reject pixels using a sigma clipping algorithm avsigclip - Reject pixels using an averaged sigma clipping algorithm pclip - Reject pixels using sigma based on percentiles blank = 0. Output value to be used when there are no pixels. scale = "none" (none|mode|median|mean|exposure) Multiplicative image scaling to be applied. The choices are none, scale by the mode, median, or mean of the specified statistics section, or scale by the exposure time given in the image header. Note that any keyword can be used for scaling and it need not actually be the exposure time. zero = "none" (none|mode|median|mean) Additive zero level image shifts to be applied. The choices are none or shift by the mode, median, or mean of the specified statistics section. statsec = "" Section of images to use in computing image statistics for scaling and weighting. If no section is given then the entire region of the input is sampled (for efficiency the images are sampled if they are big enough). lthreshold = INDEF, hthreshold = INDEF Low and high thresholds to be applied to the input pixels. This is done before any scaling, rejection, and combining. If INDEF the thresholds are not used. A set of images are combined by weighted averaging or medianing. Pixels may be rejected from the combining by using pixel masks, threshold levels, and rejection algorithms. The images may be scaled multiplicatively or additively based on image statistics or image header keywords before rejection. After all the steps of offsetting the input images, masking pixels, threshold rejection, scaling, and applying a rejection algorithm the remaining pixels are combined and output. The pixels may be combined by computing the median or by computing a weighted average. The number of images combined is recorded in the output image header under the keyword NCOMBINE. Note: In IRAF2.9, IMCOMBINE requires full size images be sumitted so that one has to make resultant-size blanks and paste the little pieces in, one per blank. Combining multiple images can require large amounts of diskspace ((images+1)* size of final image!). IRAF2.10, which NIRCOMBINE will use whenever it senses it, allows one to submit the little pieces and their offsets in the final image and requires must less room to operate. A.7 Making MASK images The UPSQIID task 'mkmask' can be used to produce bad-pixel image masks from flatfield data. Mkmask produces a two-valued image with bad-pxels set to 0 and good pixels set to 1. The task "mkmask" has a number of parameters which should be set as indicated in the "lpar" listing below. This format will be used henceforth to document the parameter default settings and the parameters of interest to the user (which are indicated with an * in the leftmost column). MKMASK: * input = "n43_flatj" Input images * output = "jmask" Clipped output images * lower_lim = 0.85 Lower limit for in/exclusion * upper_lim = 1.15 Upper limit for in/exclusion (in_value = 1.) Replacement value inside range (out_value = 0.) Replacement value outside range * (section = "[]") Image section for replacement * (trimlimits = "[2:2,18:0]") trim limits around edge (verbose = yes) Verbose output? Pixels outside the range lower_lim to upper_lim are presumbed to be bad pixels and are set to 'out_value' (0). Pixels inside the range are presumed to be good pixels and are set to 'in_value' (1). The 'trimlimits' string can be used to 'square the edges' of your mask. The syntax refers to the number of pixels on the left, right, bottom, and top edges of the input should be nulled in the output image. One adjusts the 'lower_lim' and 'upper_lim' parameters until one is happy with the result. Note that this technique is not guaranteed to correctly identify all the pixels one might want to mask (e.g. those with wildly variable dark current, etc.) The SQIID detector has a metal mask just above the detector surface to prevent radiation from entering the dectector outside the active arrary (such radiation can propagate along the readout and produce a glint across the entire array along a row or column.) This physical mask vinettes radiation a wide angles at the edge of the array. One can trust the postional information in the >10% attenuation region, but the photometric information is suspect and should be maked off by your choice of pixel mask image. "trimlimits = "[2:2,18:0]" accommodates the current position (1998) of this mask. A representative script for the raw data and oriented data is shown below: mkmask n57_flatl lmask0798 0.85 1.15 trim=[2:2,18:0] imtranspose lmask0798[*,-*] lmask0798tr The UPSQIID SETPIX operation uses these mask to set bad pixels to a predetermined value: data at bad pixel locations (as determined by the where 0 is a bad pixel and 1 is a good pixel) is replaced with the pre-detrmined value. When using SETPIX, the image mask must either be in the current directory when you use the 'UPSQIID' tasks 'upsqproc', 'zget' and 'nircombine' or a valid IRAF pathname to it must be included in the maskimage parameter name for those tasks, e.g.: mask if in current directory ../mask if one directory above data$mask if the pathname data has been defined and set in you IRAF login A.8 Re-orienting images One can simply change the orientation of IRAF images as follows: imtranspose test[-*,*] cw90 - rotate 90 deg clockwise imtranspose test[*,-*] ccw90 - rotate 90 deg counter-clockwise imcopy test[-*,-*] rot180 - rotate 180 deg imcopy test[-*,*] vflip - flip about the vertical (y) axis imcopy test[*,-*] hflip - flip about the horizontal (x) axis The same image section notation can be appended to image names in the 'display' to view images in various orientations. A.9 Keeping track of your reductions Here's some general advice which should help you to keep track of your SQIID reductions. The 'UPSQIID' package is still under development and new files are generated at each processing step. * 1) Make separate directories for each observing run and back up the text files for each run together. * 2) Place your raw image data in a subdirectory of your working directory to avoid clutter. * 3) When images have been collected into a mosaic and are no longer needed individually, move them to a separate subdirectory to avoid clutter. * 4) Normal processing generates at least three copies of each image (raw, processed, mosiac section). To conserve space, consider backing up and deleting images which are no longer necessary for quick access. (However, remember that the J channel images need to be present when you use channel offsets on wildcarded input lists ("raw/n10_001j.*.imh:h)). * 5) Do your data processing from script files which can be directly submitted to the CL by typing: * cl < filename * Put commands, one per line in a file, including the necessary inputs and explicitly setting parameters as needed. Include comment lines as needed. (A # symbol at the start of a line makes it a comment to CL and it skips over that line.) Comment out lines after they have been run. This provides a permanent record of your work and allows rapid recover from errors. Judicious use of your favorite editor vastly simplifies the amount of typing required to get the job done! Here is a sample file called "doit": # usqdark raw/n2_001j.*.imh:jhkl n32_dark10pm # usqflat @n32flatj.lis:j n32_flat n32_dark180pm # usqflat @n32flatj.lis:h n32_flat n32_dark180pm # usqflat @n32flatj.lis:k n32_flat n32_dark180pm # usqflat @n32flatj.lis:l n32_flat n32_dark180pm # usqsky @n32_013skyj.lis:jhkl n32_013sky # upsqproc raw/n2_015j.*.imh:k %n2%sn32% n32_flatk n32_015skyk orient+ # upsqproc raw/n2_015j.*.imh:h %n2%sn32% n32_flath n32_015skyh orient+ # upsqproc raw/n2_015j.*.imh:j %n2%sn32% n32_flatj n32_015skyj orient+ # usqmos @mosn32_015j.lis:k mosn32_015k 3 8 median+ subtract- # usqmos @mosn32_015j.lis:h mosn32_015h 3 8 median+ subtract- # usqmos @mosn32_015j.lis:j mosn32_015j 3 8 median+ subtract- # usqmos @mosn32_013j.lis:k mosn32_013k 3 8 median+ subtract- null="23,24" # center mosn32_015k # xylap mosn32_015k mosn32_015k.ctr.1 nx_=3 ny_=3 nxr=2 nyr=2 trim="[4:3,8:0]" # xyget mosn32_015k "10|1" out=mosn32_015k_10.xycom trim=[4:3,8:0] # xyget mosn32_015k "11|2" out=mosn32_015k_11.xycom trim=[4:3,8:0] # xyget mosn32_015k "12|3" out=mosn32_015k_12.xycom trim=[4:3,8:0] # xyget mosn32_015k "13|4" out=mosn32_015k_13.xycom trim=[4:3,8:0] # xyget mosn32_015k "14,23-24|5" out=mosn32_015k_14.xycom trim=[4:3,8:0] # xyget mosn32_015k "15|6" out=mosn32_015k_15.xycom trim=[4:3,8:0] # xyget mosn32_015k "16|7" out=mosn32_015k_16.xycom trim=[4:3,8:0] # xyget mosn32_015k "17|8" out=mosn32_015k_17.xycom trim=[4:3,8:0] # xyget mosn32_015k "18|9" out=mosn32_015k_18.xycom trim=[4:3,8:0] # mergecom @mosn32_015k.xyget mosn32_015k.xycom # xyadopt mosn32_015h "1-24|5" mosn32_015k.xycom trim=[6:2,0:9] db=htoktr.geodb # xyadopt mosn32_015j "1-24|5" mosn32_015k.xycom trim=[5:2,6:1] db=jtoktr.geodb # zget con32_015k mosn32_015k.xycom @mosn32_015k.zget set+ mask=gusq$proj/kmasktr # nircombine con32_015k con32_015k.zcom common=adopt mask=gusq$proj/kmasktr # zget con32_015h mosn32_015h.xycom @mosn32_015k.zget set+ mask=gusq$proj/hmasktr # nircombine con32_015h con32_015h.zcom common=adopt mask=gusq$proj/hmasktr # zget con32_015j mosn32_015j.xycom @mosn32_015k.zget set+ mask=gusq$proj/jmasktr # nircombine con32_015j con32_015j.zcom common=adopt mask=gusq$proj/jmasktr # xyget mosn32_018k "1|1|mosn32_015k" out=mosn32_018k_001.xycom trim=[4:3,8:0] # xyget mosn32_018k "2|2|mosn32_015k" out=mosn32_018k_002.xycom trim=[4:3,8:0] # xyget mosn32_018k "3|3|mosn32_015k" out=mosn32_018k_003.xycom trim=[4:3,8:0] # xyget mosn32_018k "4|4|mosn32_015k" out=mosn32_018k_004.xycom trim=[4:3,8:0] # xyget mosn32_018k "5|5|mosn32_015k" out=mosn32_018k_005.xycom trim=[4:3,8:0] # xyget mosn32_018k "6|6|mosn32_015k" out=mosn32_018k_006.xycom trim=[4:3,8:0] # xyget mosn32_018k "7|7|mosn32_015k" out=mosn32_018k_007.xycom trim=[4:3,8:0] # xyget mosn32_018k "8|8|mosn32_015k" out=mosn32_018k_008.xycom trim=[4:3,8:0] # xyget mosn32_018k "9|9|mosn32_015k" out=mosn32_018k_009.xycom trim=[4:3,8:0] # mergecom @mosn32_018k.xyget mosn32_018k.xycom A.10 CENTER task The CENTER task ('noao.digiphot.apphot.center') is useful for generating input lists of object coordinates. For convenience, it is also accessible (with separate parameter files) from the 'noao.imred.irred package', which is loaded by the '' package. (This way it will not collide with your choices for aperture photometry!) 'centerpars' and 'datapars' allow you to modify the associated parameter sets if required. Operationally, one displays the target image in the image display, enters the center task by typing: center image_name One then proceeds to either examine (f key) or record (space bar) the centered positions of objects under the image cursor. You get an error indicatation center position is outside 'maxshift' (typically 1 pixel). Look at the center indicated n the output line to see if the fit is acceptable. (If it isn't, you need to edit the output later by hand; use 'f' first if you are in doubt.) Be especially careful near the edges of images, especially when you are near the 1-pixel gap between adjacent subrasters within a mosaic image; the value of this region is generally quite different from the rest of the image and can strongly influence results. You quit by typing "w" to write the output and then "q" to exit. By default the positions appear in a text file 'image_name.ctr.N', where "N" increments each time a new file is produced in the current directy for the same image_name. You may wish to select a cursor marker in your display task so you can keep track of the objects as you go along. CENTER (Jul91) noao.digiphot.apphot CENTER (Jul91) CENTER computes accurate centers for a set of objects in the IRAF image 'image', whose coordinates are read from the image display interactively with the image cursor. The computed x and y coordinates, all the errors plus all the fitting commands are written to the text file 'output'. CENTER can be run either interactively or in batch mode by setting the parameter 'interactive'. In interactive mode starting x and y positions can be read directly from the image cursor; the user can examine, adjust and save the algorithm parameters. Selected Interactive Centering Commands for Image Cursor: ? Print help for CENTER commands v Verify critical CENTER parameters w Store the current CENTER parameters in the pset files q Exit spbar Fit center of current star (cursor or list), and output results f Fit center of current star (cursor or list) e Print CENTER error messages CENTER: image = "mosn7023k" Input image (datapars = "") Data dependent parameters (centerpars = "") Centering parameters (coords = "") Coordinate list (output = "default") Results (plotfile = "") Plot metacode file (graphics = "stdgraph") Graphics device (display = "stdimage") Display device (commands = "") Image cursor: [x y wcs] key [cmd] (cursor = "") Graphics cursor: [x y wcs] key [cmd] (interactive = yes) Mode of use (verify = yes) Verify critical parameters in non-interactive m (update = no) Update critical parameters in non-interactive m (radplots = no) Plot radial plots on standard graph (verbose = no) Print messages in non interactive mode CENTERPARS: (calgorithm = "centroid") Centering algorithm (cbox = 7.) Centering box width in scale units (maxshift = 1.) Maximum center shift in scale units (minsnratio = 1.) Minimum SNR ratio for centering (cmaxiter = 10) Maximum iterations for centering algorithm (clean = no) Symmetry clean before centering (rclean = 1.) Cleaning radius in scale units (rclip = 2.) Clipping radius in scale units (kclean = 3.) K-sigma rejection criterion in skysigma (mkcenter = no) Mark the computed center DATAPARS: (use at default settings) (scale = 1.) Image scale in units per pixel (fwhmpsf = 1.) FWHM of the PSF in scale units (emission = yes) Features are positive ? (sigma = INDEF) Standard deviation of background in counts (datamin = INDEF) Minimum good data value (datamax = INDEF) Maximum good data value (threshold = 0.) Detection threshold in counts above background (cthreshold = 0.) Centering threshold in counts above background Selected CENTER task parameters: image The list of images containing the objects to be centered. datapars = "" The name of the file containing the data dependent parameters. The critical parameters 'fwhmpsf', and 'sigma' are located here. If 'datapars' is null ("") then the default parameter set in the user's uparm directory is used. centerpars = "" The name of the file containing the centering algorithm parameters. The critical parameters 'calgorithm' and 'cbox' are located here. If 'centerpars' is null ("") then the default parameter set in the user's uparm directory is used. coords = "" The list of text files containing the coordinates of the objects to be centered, one object per line with the x and y coordinates in columns one and two respectively. If 'coords' is null, CENTER will either read the image cursor (default), or read the cursor commands from a text file. In the latter case the image cursor 'commands' is set to the name of a command file created for example by hand or with the RIMCURSOR task. If 'coords' is defined, CENTER will activate the image cursor and read the coordinates from either the image cursor or 'coords' (interactive mode), or compute centers using the initial postions in 'coords' if defined or those in the command file (batch mode). The number of coordinates files must be either 1 or equal to the number of image files. output = "default" The name of the results file or results directory. The string "default" indicates the current directory. If 'output' is "default", "dir$default" or a directory specification then an output name of the form dir$root.extension.version is constructed, where dir is the directory, root is the root image name, extension is "ctr" and version is the current version of the file. The number of output files must be either 1 or equal to the number of image files. If 'output' is the null string then no output file is written. In both interactive and batch mode full output is written to 'output'. In interactive mode a subset of the results is written to the standard output. A.11 The 'noao.imred.irred package The 'noao.imred.irred' contains a number of useful tasks which are either used or emulated by the 'UPSQIID' registration software. In particular, the many of the 'UPSQIID' task parameters have direct analoges in the parameters for IRMOSAIC, IRALIGN and IMSHIFT. The following is a brief description of selected tasks in this package which are of special interest to the 'UPSQIID' user: imred.irred: txdump - Select fields from the center task output text file center - Compute accurate centers for a list of objects centerpars - Edit the centering parameters datapars - Edit the data dependent parameters iralign - Align the image produced by irmosaic irmatch1d - Align and intensity match the image produced by irmosaic (1D) irmatch2d - Align and intensity match the image produced by irmosaic (2D) irmosaic - Mosaic an ordered list of images onto a grid IRMOSAIC (Oct89) noao.nproto IRMOSAIC (Oct89) IRMOSAIC takes a the list of subrasters of identical dimensions specified by 'input' and combines them into a single output image 'output'. The order in which the subrasters are placed in the output image is determined by the parameters 'corner', 'direction' and 'raster'. The orientation of each individual subraster in the output image may be altered by setting the 'trim'_'section' parameter. IRMOSAIC uses the subraster size, the number of subrasters, the 'nxoverlap' and nyoverlap 'parameters' and the 'nxsub' and 'nysub' partmeters to compute the size of the output image. An image of size larger than the minimum required can be specified by setting 'nimcols' and 'nimrows'. The pixel type of the output image is specified by 'opixtype' and undefined regions of the output image are given the value 'oval'. The median of a section each subraster may be optionally computed and placed in the database file by setting 'median'_'section'. The computed median will be subtracted from the input subrasters if 'subtract' is set to yes. Task action messages will be printed on the standard output if 'verbose' is set to yes. Selected IRMOSAIC parameters: input The list of input images to be mosaiced. The images are assumed to be ordered either by row, column, or in a raster pattern. If the image list is not in order then the iraf files task plus the editor must be used to construct an image list. The images in the input list are assumed to all be the same size. output The name of the output image. database The name of the text file listing the operations performed by irmosaic. This list can be used as input for iralign/irmatch1(2)d. nxsub The number of subrasters along a row of the output image. nysub The number of subrasters along a column of the output image. trim_section = "[*,*]" The section of the input images to be mosaiced into the output image. Section can be used to flip and/or trim the individual subrasters before adding them to the mosaic. For example if we want to flip each subraster around the y axis before adding it to the mosaic, then 'trim'_'section' = "[*,-*]". null_input = "" The list of unobserved subrasters. For example if the subrasters 3 to 5 and 10 of a sequence of observations were not observed then 'null'_'input' = "3-5,10". This parameter follows the ranges notation convention. The number of unobserved subrasters plus the number of images must equal 'nxsub' * 'nysub'. corner = "ll" The starting position in the output image. The four options are "ll" for lower left corner, "lr" for lower right corner, "ul" for upper left corner and "ur" for upper right corner. direction = "row" Add subrasters to the output image in row or column order. The options are "row" for row order and "column" for column order. raster = no (value preset by USQMOS) Add subrasters to the output image in a raster pattern or return to the start of a column or a row? nxoverlap = -1 (value preset by USQMOS) The number of columns between adjacent frames. A negative value specifies the amount of column space between adjacent subrasters. A positive value specifies the amount of column overlap on adjacent subrasters. nyoverlap = -1 (value preset by USQMOS) The number of rows between adjacent frames. A negative value specifies the amount of row space between adjacent subrasters. A positive value specifies the amount of row overlap on adjacent subrasters. oval = 0.0 The output image pixel value in regions undefined by the by the list of input images. opixtype = "r" The pixel type of the output image. The options are "s" (short integer), "i" (integer), "l" (long integer), "r" (real) and "d" for double precision. verbose = yes Print messages about task progress and actions taken. IRMATCH1(2)D (Jan90) noao.nproto IRMATCH1D (Jan90) Description of IRALIGN/IRMATCH1D/IRMATCH2D These tasks take the mosaiced image 'input', the database file 'database' generated by IRMOSAIC and a list of coordinates 'coords' and compute an output image 'output' in which all the individual subrasters are aligned. If 'alignment' = "coords", they accumulate the relative shifts between adjacent subrasters into a total shift with respect to the reference subraster. Shifts which do not correspond to adjacent subrasters are ignored. For subrasters which have no direct shift information, the tasks make a best guess at the x and y shift based on the shifts of nearby subrasters which do have direct shift information. The subrasters are inserted into the output image using the interpolation scheme defined by 'interpolant' and is made with reference to the subraster defined by 'nxrsub' and 'nyrsub', using the shifts defined by the coordinates in the file 'coords'. Subrasters are placed in the output image in the order they were inserted into the original mosaic with pixels in the most recently placed subrasters replacing those placed earlier in the overlap regions. Undefined pixels in the output image are given the value 'oval'. The position of the reference image in the output image can be adjusted by setting the parameters 'xref' and 'yref'. The edges of each subraster may be trimmed before insertion into the output image by setting the 'trimlimits' parameter. Intensities of adjacent subrasters can be matched using the 'match' parameter in IRMATCH1D/2D. At present matching is done by computing the median in the overlap region between adjacent subrasters and applying difference in these two numbers to the subrasterin question: In IRMATCH1D intensity matching is done in one dimension only with the direction of matching following the order that the individual subrasters were inserted into the mosaic. For example if IRMOSAIC was run with 'corner' = "ll", 'direction' ="row" and 'raster' = "no", then the matching would start in the lower-left corner, proceed along the first row, move to the star of the second row and so on. In IRMATCH2D intensity matching is done in two dimensions, first along the direction in which subrasters were inserted into the mosaic and then in the orthogonal dimension. For example if IRMOSAIC was run with 'corner' = "ll", 'direction' = "row" and 'raster' = "no", then the matching would proceed along each row starting with the lower-left hand corner and then along each column beginning again in the lower-left corner. Selected IRMATCH1(2)D parameters: input The mosaiced image to be aligned. This image must have been produced by the IRMOSAIC task and have an accompanying database file specified by 'database'. output The aligned image produced by IRMATCH1D. database The database file from the IRMOSAIC task. coords If 'alignment' = "coords", then 'coords' is a text file listing the coordinates of objects in the input image one object per line in the following format: 1) the x and y coordinates of the object in the first subraster 2) the x and y coordinates of the same object in the second subraster 3) the x and y coordinates of the next object in the first subraster etc. If 'alignment' = "file", then 'coords' is a text file listing the x, y and intensity shifts in columns 1, 2 and 3 respectively, of each input subraster relative to the reference subraster. The most common use of this option is to make fine adjustments by hand to the output of IRMATCH1D by editing the computed shifts slightly and rerunning IRMATCH1D with the new shifts. match = "*" Match intensities using the overlap region between adjacent subrasters. The median intensity is computed in the overlap region and the intensity scale of the current subraster is scaled to that of the previous subraster. Intensities are matched in one dimension in the order in which they are placed in the output image. The default is match everything. Those subrasters to be matched must be listed by number. For example to match intensities for subrasters 1 to 5 and 10 to 20 set match = "1-5,10-20". To match all the subrasters set match = "1-999" or match="*". nxrsub = INDEF, ls nyrsub = INDEF The column and line index of the reference subraster. This will default to the central subraster. trimlimits = "[0:0,0:0]" The number of columns and rows to be trimmed off each edge of the input subraster before it is inserted in the output image in section notation. The default is to trim 1 column or row in each direction. oval = INDEF The value of undefined pixels in the output image. The default is the value in the database file from IRMOSAIC. interpolant = linear The type of interpolant used to shift the subrasters. The options are: nearest Nearest neighbour interpolation. linear Bilinear interpolation. poly3 Bicubic polynomial interpolation. poly5 Biquintic polynomial interpolation. spline3 Bicubic spline interpolation. verbose = no Print messages on the terminal describing the progress of the task. match = "*" Match intensities using the overlap region between adjacent subrasters. The median intensity is computed in the overlap region and the intensity scale of the current subraster is scaled to that of the previous subraster. Intensities are matched in two dimensions, first in the order in which they were placed in the output image and then in the orthogonal dimension. The default is match everything. Those subrasters to be matched must be listed by number. For example to match intensities for subrasters 1 to 5 and 10 to 20 set match = "1-5,10-20". To match all the subrasters set match = "1-999" or match="*". Coordinate lists may be generated interactively on the Sun workstations using the IRAF imtool facility and centered using the APPHOT CENTER and APSELECT tasks. In addition, the perscription for image tasks employed by 'UPSQIID' follows the 'images.imshift' conventions. IMSHIFT (Feb88) images IMSHIFT (Feb88) IMSHIFT will shift an image in x and y such that: xout = xin + xshift yout = yin + yshift The output image gray levels are determined by interpolating in the input image at the positions of the shifted output pixels. IMSHIFT uses the routines in the 2-D interpolator package. Selected IMSHIFT parameters: xshift, yshift Fractional pixel shift in x and y such that xout = xin + xshift and yout = yin + yshift. interpolant_type = "linear" The choices are the following: nearest Nearest neighbour. linear Bilinear interpolation in x and y. poly3 Third order interior polynomial in x and y. poly5 Fifth order interior polynomial in x and y. spline3 Bicubic spline. boundary_type = "nearest" The choices are: nearest Use the value of the nearest boundary pixel. constant Use a constant value. reflect Generate value by reflecting about the boundary. wrap Generate a value by wrapping around to the opposite side of the image. A.12 Multi-channel registration: GEOMAP/GEOTRAN The IRAF task 'images.geomap' computes a spatial transformation function which can be used by its companion task 'images.geotrans' to transform one image to overlay another image. The effects of relative spatial offset, rotation and magnification can be corrected. The GEOMAP routine can be used to establish the geometric transformation between channel pairs from ordered lists of star coordinates for each frame. A second order linear fit (corresponding to differential offset, magnification, and rotation) appears to be sufficient for SQIID images; there is no evidence of higher order distortion in the images. Since the relative pointing of the four SQIID channels appears to be constant at the sub-pixel level, we have found that the easiest way to establish/monitor the relative pointing is by taking pictures of globular clusters; a single 60-second exposure of the core of M3/M5/M15 provides sufficient information to map the spatial registration between the channels. Any frames with stars distributed around the field can be used. If you have taken standards distributed across, their relative coordinates will do. (Either co-add them to a single frame, or make of files from individual positions. GEOMAP computes the transformation required to map the coordinate system of the reference image to the coordinate system of the input image. The coordinates of the reference points in the two images are listed in the text file(s) input one per line in the following format xref yref yin yin. The transformation has the following form: xin = f (xref, yref) yin = g (xref, yref) The parameter list is given below: GEOMAP: input = "ctr.list" Input data output = "ctr.geo.db" Output database xmin = -43. Minimum x value xmax = 300. Maximum x value ymin = -43. Minimum y value ymax = 300. Maximum y value (function = "polynomial") Surface type (xxorder = 2) Order of x fit in x (xyorder = 2) Order of x fit in y (xxterms = yes) Include cross-terms in x fit? (yxorder = 2) Order of y fit in x (yyorder = 2) Order of y fit in y (yxterms = yes) Include cross-terms in y fit? (reject = 0.) Number of sigma for rejection (calctype = "double") Computation type (real,double) (interactive = yes) Fit surface interactively (graphics = "stdgraph") Default graphics device (cursor = "") Graphics cursor input A list of text files containing the pixel coordinates of control points in the reference and input images. The control points are listed one per line in the form xref yref xin yin. By convention, we have been giving them a name ending in ".geoco" output The name of the textfile database where the computed transformation will be stored. The output database record(s) are given the names of the individual text files specified by 'input'. By convention, we have been giving them a name ending in ".geodb" xmin, xmax, ymin, ymax The range of reference coordinates over which the coordinate transformation is valid. These must be set by the user. The choices shown correctly allow for the offsets among channel centers. function = "polynomial" The type of analytic surface to be fit. The options are the following. legendre Legendre polynomials in x and y. chebyshev Chebyshev polynomials in x and y. polynomial Power series in x and y. By convention, we have been using "polynomial". xxorder = 2, xyorder = 2, yxorder = 2, yyorder = 2 The order of the polynomials in x and y for the x and y fits. The default order and cross-term settings define a linear transformation in x and y, where the 6 coefficients can be interpreted in terms of an x and y shift, an x and y scale change and a rotation of the x and y axis. interactive = yes In interactive mode the user may interact with the fitting process, e.g. change the order of the fit, reject points, displaying the data graphically until a satisfactory fit has been achieved. The 'UPSQIID' package has a few tools which make the task of getting the coordinate input files easier. The images at each channel can be loaded into the ximtool window in separate frames. Mark stars in the same sequence for each channel, choosing stars which are well distributed over the frame, unconfused and present on all frames. On Sun systems toggling the cursor readout mode (F6 key) to on allows you to mark stars with numbers and collect the positions into files. 'getcoo' then takes these positions, centers the objects, and outputs a coordinate list. Parameters for GETCOO are as follows: Both (.geoco & .geodb) files are needed in the directory where images are GEOTRANed. The task 'usqremap' is a protype which allows one to quickly ascertain the GEOMAP files for a object where the star positions at some other time are already known. It works only when the archieve data have nearly the same relative rotation as scale as the new data (which is generally the case for SQIID). The parameters for USQREMAP are as follows: USQREMAP: objimg = Input image objcoo = File with initial object X Y star coord outdb = Output database id (".geodb" will be appended) (refimg = "") Reference image (refcoo = "") File with initial reference X Y star coord (refmaster = "") Master reference image image (outco = "default") Output coordinate file (logfile = "STDOUT") Log file name (bigbox = 7.) Size of coarse search box (boxsize = 5.) Size of final centering box (xmin = INDEF) GEOMAP: minimum x value (xmax = INDEF) GEOMAP: maximum x value (ymin = INDEF) GEOMAP: minimum y value (ymax = INDEF) GEOMAP: maximum y value (zscale = yes) DISPLAY using zscale? (geometry = "rscale") fit geometry (interactive = yes) Interactive GEOMAP? (z1 = 0.) minimum greylevel to be displayed (z2 = 4000.) maximum greylevel to be displayed (verbose = yes) Verbose reporting 'objimg' is the current name of the image from the channel you wish to transform; 'objcoo' is the name of the archive co-ordinate file for that channel. 'refimg' is the name of current reference image; 'refmaster' is the name of the archive reference image and 'refcoo' is the name of its corresponding co-ordinate file. USQREMAP displays the master and current reference images and prompts you to select the same star in each so that an offset can be determined. These offsets are applied to both archive co-ordinate files and the 'proto.imcentroid' task is run to find the current positions for these stars; missing stars are eliminated. 'geomap' is then run on the new postions to generate the appropriate files. The 'UPSQIID' taks employ the 'geotran' task to transform images from one channel basis to another. GEOTRAN (Apr92) images GEOTRAN (Apr92) GEOTRAN corrects an image for geometric distortion using the coordinate transformation determined by GEOMAP. The transformation is stored as the coefficients of a polynomial surface in record 'transform', in the text file 'database'. The coordinate surface is sampled at every 'xsample' and 'ysample' pixel in x and y. The transformed coordinates at intermediate pixel values are determined by bilinear interpolation in the coordinate surface. If 'xsample' and 'ysample' = 1, the coordinate surface is evaluted at every pixel. The output image gray levels are determined by interpolating in the input image at the positions of the transformed output pixels. If the 'fluxconserve' switch is set the output pixel values are multiplied by the Jacobian of the transformation. GEOTRAN uses the routines in the 2-D interpolation package. The linear portion of the transformation may be altered after the fact by setting some or all of the parameters 'xin', 'yin', 'xout', 'yout', 'xshift', 'yshift', 'xmag', 'ymag', 'xrotation', 'yrotation'. Xin, yin, xshift, yshift, xout and yout can be used to redefine the shift. Xmag, ymag and scangle reset the scale of the transformation. Xrotation and yrotation define the orientation of the tranformation. If the CL environment variable 'nomwcs' is "no" then the world coordinate system of the input image will be modified in the output image to reflect the effects of the 'linear' portion of the geometric transformation operation. Support does not yet exist in the IRAF world coordinate system interface for the higher order distortion corrections that GEOTRAN is capable of performing. Selected GEOTRAN parameters: GEOTRAN: input = "" Input data output = "" Output data database = "" Name of GEOMAP database file transform = Names of coordinate transforms in database file (geometry = "geometric") Transformation type (linear,distortion,geometri (xin = INDEF) X origin of input frame in pixels (yin = INDEF) Y origin of input frame in pixels (xshift = INDEF) X origin shift in pixels (yshift = INDEF) Y origin shift in pixels (xout = INDEF) X origin of output frame in reference units (yout = INDEF) Y origin of output frame in reference units (xmag = INDEF) X scale of input picture in pixels per referenc (ymag = INDEF) Y scale of input picture in pixels per referenc (xrotation = INDEF) X axis rotation in degrees (yrotation = INDEF) Y axis rotation in degrees (xmin = INDEF) Minimum reference x value of output picture (xmax = INDEF) Maximum reference x value of output picture (ymin = INDEF) Minimum reference y value of output picture (ymax = INDEF) Maximum reference y value of output picture (xscale = 1.) X scale of output picture in reference units pe (yscale = 1.) Y scale of output picture in reference units pe (ncols = INDEF) Number of columns in the output picture (nlines = INDEF) Number of lines in the output picture (xsample = 1.) Coordinate surface sampling interval in x (ysample = 1.) Coordinate surface sampling interval in y (interpolant = "linear") Interpolant (nearest,linear,poly3,poly5,spline3 (boundary = "nearest") Boundary extension (nearest,constant,reflect,wr (constant = 0.) Constant boundary extension (fluxconserve = yes) Preserve image flux? (nxblock = 256) X dimension of working block size in pixels (nyblock = 256) Y dimension of working block size in pixels database The name of the text file containing the coordinate transformation (generally the database file produced by GEOMAP). If database is the null string then GEOTRAN will perform a linear transformation based the values of xin, yin, xout, yout, xshift, yshift, xmag, ymag, xrotation and yrotation. If all these parameters have their defaults values the transformation is a null transformation. If the geometric transformation is linear xin, yin, xout, yout, xshift, yshift, xmag, ymag, xrotation and yrotation can be used to override the values in the database file. transform The list of record name(s) in the file 'database' containing the desired transformations. This record name is usually the name of the text file input to geomap listing the reference and input coordinates of the control points. The number of records must 1 or equal to the number of input images. The record names may be listed in a text file 1 record per line. The transform parameter is only requested if database is not equal to the null string. geometry = "geometric" The type of geometric transformation. The geometry parameter is only requested if database is not equal to the null string. The options are: linear Perform only the linear part of the geometric transformation. geometric Compute both the linear and distortion portions of the geometric correction. xmin = INDEF, xmax = INDEF, ymin = INDEF, ymax = INDEF The minimum and maximum x and y reference values of the output image. If a database file has been defined xmin, xmax, ymin and ymax ymax default to the minimum and maximum values set by GEOMAP and may be less than but may not exceed those values. xscale = INDEF, yscale = INDEF The output picture x and y scales in units of x and y reference units per output pixel, e.g " / pixel or Angstroms / pixel if the reference coordinates are " or Angstroms. If the reference coordinates are in pixels then xscale and yscale should be 1.0 to preserve the scale of the reference image. If xscale and yscale are undefined (INDEF), xscale and yscale default to the range of the reference coordinates over the range in pixels. Xscale and yscale overide the values of ncols and nlines. interpolant = "linear" The interpolant used for rebinning the image. Same choices as IMSHIFT. boundary = "nearest" Same choices as IMSHIFT. constant = 0.0 The value of the constant for boundary extension. fluxconserve = yes Preserve the total image flux. The output pixel values are multiplied by the Jacobian of the coordinate transformation. xshift = INDEF, yshift = INDEF The shift of the input origin in pixels. If the database file is undefined then xshift and yshift determine tha shift of xin, yin. xmag = INDEF, ymag = INDEF The scale factors of the coordinate transformation in units of input pixels per reference coordinate unit. If database is undefined xmag and ymag default to 1.0; otherwise xmag and ymag default to the values found by GEOMAP. If the database file is not null then xmag and ymag override the values found by GEOMAP. xrotation = INDEF, yrotation = INDEF The rotation angles in degrees of the coordinate transformation. Positive angles are measured counter-clockwise with respect to the x axis. If database is undefined then xrotation and yrotation default to 0.0; otherwise xrotation and yrotation default to the values found by GEOMAP. If database is not NULL then xrotation and yrotation override the values found by GEOMAP. A.13 UPSQIID image registration database format The UPSQIID package generates registration databases according to a simple format. The position of each individual image is descibed by a COM-line (a lime beginning with "COM"). Commenting out a COM-line (by putting a "#" in front) eliminates that image from the final combined image. A sample database file and a description of the COM-line follows: #DBM Wed 15:56:15 15-Jan-92 MERGECOM: mosn10_024k.xycom #DBM master_file mosn10_024k.xylap #DBM master_ref mosn10_024k[258:513,258:513] #DBM merge_file mosn10_024k.xylap #DBM merge_file mosn10_024k_10.xycom #DBM merge_file mosn10_024k_11.xycom #DBM merge_file mosn10_024k_12.xycom #DBM merge_file mosn10_024k_13.xycom #DBM merge_file mosn10_024k_14.xycom #DBM merge_file mosn10_024k_15.xycom #DBM merge_file mosn10_024k_16.xycom #DBM merge_file mosn10_024k_17.xycom #DBM merge_file mosn10_024k_18.xycom #DB # Wed 14:23:16 15-Jan-92 #DB begin mosn10_024k #DB trimsection [*,*] #DB medsection [*,*] #DB ncols 256 #DB nrows 256 #DB nxsub 3 #DB nysub 8 #DB nxoverlap -1 #DB nyoverlap -1 #DB corner ll #DB order row #DB raster no #DB oval 10000. #DB nsubrasters 24 #DB Wed 14:23:37 15-Jan-92 MKMOS: #DB null_input 23-24 #DB mosaic mosn10_024k #DB median_compute yes #DB median_subtract no #DBL Wed 15:27:57 15-Jan-92 LINKLAPS: #DBL info_file _Tgmh9352es.ctr #DBL nxrsub 2 #DBL nyrsub 2 #DBL ref_image mosn10_024k[258:513,258:513] #DBL ref_nim 5 #DBG Wed 15:28:10 15-Jan-92 GETMATCH: #DBG basis_info mosn10_024k.ctr.1 #DBG lap_basis center #DBG trimlimits [4:3,8:0] #DBT mos_transform no #DBM do_tran no #DBM out_sec [1:727,1:703] #DBM overlap_sec [475:252,455:255] COM_000 mosn10_024k[258:513,258:513] 5 253 9 256 245 220 0.00 0.00 0.00 | COM_001 mosn10_024k[1:256,1:256] 5 253 9 256 21 4 0.44 -0.31 0.00 | COM_002 mosn10_024k[258:513,1:256] 5 253 9 256 241 0 0.32 0.22 0.00 | COM_003 mosn10_024k[515:770,1:256] 5 253 9 256 462 0 -0.39 -0.30 0.00 | COM_004 mosn10_024k[1:256,258:513] 5 253 9 256 25 224 -0.47 0.17 0.00 | COM_005 mosn10_024k[258:513,258:513] 5 253 9 256 245 220 0.00 0.00 0.00 | COM_006 mosn10_024k[515:770,258:513] 5 253 9 256 466 220 -0.42 -0.48 0.00 | COM_007 mosn10_024k[1:256,515:770] 5 253 9 256 28 445 -0.08 -0.19 0.00 | COM_008 mosn10_024k[258:513,515:770] 5 253 9 256 248 442 0.26 -0.13 0.00 | COM_009 mosn10_024k[515:770,515:770] 5 253 9 256 469 439 0.01 0.24 0.00 | COM_010 mosn10_024k[1:256,772:1027] 5 253 9 256 0 4 -0.34 0.11 0.00 | COM_011 mosn10_024k[258:513,772:1027] 5 253 9 256 220 1 -0.28 -0.23 0.00 | COM_012 mosn10_024k[515:770,772:1027] 5 253 9 256 439 0 0.47 0.23 0.00 | COM_013 mosn10_024k[1:256,1029:1284] 5 253 9 256 2 225 0.41 -0.39 0.00 | COM_014 mosn10_024k[258:513,1029:1284] 5 253 9 256 223 220 -0.03 0.48 0.00 | COM_015 mosn10_024k[515:770,1029:1284] 5 253 9 256 443 220 0.24 -0.07 0.00 | COM_016 mosn10_024k[1:256,1286:1541] 5 253 9 256 6 445 0.11 0.21 0.00 | COM_017 mosn10_024k[258:513,1286:1541] 5 253 9 256 226 442 0.05 0.25 0.00 | COM_018 mosn10_024k[515:770,1286:1541] 5 253 9 256 447 440 -0.15 -0.42 0.00 | Each image has a "COM-line", which is arranged as follows: Col 1: COM path ID (unless renumbered, the _XXX corresponds to the original path position in the 'usqmos' database. 'COM_000' marks the reference image. Col 2: actual name of the subraster, including image section, giving the location where this piece of the dataset can be found Col 3 - Col 6: indicate xmin, xmax, ymin, ymax - the subsection of the image subrater which will be included in the final image. The cumulative effects of the 'trimlimits' parameters are reflected here. Col 7 - Col 8: The x and y coordinate of the integer location of the subraster in the final image. The IMSHIFTED subraster will be offset to this integer location in the final image. Col 9 - Col 10: The x and y coordinate of the fraction location of the subraster in the final image. THe subraster will be IMSHIFTED by this amount prior to image combination. Col 11: The intensity offset for the subraster which will be added prior to image combination (not cumulative - applies to image only). Col 12: An end-of-line marker designed to frustrate the system demons which want to suppress any LF at the end of a line in columns 79 or 80 (thereby merging the COM-lines into one line!) Note: Buried deep in the UNIX/IRAF system is a "demon" which silently decides "The line's too long, I must silently fix it!". This "feature" is activated within the '.login' file (and elsewhere) when the environmental variable 'wrap' is set. Lines which will be 80 characters long are optionally given an automatic LF, so that you need not worry about such issues when you are typing. This feature will break COM-lines and raise all sorts of havoc within the SQIID registration database files! ------------------------------------------------------------------------ APPENDIX B. UPSQIID package NB: The UPSQIID package is not an officially released and supported IRAF package; all queries should be directed to KMM at NOAO (merrill@noao.edu; 520-318-8319). B.1 Getting and updating the UPSQIID package Current copies of the package are available from mira via ftp anonymous: The "upsqiid" IRAF package is contained within a tar file called "upsqiid.tar" within the ftp-anonymous area at mira.tuc.noao.edu. You can retrieve it by typing: ftp mira.tuc.noao.edu (aka 140.252.3.85) logging in as "anonymous" using your last name as password cd pub/software/upsqiid type "ls" to see what's there examine the "README" file for time of last update and other useful information binary get upsqiid.tar to get the tar file on your machine bye Restore the package using "tar upsqiid.tar" within your IRAF login directory and follow the "README" directions for installing the package within your system. Since the package consists entirely of IRAF cl procedures, no recompiling is needed. Two sets of four FITS format images, archival flat fields (usqflat[jhkl]fits) and bad pixel masks (usqmask[jhkl].fits) are separately available at the same site, one level up (the "software" level. They are 1MBytes each. Assuming you copy the package somewhere in a directory called "upsqiid", you would put the following definitions in you loginuser.cl file: set usqscripts = "your_pathname/upsqiid/" task upsqiid = usqscripts$upsqiid.cl You also need to edit the upsqiid.cl in the "upsqiid" package to file to point to your path instead of my path: # change upsqdir to point to UPSQIID directory in target system set upsqdir = "you_pathname/upsqiid/" When loaded (by typing 'sqiid') the 'sqiid' task loads the other IRAF packages it needs: # load necessary packages # noao # artdata is loaded to get mkpattern routine used in NIRCOMBINE artdata # imred and irred is loaded to get CENTER routine imred irred # iis is loaded to get frame routine for IRAF2.11EXPORT iis # utilities # proto # tv # local While the "upsqiid" tasks are tailored to facilitate SQIID data reduction, they are suitable for other image data as well. You can determine the date of last modification of "usqiid" tasks by typing: head taskname.cl nl=1 Contact KMM (merrill@noao.edu; 520-318-8319) when you have successfully retrieved the programs and if you have further questions. When updating the sqiid package, remember that you may have to unlearn pre-existing sqiid package parameter sets on your system. B.2 UPSQIID package listing UPSQIID package tasks are shown below (type ?usqiid after the package is loaded): UPSQIID TASKS: cl> ?usqiid usqdark closure imquadfit notchlist xyadopt usqflat expandnim imzero patproc xyget usqmask getcenters linklaps pltstat xylap usqmos getmap locate recombine xytrace usqproof getstar mergecom sqcorr zget usqsky imclip mkframelist sqremap ztrace usqtest imgraph mkmask statelist upsqproc imlinfit mkpathtbl transmat chlist imlinregress movproc where cleanup imparse nircombine which UPSQIID MENU: usqdark - Make an SQIID dark image usqflat - Make an SQIID flatfield image usqmask - Set bad pixels in an image to a selected value usqmos - Make an IRMOSAIC image and initial registration database usqproof - Make tiled image of image subsections from a list for proofing usqsky - Make an SQIID sky image usqtest - Developmental template upsqproc - Process SQIID images: sky subract, flatfield, etc. chlist - Generate list of images at fixed numerical offsets cleanup - Delete SQIID package temporary _T* files and images getmap - Produce GEOMAP between 2 (3) images interactively imclip - Clip an image outside a stated intensity interval imgraph - Produce intensity scatterplot for pixels in two images imlinfit - Pixel by pixel, linear least squares fit to Y image set imquadfit - Pixel by pixel, quadratic least squares fit to Y image set imlinregress - Pixel by pixel, linear least squares fit to XY image set imzero - Bring statistic of images in a list to a common value mergecom - Merge registration databases with common image into one database mkmask - Generate bad-pixel mask images movproc - List-process raw SQIID data using running median fro sky nircombine - Combine registration database into (masked) composite image patproc - Process raw image data generated from selected protocols pltstat - Produce graph of statistic within image subsection of a list recombine - Intensity readjust and re-imcombine of image stacks. sqcorr - Linearize raw image data sqremap - Update GEOMAP using image template from prior GEOMAP xyadopt - Create alternative channel registration database from a channel xyget - Find common stars in images and create registration database zget - Find intensity offsets from overlaps in registration database where - Outline individual pieces within displayed composite image which - At selected pixel, which images entered into composite image SUBROUTINES: closure - Find origin and minimum size for registration database expandnim - Expand range string into a list of numbers getcenters - Get nearest neighbor shifts from CENTER getstar - Locate an object in a list of images imparse - Parse image names into image root and image extension linklaps - Link mosaic CENTER registration for row/col paths locate - Locate image with respect to reference image list mkpathtbl - Convert path position to grid position (ll corner) notchlist - Make list on either side, but not including entry in list statelist - Generate lists of on or off frames for input images transmat - Translate database from old to new format xylap - Create registration database for data grid from CENTER data xytrace - Link mosaic CENTER registration for arbitrary paths ztrace - Trace ZGET intensity match paths to origin INCLUDED FILES: usqflat[jhkl].fits - representative SQIID flatfields from KPNO 2.1m usqmask[jhkl].fits - representative SQIID bad-pixel masks B.3 Reporting problems Program-related problems are most easily addressed if the appropriate info are included with the problem report: copy of command line used to call the task lpar of the task in question ( lpar task_name >> errfile ) copies of the ascii files called by that task (type filenames >> errfile) version of IRAF used (= cl.version) Send description of problem and appropriate collateral information to KMM (merrill@noao.edu; 520-318-8319). Note: unless it did not "compile" properly, the line number given in an IRAF script procedure error meesage just points past the end of the file. ------------------------------------------------------------------------ C. SQIID SPECIFIC ADDENDA C.2 SQIID image naming conventions. The IR systems at KPNO use one of the following naming conventions: 1) FORTH mode - The IR systems which employ the DEC 11/73 instrument computer have image names based on sequentially numbered files on the system disk (typically starting at 051) and saved to tape by in the CAMERA format (optionally converted to FITS format on the Sun computer) as sequentially numbered files (starting at 0001 on each tape). They can be restored to disk on your computer using "mtlocal.rcamera" or "dataio.rfits". The restored files follow the naming convention: "base_name"//XXXX (e.g., rawa0001) where "base_name" is supplied by you and XXXX is the sequential number of the files on tape. Both rfits and rcamera have an integer parameter called "offset" which can be used (use epar to set to value or append "offset = value" at the end of the command line when you invoke "rfits" or "rcamera") to restore the number sequence corresponding to the original file sequence on the FORTH system (and presumably on your log sheets! Remember that each tape resarts at 0001). To avoid confusion, it is recommended that each restoration use a uniqe base_name indentifier for each FORTH disk dump to CAMERA tape (rawa,rawb, rawc, etc.) 2) FIRE mode - The IR systems which employ the Sun computer directly, such as SQIID (from October 1991; prior observations were in FORTH mode), follow the naming convention: "base_name"//"channel_id"//".XX" (e.g., n1_001j.1) where "base_name" was supplied by you, "channel_id" is j,h,k or l, and XX is the sequential numbered until a new "base_name" is declared (typically at each new object). They have been saved to tape in FITS format as seqeuntially numbered files (starting at 0001) and can be restored to disk on your computer using "dataio.rfits". Since the specific file order on the tape is dependent on the details of how the files were saved, we strongly recommend that you restore the files to there original names by setting the "oldirafname" parameter to yes (use epar or append "old+" at the end of the command line when you invoke "rfits"). If you used SQIID before FIRE, then the images are named so that: i) Files come in sets of 4: J, H, K and L respectively. ii) The file names end with a number. Within a set the number increases by one from one image to the next. If you used SQIID with FIRE, the each channel has its own designator as noted above. The following example compares the two conventions (assuming that you followed the advice given you when the data were taken!): FORTH mode FIRE mode rawa0001 n1_001j.1 J image, set #1, object 1 rawa0002 n1_001h.1 H image, set #1, object 1 rawa0003 n1_001k.1 K image, set #1, object 1 rawa0004 n1_001l.1 L image, set #1, object 1 rawa0005 n1_001j.2 J image, set #2, object 1 rawa0006 n1_001h.2 H image, set #2, object 1 rawa0007 n1_001k.2 K image, set #2, object 1 rawa0008 n1_001l.2 L image, set #2, object 1 rawa0009 n1_002j.1 J image, set #1, object 2 rawa0010 n1_002h.1 H image, set #1, object 2 rawa0011 n1_002k.1 K image, set #1, object 2 rawa0012 n1_002l.1 L image, set #1, object 2 C.3 List tools CHLIST and COLORLIST To facilitate the handling of the large numbers of image sets typically obtained during a SQIID observing run, the SQIID software is designed to operate on IRAF "@-list" files (inside IRAF, one instructs the system to read data from the file "list" by prepending "@" to it - "@list" tells the task to read list to get the image names.) The task "chlist" is provided to create list files. (Alternatively the task "nlocal.ranges" can be used.) Generally it is only necessary to create a list file for the J channel images and refer to the other channels using the offset syntax described below to provide names for the other channels. Alternatively, the "colorlist" task can be used to generate lists for the other channels derived from the J band list. Both FORTHmode and FIREmode naming conventions are supported. When using FIREmode images, the "fire_mode" parameter in "colorlist" must be set to "yes"; for FORTHmode images, the parameter must be set to "no". Since a number of tasks call "colorlist", you need to be sure the parameter is set correctly even if you don't explicitly use "colorlist" yourself. To use "chlist" or "colorlist" the image names must end in an integer. The first useful tool is called CHLIST and its parameter list is shown below. It is used to create list of images of a given color. First_image is the name of the first image name to be in the list. Number is the total number of images in the list. Delta is the increment size. This number is added to the previous image name in the list to create the next image name of the list. CHLIST: first_image = "raw/rawa001" First image in sequentially numbered images number = 8 Number of images in output list (delta = 1) As indicated, you can include directory pathnames. Set delta = 1 for FIREmode images and delta = 4 for FORTHmode images. If you were to give IRAF the following command (with delta = 4): chlist raw001 2 > jlist then jlist would read raw001 raw005. If you were to give IRAF the following command: chlist raw003 5 > klist then klist would read raw003 raw007 raw011 raw015 raw019. If you were to give IRAF the following command (with delta = 1): chlist raw/n1_001j.1 3 > jlist then jlist would read raw/n1_001j.1 raw/n1_001j.2 raw/n1_001j.3. Now that you know how to make a list of all images of a given color, you may want to create the list of the corresponding frames for another color. One way to do that is to run CHLIST again, this time with the parameter first_image set to the first image of the new list. Another way of doing that is to use COLORLIST. This task uses the list of J images (it has to be J images) to create the corresponding list in any color. The parameter list is given below: COLORLIST: first_image = "@jlist" Image list color_image = "h" Image color: |j|h|k|l|1|2|3|4 (fire_mode = yes) FIRE mode (root || [jhkl] || .number) (delta = 4) Increment between successive images (section = "") Image subsection [xmin:xmax,ymin:ymax] First_image is the name of the file containing the list of J images. It has to be preceded by the '@' symbol otherwise it will think that you are giving it an image name. Color_image identifies the color for which to create the new list. You can either specifies it by 'j', 'h', 'k' and 'l' or by the number 1 through 4 for j to l respectively. If you want to create a list with and added image subsections then just set the parameter section to the desired value (ex.[20:180,20:180]). Example 1: Imagine that jlist contains the following raw001 raw005. Doing colorlist @jlist h > hlist (with fire_mode = no) will give you a file 'hlist' containing raw002 raw006. Example 2: Imagine that instead you did the following colorlist @jlist h section=[20:100,20:100] > hlist then the file 'hlist' would contain raw002[20:100,20:100] raw006[20:100,20:100] Example 3: Imagine that jlist contains the following raw/n1_001j.1 raw/n1_001j.2. Doing colorlist @jlist h > hlist (with fire_mode = yes) will give you a file 'hlist' containing raw/n1_001h.1 raw/n1_001h.2. CHLIST and COLORLIST have no intrinsic output files, that is why you have to use redirection. If you don't the output is printed on the terminal screen (STDOUT) but no saved. Note that CHLIST and COLORLIST act totally independently of what images you have in your directory. It does not check whether the images exist or not; they just create a list of names. For many SQIID tasks (in particular, the ones which begin with the prefix "sq") the other passband images can be referenced using the ":channel" offset syntax, "@jlist:C" where "C" is one of "j", "h", "k", or "l" and "jlist" contains the J band images list. C.4 SQIID task input image parameter The input image string parameter for SQIID tasks can be the usual "comma delimited command line lists" or "@-file" type lists. One can also use wildcards within image pathnames to operate on all images which match the template (the IRAF "section" command is invoked). For example, assuming the following images exist: raw/n1_001j.1 raw/n1_001h.1 raw/n1_001k.1 raw/n1_001j.2 raw/n1_001h.2 raw/n1_001k.2 and that jlist contains: raw/n1_001j.1 raw/n1_001j.2, and that hlist contains: raw/n1_001h.1 raw/n1_001h.2. Then, "raw/n1_001j.1,raw/n1_001j.2" "@jlist" "raw/n1_001j.*.imh" are equivalent j-image inputs. For many SQIID tasks (in particular, the ones which begin with the prefix "sq": SQDARK, SQSKY, SQFLAT, SQPROC, SQMOS) the other channel images can be invoked using the ":channel" offset syntax, "@jlist:C" where "C" is one of "j", "h", "k", or "l" and "jlist" contains the J channel images list. These tasks call "colorlist", so you need to be sure "fire_mode" parameter is set appropriately. Assuming the same images: "raw/n1_001h.1,raw/n1_001h.2" "@hlist" "@jlist:h" "raw/n1_001h.*.imh" "raw/n1_001j.*.imh:h" are equivalent h-image inputs. Note, that since the "sections" task is employed to expand the list, when using image names with wildcards and ":channel" offsets (as in the last line of the last example above), the j channel images must actually for the indirection to work. The SQDARK, SQSKY, SQMOS tasks can reference multiple passband images can be referenced using the ":channel" offset syntax, "@jlist:ABCD" where "ABCD"can be any of "j", "h", "k", or "l" and "jlist" contains the J band images list.When more then one channel is present, the tasks run through each list in turn, generating a separate image (with the channel_letter appended) for each channel requested. Thus: sqdark "@jlist:jhk" dark is equivalent to sqdark "@jlist:j" darkj sqdark "@jlist:h" darkh sqdark "@jlist:k" darkk. Within the XYGET/XYADOPT tasks, the input to images should be the name of the SQMOS mosaic rather than the list. (Although an alternative path exists for submiting image lists rather than the name of the mosaic, they are not fully debugged. If you submit the list, you need to give the full name of the SQMOS mosaic database rather than "default", because default says "append .dbmos to the image name". ------------------------------------------------------------------------ /mmerrill@noao.edu 16 February 2001/