Help Page Target Processor

The Target Processor can be found at:

• Base version
• Experimental version

The following help is available;

• 1. Overview
• 2. Preselect Target
• 3. Specify Target
• 4. Select Target
• 5a. Process
• 5b. Query
• 6. Preferences
• 7. Process Parameters
• 8. Upload Code
• 9. Job Overview

1. Overview

Below is a screenshot with the starting page of the targe processor. The target processor can be used to process data and to give an overview of what has been and has not been processed. First choose a project and instrument, not supported projects and instruments are greyed out. After selecting a project and instrument click on the preselect target link, depending on the instrument different targets can be shown. After selecting a target the target processor will step by step guide you to selecting your targets and submitting these to a cluster. Below the image of the start screen are the different links explained. The left part of the screen is always visible, the rest of the screenshots of these help pages will not show this part.

• (Contact) My email adress, mail me (Willem-Jan Vriend) with any question or comments you have about the Target Processor.
• (DB User) The (astro-wise) username that is used for interaction with the database. When you visit this page for the first time this will be the anonymous user awanonymous. You can click on it to log in with your own astro-wise username / password. In the login screen you can specify to 'remember' your username and password for future sessions. Then your username and password will encrypted and stored in a cookie.
• (Help) The help pages, will open the current page.
• (Project) The available projects for the current user.
• (Instrument) The available instrument(s) for the chosen project.
• (Preferences) This will open the preferences screen.
• (Process Parameters) Here you can view and edit the process parameters of the selected target.
• (Upload Code) With the Upload Code link you can upload your own code tree. Check the box to use the uploaded code. See the howto how to generate a zipped codefile.
• (Job overview) This link will open a window with the queued, running and finished jobs of all the dpu's. Every job can be canceled and for finished jobs the logging can be retrieved.

2. Preselect Target

This screen gives the possible targets for processing for the given instrument. The example screen below gives the possible targets for the WFI instrument, with on the left the image pipeline products and on the right the calibration.

3. Specify Target

This screen gives an overview of the possible targets grouped per night and filter or Observing Block. Specify a period and optinal a filter and click the Show button. Then click on one of the blocks to see the targets. The example screen shows the groups of targets for november 2001. Only targets with a DATE_OBS have this selection screen.

• To specify the target with manually selected input parameters click the manually link. In the preferences can be selected which screen is shown by default, this or the manually input screen.
• For some targets it is possible to specify the input. For CoaddedRegriddedFrame's it is for example possible to specify whether RegriddedFrame's or RawScienceFrame's should be as input.
• The numbers (x/y) in the legenda below the graph specify how many targets can be processed (x) and how many are already processed (y).

4. Select Target

This screen gives the possible targets which were selected in the previous screen. The user can check the targets and request these to be processed or queried. When querying it is possible to specify the query depth. When a target uses a list of objects as input the user can specify which to use. For example; for a BiasFrame the user can specify which RawBiasFrame's to use.

• Targets with a ccd attribute are plotted on one line for all ccd's. The ccd name links to a viewable version of the frame.
• Most targets have the processed column. This column shows the processed state of the target(s), hover over the value to see when the targets were processed. When a targets is processed the ccd will be printed in bold.
• When the process button is clicked the Submit screen will be shown.
• When the query button is clicked the target will be recursivly queried with the given query depth. The result will be shown as a tree strcture, giving the target and its dependencies. For larger depths this can take some time.

5a. Process

The submit screen, giving an overview of what is going to be submitted to the dpu. The user can change the following settings :

• The dpu for single and parallel jobs can be chosen. For the experimental target processor also the test dpu(s) will be available.
• The processing depth can be changed.
• Multiple jobs can be grouped in different ways on the available processing nodes. When grouped per chip all jobs for the same chip will be run on the same node. When maximized the jobs will run independent on a maximum number of nodes. Or specify the number of nodes self.
• Force the short queue option will always queue the jobs in the short queue, independent of the expected job time.
• For CoaddedRegriddedFrame's the Global Astrometry (GAS) can be turned on and off. The GAS can also chosen to be processed on the single or parallel dpu.
• With the Submit button the jobs will be send to the dpu together with a zip file with all the python code. See the Upload Code section howto provide your own zipped code file.

5b. Query

This screen shows the result of a query for a RegriddedFrame that has not been processed. See the text on screen capture for more information.

6. Preferences

The preferences screen can be opened any time and gives the following options :

• The query and process depth determine how deep the querying or processing will be done. A depth of 1 means that only the target will be processed. A depth of 2 will also query or process the dependencies of the target. In determining only calibration objects are taken into account. So Coadded-, Regridded-, Reduced- and RawScienceFrames all count as the same depth.
• Two dpu's can be chosen. One for the parallel jobs and one for the single jobs. The parallel dpu's are clusters of processing nodes, meant for (embarassingly) parallel processing. The single dpu's are single node machines optimized for a specific task, for example Coadding.
• Show process options will show a fold-out menu with options in the Query view for all dependencies.
• Show popup info will show a popup window in the Query view for all targets in the tree with information of the target.
• When set 'Start with graphical input' will show the graphical screen when specifying the target, otherwise a manually input screen will be shown.
• The set privileges determine which data will be shown and what privileges will be given to new created objects. All data with privileges equal to or higher then the set privileges will be shown. The default privileges are 1 and the maximum privileges are set by the project.

7. Process Parameters

For the selected target the process parameters will be shown and can be set. In the python classes these are known as process_parameters.

• Not all process parameters can be changed; lists and class hardcoded parameters. These are shown with a grey background.
• The recursion depth, how deep the dependencies are shown, depends on the query depth.
• To globally set a process parameter, check the option 'Change parameters globally' on top of the screen.
• The ReducedScienceFrame dependency of RegriddedFrame is not shown, because this dependency is the same as the ReducedScienceFrame dependency of AstrometricParameter.

8. Upload Code

Every time a job is submitted to a dpu all the python code is shipped also in zipped file. Normally the code from cvs is used but you can also use your own code.

9. Job Overview