Data Reduction Workshop

When on the University machines (or connected to the University VPN):

1. When starting the computer, chose to open a Linux environment (rather than windows). -OR- if on a Windows only computer search in the AppHub of "X2Go-Mira" and launch it to get a Linux environment, with a command line and terminal etc.
2. In the command line type
   xhost +
   This will allow X forwarding.
3. ssh into stargate and then zwicky:
   ssh -XC username@stargate
   then
   ssh -XC username@zwicky
   For some reason, some of you may need to use -YC rather than -XC to get the forwarding to work. If on a MAC, make sure you have xquartz installed.
4. Switch to a cshell terminal (this is needed for IRAF to work properly) by typing
   csh
   (or tcsh if you prefer).
5. Load the software modules that you need (wcstools, ds9, gaia, iraf):
   module load wcstools ds9 starlink iraf
   (NB: starlink contains gaia).
   You will need to do this every time you log in, but only load the modules that you need. If you later need additional modules use module load [name]. To see available modules/software type module available
6. Create a working directory for the data reduction course. You should have enough space for this in your home directory.
7. Download these two tarballs:
   (1) www.astro.dur.ac.uk/~simo/pg_dr_data/pg_dr_phot.tar.gz
   (2) www.astro.dur.ac.uk/~simo/pg_dr_data/pg_dr_spec.tar.gz.
   The easiest way to do this is to type wget [URL] when you are in the working directory.
8. Unpack the tarballs by typing
   tar -zxvf pg_dr_spec.tar.gz
   and
   tar -zxvf pg_dr_phot.tar.gz
   
9. Try opening some of the fits files using
   ds9 file.fits
   and
   gaia file.fits
   If nothing displays, you may need to log-in again using -YC (see point 3 above). You need to be able to run at least gaia for the workshop. If you have an error message in regards to the GAIA_DIR directory then you need to add this text to your .cshrc file:
   source /usr/local/etc/DOT_CSHRC
   (to open you .cshrc file type: vim ~/.cshrc or emacs ~/.cshrc). Once you have added the text, type:
   source ~/.cshrc
   Hopefully gaia will now work.
10. Try running a wcstools command (see below) on a fits file. e.g. imhead file.fits. If this doesn't work, add the following line to your .cshrc file:
    setenv PATH /obs/perm/ams/bin/wcstools/wcstools/bin/:{$PATH}
    open this file by typing: vim ~/.cshrc and you will need to "compile" it after editing by typing:
    source ~/.cshrc
    You need to be able to run wcstools for the workshop.
11. Make a directory in your home space called iraf:
    mkdir iraf
12. When in your iraf directory type:
    mkiraf
    and follow the instructions (choose/type xgterm when asked).
13. You now need to set-up IRAF (see below). This is already installed but needs setting up on your user environment. Add the following lines to you .cshrc file:
    (1)alias iraf '\cp /home/USERNAME/iraf/login.cl ./ ; xgterm -fg black -bg white -fn fixed -sb -e cl -title IRAF '
    (2) set path = ($path /iraf/local/bin)
    Make sure you put your username in the appropriate place and don't forget to compile(source) the .cshrc file after editing it.
14. In your home directory type source .cshrc
15. Now type
    iraf
    and hopefully a new cl window will open. In this window, test everything appears to be working by typing
    epar imarith
    and then :q to get out of that. If you get no error messages we should be ready to go!
16. Remember when in the cl window, use log to quit and exit.

Hopefully by now you are already familiar with the fits file format. This format allows one file to have multiple extensions (or "layers") of data. It is extremely useful for large tables of data and astronomical images or datacubes. Fits files also have "headers" that (can) contain extensive information about the file. For example for astronomical images the header is likely to contain information on the target, the instrumental set-up, the observing conditions, the pixel scale of the image etc. etc. This information can be absolutely vital for reducing data.

wcstools is a set of tasks that can be run from the command line for displaying and manipulating the headers of fits files. For example:
imhead filename.fits
This will display the header in the terminal. Sometimes you need to specify which extension you wish to display the header for: e.g., imhead filename.fits"[0]". You sometimes will want to display the value of a particular keyword in a header for one or several files. You can do this using gethead. For example:
gethead OBJECT *.fits
arc.fits [0] CuAr
std.fits [0] BD+28 4211
flat.fits [0] GCALflat

You will be using IRAF to do some basic data reduction during the workshops. Although IRAF is some-what out dated for most purposes, it can still be extremely powerful for performing basic data-reduction tasks and is (surprisingly!) still widely used by many astronomers (the Gemini Observatory still use this as their basis for data reduction!). The main reason we will be using IRAF is that you can do each step of the data reduction "by hand" so you can learn some of the fundamental steps. In reality you will all be using different datasets from different telescopes. Most telescopes now have automated pipelines; however, these sometimes can break, give odd results/mistakes or do not meet the level of accuracy required for some products. It is therefore useful to be able understand the basic steps and have the ability to "hack" or improve automated pipelines.

IRAF consists of several packages of tasks. For example, the images.imutil package contains the imarith task for basic image arithmetic. You can navigate to this package by typing:
images
followed by
imutil
If you want to move to the previous level type:
bye
To find out what a task does and to get help on running the task type:
help taskname
Each IRAF task has a set of parameters that need to be chosen carefully. You can see and edit the parameters of a task by typing:
epar taskname
Once you have edited the parameters you can run the tak by typing :go. If you want to leave the epar screen type :q. Tasks can also be run directly from the IRAF terminal, for example:
imarith image1.fits * image2.fits new_image.fits
This runs the imarith task and multiplies image1 by image2 to make a new image called new_image.

If you are ever in interactive mode (i.e., have a plot displayed), you can use a variety of key strokes and typed commands. You can find out what these are by typing ? when the plot is displayed. The list of instructions will display in the IRAF terminal. To get out of this type q

Some annoying quirks and get-arounds for IRAF:

• IRAF does not behave well when trying to overwrite an existing file of the same name. Therefore, it is usually best to delete/remove all files that you want IRAF to overwrite before running the task.
• When in interactive mode (when you have plots open and are interactively clicking on them) IRAF crashes if you try to close the windows by clicking the x button. DO NOT DO THIS. You will need to press q to leave interactive mode first (in most cases).
• When things are going wrong it can sometimes be useful (or required) to reset to a task's default parameters. This can be done by typing unlearn taskname. In drastic situations, this sometimes doesn't work OR tasks use sub-tasks that also need to be "unlearned". In your IRAF directory there is a sub-directory called uparm. Inside this directory are a bunch of files that store chosen parameters. In times of crisis I have deleted all of these files, but be warned, this will forget all of your previous choices of parameter values for all IRAF tasks.
• From experience I have found that IRAF doesn't cope well if you simultaneously run multiple tasks inside the same working directory. I believe this is because IRAF sometimes writes out tempory files that get mixed up if multiple tasks are running.

Some documentation:

• Several useful IRAF guides: http://iraf.net/irafdocs/. You can also find guides to the IRAF packages and tasks on this website.
• A Beginner's Guide to Using IRAF: https://iraf-community.github.io/doc/beguide.pdf
• Running IRAF: https://vitaly.neustroev.net/teaching/docs/irafmanual.pdf
• Photometry in IRAF: http://www.physics.hmc.edu/Astronomy/Ireduce.html

If so, head over to the imaging and photometry tutorial and spectroscopy tutorial after that.