Radar compositing

From Promoserv
Jump to: navigation, search


General overview of the radar composite system

In the promoserv project radar composites are generated from Dong Ha and Tam Ky radars at project server promoserv.amo.gov.vn. Data are transferred from AMO to promoserv server via ftp. The radar composite program uses ODIMHdf5 radar data files to make composites. Currently only radar data from Tam Ky and Dong Ha are used in the composite. These radars are not able to produce HDF5 files directly, but they produce UF data, which is a documented data format and can be read using tools, such as RSL library or RADX tools. The radar composite program has been originally developed at the FMI, where it has been in radar composite production for a long time.

The radar composite production comprises of the following main steps:

  • 1. UF of Dong Ha and Tam Ky radars data is transmitted from AMO to promoserv server. Due to the large file size uf files are in a comporessed form.
  • 2. UF data are uncompressed and converted to ODIM HDF5 files using a dedicated UF to ODIM HDF5 converter. The converter uses RSL library for reading the UF files and HDF5 libraries to create ODIM files.
  • 3. The converted HDF5 files are passed to radar the composite program, which produces individual composite images.

Composite process.png

Radar data folders

Incoming radar data folder (/home/ftp/radar-amo/)

  • All incoming radar data files are transferred from AMO to the folder /home/ftp/radar-amo/ at the promoserv server. This folder includes UFZ files (compressed UF files) from Tam Ky and Dong Ha, as well as other radar formats from other radars (e.g. Native EEC radar data files and radar data files from TRS-2730 radars).

UFZ folder (/home/radman/data/ufz/)

  • UFZ files (compressed UF files) are transffered to this folder from incoming radar data folder. Files older than 1 day are removed.

UF (/home/radman/data/uf/)

  • UFZ files located at UFZ folder are uncompressed to UF files and copied to this folder. Files older than 1 day are removed.

HDF5 files folder (/home/radman/data/hdf5/)

  • UF files at UF folder are converted to ODIM HDF5 files using a dedicated data conversion program and stored in this folder. Files older than 1 day are removed.

Composite files folder (/home/radman/data/)

  • HDF5 radar data at HDF5 folder are fed to the composite program, which generates composite images to this folder. Files older than 1 day are moved to archive folders /home/radman/data/2012/ etc.).

Composite software installation

1. Installation of the libraries At first you have install few libraries used by the composite software:

  • yum install hdf5
  • yum install hdf5-devel
  • yum install proj

2. Installation of the actual composite software

  • Copy the http://promoserv.amo.gov.vn/composite/devel/composite-devel-tools.tar.gz -file to your own computer.
  • After the download is finished, you can extract the whole .tar.gz -file into your home-folder.
  • Navigate to the /home/<username>/devel/composite/ -folder and issue command make clean. The command is used to clean old *.o -files.
  • In the same folder, run the make command. There may be some warnings about unused variables, but you can safely ignore them.
  • Final step is to modify testcomp_DONTKY.tcsh -file. For example Vi -editor can be used. The following variables needs to be modified:
    • $DEVELFOLDER should refer to your own home -directory (/home/<username>/devel/)
    • SET INPUTDIR = (source path of the HDF5 -files)
    • SET OUTPUTDIR = (destinaation path of the composite files)

Now the composite software is ready to use.

Composite scripts and program

In the promoserv-server, all the radar composite scripts and codes are located under the folder /home/radman/devel/

1. Script for copying and de-compressing uf-files

The Bash script /home/radman/devel/copyAndUntar.sh is used for copying and de-compressing ufz files from the incoming data folders /home/ftp/radar-amo/to ufz and uf folders /home/radman/data/ufz/ and /home/radman/data/ufz/. Before runnig the script, one has to define the source directory of ufz files using variable SOURCEDIR

Usage, for example: SOURCEDIR=/home/ftp/radar-amo/dongha/ bash copyAndUntar.sh

2. Script for converting de-compressed uf files ODIM HDF5

The Python script /home/radman/devel/uf2OdimConvert/uf2odim.py is used for converting uf files to ODIM HDF5 radar volume files.

Usage: python /home/radman/devel/uf2OdimConvert/uf2odim.py sourceFiles destinationFolder radarIndex

For example: python /home/radman/devel/uf2OdimConvert/uf2odim.py /home/radman/data/uf/DON*.uf /home/radman/data/hdf5/ 0

The radar index is needed to record radar latitude, longitude and altitude information in the HDF5 files. In the current conversion program the following indices are used: 0 (Dong Ha), 1 (Tam Ky), 2 (Nha Trang), 4 (Nha Be).

The Python script loops over source files and calls the actual uf to ODIM HDF5 conversion program /home/radman/devel/uf2OdimConvert/uf2odim for each input file and copies the converted HDF5 files to the output directoru. One can also use the uf2odim-program also for individual files:

/home/radman/devel/uf2OdimConvert/uf2odim sourceFileName targetFileName radarIndex.

As an example: /home/radman/devel/uf2OdimConvert/uf2odim DON20120917170005.uf test.hdf5 0

3. Script for generating the radar composites

The tcsh script /home/radman/devel/composite/testcomp_DONTKY.tcsh is used for generating the composite images.

Usage: Change input and output directories in the script to match the data folders on your computer and run the script using tcsh. For example, tcsh /home/radman/devel/composite/testcomp_DONTKY.tcsh .

The script loops over source HDF5 files and calls the actual composite program polarcomp.c for all HDF5 with the same timestamp with the accuracy of 10 minutes.

Personal tools