The conditonData User Command (Meta Macro)

All user commands have the form:

ldasJob { -name {} -password {} -email {} } { userCmd -opt1 {} ... }

Which is in the format of a Tcl command, ldasJob, with two required arguments:

1. A Tcl list of user information consisting of username, password, and e-mail address. All fields must be filled or the command will be rejected.
   Client software which would like to receive responses from the LDAS system on a port instead of through email can provide a hostname:port combination as the email argument. A further feature of this method is that the hostname can be replaced with the magic key !host!, and the LDAS system will use the IP address it finds by doing a peername lookup on the connected socket.
2. A user command in the form of a Tcl list, for which there exists a "meta" macro file which can be expanded into Tcl code which can then be evaluated by the LDAS system.
   The user command consists of the command name, conditionData, followed by a series of options and their values. To determine which options are required for a given process pipeline you will need to study the descriptions of the various options carefully.
   Some option fields will accept a "null" argument consisting of a matching pair of braces "{}" with no interposed space.
   Meta macros consist of a prototype declaration of the arguments for the given user command, and a template describing the calling order of API specific blocks of Tcl code which are concatenated into a larger block comprising the complete request, which can then be distributed by the assistant manager for interpretation by the low level API's.
   These API specific blocks are maintained as API specific macro files consisting of immediately interpretable Tcl code.

• conditionData

• conditionData

  Acquires input data from the frame API, metadata API, disk files and URL's and performs mathematical transformations upon it using the datacond API. The results of these transforms can be output to disk or to URL's, or sent to the metadata API for ingestion into the database.
  Data extracted from the frame API is concatenated as possible by default. This default behaviour can be modified by using the -concatenate option with an argument of 0.
  Data products can be sent to other API's by using the output() action of the -algorithms option.
  Data objects recieved by other API's can often be inspected by setting the ::DEBUG level in the appropriate API('s) to 219 (0xdb), the data will then be placed in the job result area of the LDAS installation.

  Calling convention (all on a single line):

  ldasJob  {  -name "username"  -password "********"  -email "user@foobar.edu" }  { conditionData  -inputprotocol   {Tcl list}  -returnprotocol   URL  -outputformat   data_format  -datacondtarget   {API name}  -framequery   {Tcl list}  -responsefunction   {Tcl list}  -responsefiles   {Tcl list}  -tarball   {http or ftp URL}  -aliases   {Tcl list}  -algorithms   {Tcl list}  -setsingledc   {Boolean}  }

  Option Descriptions:

  -inputprotocol:     http ftp file port

  • Optional
    The argument to the -inputprotocol option conforms to the usual browser conventions for URI's for determining the location of the results of the user request.
    When the URI is of type http or ftp the LDAS system will attempt to retrieve the data referred to by the URI description, and will make a local copy of it in the result directory assigned to the user command. On completion or failure of the user command, the local copy of the input data will be removed. When the URI is of type port, the system will attempt to read ilwd binary data from the referenced port. When the URI is of type file, the data is locally available, and will be read from the local file system if it exists and is readable. The possible formats of the argument are:

  • http://host:port/dir/...

  • ftp://host:port/dir/...

  • file:/path/... (note, only one "/")

  • port:hostname:portnumber

    NOTE: Embedded spaces in the argument to the -inputprotocol option will cause the request to fail.

  -responsefunction:     { Full Path to File }

  Optional: Deprecated in favor of -responsefiles, q.v.
  See -responsefiles option below.

  -responsefiles:     { Full Path to File(s) }
  Optional: When an external ilwd responsefile(s) is/are required, the full path to the file is specified by this option. The syntax allows for some degree of flexibility in the disposition of the files. See below.
  This option is used to specify the location and disposition of files containing data and/or coefficients which are not provided by the data as received from the frame API or which cannot be calculated from the frame derived data by the data-conditioning API or extracted from the database.
  The files referred to by this option can be injected into the data stream for the job either within the data-conditioning API by the use of the "push" option, or can be attached to the output of the data-conditioning API for transmission to the metadata or wrapper API's by use of the "pass" option. Use of the "push" option requires that an additional argument in the form of an "alias" for the data-conditioning API be provided.
  Examples:
  

  -responsefiles { 
                  file:/MayMDC/al.ilwd,push,al 
                  file:/MayMDC/bl.ilwd,push,bl
                  file:/MayMDC/am.ilwd,push,am 
                  file:/MayMDC/bm.ilwd,push,bm
                  file:/MayMDC/ah.ilwd,push,ah 
                  file:/MayMDC/bh.ilwd,push,bh
                  file:/MayMDC/resp.bin,pass
                 }
  

  Here the "push" elements are ilwd data files which are used to populate the values al, bl, am, bm, ah, and bh. The contents of the files can then be referenced in the call chain algorithm by referring to the appropriate variable.
  The "pass" element is passed on to the api pointed to by the -datacondtarget option and is not used in calculations performed by the data conditioning API.

  -datacondtarget:     {API name}

  Default: datacond

  The default API for datacond results.

  When the underscore _ is used as the protocol argument in an output() action, the value of -datacondtarget will be used as the protocol for the output.

  In a conditionData command, results are normally written to disk in plain ilwd format, or returned to the frame API for output as frames. If output actions use the _ default protocol specifier, then their output is written to disk in plain ilwd format or XML, whichever is specified via the -outputformat option, or as specified by the output() action format specifier (see the documentation for the output() action).

  NOTE:
  A straightforward method of getting wrapper formatted ilwd written to disk is to use the dataStandAlone user command.

  Examples of conditionData commands:

  Simple but complete example which reads a simple ilwd text file from the LDAS public file area as input and returns the daq rate and the data array multiplied by the daq gain found in that file.

  This example presents a Tcl script which can be run from any computer with a network connection to submit a user command to the LDAS system:

  #!/bin/sh
  # use -*-Tcl-*- \
  exec tclsh "$0" "$@"
  
  # convenient formatted command definition
  set cmd "ldasJob
     { -name foo -password **** -email foo@foobar.edu }
     {
        conditionData
           -inputprotocol {
              file:/ldas_outgoing/jobs/NORMAL2000/data.ilwd
           }   
           -aliases {
              x=channel_0:data;
              m=channel_0:gain;
              rate=channel_0:rate;
           }
           -algorithms { 
              output(rate,,rate,"daq rate from input");
              # this is a comment
              mrate = mul(m,x);
  	    output(mrate,,final,"final result");
           }
     }"
  
  # strip comments and compact cmd into user command format
  set cmd [ split $cmd "\n" ]
  foreach line $cmd {
     set line [ string trim $line ]
     if { [ regexp {^#} $line ] } {
        continue
     }
     lappend tmp $line
  }   
  set cmd [ join $tmp "\n" ]
  regsub -all -- {[\n\s]+} $cmd { } cmd
  
  # connect, issue command, print reply from manager,
  # and disconnect
  set sid [ socket ldas-dev.ligo.caltech.edu 10001 ]
  puts  $sid $cmd
  flush $sid
  puts [ read $sid ]
  close $sid
  

  Which submits this command to the system:
  • ldasJob  {  -name  "foo"  -password  "****"  -email  "foo@foobar.edu" }  { conditionData  -inputprotocol  file:/ldas_outgoing/jobs/NORMAL2000/data.ilwd  -aliases {  x=channel_0:data; m=channel_0:gain; rate=channel_0:rate } -algorithms {  output(rate,,rate,"daq rate from input"); r = mul(m,x); output(r,,final,"final result"); }  }

    Input:

    <?ilwd?>
    <ilwd name='trivial data file with one channel' size='1'>
       <ilwd name='channel_0' size='3'>
          <int_2u name='rate'>2048</int_2u>
          <int_2u name='gain'>2</int_2u>
          <int_2u name='data' dims='8'>12 12 12 13 13 12 12 11</int_2u>
       </ilwd>
    </ilwd>   
    

    Result:

    <?ilwd?>
    <ilwd size='1'>
       <ilwd size='2'>
          <int_2u name='rate' comment='daq_rate_from_input'>2048</int_2u>
          <int_2u name='data_x_gain' comment='result of multiplying the data array by the gain' dims='8'>24 24 24 26 26 24 24 22</int_2u>
       </ilwd>
    </ilwd>   
    

  -setsingledc:     {Boolean}

  • Optional, defaults to "0" (false)
    If this flag is 0 (false), then each frame formatted output from the data conditioning API will be returned as a seperate frame object. If this flag is 1 (true), then frame formatted output from the data conditioning API is combined into a single frame object.

  Frame File Naming Convention

  When the frame cache consists of a mixed collection of frames in an inconsistent hierarchy of subdirectories the appropriate frame file(s) for fulfilling a given request are determined by parsing the frame file names according to an installation-specific naming convention. The LDAS system provides a default naming convention which is described in detail in the document Naming Convention for Frame Files Which Are to be Processed by LDAS .
  In summary, the required format of frame-file names is

  • S-D-G-T.gwf

  where

  • S indicates the source of the data eg. H for Hanford frames, L for Livingston frames, G for GEO frames, HL for combined Hanford-Livingston data.
  • D is a description of the contents of a file. Some typical values include:
    • R full-rate "raw" data
    • T second-trend data
    • M minute-trend data
    • RDS_R_L1 Level 1 reduced data
  • G is the GPS time (integer sumber of seconds) at the beginning of the first frame in the file. This is either a 9- or 10-digit number.
  • T is the total time interval from G to the time-stamp of the last sample contained in the frame file, rounded up to the nearest whole second.

  Examples:

  • H-R-693960000-16.gwf
    16 seconds of raw data from Hanford.
  • L-T-693960000-60.gwf
    1 minute of second-trend data from Livingston.
  • H-RDS_L1-693960000-16.gwf
    16 seconds of Level 1 reduced data from Hanford.

  Return to Top