The getData User Commands (Meta Macros)

All user commands have the form:

ldasJob { -name {} -password {} -email {} } { userCmd -opt1 {} ... }

Which is in the format of a Tcl command, ldasJob, with two required arguments:

1. A Tcl list of user information consisting of username, password, and e-mail address. All fields must be filled or the command will be rejected.
   Client software which would like to receive responses from the LDAS system on a port instead of through email can provide a hostname:port combination as the email argument. A further feature of this method is that the hostname can be replaced with the magic key !host!, and the LDAS system will use the IP address it finds by doing a peername lookup on the connected socket.
2. A user command in the form of a Tcl list, for which there exists a "meta" macro file which can be expanded into Tcl code which can then be evaluated by the LDAS system.
   An argument need not be provided for every option. Some options have default values and these are described below as appropriate.
   Meta macros consist of a prototype declaration of the arguments for the given user command, and a template describing the calling order of API specific blocks of Tcl code which are concatenated into a larger block comprising the complete request, which can then be distributed by the assistant manager for interpretation by the low level API's.
   These API specific blocks are maintained as API specific macro files consisting of immediately interpretable Tcl code.

• getFrameData
• getMetaData

• getFrameData

  Returns full or reduced raw frame data in several formats.

  Calling convention (all on a single line):

  ldasJob   { -name   "username"   -password   "********"   -email   "user@foobar.edu"}  {getFrameData  -returnprotocol  URL   -outputformat  {Tcl list}  -frametarget  name_of_an_api  -framequery  {Tcl list} } -compressiontype   supported_compression_mode   -compressionlevel   integer   }

  Option Descriptions:

  -outputformat:     frame | {ilwd ascii} | {ilwd binary} | LIGO_LW

  Default: "{ilwd binary}"
  The argument of the -outputformat option is the data type to use in formatting the result of the user request.
  The possible output formats which the system can produce, which include binary frames, ilwd ascii, ilwd binary and LIGO_LW (an XML format) can be specified, i.e. {ilwd ascii}.

  -frametarget:     {}

  Optional: defaults to a null string
  The argument of the -target option is the name of a known LDAS API. The result output from the frame API will be registered with the named API to be used as input data for the code with the same job i.d. that the data is tagged with.
  NOTES:
  1. The target is relative to the frame API for frame commands.

  -compressiontype:     {raw, gzip, gzip_diff, zero_suppress_short, zero_suppress_int_float, or zero_suppress_otherwise_gzip}

  Optional: defaults to 'gzip'
  

  Compression Name	Description
  raw	The data is not compressed.
  gzip	The data is compressed using the gzip algorithm
  diff_gzip	The data is differentiated and then gzipped
  zero_suppress_short	Data which is composed of 2 byte integers is differentiated and then zero suppressed. All other data types are left uncompressed. This mode does not imply zero_suppress_int_float.
  zero_suppress_int_float	For 4 byte integers or float data, the data is differentiated and then zero suppressed. If the data is 2 byte integers, the compression mode is modified to be zero_suppress_short. All other data types are left uncompressed.
  zero_suppress_otherwise_gzip	All data types that support zero suppress compression are compressed using that method. All other data types are compressed using gzip.

  -compressionlevel:     {}

  Optional: defaults to '1'
  This option allows the user to specify the level of compression for compression types that support multiple levels. Gzip and gzip_diff support comprssion levels 1-6, "raw" does not support any levels. If a compression type does not support multiple levels, this option is quietly ignored.

  getMetaData

  ---

• getMetaData

  Returns meta data stored in the LDAS database in response to the query given by the user command; returns in several formats either by attaching the resulting files to an e-mail (-returnprotocol email) or by returning an e-mail with a URL pointing to the location of the results (-returnprotocol http or ftp).

  ldasJob   { -name   "username"   -password   "********"   -email   "user@foobar.edu"}   {getMetaData   -returnprotocol   URL   -outputformat   data_format   -sqlquery   {Tcl list}   -database   dsname   -metadatatarget   api   }

  Option Descriptions:

  -returnprotocol:     http ftp file port

  Default: http://some_sensible_name.ext
  The argument to -returnprotocol resembles the usual browser conventions for URL's, but is actually only a hinting mechanism for the user to get a URL back in e-mail or in a client.
  The mailto and port options are currently unsupported.

  The http, ftp, and file options cause a URL pointer to the location of the results to be returned by e-mail. Any directory structure in the location will be ignored, the filename and extension will be used unless they are found to break system conventions for mime typing.
  The possible formats of the argument are:

  • http://filename.ext

  • ftp://filename.ext

  • file:/filename.ext

    NOTES:

    1. Embedded spaces in the argument to the -returnprotocol option will cause the request to fail.

  -outputformat:     {ilwd ascii} | {ilwd binary} | LIGO_LW

  The argument of the -outputformat option is the name of the data type to use in formatting the result of the user request.
  The system supports ilwd ( defaults to binary ), ilwd ascii and LIGO_LW (an XML format) for the result rows.

  -sqlquery:     {any valid SQL statement}

  The argument of -sqlquery option is any valid sql statement supported by the LDAS database server ( IBM DB2 ). If the query turns out to be invalid, a database error message is returned.

  -database:     dsname

  The argument of -database option is a valid database dataset name for the site that the command applies e.g. cit_1 and cit_test are dataset names for site ldas-dev, lho_1 and lho_test are dataset names for site ldas-wa etc. For dsnames of sites, please visit the LDAS database link on each ldas site web page. If this option is not specified, retrieval is done from the default database.

  -metadatatarget:     api

  The argument of -metadatatarget option is a valid LDAS API name, usually metadata in which case the metadataAPI directs the output query results to file or to the ligolwAPI depending on returnformat specifications. This option need not be specified in a regular getMetaData command as the default is -metadatatarget metadata. When getMetaData is used as part of the dataPipeline command, setting -metadatatarget option to datacond and -outputformat to {ilwd binary} directs metadataAPI to send query results to the datacondAPI.

  Examples of getMetaData command:

  • for LDAS registered user foo to get all data from table sngl_burst in LIGO_LW (xml) format and have results delivered via ftp, submit:
    ldasJob   { -name   "foo"   -password   "********"   -email   "foo@foobar.edu"}   {getMetaData   -returnprotocol   "ftp://www.ldas.ligo-wa.caltech.edu/pub/incoming/getMetaData1.xml"   -outputformat   LIGO_LW   -sqlquery   {select * from frameset fetch first 100 rows only} }

    Result in LIGO_LW

  • for LDAS registered user foo to get the number of rows in gds_triggers table, with output in in LIGO_LW (xml) format and have results in http URL link format, submit:
    ldasJob   { -name   "foo"   -password   "********"   -email   "foo@foobar.edu"}   {getMetaData   -returnprotocol   http://foo.xml   -outputformat   LIGO_LW   -sqlquery   {select count(*) from gds_triggers}   }

    Result in LIGO_LW

    Note: ldas does not do the actual http put since this is not permitted. But the users can use the result links to get the files themselves.

  • for LDAS registered user foo to get the maximum mass2 in sngl_inspiral table, with output in in LIGO_LW format and have job results notification delivered by email to a specified external port on a specified host, submit:
    ldasJob   { -name   "foo"   -password   "********"   -email   "ldas-dev:63419"}   {getMetaData   -returnprotocol   "file:maxMass2.xml"   -outputformat   LIGO_LW   -sqlquery   {select max(mass2) from sngl_inspiral}   }

    Result in LIGO_LW

  • This example shows how to use database built-in functions:

    for LDAS registered user foo to get a unique Id (guranteed unique) from the database in ilwd format. Submit:

    ldasJob   { -name   "foo"   -password   "********"   -email   "foo@foobar.edu"}   {getMetaData   -returnprotocol   "foo@foobar.edu"   -outputformat   LIGO_LW   -sqlquery   {values(generate_unique())}   }

    Result in LIGO_LW

  • for LDAS registered user foo to get primary keys from segment in LIGO_LW format, submit:
    ldasJob   { -name   "foo"   -password   "********"   -email   "foo@foobar.edu"}   {getMetaData   -returnprotocol   "foo@foobar.edu"   -outputformat   LIGO_LW   -sqlquery   {select t.tabname, k.colname from syscat.tabconst t,syscat.keycoluse k where t.tabname = k.tabname and t.constname = k.constname and t.tabname = 'SEGMENT' and t.type = 'P' fetch first 10 rows only}   }

    Result in LIGO_LW

  • This an example of how to use the database option to specify a database for the command instead of retrieving from the default database

    for LDAS registered user foo to count the number of gds triggers for the DMT process glitchMon in LIGO_LW format, submit:

    ldasJob   { -name   "foo"   -password   "********"   -email   "foo@foobar.edu"}   {getMetaData   -returnprotocol   "foo@foobar.edu"   -outputformat   LIGO_LW   -sqlquery   {select count(*) from gds_trigger where process_id in (select process_id from process where program='glitchMon' }   -database   cit_1   } 

    Result in LIGO_LW

  • This an example of a complex query that could take a while to run if there are lot of rows.

    for LDAS registered user foo to further select the list of snrs selected from a subquery in LIGO_LW format, submit:

    ldasJob   { -name   "foo"   -password   "********"   -email   "foo@foobar.edu"}   {getMetaData   -returnprotocol   "file:getMeta1"   -outputformat   LIGO_LW   -sqlquery   {WITH ZZZ AS (select snr from sngl_inspiral where mass1 <= +9.89866E+008 ) select * from ZZZ where snr = (select max(snr) from ZZZ)}   }

    Result in LIGO_LW

  Examples of getMetaData command email response:

  For -returnprotocol fileURL:
  
  Message from LDAS:
  
  Your results:
  getMeta1.xml
  can be found at:
  /ldas_outgoing/jobs/NORMAL10466
  
  
  =========================================
  'LDAS API'          'CLOCK TIME(seconds)'
  =========================================
  managerAPI(queue):                0.00
  metadataAPI:                      6.48
  ligolwAPI:                        5.63
  -----------------------------------------
  managerAPI(total):               12.10
  =========================================
  
  
  
   Errors:
  
  Message from LDAS:
  
  metadataAPI:NORMAL10509: submit select * from process fetch 1 row for read only 1: Start of Error Messages
           File: ../../../../../api/metadataAPI/so/src/query.cc Line: 196
  SQLSTATE: 42601
  Native ErrorCode: -104
  [IBM][CLI Driver][DB2/SUN] SQL0104N  An unexpected token "1 row" was found following "* from process fetch" .  Expected tokens may include:  "" .  SQLSTATE=42601
  
  
  
  
  
  Return to Top