|
|
ldasJob { -name {} -password {} -email {} } { userCmd -opt1 {} ... }
Which is in the format of a Tcl command, ldasJob, with two required arguments:
ldasJob {-name mlei -password md5protocol -email 131.215.115.126:52124} {dataPipeline ... }
Then the manager returns a string consisting of the word md5salt and the md5salt that the client should use when creating the hash:
md5salt [integer]
(** Right, it's not really a salt in the normal sense, it's a
session key. The name is an artefact of an earlier implementation.
Sorry for any confusion.)
The client then appends the salt value to the user's password and calculates an md5 hash of the combined password/salt string and returns this value to the manager like this:
md5digest af2058b1b115a2aec77e76e07b85d031
And the manager validates the request.
Acquires input data from the frame API, metadata API, disk files
and URL's and performs mathematical transformations upon it using
the datacond API. The primary output sink for the datacond API
is the wrapper API, where mpi processing occurs. Secondary datacond
API results can be output to disk or to URL's in LIGO lightweight
data format, ilwd format, or frame format, or can be sent to the
metadata API for ingestion into the database.
Results of mpi processing are sent to the eventmon API, where they
are formatted for ingestion into the database via the metadata API.
Data extracted from the frame API is concatenated as possible by
default. This default behaviour can be modified by using the
-concatenate option with an argument of 0.
Data products can be sent to other API's by using the
output() action of the
-algorithms option.
Data objects recieved by other API's can often be
inspected by setting the ::DEBUG level in the appropriate
API('s) to 219 (0xdb), the data will then be placed in the
job result area of the LDAS installation.
Calling convention (all on a single line):
ldasJob { -name "username" -password "********" -email "user@foobar.edu"} {dataPipeline -inputfile URL -inputprotocol {Tcl list} -returnprotocol URL -outputformat data_format -usertype string -outputdir output_directory -usejobdirs use per-job subdirectories -subject {freeform string} -ingestdata {Tcl list} -multidimdatatarget {API name} -mpiapi {HOST,PORT} -state {Tcl list} -multidimoutput {Tcl list} -responsefunction {Tcl list} -responsefiles {Tcl list} -tarball {http or ftp URL} -autoexpand boolean -aliases {Tcl list} -algorithms {Tcl list} -framequery {String} -datacondtarget {API name} -metadatatarget {API name} -dataapi {HOST,PORT} -dbquery {Tcl list} -dbntuple {Tcl list} -dbspectrum {Tcl list} -dbqualitychannel {Tcl list} -doloadbalance {Tcl list} -concatenate {Boolean} -allowgaps {Boolean} -np {Integer} -dynlib {Full path to .so} -realtimeratio {Float} -memoryusagelimit {Float} -filterparams {Freeform String} -usertag {Freeform String} -datadistributor {WRAPPER | SEARCHMASTER} -communicateoutput {ONCE | ALWAYS} -metadataapi {API name} -resultapi {HOST,PORT} -setsingledc {Boolean} -setsingleem {Boolean} -database {String} -ligolwformat data_format }
Options Description:
-usertype: string
Allows the user to specify a type to be used in constructing
the output frame names. The specified type will be supplemented with
an underscore and integer value when more than one frame is output
by a single user command, consistent with the default naming
convention.
For example, specifying -usertype foobar will result in an
output type of foobar when only one frame file is written.
However, specifying -usertype foobar when more than one
frame is to be written will result in type specifications of the
form foobar_1, foobar_2 and so forth for each
succeeding frame file. The naming convention will be consistent
with the convention used when the option is not specified.
A request that would result in the overwriting of existing data will
raise an exception and no output will result.
Default: NONE, use default RDS output type.
Note: Specifying a unique type will help ensure that LDAS finds
the intended frames in a -framequery option of an LDAS job.
This option is very useful when creating merged RDS frames (i.e. frames
containing multiple IFO data). It will help LDAS find the correct
multi-IFO RDS frames and not other single-IFO RDS frames which may
otherwise be named similarly.
-outputdir: output_directory
This option applies when the user desires
result frames to be written to a special directory provided
for the purpose of collecting reduced data sets.
If the directory does not exist, the job fails due to
frameAPI reporting error about non-existent directory.
Default: job directory.
-usejobdirs: 0 or 1
When this flag is set to 1, there will be a new
subdirectory created under the outputdir for each new job
when the -outputdir option is also specified. When this flag
is set to '0', the files will be output directly into the
directory specified by -outputdir.
Default: 1
-np: { Integer }
-dynlib: { Filename }
Developers of dso's take note:
When pushing copies of development shared objects into the NFS
mounted directory hierarchy visible to the wrapper API, be sure
to name subsequent versions of the shared object with new, unique
names (for example, use a patchlevel number when naming the file),
otherwise there is a chance that a memory cached copy of an
older version of your dso will be used.
-mpiapi: {HOST,PORT}
-dataapi: {HOST,PORT}
-resultapi: {HOST,PORT}
-setsingledc: {Boolean}
-setsingleem: {Boolean}
-database: {String}
-ligolwformat: LIGO_LW | LIGO_LW base64
-filterparams: {Freeform cslist}
-usertag: { Freeform string }
-realtimeratio: { Float }
-memoryusagelimit: { Float }
-doloadbalance: { TRUE | FALSE }
-datadistributor: { WRAPPER, SEARCHMASTER }
This option is currently disabled.
-communicateoutput: { ONCE | ALWAYS }
-frametarget: datacond
-inputprotocol: { filenames and/or URLs }
-subject: {subject for the return e-mail}
-autoexpand: Boolean [01]
-datacondtarget: {API name}
The default API for datacond results.
When the underscore _ is used as the protocol argument in an
output()
action, the value of -datacondtarget will be used as the
protocol for the output.
In a dataPipeline, results are normally sent to the wrapper
API for mpi processing. If output actions use the _
default protocol specifier, and the user specifies
-datacondtarget datacond, then the output will go to
disk. The format of the output will be plain however, not
wrapper format. To dump wrapper formatted output to disk via
a dataPipeline command requires that the debugging flag
::DEBUG_DUMP_OBJECTS be set using the cmonClient.
A much more straightforward method of getting wrapper formatted
ilwd written to disk is to use the
dataStandAlone
user command.
-metadatatarget: metadata
-dbquery: {{{sql} push alias}...}
Example of an arbitrary query
-dbntuple: {{{sql} {} {}} ...}
Each result ilwd will be sent to the target specified by
-metadatatarget option, e.g. datacondAPI.
The second and third elements of the individual -dbntuple arguments
are currently required, and should be represented as pairs of empty curly
braces: {}.
A compound -dbntuple argument should look like:
-dbspectrum: {{param=value param=value ...}...}
Example of a spectrum query: (with embedded blanks in fields)
To enter more than 1 query, enclose each one as a Tcl list,
e.g. {{spectrumquery1} {spectrumquery2} ...}
-dbqualitychannel: {{param=value param=value}...}
The sql command must provide the four columns
StartSec (GPS time in seconds of when the interferometer achieved lock),
StartNanoSec (GPS time in nanoseconds of when the interferometer achieved lock),
StopSec (GPS time in seconds of when the interferometer losted lock),
and StopNanoSec (GPS time in nanoseconds of when the interferometer lost lock).
A fifth column may be supplied to provide data about which interferometer is
to be associated with the start/stop time pair. If this column is provided, it
must me labeled IFO. If this column is supplied, the name of the
resulting object will be appended with the IFO.
These columns can (and most likely will be) aliases of the actual database
columns. Also, the names are case insensitive. StartSec, startSec, and startsec are all equivelent.
SQL Example (no IFO column):
SQL Example (with IFO column):
Code Examples:
To enter more than 1 query, enclose each one as a Tcl list,
e.g. {{qualitychannel1} {qualitychannel2} ...}
Each result ilwd will be sent to the target specified by -metadatatarget option
e.g. datacondAPI.
-concatenate: Boolean
-allowgaps: Boolean
For example:
The information representing the gaps is retained in a history record
named gap_info.
This may be useful information for developers of filter code.
The history record contains two pieces of information.
The first is the fill method that has been applied to the gaps.
This is an enumerated type as described in the
fillgap action.
Within the history record, it is named fill_method.
The second piece of information represents the range of the gap by
a GPS start time and GPS end time pair.
Each GPS time is represented by two unsigned integers.
The first number is the seconds component of the GPS time.
The second number is the nanoseconds component of the GPS time.
These pieces are named time_range.
There may be multiple instances of time_range, each representing
a single gap.
The gap range start with the GPS start time and goes till, but does not
include, the GPS end time.
This allow for loops:
An example of the history record in ilwd form is:
-inputfile: URL
Optional: this option applies when the user desires to have the
wrapperAPI get its input from a file rather than via the
datacondAPI.
The inputfile should be a valid URL but currently full file path is
allowed, e.g.
-inputfile /ldas_outgoing/jobs/ldasmdc/mpi/test/06inspiral/input/c_1.40_1.40_11.00.ilwd
-ingestdata: { port:_null }
Optional: this option applies when there is metadata to be inserted
from the output of eventmonAPI processing.
The default, -ingestdata port:_null, should be used at all times
if included or leave out this option to use the default.
-multidimdatatarget: { API name }
Optional: this option applies when there is multi-dimension ilwd data
from eventmonAPI processing to be converted by the mddapi
to some end-user output, e.g. to LIGO_LW document by the ligolwAPI
or to frames by the frameAPI.
Valid API names are ligolw and frame.
-state: { jobid }
Optional: this option applies when there is state information
from the previous wrapperAPI job run to pass on to the next
wrapperAPI job run.
-multidimoutput: { format URL }
jobid identifies the previous MPI job that has the state information for the next run.
Currently the state is keep on file in the output directory for the job.
-responsefunction: { Full Path to File }
This option defines the format and URL of the multi-dimension data
e.g. LIGO_LW for XML output, frame output and the url for placing the output file.
Optional: Deprecated in favor of -responsefiles, q.v.
-responsefiles: { Full Path to File(s) }
See -responsefiles option below.
Optional: When an external ilwd responsefile(s) is/are required,
the full path to the file is specified by this option. The syntax
allows for some degree of flexibility in the disposition of the
files. See below.
This option is used to specify the location and disposition of
files containing data and/or coefficients which are not provided
by the data as received from the frame API or which cannot be
calculated from the frame derived data by the data-conditioning
API or extracted from the database.
The files referred to by this option can be injected into the data
stream for the job either within the data-conditioning API by the
use of the "push" option, or can be attached to the output of the
data-conditioning API for transmission to the metadata or wrapper
API's by use of the "pass" option. Use of the "push" option
requires that an additional argument in the form of an "alias" for
the data-conditioning API be provided.
Examples:
-responsefiles {
file:/MayMDC/al.ilwd,push,al
file:/MayMDC/bl.ilwd,push,bl
file:/MayMDC/am.ilwd,push,am
file:/MayMDC/bm.ilwd,push,bm
file:/MayMDC/ah.ilwd,push,ah
file:/MayMDC/bh.ilwd,push,bh
file:/MayMDC/resp.bin,pass
}
Here the "push" elements are ilwd data files which are used to
populate the values al, bl, am, bm, ah, and bh. The contents of
the files can then be referenced in the call chain algorithm by
referring to the appropriate variable.
The "pass" element is passed on to the api pointed to by the
-datacondtarget option and is not used in calculations performed by
the data conditioning API.
Optional: Number of nodes requested for MPI calculations,
defaults to 3, and installation specific limits may be enforced.
Fewer nodes or more nodes than the number requested may be assigned
according to system availability or local rules regarding the
allocation of nodes. By default, only one user may have job
processing occurring on a single node, this can be disabled by
setting the global mpi API variable ::MPI_MULTIPLE_NODES in the mpiAPI resource
file LDASmpi.rsc to "0".
Setting that value to zero is probably not what is wanted in most
cases.
The relative or absolute path to the dynamically loaded shared
object library containing the wrapperAPI search algorithms which
will be used to analyze the data.
When the path is relative (a bare filename IS a relative path),
the specified location is assumed to be relative to the directory
defined in the LDASmpi.rsc file as the variable
::DYNLIB_DIRECTORY.
A path beginning with a "/" will ALWAYS be taken as an
absolute path to the dynamic library and no attempt will be made
to match a file in the ::DYNLIB_DIRECTORY when a
leading "/" is provided.
The hostname and port number of the computer that the mpiAPI
runs on; the wrapperAPI will connect to the port on this host
to communicate with the mpiAPI.
Normally specified in the resource file for the mpi API as the
global variables ::MPIHOST and ::MPIPORT.
It may be specified in the argument list of the user command for
development or debugging purposes only.
The hostname and the port number of the platform on which the LDAS API providing
data to wrapperAPI runs on, e.g. data conditioning API host and port; the wrapperAPI
will connect to this port on this host to receive the data.
Normally specified in the resource file for the mpi API as the
global variables ::DATAHOST and ::DATAPORT.
It may be specified in the argument list of the user command for
development or debugging purposes only.
The hostname and port number of the platform on which the LDAS API
receiving data from the wrapperAPI resides on, e.g. eventmonAPI's host and port number;
the wrapper will connect to the port on this host to send its output objects.
Depending upon the -returnprotocol option,
which may be set to the name of a known API, this value
can be dynamically calculated by the mpi API. It may be
specified in the argument list of the user command for
development or debugging purposes only.
If this flag is 0 (false), then each frame formatted output from the
data conditioning API will be returned as a seperate frame object.
If this flag is 1 (true), then frame formatted output from the
data conditioning API is combined into a single frame object.
If this flag is 0 (false), then each frame formatted output from the
event monitor API will be returned as a seperate frame object.
If this flag is 1 (true), then frame formatted output from the
event monitor API is combined into a single frame object.
This option allows the metadata results produced to be inserted
into the database specified instead of the default database (the one
connected to when the metadataAPI starts up). For the list of valid
databases, please refer to the LDAS database link at your site.
e.g.
The databases for site ldas-dev.ligo.caltech.edu lists
the default database is cit_test. Using a -database cit_1 option in the
pipeline command will direct the insertion to the database cit_1
instead of cit_test.
Default: LIGO_LW
This option allows specification of base64 encoding for
Vector data in LIGO_LW documents, e.g. generated from frame ilwds. It only applies if
such data is present, otherwise base64 has no effect even if
it is specified.
List of options to pass to the shared object code specified
by the -dynlib option. The option list in this
case is not a Tcl list, but a cslist, or "comma
separated list" of values.
The syntax of the -filterparams argument is specific to the
search code being exercised. This example is from the original
ldasinspiral.so:
Argument Description type
numCoarseExch Number of coarse templates to exchange int
numPoints Number of data points in a segment int
numSegments Number of overlapping data segments int
numChisqBins Number of frequency bands for chisq veto int
deltaT Sampling interval float
ovrlap Overlap betweeen segments (# of points) int
invSpecTrunc Duration of inverse spectrum in time domain int
fLow Low frequency cut-off in inverse spectrum float
rhosqThreshold threshold for SNR float
chisqThreshold threshold for chisqr float
numTmplts number of templates int
(m1,m2;...) list of templates see below
The arguments are a comma delimited list that the wrapper parses and passes to
the shared object as strings. The first 12 arguments are simple floats or ints
that the shared object calls atof() or atoi() on.
The last argument is the list of templates. The wrapperAPI baseline
requirements state that an argment that is enclosed in paranthesis is passed
to the shared object as a single string. Thus the list of masses is passed as
a string to the shared object and parsed there.
Each template is a pair of masses m1,m2 and each template is separated by a
semicolon.
Thus a valid command line to wrapperAPI is (formatted with line-breaks for readability):
wrapperAPI
-mpiAPI="(marfik.ligo.caltech.edu,10000)"
-nodelist="(1-2)"
-dynlib="/home/duncan/lib/lalwrapper/libinspiral.so"
-dataAPI="(datahost,5678)"
-resultAPI"=(reshost, 9101)"
-filterparams="(3,1048576,1,8,0.00012207,0,0,5.0,100.0,2.1,69.0,6,(1.0,1.0;1.4,1.4;2.0,2.0;2.2,2.2;2.4,2.4;5.5,5.5))"
-nodeDutyCycle=2
-realTimeRatio=0.9
-doLoadBalance=FALSE
-dataDistributor=W
-jobID=8
-inputFile="event.ilwd"
This list of templates parsed by the shared object would be
tmplt number m1 m2
0 1.0 1.0
1 1.4 1.4
2 2.0 2.0
3 2.2 2.2
4 2.4 2.4
5 5.5 5.5
The so checks that the number of templates in the string equals
the numTmplts argument and aborts if it doesn't.
This option sets user defined tag used to group related jobs
when running database queries.
The desired ratio of the time required to process the data to the
time contained within the data segment, e.g. a value of 0.90 would
request 54 seconds be used to analyse 60 seconds worth of data.
Defaults to 1.0.
This option sets desired memory usage for each beowulf node
assigned to the job. The value is a ratio of maximum memory
amount used by the process to the total available physical memory.
Defaults to 1.0.
Determines whether the job can be dynamically load-balanced by
adding or subtracting nodes while it is running.
Defaults to FALSE
The method for distributing input data from master to slave:
Defaults to SEARCHMASTER
The method of specifying the output data object structure used by the filter algorithm:
Defaults to ALWAYS
Optional: defaults to datacond
The argument of the -frametarget option is the name of a known LDAS
API. The result output from the frame API will be registered
with the named API to be used as input data for the code with
the same job i.d. that the data is tagged with.
Optional
The -inputprotocol option is used to load data into the conditioning
API from files or URL's instead of, or in addition to data from the
frame API.
Default: "Data inserted ok."
String to be used as the subject for the e-mail returned by the
system on completion of the job.
Default: 0
When the -autoexpand flag is set, the -aliases and -algorithms
options will be used as patterns that will be expanded once for
each object recieved from the frame API.
The -aliases options used with -autoexpand will be applied in
combination with the -algorithms option to all ingested channel
data.
Example of -aliases and -algorithms options that may be auto-expanded:
-aliases sftN
-algorithms {output(sftN,,,sftN,sftN data)}
The string sftN will be replaced with the name of
each ingested frame channel, once for each channel ingested.
Implications and side-effects of auto-expansion have not been
explored in detail. There is a possibility that some combinations
of data and -aliases and -algorithms options will produce unexpected
results. Simple applications of auto-expansion will present less
opportunity for unexpected behaviour.
Default: wrapper
Default: datacond
The API which the metadata API should send it's results to.
The default option is datacond but mpi should
be used if the results are to be processed by the wrapperAPI.
If this is set to metadata the output will be dumped
to file and the data pipeline will end here.
Default: {}
This option is used to
inject metadata into the data pipeline. The first element
in the -dbquery option list must be a valid SQL statement
which results in the return of exactly one database object.
The second element must be either push or pass.
When the second element is push, the element will be made available
in the call chain of the data conditioning API using the third element of
the -dbquery option, alias, as it's variable name.
When the second element is pass, the element will be passed through
the data conditioning API and will not be available for use in the call
chain.
To enter more than 1 query, enclose each one as a Tcl list,
e.g. {{{sql1} push1 alias1} {{sql2} push2 alias2} ...}
Each result ilwd will be sent to the target specified by -metadatatarget option
e.g. datacondAPI.
Example of 2 arbitrary queries
-dbquery {{select * from sngl_inspiral fetch first 2 rows only} pass dbquery}
-dbquery {{{select * from sngl_inspiral fetch first 2 rows only} pass dbquery_1}
{{select * from sngl_burst fetch first 2 rows only} pass dbquery_2}}
Default: {}
This option is used to inject metadata into the data pipeline
for the datacondAPI to pass to the wrapperAPI only. It is
identical to the -dbquery option except the data selected
should only be types valid for the wrapperAPI, e.g. currently
wrapperAPI does not support database blobs (binary large objects)
and clobs (character large objects) so these types would be invalid
for this option.
The first element
in the -dbquery option list must be a valid SQL statement
which results in the return of exactly one data object.
The second element must be pass or {}
for the element to be passed through
from the data conditioning API to the wrapperAPI.
To enter more than 1 query, enclose each one as a Tcl list,
e.g. {{{sql1} {} {}} {{sql2} {} {}} ...}
-dbntuple { {{sql} {} {}} {{sql} {} {}} }
For example.
Default: {}
This option is used to extract spectra data into the pipeline.
The option is provided in
the form of a Tcl list of these nine elements:
Example of a spectrum query: (without embedded blanks in fields)
-dbspectrum {channel={optimally filtered cross-correlation spectrum}
spectrum_type={IFO IFO differential mode cross correlation spectrum for stochastic background search}
start_time=681932000
start_time_ns=93750000
start_frequency=0
delta_frequency=32
spectrum_length=9
pushpass=push
alias=spectrum}
-dbspectrum {channel=H2:LSC-AS_Q
spectrum_type=Welch
start_time=680938500
start_time_ns=0
start_frequency=-512
delta_frequency=.0625
spectrum_length=16384
pushpass=push
alias=spectrum}
Each result ilwd will be sent to the target specified by -metadatatarget option
e.g. datacondAPI.
-dbspectrum { { channel={optimally filtered cross-correlation spectrum}
spectrum_type={IFO IFO differential mode cross correlation spectrum for stochastic background search}
start_time=681932000
start_time_ns=93750000
start_frequency=0
delta_frequency=32
spectrum_length=9
pushpass=push
alias=spectrum1}
{ channel=H2:LSC-AS_Q
spectrum_type=Welch
start_time=680938500
start_time_ns=0
start_frequency=-512
delta_frequency=.0625
spectrum_length=16384
pushpass=push
alias=spectrum2}}
Default: {}
This option is provided for the generation of a quality channel
based on data retrieved from the database.
Multiple quality channels may
be generated by listifying successive argument groups as
demonstrated in the example. The option is provided in
the form of a Tcl list of these 8 elements:
SELECT start_time as StartSec, start_time_ns as StartNanoSec, end_time as
StopSec, end_time_ns as StopNanoSec
FROM SEGMENT
WHERE ( ( Start_Time >= 680895416 and Start_Time <= 680896086 ) OR ( end_time <= 680896086 AND end_time >= 680895416 ) ) AND ( segment_group LIKE 'H2:%' )
SELECT start_time as StartSec, start_time_ns as StartNanoSec, end_time as
StopSec, end_time_ns as StopNanoSec, substr(segment_group, 1, 2) AS IFO
FROM SEGMENT
WHERE ( ( Start_Time >= 680895416 and Start_Time <= 680896086 ) OR ( end_time <= 680896086 AND end_time >= 680895416 ) ) AND ( segment_group LIKE 'H2:%' )
-dbqualitychannel {{ 1024 680896086 0 680896086 0
{
SELECT start_time as StartSec, start_time_ns as StartNanoSec, end_time as
StopSec, end_time_ns as StopNanoSec
FROM SEGMENT
WHERE ( ( Start_Time >= 680895416 and Start_Time <= 680896086 ) OR ( end_time <= 680896086 AND end_time >= 680895416 ) ) AND ( segment_group LIKE 'H2:%' ) }
push qc1}
{ 1024 680896086 0 680896086 0 {
SELECT start_time as StartSec, start_time_ns as StartNanoSec, end_time as
StopSec, end_time_ns as StopNanoSec, substr(segment_group, 1, 2) AS IFO
FROM SEGMENT
WHERE ( ( Start_Time >= 680895416 and Start_Time <= 680896086 ) OR ( end_time <= 680896086 AND end_time >= 680895416 ) ) AND ( segment_group LIKE 'H2:%' ) }
pass qc2}
}
Default: 1
Setting this option to '1', or not declaring it explicitly (using the
default value) causes data to be serialised across frame boundaries
so that, for example, a 160 second long run of data will be packed
into a single channel object if ten 16 second frames satisfy the time
range specified in the -framequery option.
Setting the -concatenate option to '0' will cause one output object
for each channel requested from each frame satisfying the time range
specified. This is not very useful when the data required is time
domain data, but could be very useful if frequency domain data is expected
to be returned based on a time-domain request.
The frame API will normally be required to concatenate time serialised
channel data for conditioning, consequently this option will not
normally need to be explicitly declared.
Default: 0
This option applies gap-filling logic GLOBALLY over all
-framequery times specified by the user. It causes all elements
of a time range specification to be combined into a single long time range
which includes gaps due to missing data files and gaps
intentionally introduced by manipulating the format of the 'times'
spec.
-framequery { {} H {} { 600000000-600000004 600000008 } Adc(0) }
Would cause an induced gap to occur between 600000004 and 600000008
for( GPSTime t = GPSStart; t < GPSEnd; t += delta_t )
{
// Do something interesting
}
<ilwd name='LDAS_History'>
<ilwd name='gap_info' size='5'>
<int_4u name='fill_method'>0</int_4u>
<int_4u dims='4' name='time_range'>688011799 999511718 688011801 0</int_4u>
<int_4u dims='4' name='time_range'>688011802 0 688011803 0</int_4u>
<int_4u dims='4' name='time_range'>688011804 0 688011805 0</int_4u>
<int_4u dims='4' name='time_range'>688011808 0 688011809 999999999</int_4u>
</ilwd>
</ilwd>
-metadataapi: { API name | tee }
Default: metadata
The metadata results will be inserted into the database by default.
If 'ligolw' is specified e.g. -metadataapi ligolw,
and the -returnprotocol specifies a url, the metadata results will be output in the form of xml
rather than being inserted into the database i.e.
running at a site that does not have a database.
If 'tee' option is specified, e.g. -metadataapi tee, and the -returnprotocol specifies a url
the metadata results will
be output both as an xml and also be inserted
into the database.
Examples of dataPipeline commands:
Simple but complete example which concatenates the gravitational wave
channel data from one hour of frames, sends the concatenated object
to the data-conditioning API, which prepares a table of statistics
of the time series data, and puts this data into the appropriate
database.
This example shows a typical argument list fora dataPipeline user
command:
One or more products are created by the dataPipeline command and appear
in the the user's job output directory. Diagnostic ilwds may be generated
if debugging level is set for some APIs, e.g. setting DEBUG level to 1
in the eventmonAPI cause the multi-dimension ilwd and metadata ilwd, if any,
to be written to the job output directory.
Table created by dc API in ilwd ascii format:
More examples of products:
dataPipeline
-dynlib /ldcg/lib/lalwrapper/libldasinspiral.so
# the filterparams are specific coefficients relative
# to the specified shared object.
-filterparams (1048576,1,0,40.0,0,69.0,8,(64.0,0.0),(3.0,0.0),1,1,1,2,(1.400000,1.400000/1.800000,1.800000))
-aliases {gw=_ch0;darm=_ch1}
-algorithms {
rdarm = resample(darm, 1, 8);
ldarm = linfilt(bl, al, rdarm);
output(ldarm,_,mpi,darm,low bandpass);
rx = resample(gw, 1, 8);
output(rx,_,mpi,rx,resampled gw timeseries);
p = psd(rx,1048576);
output(p,_,_,psd,psd of resampled timeseries);
}
-responsefiles {
file:/MayMDC/al.ilwd,push,al
file:/MayMDC/bl.ilwd,push,bl
file:/MayMDC/am.ilwd,push,am
file:/MayMDC/bm.ilwd,push,bm
file:/MayMDC/ah.ilwd,push,ah
file:/MayMDC/bh.ilwd,push,bh
file:/MayMDC/resp.bin,pass
}
# 1024 second data segment
# the gravitation channel and an arm control channel
-framequery { {} H {} 658021000-658022023 Adc(H2:LSC-AS_Q,H2:LSC-DARM_CTRL) } }"
<ilwd size='3'>
<ilwd name='processgroup:process:table' size='10'>
<lstring name='processgroup:process:program' size='11'>datacondAPI</lstring>
<ilwd name='processgroup:process:process_id'>
<char dims='20'>process:process_id:0</char>
</ilwd>
<lstring name='processgroup:process:version' size='11'>ldas-0.0.15</lstring>
<lstring name='processgroup:process:cvs_repository' size='27'>ldas/api/datacondAPI/so/src</lstring>
<int_4s name='processgroup:process:cvs_entry_time'>0</int_4s>
<int_4s name='processgroup:process:start_time'>666000000</int_4s>
<int_4s name='processgroup:process:is_online'>0</int_4s>
<lstring name='processgroup:process:node' size='7'>datacon</lstring>
<lstring name='processgroup:process:username' size='4'>ldas</lstring>
<int_4s name='processgroup:process:unix_procid'>2378</int_4s>
</ilwd>
<ilwd name='process_paramsgroup:process_params:table' size='5'>
<lstring name='process_paramsgroup:process_params:program' size='11'>datacondAPI</lstring>
<ilwd name='process_paramsgroup:process_params:process_id'>
<char dims='20'>process:process_id:0</char>
</ilwd>
<lstring name='process_paramsgroup:process_params:param' size='6'>step-0</lstring>
<lstring name='process_paramsgroup:process_params:type' size='6'>action</lstring>
<lstring name='process_paramsgroup:process_params:value' size='8'>all(raw)</lstring>
</ilwd>
<ilwd name='summ_statisticsgroup:summ_statistics:table' size='15'>
<lstring name='summ_statisticsgroup:summ_statistics:program' size='11'>datacondAPI</lstring>
<ilwd name='processgroup:process:process_id'>
<char dims='20'>process:process_id:0</char>
</ilwd>
<int_4u name='summ_statisticsgroup:summ_statistics:start_time'>666000000</int_4u>
<int_4u name='summ_statisticsgroup:summ_statistics:start_time_ns'>0</int_4u>
<int_4u name='summ_statisticsgroup:summ_statistics:end_time'>666003600</int_4u>
<int_4u name='summ_statisticsgroup:summ_statistics:end_time_ns'>0</int_4u>
<int_4u name='summ_statisticsgroup:summ_statistics:samples'>58982400</int_4u>
<lstring name='summ_statisticsgroup:summ_statistics:channel' size='39'>H2\:LSC-AS_Q::AdcData:666000000:0:Frame</lstring>
<real_8 name='summ_statisticsgroup:summ_statistics:min_value'>-1.5082244873046875e+03</real_8>
<real_8 name='summ_statisticsgroup:summ_statistics:max_value'>5.8331585693359375e+02</real_8>
<real_8 name='summ_statisticsgroup:summ_statistics:mean'>1.1476757515331777e+00</real_8>
<real_8 name='summ_statisticsgroup:summ_statistics:rms'>1.7742118277333952e+02</real_8>
<real_8 name='summ_statisticsgroup:summ_statistics:variance'>3.1477599350458338e+04</real_8>
<real_8 name='summ_statisticsgroup:summ_statistics:skewness'>-1.9985680110792783e+00</real_8>
<real_8 name='summ_statisticsgroup:summ_statistics:kurtosis'>1.3436855850203969e+01</real_8>
</ilwd>
</ilwd>
Frame File Naming Convention
When the frame cache consists of a mixed collection of frames in
an inconsistent hierarchy of subdirectories the appropriate frame
file(s) for fulfilling a given request are determined by parsing
the frame file names according to an installation-specific naming
convention.
The LDAS system provides a default naming convention which is described
in detail in the document
Naming Convention for Frame Files Which Are to be Processed by LDAS
.
where
Examples:
Examples of the -metadataai option:
In summary, the required format of frame-file names is
16 seconds of raw data from Hanford.
1 minute of second-trend data from Livingston.
16 seconds of Level 1 reduced data from Hanford.
Subject: BOX-III98872 tfclu online running on H2:LSC-AS_Q Inserted
1 rows into ldas_tst database table process Inserted 14 rows into ldas_tst database
table process_params Inserted 1330 rows into ldas_tst database table sngl_burst
Inserted 1 rows into ldas_tst database table search_summary
Subject: BOX-III98856 tfclu online running on H2:LSC-AS_Q Your results: tfcluste--rs_result.xml
can be found at: http://131.215.114.23/ldas_outgoing/jobs/BOX-III_9/BOX-III98856
Subject: BOX-III98809 tfclu online running on H2:LSC-AS_Q Inserted 1 rows into ldas_tst database
table process Inserted 14 rows into ldas_tst database table process_params
Inserted 1330 rows into ldas_tst database table sngl_burst Inserted 1 rows into ldas_tst database
table search_summary Your results: tfclusters_result.xml can be found
at: http://131.215.114.23/ldas_outgoing/jobs/BOX-III_9/BOX-III98809
Return to Top