The wrapper API on metaserver at Caltech-Test

Legend:
green ball Normal status or debugging message
yellow ball Notable condition which may be a non-fatal error
orange ball Error condition not fatal to job
red ball Error condition fatal to job
blue ball Notable condition which is not an error
purple ball Currently undefined
email Condition requires email notification of the responsible administrator of this API
telephone Condition requires phone notification of the responsible administrator of this API

Link: API Status Page for Caltech-Test

02/21/07-13:19:33 PST 02/21/07-21:19:33 GMT 856127987 LDAS-TEST17792530 mpi::filterLogEntries 17792530.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) Rank2(metaserver) Rank3(metaserver) }
02/21/07-13:19:39 PST 02/21/07-21:19:39 GMT 856127993 LDAS-TEST17792530 mpi::filterLogEntries 17792530.1:error {Rank 1 failed with error message: ( applySearch function ): Level 1: power $Id: ApplySearch.c,v 1.7 2003/12/09 17:39:03 julien Exp $ Status code 19: File error! function LALApplySearch, file ../../../../contrib/burstwrapper/src/ApplySearch.c, line 111 }
02/21/07-13:19:39 PST 02/21/07-21:19:39 GMT 856127993 LDAS-TEST17792530 mpi::newJobCallback {Rank 1 failed with error message: ( applySearch function ): Level 1: power $Id: ApplySearch.c,v 1.7 2003/12/09 17:39:03 julien Exp $ Status code 19: File error! function LALApplySearch, file ../../../../contrib/burstwrapper/src/ApplySearch.c, line 111 }
02/21/07-13:20:26 PST 02/21/07-21:20:26 GMT 856128040 LDAS-TEST17792531 mpi::filterLogEntries 17792531.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) Rank2(metaserver) Rank3(metaserver) }
02/21/07-13:21:06 PST 02/21/07-21:21:06 GMT 856128080 LDAS-TEST17792531 mpi::filterLogEntries 17792531.1:progress 100.00%
02/21/07-13:21:06 PST 02/21/07-21:21:06 GMT 856128080 LDAS-TEST17792531 mpi::filterLogEntries 17792531.1:projected_ratio 0.01092
02/21/07-13:21:06 PST 02/21/07-21:21:06 GMT 856128080 LDAS-TEST17792531 jobstate response: '17792531.1:cont'
02/21/07-13:21:06 PST 02/21/07-21:21:06 GMT 856128080 LDAS-TEST17792531 mpi::filterLogEntries 17792531.2:progress 100.00%
02/21/07-13:21:06 PST 02/21/07-21:21:06 GMT 856128080 LDAS-TEST17792531 mpi::filterLogEntries 17792531.2:projected_ratio 0.02169
02/21/07-13:21:06 PST 02/21/07-21:21:06 GMT 856128080 LDAS-TEST17792531 mpi::filterLogEntries 17792531.3:finalizing { totalTime=40.1386 sentObjects=4; Rank0 (inPut: create=0.68755 send=0.69698 ilwd=0.54096; outPut: create=0.03236 send=0.19106 ilwd=0.08572; metadata: (un)pack=0.00030 bcast=0.03450; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12033; max memory=335872) Rank3 (inPut: create=0.92952 send=0.89049; outPut: create=0.00000 send=0.09997; metadata: (un)pack=0.00009 bcast=0.00565; IS=0.92174; CD=0.23587; AS=0.64128; FO=0.00000; FS=0.00218; LB=0.00001; max memory=729088) Rank1 (inPut: create=0.83777 send=0.55611; outPut: create=0.01590 send=0.11260; metadata: (un)pack=0.00017 bcast=0.02034; IS=0.00014; CD=0.00006; AS=35.85747; FO=0.00004; FS=0.00006; LB=0.03193; max memory=413696) Rank2 (inPut: create=0.84846 send=0.54387; outPut: create=0.03194 send=0.35656; metadata: (un)pack=0.00026 bcast=0.03503; IS=0.85176; CD=0.23676; AS=35.69731; FO=0.00008; FS=0.03074; LB=0.00001; max memory=729088) }
02/21/07-13:21:22 PST 02/21/07-21:21:22 GMT 856128096 LDAS-TEST17792532 mpi::filterLogEntries 17792532.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
02/21/07-13:21:22 PST 02/21/07-21:21:22 GMT 856128096 LDAS-TEST17792532 mpi::filterLogEntries 17792532.1:progress 100.00%
02/21/07-13:21:22 PST 02/21/07-21:21:22 GMT 856128096 LDAS-TEST17792532 mpi::filterLogEntries 17792532.1:projected_ratio 0.00097
02/21/07-13:21:22 PST 02/21/07-21:21:22 GMT 856128096 LDAS-TEST17792532 mpi::filterLogEntries 17792532.2:finalizing { totalTime=0.552625 sentObjects=3; Rank0 (inPut: create=0.04764 send=0.00635 ilwd=0.04088; outPut: create=0.01835 send=0.04959 ilwd=0.04523; metadata: (un)pack=0.00017 bcast=0.01692; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01598; max memory=3932160) Rank1 (inPut: create=0.04912 send=0.00913; outPut: create=0.01603 send=0.05140; metadata: (un)pack=0.00015 bcast=0.01767; IS=0.00011; CD=0.00784; AS=0.05827; FO=0.00021; FS=0.00004; LB=0.03826; max memory=5165056) }
02/21/07-13:21:44 PST 02/21/07-21:21:44 GMT 856128118 LDAS-TEST17792533 mpi::filterLogEntries 17792533.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
02/21/07-13:21:46 PST 02/21/07-21:21:46 GMT 856128120 LDAS-TEST17792533 mpi::filterLogEntries 17792533.1:progress 100.00%
02/21/07-13:21:46 PST 02/21/07-21:21:46 GMT 856128120 LDAS-TEST17792533 mpi::filterLogEntries 17792533.1:projected_ratio 0.00020
02/21/07-13:21:46 PST 02/21/07-21:21:46 GMT 856128120 LDAS-TEST17792533 mpi::filterLogEntries 17792533.2:finalizing { totalTime=2.36549 sentObjects=3; Rank0 (inPut: create=0.43672 send=0.16889 ilwd=0.34635; outPut: create=0.07057 send=0.62184 ilwd=0.04986; metadata: (un)pack=0.00012 bcast=0.01690; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01597; max memory=0) Rank1 (inPut: create=0.49069 send=0.17930; outPut: create=0.01593 send=0.63157; metadata: (un)pack=0.00012 bcast=0.01775; IS=0.00001; CD=0.00000; AS=0.07225; FO=0.00340; FS=0.00000; LB=0.13756; max memory=184320) }
02/21/07-13:22:33 PST 02/21/07-21:22:33 GMT 856128167 LDAS-TEST17792534 mpi::filterLogEntries 17792534.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
02/21/07-13:22:33 PST 02/21/07-21:22:33 GMT 856128167 LDAS-TEST17792534 mpi::filterLogEntries 17792534.1:progress 100.00%
02/21/07-13:22:33 PST 02/21/07-21:22:33 GMT 856128167 LDAS-TEST17792534 mpi::filterLogEntries 17792534.1:projected_ratio 0.00000
02/21/07-13:22:33 PST 02/21/07-21:22:33 GMT 856128167 LDAS-TEST17792534 mpi::filterLogEntries 17792534.2:finalizing { totalTime=0.360668 sentObjects=2; Rank0 (inPut: create=0.04309 send=0.00071 ilwd=0.03801; outPut: create=0.00000 send=0.01556 ilwd=0.04545; metadata: (un)pack=0.00010 bcast=0.00115; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01598; max memory=0) Rank1 (inPut: create=0.04178 send=0.00148; outPut: create=0.00000 send=0.04838; metadata: (un)pack=0.00016 bcast=0.00201; IS=0.00012; CD=0.00063; AS=0.00014; FO=0.00000; FS=0.00004; LB=0.03190; max memory=651264) }
02/21/07-13:22:49 PST 02/21/07-21:22:49 GMT 856128183 LDAS-TEST17792535 mpi::filterLogEntries 17792535.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
02/21/07-13:23:03 PST 02/21/07-21:23:03 GMT 856128197 LDAS-TEST17792535 mpi::filterLogEntries 17792535.1:progress 68.37%
02/21/07-13:23:03 PST 02/21/07-21:23:03 GMT 856128197 LDAS-TEST17792535 mpi::filterLogEntries 17792535.1:projected_ratio 0.03597
02/21/07-13:23:03 PST 02/21/07-21:23:03 GMT 856128197 LDAS-TEST17792535 jobstate response: '17792535.1:cont'
02/21/07-13:23:04 PST 02/21/07-21:23:04 GMT 856128198 LDAS-TEST17792535 mpi::filterLogEntries 17792535.2:progress 100.00%
02/21/07-13:23:04 PST 02/21/07-21:23:04 GMT 856128198 LDAS-TEST17792535 mpi::filterLogEntries 17792535.2:projected_ratio 0.03752
02/21/07-13:23:04 PST 02/21/07-21:23:04 GMT 856128198 LDAS-TEST17792535 mpi::filterLogEntries 17792535.3:finalizing { totalTime=15.0935 sentObjects=4; Rank0 (inPut: create=0.45947 send=0.16898 ilwd=0.34106; outPut: create=0.09436 send=0.09974 ilwd=0.34706; metadata: (un)pack=0.01203 bcast=0.06435; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.13113; max memory=5664768) Rank1 (inPut: create=0.48536 send=0.20694; outPut: create=0.07371 send=0.11046; metadata: (un)pack=0.01031 bcast=0.05003; IS=0.00009; CD=0.06331; AS=13.50616; FO=0.00210; FS=0.00374; LB=0.05131; max memory=352256) }
02/21/07-13:23:18 PST 02/21/07-21:23:18 GMT 856128212 LDAS-TEST17792536 mpi::filterLogEntries 17792536.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
02/21/07-13:23:18 PST 02/21/07-21:23:18 GMT 856128212 LDAS-TEST17792536 mpi::filterLogEntries 17792536.1:progress 100.00%
02/21/07-13:23:18 PST 02/21/07-21:23:18 GMT 856128212 LDAS-TEST17792536 mpi::filterLogEntries 17792536.1:projected_ratio 0.00001
02/21/07-13:23:18 PST 02/21/07-21:23:18 GMT 856128212 LDAS-TEST17792536 mpi::filterLogEntries 17792536.2:finalizing { totalTime=0.567735 sentObjects=3; Rank0 (inPut: create=0.04851 send=0.00290 ilwd=0.04210; outPut: create=0.01626 send=0.03116 ilwd=0.04679; metadata: (un)pack=0.00026 bcast=0.01695; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01599; max memory=0) Rank1 (inPut: create=0.04888 send=0.00457; outPut: create=0.01594 send=0.06507; metadata: (un)pack=0.00028 bcast=0.01750; IS=0.00011; CD=0.00226; AS=0.00058; FO=0.00006; FS=0.00004; LB=0.03188; max memory=2060288) }
02/21/07-13:24:16 PST 02/21/07-21:24:16 GMT 856128270 LDAS-TEST17792537 mpi::filterLogEntries 17792537.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.1:progress 0.00%
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.1:projected_ratio 0.00015
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.2:progress 10.00%
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.2:projected_ratio 0.00030
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.3:progress 20.00%
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.3:projected_ratio 0.00045
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.4:progress 30.00%
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.4:projected_ratio 0.00060
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.5:progress 40.00%
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.5:projected_ratio 0.00075
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.6:progress 50.00%
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.6:projected_ratio 0.00090
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.7:progress 60.00%
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 mpi::filterLogEntries 17792537.7:projected_ratio 0.00105
02/21/07-13:24:18 PST 02/21/07-21:24:18 GMT 856128272 LDAS-TEST17792537 jobstate response: '17792537.7:cont'
02/21/07-13:24:19 PST 02/21/07-21:24:19 GMT 856128273 LDAS-TEST17792537 mpi::filterLogEntries 17792537.8:progress 70.00%
02/21/07-13:24:19 PST 02/21/07-21:24:19 GMT 856128273 LDAS-TEST17792537 mpi::filterLogEntries 17792537.8:projected_ratio 0.00125
02/21/07-13:24:19 PST 02/21/07-21:24:19 GMT 856128273 LDAS-TEST17792537 mpi::filterLogEntries 17792537.9:progress 80.00%
02/21/07-13:24:19 PST 02/21/07-21:24:19 GMT 856128273 LDAS-TEST17792537 mpi::filterLogEntries 17792537.9:projected_ratio 0.00140
02/21/07-13:24:19 PST 02/21/07-21:24:19 GMT 856128273 LDAS-TEST17792537 mpi::filterLogEntries 17792537.10:progress 90.00%
02/21/07-13:24:19 PST 02/21/07-21:24:19 GMT 856128273 LDAS-TEST17792537 mpi::filterLogEntries 17792537.10:projected_ratio 0.00155
02/21/07-13:24:19 PST 02/21/07-21:24:19 GMT 856128273 LDAS-TEST17792537 mpi::filterLogEntries 17792537.11:progress 100.00%
02/21/07-13:24:19 PST 02/21/07-21:24:19 GMT 856128273 LDAS-TEST17792537 mpi::filterLogEntries 17792537.11:projected_ratio 0.00155
02/21/07-13:24:19 PST 02/21/07-21:24:19 GMT 856128273 LDAS-TEST17792537 mpi::filterLogEntries 17792537.12:finalizing { totalTime=3.05837 sentObjects=3; Rank0 (inPut: create=0.28464 send=0.10362 ilwd=0.22675; outPut: create=0.15287 send=0.51927 ilwd=0.11041; metadata: (un)pack=0.00040 bcast=0.15201; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.22484; max memory=483328) Rank1 (inPut: create=0.31772 send=0.09063; outPut: create=0.15176 send=0.55057; metadata: (un)pack=0.00037 bcast=0.15246; IS=0.01344; CD=0.10695; AS=1.35983; FO=0.00038; FS=0.00265; LB=0.03201; max memory=18403328) }
02/21/07-13:24:41 PST 02/21/07-21:24:41 GMT 856128295 LDAS-TEST17792538 mpi::filterLogEntries 17792538.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) Rank2(metaserver) Rank3(metaserver) }
02/21/07-13:24:45 PST 02/21/07-21:24:45 GMT 856128299 LDAS-TEST17792538 mpi::filterLogEntries 17792538.1:progress 0.00%
02/21/07-13:24:45 PST 02/21/07-21:24:45 GMT 856128299 LDAS-TEST17792538 mpi::filterLogEntries 17792538.1:projected_ratio 0.00000
02/21/07-13:24:45 PST 02/21/07-21:24:45 GMT 856128299 LDAS-TEST17792538 jobstate response: '17792538.1:cont'
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.2:progress 0.00%
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.2:projected_ratio 0.00000
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.3:progress 25.00%
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.3:projected_ratio 0.00026
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.4:progress 50.00%
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.4:projected_ratio 0.00082
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.5:progress 50.00%
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.5:projected_ratio 0.00139
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.6:progress 50.00%
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.6:projected_ratio 0.00168
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.7:progress 75.00%
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.7:projected_ratio 0.00171
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.8:progress 75.00%
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.8:projected_ratio 0.00202
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.9:progress 75.00%
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.9:projected_ratio 0.00226
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.10:progress 75.00%
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.10:projected_ratio 0.00226
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.11:progress 100.00%
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.11:projected_ratio 0.00236
02/21/07-13:24:46 PST 02/21/07-21:24:46 GMT 856128300 LDAS-TEST17792538 mpi::filterLogEntries 17792538.12:finalizing { totalTime=5.52314 sentObjects=3; Rank0 (inPut: create=0.43139 send=0.50658 ilwd=0.33740; outPut: create=0.03302 send=0.37110 ilwd=0.04521; metadata: (un)pack=0.00352 bcast=0.02663; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12042; max memory=1798144) Rank3 (inPut: create=0.58907 send=0.57780; outPut: create=0.00000 send=0.15150; metadata: (un)pack=0.00008 bcast=0.00360; IS=0.00011; CD=1.96898; AS=0.58425; FO=0.00000; FS=0.00458; LB=0.00001; max memory=1097728) Rank2 (inPut: create=0.58043 send=0.60037; outPut: create=0.00000 send=0.12078; metadata: (un)pack=0.00006 bcast=0.00550; IS=0.00012; CD=1.79919; AS=0.68708; FO=0.00000; FS=0.00461; LB=0.00001; max memory=1064960) Rank1 (inPut: create=0.59619 send=0.63234; outPut: create=0.02958 send=0.63596; metadata: (un)pack=0.00295 bcast=0.02684; IS=0.00013; CD=2.32009; AS=0.22110; FO=0.00060; FS=0.00341; LB=0.05128; max memory=344064) }
02/21/07-13:25:09 PST 02/21/07-21:25:09 GMT 856128323 LDAS-TEST17792539 mpi::filterLogEntries 17792539.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) Rank2(metaserver) }
02/21/07-13:25:25 PST 02/21/07-21:25:25 GMT 856128339 LDAS-TEST17792539 mpi::filterLogEntries 17792539.1:error {Rank 2 failed with error message: ( applySearch function ): Level 1: $Id Status code -1: Recursive error function LALApplySearch, file ../../../../contrib/waveburst/src/Search.c, line 1075 Level 2: $Id Status code -1: Recursive error function LALSlaveWaveburstSearch, file ../../../../contrib/waveburst/src/Search.c, line 210 Level 3: $Id Status code 101: cannot open /dso-test/waveburst/usertag/timeinterval/etg function LALOutputWaveburst, file ../../../../contrib/waveburst/src/Output.c, line 1037 }
02/21/07-13:25:25 PST 02/21/07-21:25:25 GMT 856128339 LDAS-TEST17792539 mpi::newJobCallback {Rank 2 failed with error message: ( applySearch function ): Level 1: $Id Status code -1: Recursive error function LALApplySearch, file ../../../../contrib/waveburst/src/Search.c, line 1075 Level 2: $Id Status code -1: Recursive error function LALSlaveWaveburstSearch, file ../../../../contrib/waveburst/src/Search.c, line 210 Level 3: $Id Status code 101: cannot open /dso-test/waveburst/usertag/timeinterval/etg function LALOutputWaveburst, file ../../../../contrib/waveburst/src/Output.c, line 1037 }
07/18/07-13:09:05 PDT 07/18/07-20:09:05 GMT 868824559 LDAS-TEST22923259 mpi::filterLogEntries 22923259.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) Rank2(metaserver) Rank3(metaserver) }
07/18/07-13:09:47 PDT 07/18/07-20:09:47 GMT 868824601 LDAS-TEST22923259 mpi::filterLogEntries 22923259.1:progress 100.00%
07/18/07-13:09:47 PDT 07/18/07-20:09:47 GMT 868824601 LDAS-TEST22923259 mpi::filterLogEntries 22923259.1:projected_ratio 0.01146
07/18/07-13:09:47 PDT 07/18/07-20:09:47 GMT 868824601 LDAS-TEST22923259 jobstate response: '22923259.1:cont'
07/18/07-13:09:47 PDT 07/18/07-20:09:47 GMT 868824601 LDAS-TEST22923259 mpi::filterLogEntries 22923259.2:progress 100.00%
07/18/07-13:09:47 PDT 07/18/07-20:09:47 GMT 868824601 LDAS-TEST22923259 mpi::filterLogEntries 22923259.2:projected_ratio 0.02251
07/18/07-13:09:47 PDT 07/18/07-20:09:47 GMT 868824601 LDAS-TEST22923259 mpi::filterLogEntries 22923259.3:finalizing { totalTime=41.7718 sentObjects=4; Rank0 (inPut: create=0.89836 send=1.13778 ilwd=0.74893; outPut: create=0.02434 send=0.07142 ilwd=0.08519; metadata: (un)pack=0.00031 bcast=0.02733; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.11633; max memory=335872) Rank3 (inPut: create=1.11846 send=0.96461; outPut: create=0.00000 send=0.01191; metadata: (un)pack=0.00012 bcast=0.00479; IS=0.64600; CD=0.43713; AS=1.53135; FO=0.00000; FS=0.00217; LB=0.00000; max memory=729088) Rank1 (inPut: create=1.14001 send=0.75301; outPut: create=0.01186 send=0.08918; metadata: (un)pack=0.00017 bcast=0.01892; IS=0.00016; CD=0.00006; AS=36.75903; FO=0.00004; FS=0.00006; LB=0.02392; max memory=413696) Rank2 (inPut: create=1.12786 send=1.18025; outPut: create=0.02393 send=0.30491; metadata: (un)pack=0.00026 bcast=0.02966; IS=0.89235; CD=0.42662; AS=36.61074; FO=0.00008; FS=0.02675; LB=0.00001; max memory=729088) }
07/18/07-13:10:02 PDT 07/18/07-20:10:02 GMT 868824616 LDAS-TEST22923260 mpi::filterLogEntries 22923260.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
07/18/07-13:10:03 PDT 07/18/07-20:10:03 GMT 868824617 LDAS-TEST22923260 mpi::filterLogEntries 22923260.1:progress 100.00%
07/18/07-13:10:03 PDT 07/18/07-20:10:03 GMT 868824617 LDAS-TEST22923260 mpi::filterLogEntries 22923260.1:projected_ratio 0.00097
07/18/07-13:10:03 PDT 07/18/07-20:10:03 GMT 868824617 LDAS-TEST22923260 mpi::filterLogEntries 22923260.2:finalizing { totalTime=0.522037 sentObjects=3; Rank0 (inPut: create=0.04693 send=0.00628 ilwd=0.04015; outPut: create=0.01435 send=0.05357 ilwd=0.04213; metadata: (un)pack=0.00018 bcast=0.01296; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01197; max memory=3932160) Rank1 (inPut: create=0.04842 send=0.00902; outPut: create=0.01218 send=0.04418; metadata: (un)pack=0.00015 bcast=0.01382; IS=0.00011; CD=0.00868; AS=0.05814; FO=0.00021; FS=0.00004; LB=0.03426; max memory=5165056) }
07/18/07-13:10:27 PDT 07/18/07-20:10:27 GMT 868824641 LDAS-TEST22923261 mpi::filterLogEntries 22923261.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
07/18/07-13:10:29 PDT 07/18/07-20:10:29 GMT 868824643 LDAS-TEST22923261 mpi::filterLogEntries 22923261.1:progress 100.00%
07/18/07-13:10:29 PDT 07/18/07-20:10:29 GMT 868824643 LDAS-TEST22923261 mpi::filterLogEntries 22923261.1:projected_ratio 0.00018
07/18/07-13:10:29 PDT 07/18/07-20:10:29 GMT 868824643 LDAS-TEST22923261 mpi::filterLogEntries 22923261.2:finalizing { totalTime=1.95953 sentObjects=3; Rank0 (inPut: create=0.47319 send=0.16780 ilwd=0.34587; outPut: create=0.06628 send=0.47381 ilwd=0.04363; metadata: (un)pack=0.00013 bcast=0.04955; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01197; max memory=0) Rank1 (inPut: create=0.52705 send=0.14133; outPut: create=0.01195 send=0.49913; metadata: (un)pack=0.00011 bcast=0.04855; IS=0.00001; CD=0.00000; AS=0.06339; FO=0.00334; FS=0.00000; LB=0.12451; max memory=184320) }
07/18/07-13:11:16 PDT 07/18/07-20:11:16 GMT 868824690 LDAS-TEST22923262 mpi::filterLogEntries 22923262.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
07/18/07-13:11:16 PDT 07/18/07-20:11:16 GMT 868824690 LDAS-TEST22923262 mpi::filterLogEntries 22923262.1:progress 100.00%
07/18/07-13:11:16 PDT 07/18/07-20:11:16 GMT 868824690 LDAS-TEST22923262 mpi::filterLogEntries 22923262.1:projected_ratio 0.00000
07/18/07-13:11:16 PDT 07/18/07-20:11:16 GMT 868824690 LDAS-TEST22923262 mpi::filterLogEntries 22923262.2:finalizing { totalTime=0.320888 sentObjects=2; Rank0 (inPut: create=0.04053 send=0.00070 ilwd=0.03550; outPut: create=0.00000 send=0.01159 ilwd=0.04352; metadata: (un)pack=0.00011 bcast=0.00115; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01198; max memory=0) Rank1 (inPut: create=0.04004 send=0.00142; outPut: create=0.00000 send=0.03892; metadata: (un)pack=0.00016 bcast=0.00194; IS=0.00012; CD=0.00057; AS=0.00014; FO=0.00000; FS=0.00004; LB=0.02352; max memory=651264) }
07/18/07-13:11:30 PDT 07/18/07-20:11:30 GMT 868824704 LDAS-TEST22923263 mpi::filterLogEntries 22923263.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
07/18/07-13:11:44 PDT 07/18/07-20:11:44 GMT 868824718 LDAS-TEST22923263 mpi::filterLogEntries 22923263.1:progress 68.37%
07/18/07-13:11:44 PDT 07/18/07-20:11:44 GMT 868824718 LDAS-TEST22923263 mpi::filterLogEntries 22923263.1:projected_ratio 0.03574
07/18/07-13:11:44 PDT 07/18/07-20:11:44 GMT 868824718 LDAS-TEST22923263 jobstate response: '22923263.1:cont'
07/18/07-13:11:44 PDT 07/18/07-20:11:44 GMT 868824718 LDAS-TEST22923263 mpi::filterLogEntries 22923263.2:progress 100.00%
07/18/07-13:11:44 PDT 07/18/07-20:11:44 GMT 868824718 LDAS-TEST22923263 mpi::filterLogEntries 22923263.2:projected_ratio 0.03731
07/18/07-13:11:45 PDT 07/18/07-20:11:45 GMT 868824719 LDAS-TEST22923263 mpi::filterLogEntries 22923263.3:finalizing { totalTime=14.9131 sentObjects=4; Rank0 (inPut: create=0.43050 send=0.16747 ilwd=0.34104; outPut: create=0.07881 send=0.07931 ilwd=0.33440; metadata: (un)pack=0.01200 bcast=0.04836; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12335; max memory=5664768) Rank1 (inPut: create=0.48415 send=0.14136; outPut: create=0.05363 send=0.09244; metadata: (un)pack=0.01126 bcast=0.02940; IS=0.00009; CD=0.06373; AS=13.43222; FO=0.00208; FS=0.00366; LB=0.04725; max memory=352256) }
07/18/07-13:11:59 PDT 07/18/07-20:11:59 GMT 868824733 LDAS-TEST22923264 mpi::filterLogEntries 22923264.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
07/18/07-13:12:00 PDT 07/18/07-20:12:00 GMT 868824734 LDAS-TEST22923264 mpi::filterLogEntries 22923264.1:progress 100.00%
07/18/07-13:12:00 PDT 07/18/07-20:12:00 GMT 868824734 LDAS-TEST22923264 mpi::filterLogEntries 22923264.1:projected_ratio 0.00001
07/18/07-13:12:00 PDT 07/18/07-20:12:00 GMT 868824734 LDAS-TEST22923264 mpi::filterLogEntries 22923264.2:finalizing { totalTime=0.862803 sentObjects=3; Rank0 (inPut: create=0.04553 send=0.00290 ilwd=0.03915; outPut: create=0.01225 send=0.02315 ilwd=0.04358; metadata: (un)pack=0.00026 bcast=0.01295; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01199; max memory=0) Rank1 (inPut: create=0.04587 send=0.00455; outPut: create=0.01537 send=0.05036; metadata: (un)pack=0.00029 bcast=0.01694; IS=0.00011; CD=0.00226; AS=0.00058; FO=0.00006; FS=0.00004; LB=0.02407; max memory=2060288) }
07/18/07-13:12:50 PDT 07/18/07-20:12:50 GMT 868824784 LDAS-TEST22923265 mpi::filterLogEntries 22923265.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.1:progress 0.00%
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.1:projected_ratio 0.00016
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.2:progress 10.00%
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.2:projected_ratio 0.00031
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.3:progress 20.00%
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.3:projected_ratio 0.00046
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.4:progress 30.00%
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.4:projected_ratio 0.00061
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.5:progress 40.00%
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.5:projected_ratio 0.00076
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.6:progress 50.00%
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.6:projected_ratio 0.00091
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.7:progress 60.00%
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 mpi::filterLogEntries 22923265.7:projected_ratio 0.00106
07/18/07-13:12:52 PDT 07/18/07-20:12:52 GMT 868824786 LDAS-TEST22923265 jobstate response: '22923265.7:cont'
07/18/07-13:12:53 PDT 07/18/07-20:12:53 GMT 868824787 LDAS-TEST22923265 mpi::filterLogEntries 22923265.8:progress 70.00%
07/18/07-13:12:53 PDT 07/18/07-20:12:53 GMT 868824787 LDAS-TEST22923265 mpi::filterLogEntries 22923265.8:projected_ratio 0.00123
07/18/07-13:12:53 PDT 07/18/07-20:12:53 GMT 868824787 LDAS-TEST22923265 mpi::filterLogEntries 22923265.9:progress 80.00%
07/18/07-13:12:53 PDT 07/18/07-20:12:53 GMT 868824787 LDAS-TEST22923265 mpi::filterLogEntries 22923265.9:projected_ratio 0.00138
07/18/07-13:12:53 PDT 07/18/07-20:12:53 GMT 868824787 LDAS-TEST22923265 mpi::filterLogEntries 22923265.10:progress 90.00%
07/18/07-13:12:53 PDT 07/18/07-20:12:53 GMT 868824787 LDAS-TEST22923265 mpi::filterLogEntries 22923265.10:projected_ratio 0.00153
07/18/07-13:12:53 PDT 07/18/07-20:12:53 GMT 868824787 LDAS-TEST22923265 mpi::filterLogEntries 22923265.11:progress 100.00%
07/18/07-13:12:53 PDT 07/18/07-20:12:53 GMT 868824787 LDAS-TEST22923265 mpi::filterLogEntries 22923265.11:projected_ratio 0.00153
07/18/07-13:12:53 PDT 07/18/07-20:12:53 GMT 868824787 LDAS-TEST22923265 mpi::filterLogEntries 22923265.12:finalizing { totalTime=2.87705 sentObjects=3; Rank0 (inPut: create=0.28445 send=0.10223 ilwd=0.22721; outPut: create=0.12047 send=0.42353 ilwd=0.04267; metadata: (un)pack=0.00041 bcast=0.11964; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.22088; max memory=483328) Rank1 (inPut: create=0.31721 send=0.08888; outPut: create=0.12402 send=0.46297; metadata: (un)pack=0.00036 bcast=0.12476; IS=0.01338; CD=0.10646; AS=1.34104; FO=0.00036; FS=0.00260; LB=0.02399; max memory=18403328) }
07/18/07-13:13:15 PDT 07/18/07-20:13:15 GMT 868824809 LDAS-TEST22923266 mpi::filterLogEntries 22923266.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) Rank2(metaserver) Rank3(metaserver) }
07/18/07-13:13:19 PDT 07/18/07-20:13:19 GMT 868824813 LDAS-TEST22923266 mpi::filterLogEntries 22923266.1:progress 0.00%
07/18/07-13:13:19 PDT 07/18/07-20:13:19 GMT 868824813 LDAS-TEST22923266 mpi::filterLogEntries 22923266.1:projected_ratio 0.00000
07/18/07-13:13:19 PDT 07/18/07-20:13:19 GMT 868824813 LDAS-TEST22923266 jobstate response: '22923266.1:cont'
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.2:progress 0.00%
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.2:projected_ratio 0.00044
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.3:progress 25.00%
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.3:projected_ratio 0.00096
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.4:progress 50.00%
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.4:projected_ratio 0.00153
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.5:progress 50.00%
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.5:projected_ratio 0.00212
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.6:progress 50.00%
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.6:projected_ratio 0.00247
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.7:progress 75.00%
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.7:projected_ratio 0.00250
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.8:progress 75.00%
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.8:projected_ratio 0.00282
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.9:progress 75.00%
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.9:projected_ratio 0.00309
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.10:progress 75.00%
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.10:projected_ratio 0.00312
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.11:progress 100.00%
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.11:projected_ratio 0.00331
07/18/07-13:13:20 PDT 07/18/07-20:13:20 GMT 868824814 LDAS-TEST22923266 mpi::filterLogEntries 22923266.12:finalizing { totalTime=5.53809 sentObjects=3; Rank0 (inPut: create=0.48697 send=0.75644 ilwd=0.34207; outPut: create=0.03761 send=0.27693 ilwd=0.04401; metadata: (un)pack=0.00352 bcast=0.03176; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12067; max memory=1798144) Rank3 (inPut: create=0.59799 send=0.37474; outPut: create=0.00000 send=0.17763; metadata: (un)pack=0.00007 bcast=0.00225; IS=0.00013; CD=2.05047; AS=0.61859; FO=0.00000; FS=0.00478; LB=0.00001; max memory=913408) Rank2 (inPut: create=0.59526 send=0.33909; outPut: create=0.00000 send=0.10917; metadata: (un)pack=0.00006 bcast=0.00495; IS=0.00011; CD=2.43888; AS=0.76147; FO=0.00000; FS=0.00454; LB=0.00001; max memory=1093632) Rank1 (inPut: create=0.48890 send=0.79140; outPut: create=0.03360 send=0.42304; metadata: (un)pack=0.00296 bcast=0.03029; IS=0.00012; CD=1.69662; AS=0.50917; FO=0.00060; FS=0.00355; LB=0.04755; max memory=344064) }
07/18/07-13:13:40 PDT 07/18/07-20:13:40 GMT 868824834 LDAS-TEST22923267 mpi::filterLogEntries 22923267.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) Rank2(metaserver) }
07/18/07-13:13:56 PDT 07/18/07-20:13:56 GMT 868824850 LDAS-TEST22923267 mpi::filterLogEntries 22923267.1:error {Rank 2 failed with error message: ( applySearch function ): Level 1: $Id Status code -1: Recursive error function LALApplySearch, file ../../../../contrib/waveburst/src/Search.c, line 1075 Level 2: $Id Status code -1: Recursive error function LALSlaveWaveburstSearch, file ../../../../contrib/waveburst/src/Search.c, line 210 Level 3: $Id Status code 101: cannot open /dso-test/waveburst/usertag/timeinterval/etg function LALOutputWaveburst, file ../../../../contrib/waveburst/src/Output.c, line 1037 }
07/18/07-13:13:56 PDT 07/18/07-20:13:56 GMT 868824850 LDAS-TEST22923267 mpi::newJobCallback {Rank 2 failed with error message: ( applySearch function ): Level 1: $Id Status code -1: Recursive error function LALApplySearch, file ../../../../contrib/waveburst/src/Search.c, line 1075 Level 2: $Id Status code -1: Recursive error function LALSlaveWaveburstSearch, file ../../../../contrib/waveburst/src/Search.c, line 210 Level 3: $Id Status code 101: cannot open /dso-test/waveburst/usertag/timeinterval/etg function LALOutputWaveburst, file ../../../../contrib/waveburst/src/Output.c, line 1037 }
07/18/07-14:57:04 PDT 07/18/07-21:57:04 GMT 868831038 LDAS-TEST22923320 mpi::filterLogEntries 22923320.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) Rank2(metaserver) Rank3(metaserver) }
07/18/07-14:57:44 PDT 07/18/07-21:57:44 GMT 868831078 LDAS-TEST22923320 mpi::filterLogEntries 22923320.1:progress 100.00%
07/18/07-14:57:44 PDT 07/18/07-21:57:44 GMT 868831078 LDAS-TEST22923320 mpi::filterLogEntries 22923320.1:projected_ratio 0.01128
07/18/07-14:57:44 PDT 07/18/07-21:57:44 GMT 868831078 LDAS-TEST22923320 jobstate response: '22923320.1:cont'
07/18/07-14:57:45 PDT 07/18/07-21:57:45 GMT 868831079 LDAS-TEST22923320 mpi::filterLogEntries 22923320.2:progress 100.00%
07/18/07-14:57:45 PDT 07/18/07-21:57:45 GMT 868831079 LDAS-TEST22923320 mpi::filterLogEntries 22923320.2:projected_ratio 0.02232
07/18/07-14:57:45 PDT 07/18/07-21:57:45 GMT 868831079 LDAS-TEST22923320 mpi::filterLogEntries 22923320.3:finalizing { totalTime=40.9899 sentObjects=4; Rank0 (inPut: create=0.68189 send=1.15279 ilwd=0.53750; outPut: create=0.03232 send=0.09528 ilwd=0.08874; metadata: (un)pack=0.00031 bcast=0.03452; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12035; max memory=335872) Rank3 (inPut: create=0.92098 send=0.98358; outPut: create=0.00000 send=0.01591; metadata: (un)pack=0.00011 bcast=0.00365; IS=0.42110; CD=0.23499; AS=1.01093; FO=0.00000; FS=0.00217; LB=0.00000; max memory=729088) Rank1 (inPut: create=0.92553 send=0.49129; outPut: create=0.01586 send=0.09276; metadata: (un)pack=0.00017 bcast=0.02134; IS=0.00015; CD=0.00005; AS=36.71918; FO=0.00004; FS=0.00006; LB=0.03192; max memory=413696) Rank2 (inPut: create=0.92536 send=1.19558; outPut: create=0.03197 send=0.35246; metadata: (un)pack=0.00025 bcast=0.03611; IS=0.42134; CD=0.24288; AS=36.54305; FO=0.00008; FS=0.02505; LB=0.00001; max memory=729088) }
07/18/07-14:57:57 PDT 07/18/07-21:57:57 GMT 868831091 LDAS-TEST22923321 mpi::filterLogEntries 22923321.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
07/18/07-14:57:58 PDT 07/18/07-21:57:58 GMT 868831092 LDAS-TEST22923321 mpi::filterLogEntries 22923321.1:progress 100.00%
07/18/07-14:57:58 PDT 07/18/07-21:57:58 GMT 868831092 LDAS-TEST22923321 mpi::filterLogEntries 22923321.1:projected_ratio 0.00097
07/18/07-14:57:58 PDT 07/18/07-21:57:58 GMT 868831092 LDAS-TEST22923321 mpi::filterLogEntries 22923321.2:finalizing { totalTime=0.521042 sentObjects=3; Rank0 (inPut: create=0.04905 send=0.00632 ilwd=0.04229; outPut: create=0.01838 send=0.04953 ilwd=0.04439; metadata: (un)pack=0.00018 bcast=0.01698; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01600; max memory=3932160) Rank1 (inPut: create=0.04637 send=0.00904; outPut: create=0.01595 send=0.05052; metadata: (un)pack=0.00015 bcast=0.01762; IS=0.00011; CD=0.00874; AS=0.05831; FO=0.00020; FS=0.00004; LB=0.03830; max memory=5165056) }
07/18/07-14:58:18 PDT 07/18/07-21:58:18 GMT 868831112 LDAS-TEST22923322 mpi::filterLogEntries 22923322.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
07/18/07-14:58:20 PDT 07/18/07-21:58:20 GMT 868831114 LDAS-TEST22923322 mpi::filterLogEntries 22923322.1:progress 100.00%
07/18/07-14:58:20 PDT 07/18/07-21:58:20 GMT 868831114 LDAS-TEST22923322 mpi::filterLogEntries 22923322.1:projected_ratio 0.00018
07/18/07-14:58:20 PDT 07/18/07-21:58:20 GMT 868831114 LDAS-TEST22923322 mpi::filterLogEntries 22923322.2:finalizing { totalTime=1.9478 sentObjects=3; Rank0 (inPut: create=0.43182 send=0.16899 ilwd=0.34228; outPut: create=0.08626 send=0.46934 ilwd=0.04276; metadata: (un)pack=0.00012 bcast=0.01702; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01999; max memory=0) Rank1 (inPut: create=0.48543 send=0.19975; outPut: create=0.01590 send=0.50722; metadata: (un)pack=0.00011 bcast=0.01772; IS=0.00001; CD=0.00000; AS=0.06424; FO=0.00332; FS=0.00000; LB=0.13254; max memory=184320) }
07/18/07-14:59:04 PDT 07/18/07-21:59:04 GMT 868831158 LDAS-TEST22923323 mpi::filterLogEntries 22923323.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
07/18/07-14:59:05 PDT 07/18/07-21:59:05 GMT 868831159 LDAS-TEST22923323 mpi::filterLogEntries 22923323.1:progress 100.00%
07/18/07-14:59:05 PDT 07/18/07-21:59:05 GMT 868831159 LDAS-TEST22923323 mpi::filterLogEntries 22923323.1:projected_ratio 0.00000
07/18/07-14:59:05 PDT 07/18/07-21:59:05 GMT 868831159 LDAS-TEST22923323 mpi::filterLogEntries 22923323.2:finalizing { totalTime=0.488818 sentObjects=2; Rank0 (inPut: create=0.18090 send=0.00069 ilwd=0.17589; outPut: create=0.00000 send=0.01555 ilwd=0.04352; metadata: (un)pack=0.00010 bcast=0.00112; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01599; max memory=0) Rank1 (inPut: create=0.17618 send=0.00143; outPut: create=0.00000 send=0.04642; metadata: (un)pack=0.00016 bcast=0.00193; IS=0.00013; CD=0.00057; AS=0.00014; FO=0.00000; FS=0.00004; LB=0.03190; max memory=651264) }
07/18/07-14:59:19 PDT 07/18/07-21:59:19 GMT 868831173 LDAS-TEST22923324 mpi::filterLogEntries 22923324.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
07/18/07-14:59:33 PDT 07/18/07-21:59:33 GMT 868831187 LDAS-TEST22923324 mpi::filterLogEntries 22923324.1:progress 68.37%
07/18/07-14:59:33 PDT 07/18/07-21:59:33 GMT 868831187 LDAS-TEST22923324 mpi::filterLogEntries 22923324.1:projected_ratio 0.03568
07/18/07-14:59:33 PDT 07/18/07-21:59:33 GMT 868831187 LDAS-TEST22923324 jobstate response: '22923324.1:cont'
07/18/07-14:59:34 PDT 07/18/07-21:59:34 GMT 868831188 LDAS-TEST22923324 mpi::filterLogEntries 22923324.2:progress 100.00%
07/18/07-14:59:34 PDT 07/18/07-21:59:34 GMT 868831188 LDAS-TEST22923324 mpi::filterLogEntries 22923324.2:projected_ratio 0.03722
07/18/07-14:59:34 PDT 07/18/07-21:59:34 GMT 868831188 LDAS-TEST22923324 mpi::filterLogEntries 22923324.3:finalizing { totalTime=14.9286 sentObjects=4; Rank0 (inPut: create=0.43322 send=0.17089 ilwd=0.34372; outPut: create=0.09489 send=0.09922 ilwd=0.32933; metadata: (un)pack=0.01198 bcast=0.06436; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.13057; max memory=5664768) Rank1 (inPut: create=0.48266 send=0.14322; outPut: create=0.07355 send=0.11550; metadata: (un)pack=0.01058 bcast=0.05011; IS=0.00010; CD=0.06290; AS=13.40085; FO=0.00210; FS=0.00368; LB=0.05126; max memory=352256) }
07/18/07-14:59:50 PDT 07/18/07-21:59:50 GMT 868831204 LDAS-TEST22923325 mpi::filterLogEntries 22923325.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
07/18/07-14:59:50 PDT 07/18/07-21:59:50 GMT 868831204 LDAS-TEST22923325 mpi::filterLogEntries 22923325.1:progress 100.00%
07/18/07-14:59:50 PDT 07/18/07-21:59:50 GMT 868831204 LDAS-TEST22923325 mpi::filterLogEntries 22923325.1:projected_ratio 0.00001
07/18/07-14:59:51 PDT 07/18/07-21:59:51 GMT 868831205 LDAS-TEST22923325 mpi::filterLogEntries 22923325.2:finalizing { totalTime=0.439639 sentObjects=3; Rank0 (inPut: create=0.05144 send=0.00287 ilwd=0.04505; outPut: create=0.01629 send=0.03114 ilwd=0.04528; metadata: (un)pack=0.00025 bcast=0.01697; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01599; max memory=0) Rank1 (inPut: create=0.04774 send=0.00453; outPut: create=0.01597 send=0.06318; metadata: (un)pack=0.00028 bcast=0.01754; IS=0.00011; CD=0.00229; AS=0.00058; FO=0.00006; FS=0.00005; LB=0.03188; max memory=2060288) }
07/18/07-15:00:29 PDT 07/18/07-22:00:29 GMT 868831243 LDAS-TEST22923326 mpi::filterLogEntries 22923326.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) }
07/18/07-15:00:31 PDT 07/18/07-22:00:31 GMT 868831245 LDAS-TEST22923326 mpi::filterLogEntries 22923326.1:progress 0.00%
07/18/07-15:00:31 PDT 07/18/07-22:00:31 GMT 868831245 LDAS-TEST22923326 mpi::filterLogEntries 22923326.1:projected_ratio 0.00015
07/18/07-15:00:31 PDT 07/18/07-22:00:31 GMT 868831245 LDAS-TEST22923326 mpi::filterLogEntries 22923326.2:progress 10.00%
07/18/07-15:00:31 PDT 07/18/07-22:00:31 GMT 868831245 LDAS-TEST22923326 mpi::filterLogEntries 22923326.2:projected_ratio 0.00030
07/18/07-15:00:31 PDT 07/18/07-22:00:31 GMT 868831245 LDAS-TEST22923326 mpi::filterLogEntries 22923326.3:progress 20.00%
07/18/07-15:00:31 PDT 07/18/07-22:00:31 GMT 868831245 LDAS-TEST22923326 mpi::filterLogEntries 22923326.3:projected_ratio 0.00045
07/18/07-15:00:31 PDT 07/18/07-22:00:31 GMT 868831245 LDAS-TEST22923326 mpi::filterLogEntries 22923326.4:progress 30.00%
07/18/07-15:00:31 PDT 07/18/07-22:00:31 GMT 868831245 LDAS-TEST22923326 mpi::filterLogEntries 22923326.4:projected_ratio 0.00060
07/18/07-15:00:31 PDT 07/18/07-22:00:31 GMT 868831245 LDAS-TEST22923326 mpi::filterLogEntries 22923326.5:progress 40.00%
07/18/07-15:00:31 PDT 07/18/07-22:00:31 GMT 868831245 LDAS-TEST22923326 mpi::filterLogEntries 22923326.5:projected_ratio 0.00075
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.6:progress 50.00%
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.6:projected_ratio 0.00090
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.7:progress 60.00%
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.7:projected_ratio 0.00105
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 jobstate response: '22923326.7:cont'
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.8:progress 70.00%
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.8:projected_ratio 0.00123
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.9:progress 80.00%
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.9:projected_ratio 0.00138
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.10:progress 90.00%
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.10:projected_ratio 0.00153
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.11:progress 100.00%
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.11:projected_ratio 0.00153
07/18/07-15:00:32 PDT 07/18/07-22:00:32 GMT 868831246 LDAS-TEST22923326 mpi::filterLogEntries 22923326.12:finalizing { totalTime=3.06411 sentObjects=3; Rank0 (inPut: create=0.34442 send=0.10268 ilwd=0.23289; outPut: create=0.16046 send=0.56348 ilwd=0.02425; metadata: (un)pack=0.00041 bcast=0.21375; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.23045; max memory=540672) Rank1 (inPut: create=0.37708 send=0.08918; outPut: create=0.16018 send=0.62931; metadata: (un)pack=0.00036 bcast=0.18058; IS=0.01327; CD=0.10597; AS=1.34096; FO=0.00035; FS=0.00260; LB=0.03197; max memory=18403328) }
07/18/07-15:00:52 PDT 07/18/07-22:00:52 GMT 868831266 LDAS-TEST22923327 mpi::filterLogEntries 22923327.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) Rank2(metaserver) Rank3(metaserver) }
07/18/07-15:00:56 PDT 07/18/07-22:00:56 GMT 868831270 LDAS-TEST22923327 mpi::filterLogEntries 22923327.1:progress 0.00%
07/18/07-15:00:56 PDT 07/18/07-22:00:56 GMT 868831270 LDAS-TEST22923327 mpi::filterLogEntries 22923327.1:projected_ratio 0.00001
07/18/07-15:00:56 PDT 07/18/07-22:00:56 GMT 868831270 LDAS-TEST22923327 jobstate response: '22923327.1:cont'
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.2:progress 0.00%
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.2:projected_ratio 0.00044
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.3:progress 25.00%
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.3:projected_ratio 0.00087
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.4:progress 50.00%
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.4:projected_ratio 0.00144
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.5:progress 50.00%
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.5:projected_ratio 0.00202
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.6:progress 50.00%
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.6:projected_ratio 0.00234
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.7:progress 50.00%
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.7:projected_ratio 0.00234
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.8:progress 75.00%
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.8:projected_ratio 0.00256
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.9:progress 75.00%
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.9:projected_ratio 0.00296
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.10:progress 75.00%
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.10:projected_ratio 0.00296
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.11:progress 100.00%
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.11:projected_ratio 0.00301
07/18/07-15:00:58 PDT 07/18/07-22:00:58 GMT 868831272 LDAS-TEST22923327 mpi::filterLogEntries 22923327.12:finalizing { totalTime=5.44814 sentObjects=3; Rank0 (inPut: create=0.43536 send=0.63749 ilwd=0.34219; outPut: create=0.04102 send=0.32383 ilwd=0.04244; metadata: (un)pack=0.00350 bcast=0.03461; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12483; max memory=1798144) Rank2 (inPut: create=0.54085 send=0.52226; outPut: create=0.00000 send=0.15206; metadata: (un)pack=0.00008 bcast=0.00358; IS=0.00012; CD=0.78101; AS=0.57486; FO=0.00000; FS=0.00453; LB=0.00001; max memory=1110016) Rank3 (inPut: create=0.59846 send=0.79255; outPut: create=0.00000 send=0.20763; metadata: (un)pack=0.00006 bcast=0.00545; IS=0.00013; CD=1.22073; AS=0.68751; FO=0.00000; FS=0.00452; LB=0.00001; max memory=1064960) Rank1 (inPut: create=0.54361 send=0.66601; outPut: create=0.03756 send=0.48106; metadata: (un)pack=0.00298 bcast=0.03480; IS=0.00012; CD=1.56997; AS=0.47950; FO=0.00060; FS=0.00354; LB=0.05526; max memory=344064) }
07/18/07-15:01:20 PDT 07/18/07-22:01:20 GMT 868831294 LDAS-TEST22923328 mpi::filterLogEntries 22923328.0:initializing { Master-Rank0(metaserver) Rank1(metaserver) Rank2(metaserver) }
07/18/07-15:01:35 PDT 07/18/07-22:01:35 GMT 868831309 LDAS-TEST22923328 mpi::filterLogEntries 22923328.1:error {Rank 2 failed with error message: ( applySearch function ): Level 1: $Id Status code -1: Recursive error function LALApplySearch, file ../../../../contrib/waveburst/src/Search.c, line 1075 Level 2: $Id Status code -1: Recursive error function LALSlaveWaveburstSearch, file ../../../../contrib/waveburst/src/Search.c, line 210 Level 3: $Id Status code 101: cannot open /dso-test/waveburst/usertag/timeinterval/etg function LALOutputWaveburst, file ../../../../contrib/waveburst/src/Output.c, line 1037 }
07/18/07-15:01:35 PDT 07/18/07-22:01:35 GMT 868831309 LDAS-TEST22923328 mpi::newJobCallback {Rank 2 failed with error message: ( applySearch function ): Level 1: $Id Status code -1: Recursive error function LALApplySearch, file ../../../../contrib/waveburst/src/Search.c, line 1075 Level 2: $Id Status code -1: Recursive error function LALSlaveWaveburstSearch, file ../../../../contrib/waveburst/src/Search.c, line 210 Level 3: $Id Status code 101: cannot open /dso-test/waveburst/usertag/timeinterval/etg function LALOutputWaveburst, file ../../../../contrib/waveburst/src/Output.c, line 1037 }