The mpi API on metaserver at Caltech-Test

Link: API Status Page for Caltech-Test

903132933 STARTUP archiveLog file "/ldas_outgoing/logs/LDASmpi.log.html" already closed. (archived as /ldas_outgoing/logs/archive/mpiAPI/LDASmpi.903132491)
903132933 STARTUP execOverload overloaded exec
903132933 STARTUP closeListenSock no cid registered for service 'data'
903132933 STARTUP mpi::init unused data port 10024 closed
903132933 STARTUP mpi::initNodes *** no slave nodes configured on this system ***
903132933 STARTUP mpi::init port 10024 (jobstate) opened on metaserver as sock5
903132933 STARTUP bgLoop Looping process watchlogs started
903132933 STARTUP openListenSock port 10022 (operator) opened on metaserver as sock6
903132933 STARTUP openListenSock port 10023 (emergency) opened on metaserver as sock7
903132933 STARTUP setResourceLimit vmemoryuse=unlimited; datasize=unlimited; core=unlimited; maxproc=32767; descriptors=1024; memorylocked=32768; filesize=unlimited; cputime=unlimited
903132933 STARTUP leakLogger inital size of mpi API: 12808 kB
903132933 STARTUP bgLoop Looping process etchosts started
903132933 STARTUP mpi Trying API
903132933 STARTUP mpi API yes
903132933 STARTUP mpi Trying LDAS_SYSTEM
903132933 STARTUP mpi LDAS_SYSTEM yes
903132933 IDLE bgLoop Looping process statpagefile started
903132933 IDLE bgLoop Looping process killedjobreaper started
903132933 IDLE bgLoop Looping process logrotate started
903132938 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.115.235') (::FTPDIR '') (::HTTPURL 'http://131.215.115.235/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas-test 131.215.115.235') (::LDAS_SYSTEM 'ldas-test') (::RUNCODE 'LDAS-TEST')
903132938 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.115.235') (::FTPDIR '') (::HTTPURL 'http://131.215.115.235/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas-test 131.215.115.235') (::LDAS_SYSTEM 'ldas-test') (::RUNCODE 'LDAS-TEST')
903132938 STARTUP mpi::killAllMpirun cleaning up for user ldas
903132940 STARTUP mpi::killAllMpirun ran kill 2 times in 1.700 seconds
903132940 STARTUP mpi::prestartLamds running lamboot for user search01
903132941 STARTUP mpi::prestartLamds running lamboot for user search02
903132942 STARTUP mpi::prestartLamds running lamboot for user search03
903132943 STARTUP mpi::prestartLamds running lamboot for user search04
903132944 STARTUP mpi::prestartLamds running lamboot for user search05
903132945 STARTUP mpi::prestartLamds running lamboot for user search06
903132946 STARTUP mpi::prestartLamds running lamboot for user search07
903132947 STARTUP mpi::prestartLamds running lamboot for user search08
903132948 STARTUP mpi::prestartLamds running lamboot for user search09
903132949 STARTUP mpi::prestartLamds running lamboot for user search10
903132950 STARTUP mpi::prestartLamds running lamboot for user search11
903132951 STARTUP mpi::prestartLamds running lamboot for user search12
903132952 STARTUP mpi::prestartLamds running lamboot for user search13
903132953 STARTUP mpi::prestartLamds running lamboot for user search14
903132954 STARTUP mpi::prestartLamds running lamboot for user search15
903132956 STARTUP mpi::prestartLamds running lamboot for user search16
903132957 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'metaserver'
903132957 STARTUP mpi::prestartLamds STARTUP search01 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search02 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search03 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search04 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search05 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search06 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search07 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search08 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search09 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search10 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search11 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search12 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search13 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search14 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 STARTUP mpi::prestartLamds STARTUP search15 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132957 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'metaserver'
903132958 STARTUP mpi::prestartLamds STARTUP search16 metaserver {lamboot errors: couldn't execute "recon": no such file or directory}
903132958 STARTUP mpi::killAllMpirun {ldas@metaserver:mpirun: sudo: sorry, you must have a tty to run sudo} {ldas@metaserver:wrapperAPI: sudo: sorry, you must have a tty to run sudo} {ldas@metaserver:lamd: sudo: sorry, you must have a tty to run sudo}
903133308 SHUTDOWN closeListenSock port 10022 (sock6) (operator) closed on metaserver
903133308 SHUTDOWN mpi::sHuTdOwN Subject: LDAS Caltech-Test mpi shutdown at 903133308 ( 08/18/2008 03:21:34 PM PDT ); Body: mpi shutting down NOW , nocore
903133308 search12 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search04 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search14 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search06 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search16 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search08 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search01 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search11 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search03 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search13 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search05 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search15 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search07 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search10 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search09 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 search02 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {couldn't open socket: connection refused}
903133308 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search01'
903133308 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133308 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search02'
903133309 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133309 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search03'
903133309 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133309 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search04'
903133309 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133309 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search05'
903133310 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133310 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search06'
903133310 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133310 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search07'
903133310 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133310 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search08'
903133311 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133311 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search09'
903133311 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133311 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search10'
903133311 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133311 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search11'
903133312 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133312 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search12'
903133312 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133312 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search13'
903133312 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133312 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search14'
903133313 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133313 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search15'
903133313 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133313 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search16'
903133313 SHUTDOWN lam::halt LAM 7.1.2/MPI 2 C++/ROMIO - Indiana University ----------------------------------------------------------------------------- It seems that there is no lamd running on the host metaserver. This indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "lamhalt" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. -----------------------------------------------------------------------------
903133313 SHUTDOWN closeListenSock port 10023 (sock7) (emergency) closed on metaserver
903133313 SHUTDOWN closeListenSock no cid registered for service 'data'
903133313 SHUTDOWN closeLog /ldas_outgoing/logs/LDASmpi.log.html (file4) closed