Class Kickstart
- java.lang.Object
-
- edu.isi.pegasus.planner.code.gridstart.Kickstart
-
- All Implemented Interfaces:
GridStart
public class Kickstart extends java.lang.Object implements GridStart
This enables a constituentJob to be run on the grid, by launching it through kickstart. The kickstart executable is a light-weight program which connects the stdin, stdout and stderr filehandles for Pegasus jobs on the remote site.Sitting in between the remote scheduler and the executable, it is possible for kickstart to gather additional information about the executable run-time behavior, including the exit status of jobs.
Kickstart is an executable distributed with Pegasus that can generally be found at $PEGASUS_HOME/bin/kickstart
- Version:
- $Revision$
- Author:
- Karan Vahi vahi@isi.edu
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
CLASSNAME
The basename of the class that is implmenting this.static java.lang.String
COMPLETE_TRANSFORMATION_NAME
The complete TC name for kickstart.static java.lang.String
EXECUTABLE_BASENAME
The basename of the kickstart executable.static java.lang.String
KICKSTART_CLEANUP
The environment variable used to the set Kickstart CLEANUP JOB.static java.lang.String
KICKSTART_INPUT_SUFFIX
The suffix for the kickstart input file, that is generated to use invoke at the remote end.static java.lang.String
KICKSTART_POSTJOB
The environment variable used to the set Kickstart POSTJOB.static java.lang.String
KICKSTART_PREJOB
The environment variable used to the set Kickstart PREJOB.static java.lang.String
KICKSTART_SETUP
The environment variable used to the set Kickstart SETUP JOB.private ADag
mConcDAG
The handle to the workflow that is being enabled.private boolean
mDisableInvokeFunctionality
Boolean indicating whether to disable invoke functionality.private boolean
mDoStat
A boolean indicating whether to stat files or not.private boolean
mDynamicDeployment
A boolean indicating whether kickstart is deployed dynamically or not.private boolean
mEnablingPartOfAggregatedJob
An instance variable to track if enabling is happening as part of a clustered constituentJob.private boolean
mGenerateLOF
A boolean indicating whether to generate lof files or not.private boolean
mInvokeAlways
A boolean indicating whether to use invoke always or not.private long
mInvokeLength
The invoke limit trigger.private java.lang.String
mKickstartLabel
The label that is passed to kickstart.private LogManager
mLogger
The LogManager object which is used to log all the messages.private GridStart
mNoGridStartImpl
Handle to NoGridStart implementation.private PlannerOptions
mPOptions
The options passed to the planner.private PegasusProperties
mProps
The object holding all the properties pertaining to Pegasus.private boolean
mSetXBit
Whether kickstart should set the X Bit on the staged executables.private SiteStore
mSiteStore
Handle to the site catalog store.private SLS
mSLS
The handle to the SLS implementorprivate java.lang.String
mSubmitDir
The submit exectionSiteDirectory where the submit files are being generated for the workflow.private TransformationCatalog
mTCHandle
Handle to Transformation Catalog.private boolean
mUseFullPathToGridStart
Boolean indicating whether to use full path or not(package private) boolean
mWorkerNodeExecution
A boolean indicating whether to have worker node execution or not.static java.lang.String
SHORT_NAME
The SHORTNAME for this implementation.static java.lang.String
TRANSFORMATION_NAME
The logical name of kickstartstatic java.lang.String
TRANSFORMATION_NAMESPACE
The transformation namespace for the kickstartstatic java.lang.String
TRANSFORMATION_VERSION
The version number for kickstart.-
Fields inherited from interface edu.isi.pegasus.planner.code.GridStart
mSeparator, VERSION
-
-
Constructor Summary
Constructors Constructor Description Kickstart()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description private void
addCleanupPostScript(Job job, java.util.List files)
Adds a /bin/rm post constituentJob to kickstart that removes the files passed.boolean
canSetXBit()
Indicates whether the enabling mechanism can set the X bit on the executable on the remote grid site, in addition to launching it on the remote grid site.private void
construct(Job job, java.lang.String key, java.lang.String value)
Constructs a condor variable in the condor profile namespace associated with the constituentJob.protected java.lang.String
constructCleanupJob(Job job, java.lang.String workerNodeTmp)
Constructs a kickstart setup constituentJobprotected java.lang.String
constructPREJob(Job job, java.lang.String headNodeURLPrefix, java.lang.String headNodeDirectory, java.lang.String workerNodeDirectory, java.lang.String slsFile)
Constructs the prejob that fetches sls file, and then invokes transfer again.protected java.lang.String
constructSetupJob(Job job, java.lang.String workerNodeTmp)
Constructs a kickstart setup constituentJobjava.lang.String
defaultPOSTScript()
Returns the SHORT_NAME for the POSTScript implementation that is used to be as default with this GridStart implementation.boolean
enable(AggregatedJob job, boolean isGlobusJob)
Enables a constituentJob to run on the grid.boolean
enable(Job job, boolean isGlobusJob)
Enables a constituentJob to run on the grid by launching it through kickstart.protected boolean
enable(Job job, boolean isGlobusJob, boolean stat, boolean addPostScript, boolean partOfClusteredJob)
Enables a constituentJob to run on the grid by launching it through kickstart.java.lang.String
generateListofFilenamesFile(java.util.Set files, java.lang.String basename)
Writes out the list of filenames file for the constituentJob.protected java.lang.String
getDirectory(Job job)
Returns the directory in which the job should run.private java.lang.String
getDirectoryKey(Job job)
Returns the exectionSiteDirectory that is associated with the constituentJob to specify the exectionSiteDirectory in which the constituentJob needs to runjava.lang.String
getKickstartPath(SiteCatalogEntry site)
Returns the default path to kickstart as constructed from the environment variable associated with a site in the site catalogTransformationCatalogEntry
getTransformationCatalogEntry(java.lang.String site)
Returns the transformation catalog entry for kickstart on a sitejava.lang.String
getVDSKeyValue()
Returns the value of the vds profile with key as Pegasus.GRIDSTART_KEY, that would result in the loading of this particular implementation.java.lang.String
getWorkerNodeDirectory(Job job)
Returns the exectionSiteDirectory in which the constituentJob executes on the worker node.protected java.lang.String
handleTransferOfExecutable(Job job, java.lang.String path)
It changes the paths to the executable depending on whether we want to transfer the executable or not.void
initialize(PegasusBag bag, ADag dag)
Initializes the GridStart implementation.private java.lang.String
quote(java.lang.String string)
Condor Quotes a stringprotected boolean
requiresToSetDirectory(Job job)
Returns a boolean indicating whether we need to set the directory for the job or not.java.lang.String
shortDescribe()
Returns a short textual description in the form of the name of the class.void
useFullPathToGridStarts(boolean fullPath)
Setter method to control whether a full path to Gridstart should be returned while wrapping a job or not.private boolean
useInvoke(Job job, java.lang.String executable, java.lang.StringBuffer args)
Triggers the creation of the kickstart input file, that contains the the remote executable and the arguments with which it has to be invoked.
-
-
-
Field Detail
-
TRANSFORMATION_NAMESPACE
public static final java.lang.String TRANSFORMATION_NAMESPACE
The transformation namespace for the kickstart- See Also:
- Constant Field Values
-
TRANSFORMATION_NAME
public static final java.lang.String TRANSFORMATION_NAME
The logical name of kickstart- See Also:
- Constant Field Values
-
TRANSFORMATION_VERSION
public static final java.lang.String TRANSFORMATION_VERSION
The version number for kickstart.
-
EXECUTABLE_BASENAME
public static final java.lang.String EXECUTABLE_BASENAME
The basename of the kickstart executable.- See Also:
- Constant Field Values
-
COMPLETE_TRANSFORMATION_NAME
public static final java.lang.String COMPLETE_TRANSFORMATION_NAME
The complete TC name for kickstart.
-
KICKSTART_INPUT_SUFFIX
public static final java.lang.String KICKSTART_INPUT_SUFFIX
The suffix for the kickstart input file, that is generated to use invoke at the remote end.- See Also:
- Constant Field Values
-
CLASSNAME
public static final java.lang.String CLASSNAME
The basename of the class that is implmenting this. Could have been determined by reflection.- See Also:
- Constant Field Values
-
SHORT_NAME
public static final java.lang.String SHORT_NAME
The SHORTNAME for this implementation.- See Also:
- Constant Field Values
-
KICKSTART_SETUP
public static final java.lang.String KICKSTART_SETUP
The environment variable used to the set Kickstart SETUP JOB.- See Also:
- Constant Field Values
-
KICKSTART_PREJOB
public static final java.lang.String KICKSTART_PREJOB
The environment variable used to the set Kickstart PREJOB.- See Also:
- Constant Field Values
-
KICKSTART_POSTJOB
public static final java.lang.String KICKSTART_POSTJOB
The environment variable used to the set Kickstart POSTJOB.- See Also:
- Constant Field Values
-
KICKSTART_CLEANUP
public static final java.lang.String KICKSTART_CLEANUP
The environment variable used to the set Kickstart CLEANUP JOB.- See Also:
- Constant Field Values
-
mLogger
private LogManager mLogger
The LogManager object which is used to log all the messages.
-
mProps
private PegasusProperties mProps
The object holding all the properties pertaining to Pegasus.
-
mPOptions
private PlannerOptions mPOptions
The options passed to the planner.
-
mConcDAG
private ADag mConcDAG
The handle to the workflow that is being enabled.
-
mSiteStore
private SiteStore mSiteStore
Handle to the site catalog store.
-
mTCHandle
private TransformationCatalog mTCHandle
Handle to Transformation Catalog.
-
mSubmitDir
private java.lang.String mSubmitDir
The submit exectionSiteDirectory where the submit files are being generated for the workflow.
-
mInvokeAlways
private boolean mInvokeAlways
A boolean indicating whether to use invoke always or not.
-
mDoStat
private boolean mDoStat
A boolean indicating whether to stat files or not.
-
mGenerateLOF
private boolean mGenerateLOF
A boolean indicating whether to generate lof files or not.
-
mInvokeLength
private long mInvokeLength
The invoke limit trigger.
-
mWorkerNodeExecution
boolean mWorkerNodeExecution
A boolean indicating whether to have worker node execution or not.
-
mSLS
private SLS mSLS
The handle to the SLS implementor
-
mEnablingPartOfAggregatedJob
private boolean mEnablingPartOfAggregatedJob
An instance variable to track if enabling is happening as part of a clustered constituentJob. See Bug 21 comments on Pegasus Bugzilla
-
mDynamicDeployment
private boolean mDynamicDeployment
A boolean indicating whether kickstart is deployed dynamically or not.
-
mKickstartLabel
private java.lang.String mKickstartLabel
The label that is passed to kickstart.
-
mSetXBit
private boolean mSetXBit
Whether kickstart should set the X Bit on the staged executables.
-
mNoGridStartImpl
private GridStart mNoGridStartImpl
Handle to NoGridStart implementation.
-
mUseFullPathToGridStart
private boolean mUseFullPathToGridStart
Boolean indicating whether to use full path or not
-
mDisableInvokeFunctionality
private boolean mDisableInvokeFunctionality
Boolean indicating whether to disable invoke functionality.
-
-
Method Detail
-
initialize
public void initialize(PegasusBag bag, ADag dag)
Initializes the GridStart implementation.- Specified by:
initialize
in interfaceGridStart
- Parameters:
bag
- the bag of objects that is used for initialization.dag
- the concrete dag so far.
-
useFullPathToGridStarts
public void useFullPathToGridStarts(boolean fullPath)
Setter method to control whether a full path to Gridstart should be returned while wrapping a job or not.- Specified by:
useFullPathToGridStarts
in interfaceGridStart
- Parameters:
fullPath
- if set to true, indicates that full path would be used.
-
enable
public boolean enable(AggregatedJob job, boolean isGlobusJob)
Enables a constituentJob to run on the grid. This also determines how the stdin,stderr and stdout of the constituentJob are to be propogated. To grid enable a constituentJob, the constituentJob may need to be wrapped into another constituentJob, that actually launches the constituentJob. It usually results in the constituentJob description passed being modified modified.- Specified by:
enable
in interfaceGridStart
- Parameters:
constituentJob
- theJob
object containing the constituentJob description of the constituentJob that has to be enabled on the grid.isGlobusJob
- istrue
, if the constituentJob generated a lineuniverse = globus
, and thus runs remotely. Set tofalse
, if the constituentJob runs on the submit host in any way.- Returns:
- boolean true if enabling was successful,else false.
-
enable
public boolean enable(Job job, boolean isGlobusJob)
Enables a constituentJob to run on the grid by launching it through kickstart. Does the stdio, and stderr handling of the constituentJob to be run on the grid. It modifies the constituentJob description, and also constructs all the valid option to be passed to kickstart for launching the executable.- Specified by:
enable
in interfaceGridStart
- Parameters:
constituentJob
- theJob
object containing the constituentJob description of the constituentJob that has to be enabled on the grid.isGlobusJob
- istrue
, if the constituentJob generated a lineuniverse = globus
, and thus runs remotely. Set tofalse
, if the constituentJob runs on the submit host in any way.- Returns:
- boolean true if enabling was successful,else false in case when the path to kickstart could not be determined on the site where the constituentJob is scheduled.
-
enable
protected boolean enable(Job job, boolean isGlobusJob, boolean stat, boolean addPostScript, boolean partOfClusteredJob)
Enables a constituentJob to run on the grid by launching it through kickstart. Does the stdio, and stderr handling of the constituentJob to be run on the grid. It modifies the constituentJob description, and also constructs all the valid option to be passed to kickstart for launching the executable.- Parameters:
constituentJob
- theJob
object containing the constituentJob description of the constituentJob that has to be enabled on the grid.isGlobusJob
- istrue
, if the constituentJob generated a lineuniverse = globus
, and thus runs remotely. Set tofalse
, if the constituentJob runs on the submit host in any way.stat
- boolean indicating whether to generate the lof files for kickstart stat option or not.addPostScript
- boolean indicating whether to add a postscript or not.partOfClusteredJob
- boolean indicating whether the job being enabled is part of a clustered job or not.- Returns:
- boolean true if enabling was successful,else false in case when the path to kickstart could not be determined on the site where the constituentJob is scheduled.
-
handleTransferOfExecutable
protected java.lang.String handleTransferOfExecutable(Job job, java.lang.String path)
It changes the paths to the executable depending on whether we want to transfer the executable or not. If the transfer_executable is set to true, then the executable needs to be shipped from the submit host meaning the local pool. This function changes the path of the executable to the one on the local pool, so that it can be shipped. If the worker package is being deployed dynamically, then the path is set to the exectionSiteDirectory where the worker package is deployed. Else, we pick up the path from the site catalog that is passed as input- Parameters:
constituentJob
- theJob
containing the constituentJob description.path
- the path to kickstart on the remote compute site, as determined from the site catalog.- Returns:
- the path that needs to be set as the executable
-
getTransformationCatalogEntry
public TransformationCatalogEntry getTransformationCatalogEntry(java.lang.String site)
Returns the transformation catalog entry for kickstart on a site- Parameters:
site
- the site on which the entry is required- Returns:
- the entry if found else null
-
getKickstartPath
public java.lang.String getKickstartPath(SiteCatalogEntry site)
Returns the default path to kickstart as constructed from the environment variable associated with a site in the site catalog- Parameters:
site
- the SiteCatalogEntry object for the site.- Returns:
- value if set else null
-
getWorkerNodeDirectory
public java.lang.String getWorkerNodeDirectory(Job job)
Returns the exectionSiteDirectory in which the constituentJob executes on the worker node.- Specified by:
getWorkerNodeDirectory
in interfaceGridStart
- Parameters:
constituentJob
-- Returns:
- the full path to the exectionSiteDirectory where the constituentJob executes
-
canSetXBit
public boolean canSetXBit()
Indicates whether the enabling mechanism can set the X bit on the executable on the remote grid site, in addition to launching it on the remote grid site.- Specified by:
canSetXBit
in interfaceGridStart
- Returns:
- true indicating Kickstart can set the X bit or not.
-
getVDSKeyValue
public java.lang.String getVDSKeyValue()
Returns the value of the vds profile with key as Pegasus.GRIDSTART_KEY, that would result in the loading of this particular implementation. It is usually the name of the implementing class without the package name.- Specified by:
getVDSKeyValue
in interfaceGridStart
- Returns:
- the value of the profile key.
- See Also:
org.griphyn.cPlanner.namespace.Pegasus#GRIDSTART_KEY
-
shortDescribe
public java.lang.String shortDescribe()
Returns a short textual description in the form of the name of the class.- Specified by:
shortDescribe
in interfaceGridStart
- Returns:
- short textual description.
-
defaultPOSTScript
public java.lang.String defaultPOSTScript()
Returns the SHORT_NAME for the POSTScript implementation that is used to be as default with this GridStart implementation.- Specified by:
defaultPOSTScript
in interfaceGridStart
- Returns:
- the identifier for the PegasusExitCode POSTScript implementation.
- See Also:
POSTScript.shortDescribe()
-
requiresToSetDirectory
protected boolean requiresToSetDirectory(Job job)
Returns a boolean indicating whether we need to set the directory for the job or not.- Parameters:
job
- the job for which to set directory.- Returns:
-
getDirectory
protected java.lang.String getDirectory(Job job)
Returns the directory in which the job should run.- Parameters:
job
- the job in which the directory has to run.- Returns:
-
getDirectoryKey
private java.lang.String getDirectoryKey(Job job)
Returns the exectionSiteDirectory that is associated with the constituentJob to specify the exectionSiteDirectory in which the constituentJob needs to run- Parameters:
constituentJob
- the constituentJob- Returns:
- the condor key . can be initialdir or remote_initialdir
-
useInvoke
private boolean useInvoke(Job job, java.lang.String executable, java.lang.StringBuffer args)
Triggers the creation of the kickstart input file, that contains the the remote executable and the arguments with which it has to be invoked. The kickstart input file is created in the submit directory.- Parameters:
constituentJob
- theJob
object containing the constituentJob description.executable
- the path to the executable used.args
- the arguments buffer for gridstart invocation so far.- Returns:
- boolean indicating whether kickstart input file was generated or not. false in case of any error.
-
constructSetupJob
protected java.lang.String constructSetupJob(Job job, java.lang.String workerNodeTmp)
Constructs a kickstart setup constituentJob- Parameters:
constituentJob
- the constituentJob to be run.workerNodeTmp
- the worker node tmp to run the constituentJob in.- Returns:
- String
-
constructCleanupJob
protected java.lang.String constructCleanupJob(Job job, java.lang.String workerNodeTmp)
Constructs a kickstart setup constituentJob- Parameters:
constituentJob
- the constituentJob to be run.workerNodeTmp
- the worker node tmp to run the constituentJob in.- Returns:
- String
-
constructPREJob
protected java.lang.String constructPREJob(Job job, java.lang.String headNodeURLPrefix, java.lang.String headNodeDirectory, java.lang.String workerNodeDirectory, java.lang.String slsFile)
Constructs the prejob that fetches sls file, and then invokes transfer again.- Parameters:
constituentJob
- the constituentJob for which the prejob is being createdheadNodeURLPrefix
- StringheadNodeDirectory
- StringworkerNodeDirectory
- StringslsFile
- String- Returns:
- String containing the prescript invocation
-
generateListofFilenamesFile
public java.lang.String generateListofFilenamesFile(java.util.Set files, java.lang.String basename)
Writes out the list of filenames file for the constituentJob.- Parameters:
files
- the list ofPegasusFile
objects contains the files whose stat information is required.basename
- the basename of the file that is to be created- Returns:
- the full path to lof file created, else null if no file is written out.
-
construct
private void construct(Job job, java.lang.String key, java.lang.String value)
Constructs a condor variable in the condor profile namespace associated with the constituentJob. Overrides any preexisting key values.- Parameters:
constituentJob
- contains the constituentJob description.key
- the key of the profile.value
- the associated value.
-
quote
private java.lang.String quote(java.lang.String string)
Condor Quotes a string- Parameters:
string
- the string to be quoted.- Returns:
- quoted string.
-
addCleanupPostScript
private void addCleanupPostScript(Job job, java.util.List files)
Adds a /bin/rm post constituentJob to kickstart that removes the files passed. The post jobs is added as an environment variable.- Parameters:
constituentJob
- the constituentJob in which the post constituentJob needs to be added.files
- the files to be deleted.
-
-