Class Kickstart

  • All Implemented Interfaces:
    GridStart

    public class Kickstart
    extends java.lang.Object
    implements GridStart
    This enables a constituentJob to be run on the grid, by launching it through kickstart. The kickstart executable is a light-weight program which connects the stdin, stdout and stderr filehandles for Pegasus jobs on the remote site.

    Sitting in between the remote scheduler and the executable, it is possible for kickstart to gather additional information about the executable run-time behavior, including the exit status of jobs.

    Kickstart is an executable distributed with Pegasus that can generally be found at $PEGASUS_HOME/bin/kickstart

    Version:
    $Revision$
    Author:
    Karan Vahi vahi@isi.edu
    • Field Summary

      Fields 
      Modifier and Type Field Description
      static java.lang.String CLASSNAME
      The basename of the class that is implmenting this.
      static java.lang.String COMPLETE_TRANSFORMATION_NAME
      The complete TC name for kickstart.
      static java.lang.String EXECUTABLE_BASENAME
      The basename of the kickstart executable.
      static java.lang.String KICKSTART_CLEANUP
      The environment variable used to the set Kickstart CLEANUP JOB.
      static java.lang.String KICKSTART_INPUT_SUFFIX
      The suffix for the kickstart input file, that is generated to use invoke at the remote end.
      static java.lang.String KICKSTART_POSTJOB
      The environment variable used to the set Kickstart POSTJOB.
      static java.lang.String KICKSTART_PREJOB
      The environment variable used to the set Kickstart PREJOB.
      static java.lang.String KICKSTART_SETUP
      The environment variable used to the set Kickstart SETUP JOB.
      private ADag mConcDAG
      The handle to the workflow that is being enabled.
      private boolean mDisableInvokeFunctionality
      Boolean indicating whether to disable invoke functionality.
      private boolean mDoStat
      A boolean indicating whether to stat files or not.
      private boolean mDynamicDeployment
      A boolean indicating whether kickstart is deployed dynamically or not.
      private boolean mEnablingPartOfAggregatedJob
      An instance variable to track if enabling is happening as part of a clustered constituentJob.
      private boolean mGenerateLOF
      A boolean indicating whether to generate lof files or not.
      private boolean mInvokeAlways
      A boolean indicating whether to use invoke always or not.
      private long mInvokeLength
      The invoke limit trigger.
      private java.lang.String mKickstartLabel
      The label that is passed to kickstart.
      private LogManager mLogger
      The LogManager object which is used to log all the messages.
      private GridStart mNoGridStartImpl
      Handle to NoGridStart implementation.
      private PlannerOptions mPOptions
      The options passed to the planner.
      private PegasusProperties mProps
      The object holding all the properties pertaining to Pegasus.
      private boolean mSetXBit
      Whether kickstart should set the X Bit on the staged executables.
      private SiteStore mSiteStore
      Handle to the site catalog store.
      private SLS mSLS
      The handle to the SLS implementor
      private java.lang.String mSubmitDir
      The submit exectionSiteDirectory where the submit files are being generated for the workflow.
      private TransformationCatalog mTCHandle
      Handle to Transformation Catalog.
      private boolean mUseFullPathToGridStart
      Boolean indicating whether to use full path or not
      (package private) boolean mWorkerNodeExecution
      A boolean indicating whether to have worker node execution or not.
      static java.lang.String SHORT_NAME
      The SHORTNAME for this implementation.
      static java.lang.String TRANSFORMATION_NAME
      The logical name of kickstart
      static java.lang.String TRANSFORMATION_NAMESPACE
      The transformation namespace for the kickstart
      static java.lang.String TRANSFORMATION_VERSION
      The version number for kickstart.
    • Constructor Summary

      Constructors 
      Constructor Description
      Kickstart()  
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      private void addCleanupPostScript​(Job job, java.util.List files)
      Adds a /bin/rm post constituentJob to kickstart that removes the files passed.
      boolean canSetXBit()
      Indicates whether the enabling mechanism can set the X bit on the executable on the remote grid site, in addition to launching it on the remote grid site.
      private void construct​(Job job, java.lang.String key, java.lang.String value)
      Constructs a condor variable in the condor profile namespace associated with the constituentJob.
      protected java.lang.String constructCleanupJob​(Job job, java.lang.String workerNodeTmp)
      Constructs a kickstart setup constituentJob
      protected java.lang.String constructPREJob​(Job job, java.lang.String headNodeURLPrefix, java.lang.String headNodeDirectory, java.lang.String workerNodeDirectory, java.lang.String slsFile)
      Constructs the prejob that fetches sls file, and then invokes transfer again.
      protected java.lang.String constructSetupJob​(Job job, java.lang.String workerNodeTmp)
      Constructs a kickstart setup constituentJob
      java.lang.String defaultPOSTScript()
      Returns the SHORT_NAME for the POSTScript implementation that is used to be as default with this GridStart implementation.
      boolean enable​(AggregatedJob job, boolean isGlobusJob)
      Enables a constituentJob to run on the grid.
      boolean enable​(Job job, boolean isGlobusJob)
      Enables a constituentJob to run on the grid by launching it through kickstart.
      protected boolean enable​(Job job, boolean isGlobusJob, boolean stat, boolean addPostScript, boolean partOfClusteredJob)
      Enables a constituentJob to run on the grid by launching it through kickstart.
      java.lang.String generateListofFilenamesFile​(java.util.Set files, java.lang.String basename)
      Writes out the list of filenames file for the constituentJob.
      protected java.lang.String getDirectory​(Job job)
      Returns the directory in which the job should run.
      private java.lang.String getDirectoryKey​(Job job)
      Returns the exectionSiteDirectory that is associated with the constituentJob to specify the exectionSiteDirectory in which the constituentJob needs to run
      java.lang.String getKickstartPath​(SiteCatalogEntry site)
      Returns the default path to kickstart as constructed from the environment variable associated with a site in the site catalog
      TransformationCatalogEntry getTransformationCatalogEntry​(java.lang.String site)
      Returns the transformation catalog entry for kickstart on a site
      java.lang.String getVDSKeyValue()
      Returns the value of the vds profile with key as Pegasus.GRIDSTART_KEY, that would result in the loading of this particular implementation.
      java.lang.String getWorkerNodeDirectory​(Job job)
      Returns the exectionSiteDirectory in which the constituentJob executes on the worker node.
      protected java.lang.String handleTransferOfExecutable​(Job job, java.lang.String path)
      It changes the paths to the executable depending on whether we want to transfer the executable or not.
      void initialize​(PegasusBag bag, ADag dag)
      Initializes the GridStart implementation.
      private java.lang.String quote​(java.lang.String string)
      Condor Quotes a string
      protected boolean requiresToSetDirectory​(Job job)
      Returns a boolean indicating whether we need to set the directory for the job or not.
      java.lang.String shortDescribe()
      Returns a short textual description in the form of the name of the class.
      void useFullPathToGridStarts​(boolean fullPath)
      Setter method to control whether a full path to Gridstart should be returned while wrapping a job or not.
      private boolean useInvoke​(Job job, java.lang.String executable, java.lang.StringBuffer args)
      Triggers the creation of the kickstart input file, that contains the the remote executable and the arguments with which it has to be invoked.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Field Detail

      • TRANSFORMATION_NAMESPACE

        public static final java.lang.String TRANSFORMATION_NAMESPACE
        The transformation namespace for the kickstart
        See Also:
        Constant Field Values
      • TRANSFORMATION_NAME

        public static final java.lang.String TRANSFORMATION_NAME
        The logical name of kickstart
        See Also:
        Constant Field Values
      • TRANSFORMATION_VERSION

        public static final java.lang.String TRANSFORMATION_VERSION
        The version number for kickstart.
      • EXECUTABLE_BASENAME

        public static final java.lang.String EXECUTABLE_BASENAME
        The basename of the kickstart executable.
        See Also:
        Constant Field Values
      • COMPLETE_TRANSFORMATION_NAME

        public static final java.lang.String COMPLETE_TRANSFORMATION_NAME
        The complete TC name for kickstart.
      • KICKSTART_INPUT_SUFFIX

        public static final java.lang.String KICKSTART_INPUT_SUFFIX
        The suffix for the kickstart input file, that is generated to use invoke at the remote end.
        See Also:
        Constant Field Values
      • CLASSNAME

        public static final java.lang.String CLASSNAME
        The basename of the class that is implmenting this. Could have been determined by reflection.
        See Also:
        Constant Field Values
      • SHORT_NAME

        public static final java.lang.String SHORT_NAME
        The SHORTNAME for this implementation.
        See Also:
        Constant Field Values
      • KICKSTART_SETUP

        public static final java.lang.String KICKSTART_SETUP
        The environment variable used to the set Kickstart SETUP JOB.
        See Also:
        Constant Field Values
      • KICKSTART_PREJOB

        public static final java.lang.String KICKSTART_PREJOB
        The environment variable used to the set Kickstart PREJOB.
        See Also:
        Constant Field Values
      • KICKSTART_POSTJOB

        public static final java.lang.String KICKSTART_POSTJOB
        The environment variable used to the set Kickstart POSTJOB.
        See Also:
        Constant Field Values
      • KICKSTART_CLEANUP

        public static final java.lang.String KICKSTART_CLEANUP
        The environment variable used to the set Kickstart CLEANUP JOB.
        See Also:
        Constant Field Values
      • mLogger

        private LogManager mLogger
        The LogManager object which is used to log all the messages.
      • mProps

        private PegasusProperties mProps
        The object holding all the properties pertaining to Pegasus.
      • mPOptions

        private PlannerOptions mPOptions
        The options passed to the planner.
      • mConcDAG

        private ADag mConcDAG
        The handle to the workflow that is being enabled.
      • mSiteStore

        private SiteStore mSiteStore
        Handle to the site catalog store.
      • mSubmitDir

        private java.lang.String mSubmitDir
        The submit exectionSiteDirectory where the submit files are being generated for the workflow.
      • mInvokeAlways

        private boolean mInvokeAlways
        A boolean indicating whether to use invoke always or not.
      • mDoStat

        private boolean mDoStat
        A boolean indicating whether to stat files or not.
      • mGenerateLOF

        private boolean mGenerateLOF
        A boolean indicating whether to generate lof files or not.
      • mInvokeLength

        private long mInvokeLength
        The invoke limit trigger.
      • mWorkerNodeExecution

        boolean mWorkerNodeExecution
        A boolean indicating whether to have worker node execution or not.
      • mSLS

        private SLS mSLS
        The handle to the SLS implementor
      • mEnablingPartOfAggregatedJob

        private boolean mEnablingPartOfAggregatedJob
        An instance variable to track if enabling is happening as part of a clustered constituentJob. See Bug 21 comments on Pegasus Bugzilla
      • mDynamicDeployment

        private boolean mDynamicDeployment
        A boolean indicating whether kickstart is deployed dynamically or not.
      • mKickstartLabel

        private java.lang.String mKickstartLabel
        The label that is passed to kickstart.
      • mSetXBit

        private boolean mSetXBit
        Whether kickstart should set the X Bit on the staged executables.
      • mNoGridStartImpl

        private GridStart mNoGridStartImpl
        Handle to NoGridStart implementation.
      • mUseFullPathToGridStart

        private boolean mUseFullPathToGridStart
        Boolean indicating whether to use full path or not
      • mDisableInvokeFunctionality

        private boolean mDisableInvokeFunctionality
        Boolean indicating whether to disable invoke functionality.
    • Constructor Detail

      • Kickstart

        public Kickstart()
    • Method Detail

      • initialize

        public void initialize​(PegasusBag bag,
                               ADag dag)
        Initializes the GridStart implementation.
        Specified by:
        initialize in interface GridStart
        Parameters:
        bag - the bag of objects that is used for initialization.
        dag - the concrete dag so far.
      • useFullPathToGridStarts

        public void useFullPathToGridStarts​(boolean fullPath)
        Setter method to control whether a full path to Gridstart should be returned while wrapping a job or not.
        Specified by:
        useFullPathToGridStarts in interface GridStart
        Parameters:
        fullPath - if set to true, indicates that full path would be used.
      • enable

        public boolean enable​(AggregatedJob job,
                              boolean isGlobusJob)
        Enables a constituentJob to run on the grid. This also determines how the stdin,stderr and stdout of the constituentJob are to be propogated. To grid enable a constituentJob, the constituentJob may need to be wrapped into another constituentJob, that actually launches the constituentJob. It usually results in the constituentJob description passed being modified modified.
        Specified by:
        enable in interface GridStart
        Parameters:
        constituentJob - the Job object containing the constituentJob description of the constituentJob that has to be enabled on the grid.
        isGlobusJob - is true, if the constituentJob generated a line universe = globus, and thus runs remotely. Set to false, if the constituentJob runs on the submit host in any way.
        Returns:
        boolean true if enabling was successful,else false.
      • enable

        public boolean enable​(Job job,
                              boolean isGlobusJob)
        Enables a constituentJob to run on the grid by launching it through kickstart. Does the stdio, and stderr handling of the constituentJob to be run on the grid. It modifies the constituentJob description, and also constructs all the valid option to be passed to kickstart for launching the executable.
        Specified by:
        enable in interface GridStart
        Parameters:
        constituentJob - the Job object containing the constituentJob description of the constituentJob that has to be enabled on the grid.
        isGlobusJob - is true, if the constituentJob generated a line universe = globus, and thus runs remotely. Set to false, if the constituentJob runs on the submit host in any way.
        Returns:
        boolean true if enabling was successful,else false in case when the path to kickstart could not be determined on the site where the constituentJob is scheduled.
      • enable

        protected boolean enable​(Job job,
                                 boolean isGlobusJob,
                                 boolean stat,
                                 boolean addPostScript,
                                 boolean partOfClusteredJob)
        Enables a constituentJob to run on the grid by launching it through kickstart. Does the stdio, and stderr handling of the constituentJob to be run on the grid. It modifies the constituentJob description, and also constructs all the valid option to be passed to kickstart for launching the executable.
        Parameters:
        constituentJob - the Job object containing the constituentJob description of the constituentJob that has to be enabled on the grid.
        isGlobusJob - is true, if the constituentJob generated a line universe = globus, and thus runs remotely. Set to false, if the constituentJob runs on the submit host in any way.
        stat - boolean indicating whether to generate the lof files for kickstart stat option or not.
        addPostScript - boolean indicating whether to add a postscript or not.
        partOfClusteredJob - boolean indicating whether the job being enabled is part of a clustered job or not.
        Returns:
        boolean true if enabling was successful,else false in case when the path to kickstart could not be determined on the site where the constituentJob is scheduled.
      • handleTransferOfExecutable

        protected java.lang.String handleTransferOfExecutable​(Job job,
                                                              java.lang.String path)
        It changes the paths to the executable depending on whether we want to transfer the executable or not. If the transfer_executable is set to true, then the executable needs to be shipped from the submit host meaning the local pool. This function changes the path of the executable to the one on the local pool, so that it can be shipped. If the worker package is being deployed dynamically, then the path is set to the exectionSiteDirectory where the worker package is deployed. Else, we pick up the path from the site catalog that is passed as input
        Parameters:
        constituentJob - the Job containing the constituentJob description.
        path - the path to kickstart on the remote compute site, as determined from the site catalog.
        Returns:
        the path that needs to be set as the executable
      • getTransformationCatalogEntry

        public TransformationCatalogEntry getTransformationCatalogEntry​(java.lang.String site)
        Returns the transformation catalog entry for kickstart on a site
        Parameters:
        site - the site on which the entry is required
        Returns:
        the entry if found else null
      • getKickstartPath

        public java.lang.String getKickstartPath​(SiteCatalogEntry site)
        Returns the default path to kickstart as constructed from the environment variable associated with a site in the site catalog
        Parameters:
        site - the SiteCatalogEntry object for the site.
        Returns:
        value if set else null
      • getWorkerNodeDirectory

        public java.lang.String getWorkerNodeDirectory​(Job job)
        Returns the exectionSiteDirectory in which the constituentJob executes on the worker node.
        Specified by:
        getWorkerNodeDirectory in interface GridStart
        Parameters:
        constituentJob -
        Returns:
        the full path to the exectionSiteDirectory where the constituentJob executes
      • canSetXBit

        public boolean canSetXBit()
        Indicates whether the enabling mechanism can set the X bit on the executable on the remote grid site, in addition to launching it on the remote grid site.
        Specified by:
        canSetXBit in interface GridStart
        Returns:
        true indicating Kickstart can set the X bit or not.
      • getVDSKeyValue

        public java.lang.String getVDSKeyValue()
        Returns the value of the vds profile with key as Pegasus.GRIDSTART_KEY, that would result in the loading of this particular implementation. It is usually the name of the implementing class without the package name.
        Specified by:
        getVDSKeyValue in interface GridStart
        Returns:
        the value of the profile key.
        See Also:
        org.griphyn.cPlanner.namespace.Pegasus#GRIDSTART_KEY
      • shortDescribe

        public java.lang.String shortDescribe()
        Returns a short textual description in the form of the name of the class.
        Specified by:
        shortDescribe in interface GridStart
        Returns:
        short textual description.
      • defaultPOSTScript

        public java.lang.String defaultPOSTScript()
        Returns the SHORT_NAME for the POSTScript implementation that is used to be as default with this GridStart implementation.
        Specified by:
        defaultPOSTScript in interface GridStart
        Returns:
        the identifier for the PegasusExitCode POSTScript implementation.
        See Also:
        POSTScript.shortDescribe()
      • requiresToSetDirectory

        protected boolean requiresToSetDirectory​(Job job)
        Returns a boolean indicating whether we need to set the directory for the job or not.
        Parameters:
        job - the job for which to set directory.
        Returns:
      • getDirectory

        protected java.lang.String getDirectory​(Job job)
        Returns the directory in which the job should run.
        Parameters:
        job - the job in which the directory has to run.
        Returns:
      • getDirectoryKey

        private java.lang.String getDirectoryKey​(Job job)
        Returns the exectionSiteDirectory that is associated with the constituentJob to specify the exectionSiteDirectory in which the constituentJob needs to run
        Parameters:
        constituentJob - the constituentJob
        Returns:
        the condor key . can be initialdir or remote_initialdir
      • useInvoke

        private boolean useInvoke​(Job job,
                                  java.lang.String executable,
                                  java.lang.StringBuffer args)
        Triggers the creation of the kickstart input file, that contains the the remote executable and the arguments with which it has to be invoked. The kickstart input file is created in the submit directory.
        Parameters:
        constituentJob - the Job object containing the constituentJob description.
        executable - the path to the executable used.
        args - the arguments buffer for gridstart invocation so far.
        Returns:
        boolean indicating whether kickstart input file was generated or not. false in case of any error.
      • constructSetupJob

        protected java.lang.String constructSetupJob​(Job job,
                                                     java.lang.String workerNodeTmp)
        Constructs a kickstart setup constituentJob
        Parameters:
        constituentJob - the constituentJob to be run.
        workerNodeTmp - the worker node tmp to run the constituentJob in.
        Returns:
        String
      • constructCleanupJob

        protected java.lang.String constructCleanupJob​(Job job,
                                                       java.lang.String workerNodeTmp)
        Constructs a kickstart setup constituentJob
        Parameters:
        constituentJob - the constituentJob to be run.
        workerNodeTmp - the worker node tmp to run the constituentJob in.
        Returns:
        String
      • constructPREJob

        protected java.lang.String constructPREJob​(Job job,
                                                   java.lang.String headNodeURLPrefix,
                                                   java.lang.String headNodeDirectory,
                                                   java.lang.String workerNodeDirectory,
                                                   java.lang.String slsFile)
        Constructs the prejob that fetches sls file, and then invokes transfer again.
        Parameters:
        constituentJob - the constituentJob for which the prejob is being created
        headNodeURLPrefix - String
        headNodeDirectory - String
        workerNodeDirectory - String
        slsFile - String
        Returns:
        String containing the prescript invocation
      • generateListofFilenamesFile

        public java.lang.String generateListofFilenamesFile​(java.util.Set files,
                                                            java.lang.String basename)
        Writes out the list of filenames file for the constituentJob.
        Parameters:
        files - the list of PegasusFile objects contains the files whose stat information is required.
        basename - the basename of the file that is to be created
        Returns:
        the full path to lof file created, else null if no file is written out.
      • construct

        private void construct​(Job job,
                               java.lang.String key,
                               java.lang.String value)
        Constructs a condor variable in the condor profile namespace associated with the constituentJob. Overrides any preexisting key values.
        Parameters:
        constituentJob - contains the constituentJob description.
        key - the key of the profile.
        value - the associated value.
      • quote

        private java.lang.String quote​(java.lang.String string)
        Condor Quotes a string
        Parameters:
        string - the string to be quoted.
        Returns:
        quoted string.
      • addCleanupPostScript

        private void addCleanupPostScript​(Job job,
                                          java.util.List files)
        Adds a /bin/rm post constituentJob to kickstart that removes the files passed. The post jobs is added as an environment variable.
        Parameters:
        constituentJob - the constituentJob in which the post constituentJob needs to be added.
        files - the files to be deleted.