Class DeployWorkerPackage


  • public class DeployWorkerPackage
    extends Engine
    The refiner that is responsible for adding - setup nodes that deploy a worker package on each deployment site at start of workflow execution - cleanup nodes that undeploy a worker package on each deployment site at end workflow execution
    Version:
    $Revision: 538 $
    Author:
    Karan Vahi, Gaurang Mehta
    • Field Detail

      • DEPLOY_WORKER_PREFIX

        public static final java.lang.String DEPLOY_WORKER_PREFIX
        Constant suffix for the names of the deployment nodes.
        See Also:
        Constant Field Values
      • UNTAR_PREFIX

        public static final java.lang.String UNTAR_PREFIX
        Constant suffix for the names of the deployment nodes.
        See Also:
        Constant Field Values
      • CLEANUP_PREFIX

        public static final java.lang.String CLEANUP_PREFIX
        Constant suffix for the names of the deployment nodes.
        See Also:
        Constant Field Values
      • POSTSCRIPT_ARGUMENTS_FOR_ONLY_ROTATING_LOG_FILE

        public static final java.lang.String POSTSCRIPT_ARGUMENTS_FOR_ONLY_ROTATING_LOG_FILE
        The arguments for pegasus-exitcode when you only want the log files to be rotated.
        See Also:
        Constant Field Values
      • PEGASUS_WORKER_EXECUTABLES

        public static final java.lang.String[][] PEGASUS_WORKER_EXECUTABLES
        Array storing the names of the executables in the $PEGASUS_HOME/bin directory Associates the transformation name with the executable basenames
      • mRegexExpression

        private static final java.lang.String mRegexExpression
        Store the regular expressions necessary to parse the basename from the worker package url to retrieve the version of pegasus.
        See Also:
        Constant Field Values
      • PATH_VALUE

        public static final java.lang.String PATH_VALUE
        The path to be set for create dir jobs.
        See Also:
        Constant Field Values
      • DEFAULT_REFINER

        public static final java.lang.String DEFAULT_REFINER
        The default transfer refiner name.
        See Also:
        Constant Field Values
      • TRANSFORMATION_NAMESPACE

        public static final java.lang.String TRANSFORMATION_NAMESPACE
        The transformation namespace for the worker package
        See Also:
        Constant Field Values
      • TRANSFORMATION_NAME

        public static final java.lang.String TRANSFORMATION_NAME
        The logical name of the worker package
        See Also:
        Constant Field Values
      • TRANSFORMATION_VERSION

        public static final java.lang.String TRANSFORMATION_VERSION
        The version number for the worker package.
      • UNTAR_TRANSFORMATION_NAMESPACE

        public static final java.lang.String UNTAR_TRANSFORMATION_NAMESPACE
        The transformation namespace for the worker package
      • UNTAR_TRANSFORMATION_NAME

        public static final java.lang.String UNTAR_TRANSFORMATION_NAME
        The logical name of the worker package
        See Also:
        Constant Field Values
      • UNTAR_TRANSFORMATION_VERSION

        public static final java.lang.String UNTAR_TRANSFORMATION_VERSION
        The version number for the worker package.
      • COMPLETE_UNTAR_TRANSFORMATION_NAME

        public static final java.lang.String COMPLETE_UNTAR_TRANSFORMATION_NAME
        The complete TC name for untar.
      • COMPLETE_TRANSFORMATION_NAME

        public static final java.lang.String COMPLETE_TRANSFORMATION_NAME
        The complete TC name for pegasus worker package.
      • DERIVATION_NAMESPACE

        public static final java.lang.String DERIVATION_NAMESPACE
        The derivation namespace for the worker package.
        See Also:
        Constant Field Values
      • DERIVATION_NAME

        public static final java.lang.String DERIVATION_NAME
        The logical name of the transformation for the worker package
        See Also:
        Constant Field Values
      • DERIVATION_VERSION

        public static final java.lang.String DERIVATION_VERSION
        The version number for the derivations for worker package.
        See Also:
        Constant Field Values
      • UNTAR_DERIVATION_NAMESPACE

        public static final java.lang.String UNTAR_DERIVATION_NAMESPACE
        The derivation namespace for the untar job.
      • UNTAR_DERIVATION_NAME

        public static final java.lang.String UNTAR_DERIVATION_NAME
        The logical name of the transformation for the untar job.
        See Also:
        Constant Field Values
      • UNTAR_DERIVATION_VERSION

        public static final java.lang.String UNTAR_DERIVATION_VERSION
        The version number for the derivations for untar job.
        See Also:
        Constant Field Values
      • PACKAGE_NAME

        public static final java.lang.String PACKAGE_NAME
        The name of the package in which all the implementing classes are.
        See Also:
        Constant Field Values
      • BASE_BUILD_DIRECTORY_URL

        public static final java.lang.String BASE_BUILD_DIRECTORY_URL
        The base directory URL for the builds.
        See Also:
        Constant Field Values
      • PEGASUS_VERSION

        public static final java.lang.String PEGASUS_VERSION
        The version of pegasus matching the planner.
      • REFINER_NAME

        public static final java.lang.String REFINER_NAME
        The name of the refiner for purposes of error logging
        See Also:
        Constant Field Values
      • mPattern

        private static java.util.regex.Pattern mPattern
        Stores compiled patterns at first use, quasi-Singleton.
      • mOSToNMIOSReleaseAndVersion

        private static java.util.Map<SysInfo.OS,​java.lang.String> mOSToNMIOSReleaseAndVersion
        The map storing OS to corresponding NMI OS platforms.
      • mSupportedOSReleaseVersions

        private static java.util.Set<java.lang.String> mSupportedOSReleaseVersions
        A set of supported OS release and versions that our build process builds for.
      • mCurrentDag

        protected ADag mCurrentDag
        It is a reference to the Concrete Dag so far.
      • mJobPrefix

        protected java.lang.String mJobPrefix
        The job prefix that needs to be applied to the job file basenames.
      • mSetupTransferImplementation

        protected Implementation mSetupTransferImplementation
        The transfer implementation to be used for staging in the data as part of setup job.
      • mFTMap

        protected java.util.Map<java.lang.String,​FileTransfer> mFTMap
        The FileTransfer map indexed by site id.
      • mLocalTransfers

        protected java.util.Map<java.lang.String,​java.lang.Boolean> mLocalTransfers
        Map that indicates whether we need local setup transfer jobs for a site or not.
      • mSiteToPegasusHomeMap

        protected java.util.Map<java.lang.String,​java.lang.String> mSiteToPegasusHomeMap
        Maps a site to the the directory where the pegasus worker package has been untarred during workflow execution.
      • mUserSpecifiedSourceLocation

        protected java.lang.String mUserSpecifiedSourceLocation
        The user specified location from where to stage the worker packages.
      • mUseUserSpecifiedSourceLocation

        protected boolean mUseUserSpecifiedSourceLocation
        Boolean indicating whether to use the user specified location or not
      • mTransferWorkerPackage

        protected boolean mTransferWorkerPackage
        Boolean indicating whether user wants the worker package to be transferred or not.
      • mWorkerNodeExecution

        protected boolean mWorkerNodeExecution
        Boolean indicating worker node execution.
    • Constructor Detail

      • DeployWorkerPackage

        public DeployWorkerPackage​(PegasusBag bag)
        A pratically nothing constructor !
        Parameters:
        bag - bag of initialization objects
    • Method Detail

      • osToOSReleaseAndVersion

        private static java.util.Map<SysInfo.OS,​java.lang.String> osToOSReleaseAndVersion()
        Maps each to OS to a specific OS release for purposes of picking up the correct worker package for a site. The mapping is to be kept consistent with the NMI builds for the releases.
        Returns:
        map
      • supportedOSReleaseAndVersions

        private static java.util.Set<java.lang.String> supportedOSReleaseAndVersions()
        A set of OS release and version combinations for which our build processes build Pegasus binaries.
        Returns:
      • loadDeployWorkerPackage

        public static DeployWorkerPackage loadDeployWorkerPackage​(PegasusBag bag)
                                                           throws FactoryException
        Loads the implementing class corresponding to the mode specified by the user at runtime.
        Parameters:
        bag - bag of initialization objects
        Returns:
        instance of a DeployWorkerPackage implementation
        Throws:
        FactoryException - that nests any error that might occur during the instantiation of the implementation.
      • initialize

        public void initialize​(ADag scheduledDAG)
        Initialize with the scheduled graph. Results in the appropriate population of the transformation catalog with pegasus-worker executables.
        Parameters:
        scheduledDAG - the scheduled workflow.
      • runTransferOnLocalSite

        public boolean runTransferOnLocalSite​(Refiner refiner,
                                              java.lang.String site,
                                              java.lang.String destinationURL,
                                              int type)
        Returns whether to run a transfer job on local site or not.
        Parameters:
        site - the site handle associated with the destination URL.
        destURL - the destination URL
        type - the type of transfer job for which the URL is being constructed.
        Returns:
        true indicating if the associated transfer job should run on local site or not.
      • getRootDirectoryNameForPegasus

        protected java.lang.String getRootDirectoryNameForPegasus​(java.lang.String url)
        Does regex magic to figure out the version of pegasus from the url, and use it to construct the name of pegasus directory, when worker package is untarred.
        Parameters:
        url - the url.
        Returns:
        basename for pegasus directory
      • addSetupNodes

        public ADag addSetupNodes​(ADag dag)
        Adds a setup node per execution site in the workflow that will stage the worker node executables to the workdirectory on the sites the workflow has been scheduled to.
        Parameters:
        dag - the scheduled workflow.
        Returns:
        the workflow with setup jobs added
      • addSetupNodesWithUntarNodes

        private ADag addSetupNodesWithUntarNodes​(ADag dag,
                                                 java.util.Set<java.lang.String> deploymentSites)
        Adds untar nodes to the workflow, in addition to the stage worker nodes
        Parameters:
        dag - the dag
        deploymentSites - the sites for which the worker package has to be deployed
        Returns:
        the workflow in the graph representation with the nodes added.
      • addSetupNodesWithoutUntarNodes

        private ADag addSetupNodesWithoutUntarNodes​(ADag dag,
                                                    java.util.Set<java.lang.String> deploymentSites)
        Adds only the stage worker nodes to the workflow. This is used when Pegasus Lite is used to launch the jobs on the execution sites.
        Parameters:
        dag - the dag
        deploymentSites - the sites for which the worker package has to be deployed
        Returns:
        the workflow in the graph representation with the nodes added.
      • addCleanupNodesForWorkerPackage

        public ADag addCleanupNodesForWorkerPackage​(ADag dag)
        Adds cleanup nodes in the workflow for sites specified.
        Parameters:
        dag - the workflow
        Returns:
        workflow with cleanup jobs added
      • getDeploymentSites

        protected java.util.Set getDeploymentSites​(ADag dag)
        Retrieves the sites for which the deployment jobs need to be created.
        Parameters:
        dag - the dag on which the jobs need to execute.
        Returns:
        a Set containing a list of siteID's of the sites where the dag has to be run.
      • getDeployJobName

        protected java.lang.String getDeployJobName​(ADag dag,
                                                    java.lang.String site,
                                                    boolean localTransfer)
        It returns the name of the deployment job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.
        Parameters:
        dag - the workflow so far.
        site - the execution pool for which the create directory job is responsible.
        localTransfer - whether the transfer needs to run locally or not.
        Returns:
        String corresponding to the name of the job.
      • getUntarJobName

        protected java.lang.String getUntarJobName​(ADag dag,
                                                   java.lang.String site)
        It returns the name of the untar job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.
        Parameters:
        dag - the workflow so far.
        site - the execution pool for which the create directory job is responsible.
        Returns:
        String corresponding to the name of the job.
      • getCleanupJobname

        protected java.lang.String getCleanupJobname​(ADag dag,
                                                     java.lang.String site)
        It returns the name of the untar job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.
        Parameters:
        dag - the workflow so far.
        site - the execution pool for which the create directory job is responsible.
        Returns:
        String corresponding to the name of the job.
      • makeUntarJob

        protected Job makeUntarJob​(java.lang.String site,
                                   java.lang.String jobName,
                                   java.lang.String wpBasename)
        It creates a untar job , that untars the worker package that is staged by the setup transfer job.
        Parameters:
        site - the execution pool for which the create dir job is to be created.
        jobName - the name that is to be assigned to the job.
        wpBasename - the basename of the worker package that is staged to remote site.
        Returns:
        create dir job.
      • addDefaultTCEntry

        private TransformationCatalogEntry addDefaultTCEntry​(java.lang.String site,
                                                             java.lang.String pegasusHome,
                                                             SysInfo sysinfo,
                                                             boolean useFullPath,
                                                             java.lang.String name,
                                                             java.lang.String executable)
        Returns a default TC entry to be used in case entry is not found in the transformation catalog. It also attempts to add the transformation catalog entry to the underlying TC store.
        Parameters:
        site - the site for which the default entry is required.
        pegasusHome - the path to deployed worker package
        sysinfo - the system information of that site.
        useFullPath - boolean indicating whether to use just the basename or the full path
        name - the logical name of the transformation
        executable - the basename of the executable
        Returns:
        the default entry.
      • addDefaultTCEntryForPegasusWebsite

        protected TransformationCatalogEntry addDefaultTCEntryForPegasusWebsite​(java.lang.String site,
                                                                                java.lang.String name)
        Returns a default TC entry for the pegasus site. The entry points to the http webserver on the pegasus website. It also attempts to add the transformation catalog entry to the TC store.
        Parameters:
        site - the execution site for which we need a matching static binary.
        name - logical name of the transformation
        Returns:
        the default entry.
      • constructDefaultURLToPegasusWorkerPackage

        protected java.lang.String constructDefaultURLToPegasusWorkerPackage​(java.lang.String name,
                                                                             SysInfo sysinfo)
        Constructs the default URL's for the pegasus worker package. If the user has not specified the URL to the source directory in Pegaus Properties then the URL constructed points to the pegasus website. The version of Pegasus retrieved is the one against which the planner is executing.
        Parameters:
        name - the logical name of the executable, usually worker|binary.
        sysinfo - the sysinfo for which the path is required.
        Returns:
        url
      • defaultUntarTCEntry

        private TransformationCatalogEntry defaultUntarTCEntry​(SiteCatalogEntry site)
        Returns a default TC entry to be used in case entry is not found in the transformation catalog.
        Parameters:
        site - the site for which the default entry is required.
        Returns:
        the default entry.
      • getBasename

        protected java.lang.String getBasename​(java.lang.String url)
        Returns the basename of the URL using substring.
        Parameters:
        url -
        Returns:
        basename
      • getStagingSite

        private java.lang.String getStagingSite​(java.lang.String site)
        Returns the staging site for a particular execution site. If worker node execution is enabled, then the staging site is the submit directory for the workflow on the local sit.e
        Parameters:
        site - the execution site.
        Returns:
        the staging site