Class CPlanner


  • public class CPlanner
    extends Executable
    This is the main program for the Pegasus. It parses the options specified by the user and calls out to the appropriate components to parse the abstract plan, concretize it and then write the submit files.
    Version:
    $Revision$
    Author:
    Gaurang Mehta, Karan Vahi
    • Field Summary

      Fields 
      Modifier and Type Field Description
      static java.lang.String CLEANUP_DIR
      The basename of the directory that contains the submit files for the cleanup DAG that for the concrete dag generated for the workflow.
      static java.lang.String DEFAULT_MEGADAG_MODE
      The default megadag mode that is used for generation of megadags in deferred planning.
      private static java.lang.String EMPTY_FINAL_WORKFLOW_MESSAGE
      The final successful message that is to be logged.
      static java.lang.String JAVA_COMMAND_LINE_PROPERTY_REGEX
      The regex used to match against a java property that is set using -Dpropertyname=value in the argument string
      private PegasusBag mBag
      The object containing the bag of pegasus objects
      private java.text.NumberFormat mNumFormatter
      The number formatter to format the run submit dir entries.
      private PlannerMetrics mPMetrics
      The PlannerMetrics object storing the metrics about this planning instance.
      private PlannerOptions mPOptions
      The object containing all the options passed to the Concrete Planner.
      private boolean mSendMetrics
      A boolean indicating whether metrics should be sent to metrics server or not
      private java.lang.String mUser
      The user name of the user running Pegasus.
      static java.lang.String NOOP_PREFIX
      The prefix for the NoOP jobs that are created.
      static java.lang.String PEGASUS_MONITORD_LAUNCH_PROPERTY_KEY
      The name of the property key that determines whether pegasus-run should monitord or not.
      private static java.lang.String SUCCESS_MESSAGE
      The message to be logged in case of empty executable workflow.
    • Constructor Summary

      Constructors 
      Constructor Description
      CPlanner()
      Default constructor.
      CPlanner​(LogManager logger)
      The overload constructor.
    • Field Detail

      • DEFAULT_MEGADAG_MODE

        public static final java.lang.String DEFAULT_MEGADAG_MODE
        The default megadag mode that is used for generation of megadags in deferred planning.
        See Also:
        Constant Field Values
      • CLEANUP_DIR

        public static final java.lang.String CLEANUP_DIR
        The basename of the directory that contains the submit files for the cleanup DAG that for the concrete dag generated for the workflow.
        See Also:
        Constant Field Values
      • NOOP_PREFIX

        public static final java.lang.String NOOP_PREFIX
        The prefix for the NoOP jobs that are created.
        See Also:
        Constant Field Values
      • PEGASUS_MONITORD_LAUNCH_PROPERTY_KEY

        public static final java.lang.String PEGASUS_MONITORD_LAUNCH_PROPERTY_KEY
        The name of the property key that determines whether pegasus-run should monitord or not.
        See Also:
        Constant Field Values
      • JAVA_COMMAND_LINE_PROPERTY_REGEX

        public static final java.lang.String JAVA_COMMAND_LINE_PROPERTY_REGEX
        The regex used to match against a java property that is set using -Dpropertyname=value in the argument string
        See Also:
        Constant Field Values
      • EMPTY_FINAL_WORKFLOW_MESSAGE

        private static final java.lang.String EMPTY_FINAL_WORKFLOW_MESSAGE
        The final successful message that is to be logged.
        See Also:
        Constant Field Values
      • SUCCESS_MESSAGE

        private static final java.lang.String SUCCESS_MESSAGE
        The message to be logged in case of empty executable workflow.
        See Also:
        Constant Field Values
      • mPOptions

        private PlannerOptions mPOptions
        The object containing all the options passed to the Concrete Planner.
      • mBag

        private PegasusBag mBag
        The object containing the bag of pegasus objects
      • mPMetrics

        private PlannerMetrics mPMetrics
        The PlannerMetrics object storing the metrics about this planning instance.
      • mNumFormatter

        private java.text.NumberFormat mNumFormatter
        The number formatter to format the run submit dir entries.
      • mUser

        private java.lang.String mUser
        The user name of the user running Pegasus.
      • mSendMetrics

        private boolean mSendMetrics
        A boolean indicating whether metrics should be sent to metrics server or not
    • Constructor Detail

      • CPlanner

        public CPlanner()
        Default constructor.
      • CPlanner

        public CPlanner​(LogManager logger)
        The overload constructor.
        Parameters:
        logger - the logger object to use. can be null.
    • Method Detail

      • initialize

        public void initialize​(java.lang.String[] opts,
                               char confChar)
        Description copied from class: Executable
        Initialize the executable object
        Overrides:
        initialize in class Executable
        Parameters:
        opts - the command line argument passed by the user
        confChar - the short option corresponding the conf property.
      • main

        public static void main​(java.lang.String[] args)
        The main program for the CPlanner.
        Parameters:
        args - the main arguments passed to the planner.
      • loadProperties

        public void loadProperties()
        Loads all the properties that are needed by this class.
        Specified by:
        loadProperties in class Executable
      • executeCommand

        public void executeCommand()
        Executes the command on the basis of the options specified.
        Parameters:
        args - the command line options.
      • executeCommand

        public java.util.Collection<java.io.File> executeCommand​(PlannerOptions options)
        Executes the command on the basis of the options specified.
        Parameters:
        options - the command line options.
        Returns:
        the Collection of File objects for the files written out.
      • getNOOPJobName

        public java.lang.String getNOOPJobName​(ADag dag)
        Returns the name of the noop job.
        Parameters:
        dag - the workflow
        Returns:
        the name
      • createNoOPJob

        protected Job createNoOPJob​(java.lang.String name)
        It creates a NoOP job that runs on the submit host.
        Parameters:
        name - the name to be assigned to the noop job
        Returns:
        the noop job.
      • construct

        protected void construct​(Job job,
                                 java.lang.String key,
                                 java.lang.String value)
        Constructs a condor variable in the condor profile namespace associated with the job. Overrides any preexisting key values.
        Parameters:
        job - contains the job description.
        key - the key of the profile.
        value - the associated value.
      • parseCommandLineArguments

        public PlannerOptions parseCommandLineArguments​(java.lang.String[] args)
        Parses the command line arguments using GetOpt and returns a PlannerOptions contains all the options passed by the user at the command line.
        Parameters:
        args - the arguments passed by the user at command line.
        Returns:
        the options.
      • parseCommandLineArguments

        public PlannerOptions parseCommandLineArguments​(java.lang.String[] args,
                                                        boolean sanitizePath)
        Parses the command line arguments using GetOpt and returns a PlannerOptions contains all the options passed by the user at the command line.
        Parameters:
        args - the arguments passed by the user at command line.
        sanitizePath - whether to sanitize path during construction of options
        Returns:
        the options.
      • submitWorkflow

        public boolean submitWorkflow​(java.lang.String invocation)
        Submits the workflow for execution using pegasus-run, a wrapper around pegasus-submit-dag.
        Parameters:
        invocation - the pegasus run invocation
        Returns:
        boolean indicating whether could successfully submit the workflow or not.
      • getRandomDirectory

        protected java.lang.String getRandomDirectory​(ADag dag)
        Sets the basename of the random directory that is created on the remote sites per workflow. The name is generated by default from teh flow ID, unless a basename prefix is specifed at runtime in the planner options.
        Parameters:
        dag - the DAG containing the abstract workflow.
        Returns:
        the basename of the random directory.
      • generateValidOptions

        public gnu.getopt.LongOpt[] generateValidOptions()
        Tt generates the LongOpt which contain the valid options that the command will accept.
        Specified by:
        generateValidOptions in class Executable
        Returns:
        array of LongOpt objects , corresponding to the valid options
      • printShortVersion

        public void printShortVersion()
        Prints out a short description of what the command does.
        Specified by:
        printShortVersion in class Executable
      • printLongVersion

        public void printLongVersion()
        Prints the long description, displaying in detail what the various options to the command stand for.
        Specified by:
        printLongVersion in class Executable
      • determineRootWorkflowUUID

        private java.lang.String determineRootWorkflowUUID​(ADag dag,
                                                           PlannerOptions options,
                                                           PegasusProperties properties)
        Determines the workflow uuid for a workflow
        Parameters:
        dag - the workflow
        options - the options passed to the planner
        properties - the properties passed to the planner
        Returns:
        uuid for the root workflow instance
      • determineRelativeSubmitDirectory

        protected java.lang.String determineRelativeSubmitDirectory​(ADag dag,
                                                                    java.lang.String dir,
                                                                    java.lang.String user,
                                                                    java.lang.String vogroup,
                                                                    boolean timestampBased)
                                                             throws java.io.IOException
        Creates the submit directory for the workflow. This is not thread safe.
        Parameters:
        dag - the workflow being worked upon.
        dir - the base directory specified by the user.
        user - the username of the user.
        vogroup - the vogroup to which the user belongs to.
        timestampBased - boolean indicating whether to have a timestamp based dir or not
        Returns:
        the directory name created relative to the base directory passed as input.
        Throws:
        java.io.IOException - in case of unable to create submit directory.
      • determineRelativeSubmitDirectory

        protected java.lang.String determineRelativeSubmitDirectory​(java.lang.String label,
                                                                    java.lang.String dir,
                                                                    java.lang.String user,
                                                                    java.lang.String vogroup,
                                                                    boolean timestampBased)
                                                             throws java.io.IOException
        Creates the submit directory for the workflow. This is not thread safe.
        Parameters:
        label - the label of the workflow
        dir - the base directory specified by the user.
        user - the username of the user.
        vogroup - the vogroup to which the user belongs to.
        timestampBased - boolean indicating whether to have a timestamp based dir or not
        Returns:
        the directory name created relative to the base directory passed as input.
        Throws:
        java.io.IOException - in case of unable to create submit directory.
      • sanityCheck

        protected static void sanityCheck​(java.io.File dir)
                                   throws java.io.IOException
        Checks the destination location for existence, if it can be created, if it is writable etc.
        Parameters:
        dir - is the new base directory to optionally create.
        Throws:
        java.io.IOException - in case of error while writing out files.
      • getDAGFilename

        protected java.lang.String getDAGFilename​(ADag dag,
                                                  PlannerOptions options)
        Returns the basename of the dag file
        Parameters:
        dag - the dag that was parsed.
        options - the planner options
        Returns:
        boolean true means submit the rescue false do the planning operation
      • handleRescueDAG

        protected boolean handleRescueDAG​(ADag dag,
                                          PlannerOptions options)
        Checks for rescue dags, and determines whether to plan or not.
        Parameters:
        dag - the dag that was parsed.
        options - the planner options
        Returns:
        boolean true means submit the rescue false do the planning operation
      • handleRescueDAG

        protected boolean handleRescueDAG​(java.lang.String dag,
                                          java.lang.String dir,
                                          int numOfRescues)
        Checks for rescue dags, and determines whether to submit a rescue dag or not.
        Parameters:
        dag - the dag file for the dax
        dir - the submit directory.
        numOfRescues - the number of rescues to handle.
        Returns:
        true means submit the rescue false do the planning operation
      • createSymbolicLink

        protected boolean createSymbolicLink​(java.lang.String source,
                                             java.lang.String destination,
                                             java.io.File directory,
                                             boolean logErrorToDebug)
        This method generates a symlink between two files
        Parameters:
        source - the file that has to be symlinked
        destination - the destination of the symlink
        directory - the directory in which to execute the command
        logErrorToDebug - whether to log messeage to debug or not
        Returns:
        boolean indicating if creation of symlink was successful or not
      • generateStampedeEventsForAbstractWorkflow

        private void generateStampedeEventsForAbstractWorkflow​(ADag workflow,
                                                               PegasusBag bag)
        Generates events for the abstract workflow.
        Parameters:
        workflow - the parsed dax
        bag - the initialized object bag
      • loadSiteStore

        private SiteStore loadSiteStore​(java.util.Set<java.lang.String> sites)
        Loads the sites from the site catalog into the site store
        Parameters:
        sites -
        Returns:
        SiteStore object containing the information about the sites.
      • logSuccessfulCompletion

        private void logSuccessfulCompletion​(boolean emptyWorkflow)
        Logs the successful completion message.
        Parameters:
        emptyWorkflow - indicates whether the workflow created was empty or not.
      • getPegasusRunInvocation

        private java.lang.String getPegasusRunInvocation()
        Returns the pegasus-run invocation on the workflow planned.
        Returns:
        the pegasus-run invocation
      • doBackupAndCreateSymbolicLinkForSubmitDirectory

        protected java.lang.String doBackupAndCreateSymbolicLinkForSubmitDirectory​(java.lang.String baseDir,
                                                                                   java.lang.String relativeSubmitDir)
                                                                            throws java.io.IOException
        Throws:
        java.io.IOException
      • logMemoryUsage

        private void logMemoryUsage()
        Logs memory usage of the JVM
      • parseDAX

        private ADag parseDAX​(java.lang.String dax)
        Parses the DAX and returns the associated ADag object
        Parameters:
        dax - path to the DAX file.
        Returns: