Package edu.isi.pegasus.planner.client
Class CPlanner
- java.lang.Object
-
- edu.isi.pegasus.planner.client.Executable
-
- edu.isi.pegasus.planner.client.CPlanner
-
public class CPlanner extends Executable
This is the main program for the Pegasus. It parses the options specified by the user and calls out to the appropriate components to parse the abstract plan, concretize it and then write the submit files.- Version:
- $Revision$
- Author:
- Gaurang Mehta, Karan Vahi
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
CLEANUP_DIR
The basename of the directory that contains the submit files for the cleanup DAG that for the concrete dag generated for the workflow.static java.lang.String
DEFAULT_MEGADAG_MODE
The default megadag mode that is used for generation of megadags in deferred planning.private static java.lang.String
EMPTY_FINAL_WORKFLOW_MESSAGE
The final successful message that is to be logged.static java.lang.String
JAVA_COMMAND_LINE_PROPERTY_REGEX
The regex used to match against a java property that is set using -Dpropertyname=value in the argument stringprivate PegasusBag
mBag
The object containing the bag of pegasus objectsprivate java.text.NumberFormat
mNumFormatter
The number formatter to format the run submit dir entries.private PlannerMetrics
mPMetrics
The PlannerMetrics object storing the metrics about this planning instance.private PlannerOptions
mPOptions
The object containing all the options passed to the Concrete Planner.private boolean
mSendMetrics
A boolean indicating whether metrics should be sent to metrics server or notprivate java.lang.String
mUser
The user name of the user running Pegasus.static java.lang.String
NOOP_PREFIX
The prefix for the NoOP jobs that are created.static java.lang.String
PEGASUS_MONITORD_LAUNCH_PROPERTY_KEY
The name of the property key that determines whether pegasus-run should monitord or not.private static java.lang.String
SUCCESS_MESSAGE
The message to be logged in case of empty executable workflow.-
Fields inherited from class edu.isi.pegasus.planner.client.Executable
mLogger, mLogMsg, mProps, mVersion
-
-
Constructor Summary
Constructors Constructor Description CPlanner()
Default constructor.CPlanner(LogManager logger)
The overload constructor.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
construct(Job job, java.lang.String key, java.lang.String value)
Constructs a condor variable in the condor profile namespace associated with the job.protected Job
createNoOPJob(java.lang.String name)
It creates a NoOP job that runs on the submit host.protected boolean
createSymbolicLink(java.lang.String source, java.lang.String destination, java.io.File directory, boolean logErrorToDebug)
This method generates a symlink between two filesprotected java.lang.String
determineRelativeSubmitDirectory(ADag dag, java.lang.String dir, java.lang.String user, java.lang.String vogroup, boolean timestampBased)
Creates the submit directory for the workflow.protected java.lang.String
determineRelativeSubmitDirectory(java.lang.String label, java.lang.String dir, java.lang.String user, java.lang.String vogroup, boolean timestampBased)
Creates the submit directory for the workflow.private java.lang.String
determineRootWorkflowUUID(ADag dag, PlannerOptions options, PegasusProperties properties)
Determines the workflow uuid for a workflowprotected java.lang.String
doBackupAndCreateSymbolicLinkForSubmitDirectory(java.lang.String baseDir, java.lang.String relativeSubmitDir)
void
executeCommand()
Executes the command on the basis of the options specified.java.util.Collection<java.io.File>
executeCommand(PlannerOptions options)
Executes the command on the basis of the options specified.private void
generateStampedeEventsForAbstractWorkflow(ADag workflow, PegasusBag bag)
Generates events for the abstract workflow.gnu.getopt.LongOpt[]
generateValidOptions()
Tt generates the LongOpt which contain the valid options that the command will accept.protected java.lang.String
getDAGFilename(ADag dag, PlannerOptions options)
Returns the basename of the dag filejava.lang.String
getNOOPJobName(ADag dag)
Returns the name of the noop job.private java.lang.String
getPegasusRunInvocation()
Returns the pegasus-run invocation on the workflow planned.protected java.lang.String
getRandomDirectory(ADag dag)
Sets the basename of the random directory that is created on the remote sites per workflow.protected boolean
handleRescueDAG(ADag dag, PlannerOptions options)
Checks for rescue dags, and determines whether to plan or not.protected boolean
handleRescueDAG(java.lang.String dag, java.lang.String dir, int numOfRescues)
Checks for rescue dags, and determines whether to submit a rescue dag or not.void
initialize(java.lang.String[] opts, char confChar)
Initialize the executable objectvoid
loadProperties()
Loads all the properties that are needed by this class.private SiteStore
loadSiteStore(java.util.Set<java.lang.String> sites)
Loads the sites from the site catalog into the site storeprivate void
logMemoryUsage()
Logs memory usage of the JVMprivate void
logSuccessfulCompletion(boolean emptyWorkflow)
Logs the successful completion message.static void
main(java.lang.String[] args)
The main program for the CPlanner.PlannerOptions
parseCommandLineArguments(java.lang.String[] args)
Parses the command line arguments using GetOpt and returns aPlannerOptions
contains all the options passed by the user at the command line.PlannerOptions
parseCommandLineArguments(java.lang.String[] args, boolean sanitizePath)
Parses the command line arguments using GetOpt and returns aPlannerOptions
contains all the options passed by the user at the command line.private ADag
parseDAX(java.lang.String dax)
Parses the DAX and returns the associated ADag objectvoid
printLongVersion()
Prints the long description, displaying in detail what the various options to the command stand for.void
printShortVersion()
Prints out a short description of what the command does.protected static void
sanityCheck(java.io.File dir)
Checks the destination location for existence, if it can be created, if it is writable etc.boolean
submitWorkflow(java.lang.String invocation)
Submits the workflow for execution using pegasus-run, a wrapper around pegasus-submit-dag.-
Methods inherited from class edu.isi.pegasus.planner.client.Executable
convertException, convertException, getCommandLineOptions, getEnvValue, getGVDSVersion, initialize, log, lookupConfProperty, sanityCheckOnProperties, setupLogging
-
-
-
-
Field Detail
-
DEFAULT_MEGADAG_MODE
public static final java.lang.String DEFAULT_MEGADAG_MODE
The default megadag mode that is used for generation of megadags in deferred planning.- See Also:
- Constant Field Values
-
CLEANUP_DIR
public static final java.lang.String CLEANUP_DIR
The basename of the directory that contains the submit files for the cleanup DAG that for the concrete dag generated for the workflow.- See Also:
- Constant Field Values
-
NOOP_PREFIX
public static final java.lang.String NOOP_PREFIX
The prefix for the NoOP jobs that are created.- See Also:
- Constant Field Values
-
PEGASUS_MONITORD_LAUNCH_PROPERTY_KEY
public static final java.lang.String PEGASUS_MONITORD_LAUNCH_PROPERTY_KEY
The name of the property key that determines whether pegasus-run should monitord or not.- See Also:
- Constant Field Values
-
JAVA_COMMAND_LINE_PROPERTY_REGEX
public static final java.lang.String JAVA_COMMAND_LINE_PROPERTY_REGEX
The regex used to match against a java property that is set using -Dpropertyname=value in the argument string- See Also:
- Constant Field Values
-
EMPTY_FINAL_WORKFLOW_MESSAGE
private static final java.lang.String EMPTY_FINAL_WORKFLOW_MESSAGE
The final successful message that is to be logged.- See Also:
- Constant Field Values
-
SUCCESS_MESSAGE
private static final java.lang.String SUCCESS_MESSAGE
The message to be logged in case of empty executable workflow.- See Also:
- Constant Field Values
-
mPOptions
private PlannerOptions mPOptions
The object containing all the options passed to the Concrete Planner.
-
mBag
private PegasusBag mBag
The object containing the bag of pegasus objects
-
mPMetrics
private PlannerMetrics mPMetrics
The PlannerMetrics object storing the metrics about this planning instance.
-
mNumFormatter
private java.text.NumberFormat mNumFormatter
The number formatter to format the run submit dir entries.
-
mUser
private java.lang.String mUser
The user name of the user running Pegasus.
-
mSendMetrics
private boolean mSendMetrics
A boolean indicating whether metrics should be sent to metrics server or not
-
-
Constructor Detail
-
CPlanner
public CPlanner()
Default constructor.
-
CPlanner
public CPlanner(LogManager logger)
The overload constructor.- Parameters:
logger
- the logger object to use. can be null.
-
-
Method Detail
-
initialize
public void initialize(java.lang.String[] opts, char confChar)
Description copied from class:Executable
Initialize the executable object- Overrides:
initialize
in classExecutable
- Parameters:
opts
- the command line argument passed by the userconfChar
- the short option corresponding the conf property.
-
main
public static void main(java.lang.String[] args)
The main program for the CPlanner.- Parameters:
args
- the main arguments passed to the planner.
-
loadProperties
public void loadProperties()
Loads all the properties that are needed by this class.- Specified by:
loadProperties
in classExecutable
-
executeCommand
public void executeCommand()
Executes the command on the basis of the options specified.- Parameters:
args
- the command line options.
-
executeCommand
public java.util.Collection<java.io.File> executeCommand(PlannerOptions options)
Executes the command on the basis of the options specified.- Parameters:
options
- the command line options.- Returns:
- the Collection of
File
objects for the files written out.
-
getNOOPJobName
public java.lang.String getNOOPJobName(ADag dag)
Returns the name of the noop job.- Parameters:
dag
- the workflow- Returns:
- the name
-
createNoOPJob
protected Job createNoOPJob(java.lang.String name)
It creates a NoOP job that runs on the submit host.- Parameters:
name
- the name to be assigned to the noop job- Returns:
- the noop job.
-
construct
protected void construct(Job job, java.lang.String key, java.lang.String value)
Constructs a condor variable in the condor profile namespace associated with the job. Overrides any preexisting key values.- Parameters:
job
- contains the job description.key
- the key of the profile.value
- the associated value.
-
parseCommandLineArguments
public PlannerOptions parseCommandLineArguments(java.lang.String[] args)
Parses the command line arguments using GetOpt and returns aPlannerOptions
contains all the options passed by the user at the command line.- Parameters:
args
- the arguments passed by the user at command line.- Returns:
- the options.
-
parseCommandLineArguments
public PlannerOptions parseCommandLineArguments(java.lang.String[] args, boolean sanitizePath)
Parses the command line arguments using GetOpt and returns aPlannerOptions
contains all the options passed by the user at the command line.- Parameters:
args
- the arguments passed by the user at command line.sanitizePath
- whether to sanitize path during construction of options- Returns:
- the options.
-
submitWorkflow
public boolean submitWorkflow(java.lang.String invocation)
Submits the workflow for execution using pegasus-run, a wrapper around pegasus-submit-dag.- Parameters:
invocation
- the pegasus run invocation- Returns:
- boolean indicating whether could successfully submit the workflow or not.
-
getRandomDirectory
protected java.lang.String getRandomDirectory(ADag dag)
Sets the basename of the random directory that is created on the remote sites per workflow. The name is generated by default from teh flow ID, unless a basename prefix is specifed at runtime in the planner options.- Parameters:
dag
- the DAG containing the abstract workflow.- Returns:
- the basename of the random directory.
-
generateValidOptions
public gnu.getopt.LongOpt[] generateValidOptions()
Tt generates the LongOpt which contain the valid options that the command will accept.- Specified by:
generateValidOptions
in classExecutable
- Returns:
- array of
LongOpt
objects , corresponding to the valid options
-
printShortVersion
public void printShortVersion()
Prints out a short description of what the command does.- Specified by:
printShortVersion
in classExecutable
-
printLongVersion
public void printLongVersion()
Prints the long description, displaying in detail what the various options to the command stand for.- Specified by:
printLongVersion
in classExecutable
-
determineRootWorkflowUUID
private java.lang.String determineRootWorkflowUUID(ADag dag, PlannerOptions options, PegasusProperties properties)
Determines the workflow uuid for a workflow- Parameters:
dag
- the workflowoptions
- the options passed to the plannerproperties
- the properties passed to the planner- Returns:
- uuid for the root workflow instance
-
determineRelativeSubmitDirectory
protected java.lang.String determineRelativeSubmitDirectory(ADag dag, java.lang.String dir, java.lang.String user, java.lang.String vogroup, boolean timestampBased) throws java.io.IOException
Creates the submit directory for the workflow. This is not thread safe.- Parameters:
dag
- the workflow being worked upon.dir
- the base directory specified by the user.user
- the username of the user.vogroup
- the vogroup to which the user belongs to.timestampBased
- boolean indicating whether to have a timestamp based dir or not- Returns:
- the directory name created relative to the base directory passed as input.
- Throws:
java.io.IOException
- in case of unable to create submit directory.
-
determineRelativeSubmitDirectory
protected java.lang.String determineRelativeSubmitDirectory(java.lang.String label, java.lang.String dir, java.lang.String user, java.lang.String vogroup, boolean timestampBased) throws java.io.IOException
Creates the submit directory for the workflow. This is not thread safe.- Parameters:
label
- the label of the workflowdir
- the base directory specified by the user.user
- the username of the user.vogroup
- the vogroup to which the user belongs to.timestampBased
- boolean indicating whether to have a timestamp based dir or not- Returns:
- the directory name created relative to the base directory passed as input.
- Throws:
java.io.IOException
- in case of unable to create submit directory.
-
sanityCheck
protected static void sanityCheck(java.io.File dir) throws java.io.IOException
Checks the destination location for existence, if it can be created, if it is writable etc.- Parameters:
dir
- is the new base directory to optionally create.- Throws:
java.io.IOException
- in case of error while writing out files.
-
getDAGFilename
protected java.lang.String getDAGFilename(ADag dag, PlannerOptions options)
Returns the basename of the dag file- Parameters:
dag
- the dag that was parsed.options
- the planner options- Returns:
- boolean true means submit the rescue false do the planning operation
-
handleRescueDAG
protected boolean handleRescueDAG(ADag dag, PlannerOptions options)
Checks for rescue dags, and determines whether to plan or not.- Parameters:
dag
- the dag that was parsed.options
- the planner options- Returns:
- boolean true means submit the rescue false do the planning operation
-
handleRescueDAG
protected boolean handleRescueDAG(java.lang.String dag, java.lang.String dir, int numOfRescues)
Checks for rescue dags, and determines whether to submit a rescue dag or not.- Parameters:
dag
- the dag file for the daxdir
- the submit directory.numOfRescues
- the number of rescues to handle.- Returns:
- true means submit the rescue false do the planning operation
-
createSymbolicLink
protected boolean createSymbolicLink(java.lang.String source, java.lang.String destination, java.io.File directory, boolean logErrorToDebug)
This method generates a symlink between two files- Parameters:
source
- the file that has to be symlinkeddestination
- the destination of the symlinkdirectory
- the directory in which to execute the commandlogErrorToDebug
- whether to log messeage to debug or not- Returns:
- boolean indicating if creation of symlink was successful or not
-
generateStampedeEventsForAbstractWorkflow
private void generateStampedeEventsForAbstractWorkflow(ADag workflow, PegasusBag bag)
Generates events for the abstract workflow.- Parameters:
workflow
- the parsed daxbag
- the initialized object bag
-
loadSiteStore
private SiteStore loadSiteStore(java.util.Set<java.lang.String> sites)
Loads the sites from the site catalog into the site store- Parameters:
sites
-- Returns:
- SiteStore object containing the information about the sites.
-
logSuccessfulCompletion
private void logSuccessfulCompletion(boolean emptyWorkflow)
Logs the successful completion message.- Parameters:
emptyWorkflow
- indicates whether the workflow created was empty or not.
-
getPegasusRunInvocation
private java.lang.String getPegasusRunInvocation()
Returns the pegasus-run invocation on the workflow planned.- Returns:
- the pegasus-run invocation
-
doBackupAndCreateSymbolicLinkForSubmitDirectory
protected java.lang.String doBackupAndCreateSymbolicLinkForSubmitDirectory(java.lang.String baseDir, java.lang.String relativeSubmitDir) throws java.io.IOException
- Throws:
java.io.IOException
-
logMemoryUsage
private void logMemoryUsage()
Logs memory usage of the JVM
-
parseDAX
private ADag parseDAX(java.lang.String dax)
Parses the DAX and returns the associated ADag object- Parameters:
dax
- path to the DAX file.- Returns:
-
-