Class Transfer
- java.lang.Object
-
- edu.isi.pegasus.planner.transfer.sls.Transfer
-
-
Field Summary
Fields Modifier and Type Field Description static int
DEFAULT_NUMBER_OF_THREADS
The default number of threads pegasus-transfer usesstatic java.lang.String
DERIVATION_NAME
The name of the underlying derivation.static java.lang.String
DERIVATION_NAMESPACE
The derivation namespace for for the transfer job.static java.lang.String
DERIVATION_VERSION
The derivation version number for the transfer job.static java.lang.String
DESCRIPTION
A short description of the transfer implementation.static java.lang.String
EXECUTABLE_BASENAME
The executable basenameprivate boolean
mBypassStagingForInputs
A boolean indicating whether to bypass first level staging for inputsprotected java.lang.String
mExtraArguments
Any extra arguments that need to be passed ahead to the s3 client invocation.protected LogManager
mLogger
The handle to the logging manager.private PlannerCache
mPlannerCache
A SimpleFile Replica Catalog, that tracks all the files that are being materialized as part of workflow execution.protected PegasusProperties
mProps
The handle to the properties.protected boolean
mSeqExecGridStartUsed
Boolean to track whether the gridstart used in PegasusLite or notprotected SiteStore
mSiteStore
The handle to the site catalog.protected boolean
mStageSLSFile
Boolean to track whether to stage sls file or notprotected TransformationCatalog
mTCHandle
The handle to the transformation catalog.protected boolean
mUseSymLinks
This member variable if set causes the destination URL for the symlink jobs to have symlink:// url if the pool attributed associated with the pfn is same as a particular jobs execution pool.static java.lang.String
TRANSFORMATION_NAME
The name of the underlying transformation that is queried for in the Transformation Catalog.static java.lang.String
TRANSFORMATION_NAMESPACE
The transformation namespace for the transfer job.static java.lang.String
TRANSFORMATION_VERSION
The version number for the transfer job.
-
Constructor Summary
Constructors Constructor Description Transfer()
The default constructor.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
complainForHeadNodeURLPrefix(Job job, java.lang.String site)
Complains for head node url prefix not specifiedprotected void
complainForHeadNodeURLPrefix(java.lang.String site)
Complains for head node url prefix not specifiedprotected TransformationCatalogEntry
defaultTCEntry(java.lang.String namespace, java.lang.String name, java.lang.String version, java.lang.String executableBasename, java.lang.String site)
Returns a default TC entry to be used in case entry is not found in the transformation catalog.java.util.Collection<FileTransfer>
determineSLSInputTransfers(Job job, java.lang.String fileName, FileServer stagingSiteServer, java.lang.String stagingSiteDirectory, java.lang.String workerNodeDirectory)
Generates a second level staging file of the input files to the worker node directory.java.util.Collection<FileTransfer>
determineSLSOutputTransfers(Job job, java.lang.String fileName, FileServer stagingSiteServer, java.lang.String stagingSiteDirectory, java.lang.String workerNodeDirectory)
Generates a second level staging file of the input files to the worker node directory.boolean
doesCondorModifications()
Returns a boolean whether the SLS implementation does a condor based modification or not.protected java.util.List
getEnvironmentVariables(java.lang.String site)
Returns the environment profiles that are required for the default entry to sensibly work.protected java.lang.String
getExecutableBasename()
Return the executable basename for transfer executable used.java.lang.String
getSLSInputLFN(Job job)
Returns the LFN of sls input file.java.lang.String
getSLSOutputLFN(Job job)
Returns the LFN of sls output file.TransformationCatalogEntry
getTransformationCatalogEntry(java.lang.String siteHandle)
Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation.void
initialize(PegasusBag bag)
Initializes the SLS implementation.java.lang.String
invocationString(Job job, java.io.File slsFile)
Constructs a command line invocation for a job, with a given sls file.boolean
modifyJobForWorkerNodeExecution(Job job, java.lang.String stagingSiteURLPrefix, java.lang.String stagingSitedirectory, java.lang.String workerNodeDirectory)
Modifies a compute job for second level staging.boolean
needsSLSInputTransfers(Job job)
Returns a boolean indicating whether it will an input file for a job to do the transfers.boolean
needsSLSOutputTransfers(Job job)
Returns a boolean indicating whether it will an output file for a job to do the transfers.
-
-
-
Field Detail
-
TRANSFORMATION_NAMESPACE
public static final java.lang.String TRANSFORMATION_NAMESPACE
The transformation namespace for the transfer job.- See Also:
- Constant Field Values
-
TRANSFORMATION_NAME
public static final java.lang.String TRANSFORMATION_NAME
The name of the underlying transformation that is queried for in the Transformation Catalog.- See Also:
- Constant Field Values
-
TRANSFORMATION_VERSION
public static final java.lang.String TRANSFORMATION_VERSION
The version number for the transfer job.
-
DERIVATION_NAMESPACE
public static final java.lang.String DERIVATION_NAMESPACE
The derivation namespace for for the transfer job.- See Also:
- Constant Field Values
-
DERIVATION_NAME
public static final java.lang.String DERIVATION_NAME
The name of the underlying derivation.- See Also:
- Constant Field Values
-
DERIVATION_VERSION
public static final java.lang.String DERIVATION_VERSION
The derivation version number for the transfer job.- See Also:
- Constant Field Values
-
DEFAULT_NUMBER_OF_THREADS
public static final int DEFAULT_NUMBER_OF_THREADS
The default number of threads pegasus-transfer uses- See Also:
- Constant Field Values
-
DESCRIPTION
public static final java.lang.String DESCRIPTION
A short description of the transfer implementation.- See Also:
- Constant Field Values
-
EXECUTABLE_BASENAME
public static final java.lang.String EXECUTABLE_BASENAME
The executable basename- See Also:
- Constant Field Values
-
mSiteStore
protected SiteStore mSiteStore
The handle to the site catalog.
-
mTCHandle
protected TransformationCatalog mTCHandle
The handle to the transformation catalog.
-
mProps
protected PegasusProperties mProps
The handle to the properties.
-
mLogger
protected LogManager mLogger
The handle to the logging manager.
-
mExtraArguments
protected java.lang.String mExtraArguments
Any extra arguments that need to be passed ahead to the s3 client invocation.
-
mStageSLSFile
protected boolean mStageSLSFile
Boolean to track whether to stage sls file or not
-
mSeqExecGridStartUsed
protected boolean mSeqExecGridStartUsed
Boolean to track whether the gridstart used in PegasusLite or not
-
mBypassStagingForInputs
private boolean mBypassStagingForInputs
A boolean indicating whether to bypass first level staging for inputs
-
mPlannerCache
private PlannerCache mPlannerCache
A SimpleFile Replica Catalog, that tracks all the files that are being materialized as part of workflow execution.
-
mUseSymLinks
protected boolean mUseSymLinks
This member variable if set causes the destination URL for the symlink jobs to have symlink:// url if the pool attributed associated with the pfn is same as a particular jobs execution pool.
-
-
Method Detail
-
initialize
public void initialize(PegasusBag bag)
Initializes the SLS implementation.- Specified by:
initialize
in interfaceSLS
- Parameters:
bag
- the bag of objects. Contains access to catalogs etc.
-
doesCondorModifications
public boolean doesCondorModifications()
Returns a boolean whether the SLS implementation does a condor based modification or not. By condor based modification we mean whether it uses condor specific classads to achieve the second level staging or not.- Specified by:
doesCondorModifications
in interfaceSLS
- Returns:
- false
-
invocationString
public java.lang.String invocationString(Job job, java.io.File slsFile)
Constructs a command line invocation for a job, with a given sls file. The SLS maybe null. In the case where SLS impl does not read from a file, it is advised to create a file in generateSLSXXX methods, and then read the file in this function and put it on the command line.- Specified by:
invocationString
in interfaceSLS
- Parameters:
job
- the job that is being sls enabledslsFile
- the slsFile can be null- Returns:
- invocation string
-
needsSLSInputTransfers
public boolean needsSLSInputTransfers(Job job)
Returns a boolean indicating whether it will an input file for a job to do the transfers. Transfer reads from stdin the file transfers that it needs to do. Always returns true, as we need to transfer the proxy always.- Specified by:
needsSLSInputTransfers
in interfaceSLS
- Parameters:
job
- the job being detected.- Returns:
- true
-
needsSLSOutputTransfers
public boolean needsSLSOutputTransfers(Job job)
Returns a boolean indicating whether it will an output file for a job to do the transfers. Transfer reads from stdin the file transfers that it needs to do.- Specified by:
needsSLSOutputTransfers
in interfaceSLS
- Parameters:
job
- the job being detected.- Returns:
- true
-
getSLSInputLFN
public java.lang.String getSLSInputLFN(Job job)
Returns the LFN of sls input file.- Specified by:
getSLSInputLFN
in interfaceSLS
- Parameters:
job
- Job- Returns:
- the name of the sls input file.
-
getSLSOutputLFN
public java.lang.String getSLSOutputLFN(Job job)
Returns the LFN of sls output file.- Specified by:
getSLSOutputLFN
in interfaceSLS
- Parameters:
job
- Job- Returns:
- the name of the sls input file.
-
determineSLSInputTransfers
public java.util.Collection<FileTransfer> determineSLSInputTransfers(Job job, java.lang.String fileName, FileServer stagingSiteServer, java.lang.String stagingSiteDirectory, java.lang.String workerNodeDirectory)
Generates a second level staging file of the input files to the worker node directory.- Specified by:
determineSLSInputTransfers
in interfaceSLS
- Parameters:
job
- job for which the file is being createdfileName
- name of the file that needs to be written out.stagingSiteServer
- the file server on the staging site to be used for retrieval of files i.e the get operationstagingSiteDirectory
- directory on the head node of the staging site.workerNodeDirectory
- worker node directory- Returns:
- a Collection of FileTransfer objects listing the transfers that need to be done.
- See Also:
needsSLSInputTransfers( Job)
-
determineSLSOutputTransfers
public java.util.Collection<FileTransfer> determineSLSOutputTransfers(Job job, java.lang.String fileName, FileServer stagingSiteServer, java.lang.String stagingSiteDirectory, java.lang.String workerNodeDirectory)
Generates a second level staging file of the input files to the worker node directory.- Specified by:
determineSLSOutputTransfers
in interfaceSLS
- Parameters:
job
- the job for which the file is being createdfileName
- the name of the file that needs to be written out.stagingSiteServer
- the file server on the staging site to be used for retrieval of files i.e the put operationstagingSiteDirectory
- the directory on the head node of the staging site.workerNodeDirectory
- the worker node directory- Returns:
- a Collection of FileTransfer objects listing the transfers that need to be done.
- See Also:
needsSLSOutputTransfers( Job)
-
modifyJobForWorkerNodeExecution
public boolean modifyJobForWorkerNodeExecution(Job job, java.lang.String stagingSiteURLPrefix, java.lang.String stagingSitedirectory, java.lang.String workerNodeDirectory)
Modifies a compute job for second level staging. The only modification it does is add the appropriate environment varialbes to the job- Specified by:
modifyJobForWorkerNodeExecution
in interfaceSLS
- Parameters:
job
- the job to be modified.stagingSiteURLPrefix
- the url prefix for the server on the staging sitestagingSitedirectory
- the directory on the staging site, where the inpworkerNodeDirectory
- the directory in the worker node tmp- Returns:
- boolean indicating whether job was successfully modified or not.
-
getTransformationCatalogEntry
public TransformationCatalogEntry getTransformationCatalogEntry(java.lang.String siteHandle)
Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation. If an entry is not specified in the Transformation Catalog, then null is returned.- Parameters:
siteHandle
- the handle of the site where the transformation is to be searched.- Returns:
- the transformation catalog entry if found, else null.
-
defaultTCEntry
protected TransformationCatalogEntry defaultTCEntry(java.lang.String namespace, java.lang.String name, java.lang.String version, java.lang.String executableBasename, java.lang.String site)
Returns a default TC entry to be used in case entry is not found in the transformation catalog.- Parameters:
namespace
- the namespace of the transfer transformationname
- the logical name of the transfer transformationversion
- the version of the transfer transformationexecutableBasename
- the basename of the executablesite
- the site for which the default entry is required.- Returns:
- the default entry.
-
getEnvironmentVariables
protected java.util.List getEnvironmentVariables(java.lang.String site)
Returns the environment profiles that are required for the default entry to sensibly work. Tries to retrieve the following variablesPEGASUS_HOME GLOBUS_LOCATION LD_LIBRARY_PATH
- Parameters:
site
- the site where the job is going to run.- Returns:
- List of environment variables, else empty list if none are found
-
getExecutableBasename
protected java.lang.String getExecutableBasename()
Return the executable basename for transfer executable used.- Returns:
- the executable basename.
-
complainForHeadNodeURLPrefix
protected void complainForHeadNodeURLPrefix(java.lang.String site)
Complains for head node url prefix not specified- Parameters:
site
- the site handle- Throws:
java.lang.RuntimeException
- when URL Prefix cannot be determined for various reason.
-
complainForHeadNodeURLPrefix
protected void complainForHeadNodeURLPrefix(Job job, java.lang.String site)
Complains for head node url prefix not specified- Parameters:
job
- the related job if anysite
- the site handle- Throws:
java.lang.RuntimeException
- when URL Prefix cannot be determined for various reason.
-
-