Class Transfer

  • All Implemented Interfaces:
    Implementation, MultipleFTPerXFERJob

    public class Transfer
    extends AbstractMultipleFTPerXFERJob
    The implementation that creates transfer jobs referring to the python based transfer script distributed with Pegasus since version 3.0

    Transfer is distributed as part of the Pegasus worker package and can be found at $PEGASUS_HOME/bin/pegasus-transfer.

    It leads to the creation of the setup chmod jobs to the workflow, that appear as parents to compute jobs in case the transfer implementation does not preserve the X bit on the file being transferred. This is required for staging of executables as part of the workflow. The setup jobs are only added as children to the stage in jobs.

    In order to use the transfer implementation implemented by this class, the property pegasus.transfer.*.impl must be set to value Transfer. The arguments with which the pegasus-transfer client is invoked can be specified

           - by specifying the property pegasus.transfer.arguments
           - associating the Pegasus profile key transfer.arguments
     
    Version:
    $Revision$
    Author:
    Karan Vahi
    • Field Detail

      • TRANSFORMATION_NAMESPACE

        public static final java.lang.String TRANSFORMATION_NAMESPACE
        The transformation namespace for the transfer job.
        See Also:
        Constant Field Values
      • TRANSFORMATION_NAME

        public static final java.lang.String TRANSFORMATION_NAME
        The name of the underlying transformation that is queried for in the Transformation Catalog.
        See Also:
        Constant Field Values
      • TRANSFORMATION_VERSION

        public static final java.lang.String TRANSFORMATION_VERSION
        The version number for the transfer job.
      • DERIVATION_NAMESPACE

        public static final java.lang.String DERIVATION_NAMESPACE
        The derivation namespace for for the transfer job.
        See Also:
        Constant Field Values
      • DERIVATION_NAME

        public static final java.lang.String DERIVATION_NAME
        The name of the underlying derivation.
        See Also:
        Constant Field Values
      • DERIVATION_VERSION

        public static final java.lang.String DERIVATION_VERSION
        The derivation version number for the transfer job.
        See Also:
        Constant Field Values
      • DEFAULT_NUMBER_OF_THREADS

        public static final int DEFAULT_NUMBER_OF_THREADS
        The default number of threads pegasus-transfer uses
        See Also:
        Constant Field Values
      • DESCRIPTION

        public static final java.lang.String DESCRIPTION
        A short description of the transfer implementation.
        See Also:
        Constant Field Values
      • EXECUTABLE_BASENAME

        public static final java.lang.String EXECUTABLE_BASENAME
        The executable basename for the transfer executable.
        See Also:
        Constant Field Values
    • Constructor Detail

      • Transfer

        public Transfer​(PegasusBag bag)
        The overloaded constructor, that is called by the Factory to load the class.
        Parameters:
        bag - the bag of initialization objects.
    • Method Detail

      • useThirdPartyTransferAlways

        public boolean useThirdPartyTransferAlways()
        Return a boolean indicating whether the transfers to be done always in a third party transfer mode. A value of false, results in the direct or peer to peer transfers being done.

        A value of false does not preclude third party transfers. They still can be done, by setting the property "pegasus.transfer.*.thirdparty.sites".

        Returns:
        boolean indicating whether to always use third party transfers or not.
        See Also:
        PegasusProperties.getThirdPartySites(String)
      • doesPreserveXBit

        public boolean doesPreserveXBit()
        Returns a boolean indicating whether the transfer protocol being used by the implementation preserves the X Bit or not while staging.
        Returns:
        boolean
      • getDescription

        public java.lang.String getDescription()
        Returns a textual description of the transfer implementation.
        Returns:
        a short textual description
      • getTransformationCatalogEntry

        public TransformationCatalogEntry getTransformationCatalogEntry​(java.lang.String siteHandle,
                                                                        int jobClass)
        Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation.
        Parameters:
        siteHandle - the handle of the site where the transformation is to be searched.
        jobClass - the job Class for the newly added job. Can be one of the following: stage-in stage-out inter-pool transfer stage-in worker transfer
        Returns:
        the transformation catalog entry if found, else null.
      • postProcess

        public void postProcess​(TransferJob job)
        An optional method that allows the derived classes to do their own post processing on the the transfer job before it is returned to the calling module.
        Overrides:
        postProcess in class AbstractMultipleFTPerXFERJob
        Parameters:
        job - the TransferJob that has been created.
      • getEnvironmentVariables

        protected java.util.List getEnvironmentVariables​(java.lang.String site)
        Returns the environment profiles that are required for the default entry to sensibly work. Tries to retrieve the following variables
         PEGASUS_HOME
         GLOBUS_LOCATION
         LD_LIBRARY_PATH
         
        Specified by:
        getEnvironmentVariables in class AbstractMultipleFTPerXFERJob
        Parameters:
        site - the site where the job is going to run.
        Returns:
        List of environment variables, else empty list if none are found
      • getDerivationNamespace

        protected java.lang.String getDerivationNamespace()
        Returns the namespace of the derivation that this implementation refers to.
        Specified by:
        getDerivationNamespace in class AbstractMultipleFTPerXFERJob
        Returns:
        the namespace of the derivation.
      • getDerivationName

        protected java.lang.String getDerivationName()
        Returns the logical name of the derivation that this implementation refers to.
        Specified by:
        getDerivationName in class AbstractMultipleFTPerXFERJob
        Returns:
        the name of the derivation.
      • getDerivationVersion

        protected java.lang.String getDerivationVersion()
        Returns the version of the derivation that this implementation refers to.
        Specified by:
        getDerivationVersion in class AbstractMultipleFTPerXFERJob
        Returns:
        the version of the derivation.
      • generateArgumentString

        protected java.lang.String generateArgumentString​(TransferJob job)
        It constructs the arguments to the transfer executable that need to be passed to the executable referred to in this transfer mode.
        Specified by:
        generateArgumentString in class AbstractMultipleFTPerXFERJob
        Parameters:
        job - the object containing the transfer node.
        Returns:
        the argument string
      • writeStdInAndAssociateCredentials

        protected void writeStdInAndAssociateCredentials​(TransferJob job,
                                                         java.io.FileWriter writer,
                                                         java.util.Collection files,
                                                         java.lang.String stagingSite,
                                                         int jobClass)
                                                  throws java.lang.Exception
        Writes to a FileWriter stream the stdin which goes into the magic script via standard input
        Specified by:
        writeStdInAndAssociateCredentials in class AbstractMultipleFTPerXFERJob
        Parameters:
        job - the transfer job
        writer - the writer to the stdin file.
        files - Collection of FileTransfer objects containing the information about sourceam fin and destURL's.
        stagingSite - the site where the data will be populated by first level staging jobs.
        jobClass - the job Class for the newly added job. Can be one of the following: stage-in stage-out inter-pool transfer
        Throws:
        java.lang.Exception