Class Directory

  • All Implemented Interfaces:
    Catalog, ReplicaCatalog

    public class Directory
    extends java.lang.Object
    implements ReplicaCatalog
    This class implements a replica catalog on top of a directory. This implementation does a directory listing to build up the lfn, pfn mappings for the Replica Catalog. To connect to this implementation, in Pegasus Properties set pegasus.catalog.replica Directory The site attribute defaults to local unless specified in Pegasus Properties by specifying the property
          pegasus.catalog.replica.directory.site 
     
    The URL prefix for the PFN's defaults to file:// unless specified in Pegasus Properties by specifying the property
          pegasus.catalog.replica.directory.url.prefix
     
    By default, deep LFN's are constructed while traversing through the directory, unless the following property is set to true
         pegasus.catalog.replica.directory.flat.lfn
     
    Version:
    $Revision$
    Author:
    Karan Vahi
    • Field Summary

      Fields 
      Modifier and Type Field Description
      static java.lang.String DEFAULT_SITE_HANDLE
      The default site handle to use.
      static java.lang.String DEFAULT_URL_PREFIX
      The default URL Prefix to use.
      static java.lang.String DIRECTORY_PROPERTY_KEY
      The name of the key that specifies the path to directory.
      static java.lang.String FLAT_LFN_PROPERTY_KEY
      the name of the key that specifies whether we want flat lfns or not.
      (package private) boolean mConstructFlatLFN
      A boolean indicating whether the catalog is to construct flat lfns or not
      protected java.lang.String mDirectory
      Records the name of the on-disk representation.
      protected java.util.Map mLFNMap
      Maintains a memory slurp of the file representation.
      (package private) boolean mReadOnly
      A boolean indicating whether the catalog is read only or not.
      protected java.lang.String mSiteHandle
      The site handle to use.
      protected java.lang.String mURLPrefix
      The URL prefix to use
      static java.lang.String SITE_PROPERTY_KEY
      The name of the key that specifies the site attribute to be associated
      static java.lang.String URL_PRFIX_PROPERTY_KEY
      The name of the key that specifies the url prefix to be associated with the PFN's
    • Constructor Summary

      Constructors 
      Constructor Description
      Directory()
      Default empty constructor creates an object that is not yet connected to any database.
    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      int clear()
      Removes everything.
      void close()
      This operation will dump the in-memory representation back onto disk.
      boolean connect​(java.lang.String directory)
      Does the file listing on the directory to create the mappings in memory.
      boolean connect​(java.util.Properties props)
      Establishes a connection to the database from the properties.
      int delete​(java.lang.String lfn, ReplicaCatalogEntry tuple)
      Deletes a very specific mapping from the replica catalog.
      int delete​(java.lang.String lfn, java.lang.String pfn)
      Deletes a specific mapping from the replica catalog.
      int delete​(java.lang.String lfn, java.lang.String name, java.lang.Object value)
      Deletes all PFN entries for a given LFN from the replica catalog where the PFN attribute is found, and matches exactly the object value.
      int delete​(java.util.Map x, boolean matchAttributes)
      Deletes multiple mappings into the replica catalog.
      int deleteByResource​(java.lang.String lfn, java.lang.String handle)
      Deletes all PFN entries for a given LFN from the replica catalog where the resource handle is found.
      private boolean hasMatchingAttr​(ReplicaCatalogEntry rce, java.lang.String name, java.lang.Object value)
      Looks for a match of an attribute value in a replica catalog entry.
      int insert​(java.lang.String lfn, ReplicaCatalogEntry tuple)
      Inserts a new mapping into the replica catalog.
      int insert​(java.lang.String lfn, java.lang.String pfn, java.lang.String handle)
      Inserts a new mapping into the replica catalog.
      int insert​(java.util.Map x)
      Inserts multiple mappings into the replica catalog.
      boolean isClosed()
      Predicate to check, if the connection with the catalog's implementation is still active.
      java.util.Set list()
      Lists all logical filenames in the catalog.
      java.util.Set list​(java.lang.String constraint)
      Lists a subset of all logical filenames in the catalog.
      java.util.Collection lookup​(java.lang.String lfn)
      Retrieves all entries for a given LFN from the replica catalog.
      java.lang.String lookup​(java.lang.String lfn, java.lang.String handle)
      Retrieves the entry for a given directory and site handle from the replica catalog.
      java.util.Map lookup​(java.util.Map constraints)
      Retrieves multiple entries for a given logical directory, up to the complete catalog.
      java.util.Map lookup​(java.util.Set lfns)
      Retrieves multiple entries for a given logical directory, up to the complete catalog.
      java.util.Map lookup​(java.util.Set lfns, java.lang.String handle)
      Retrieves multiple entries for a given logical directory, up to the complete catalog.
      java.util.Set lookupNoAttributes​(java.lang.String lfn)
      Retrieves all entries for a given LFN from the replica catalog.
      java.util.Map lookupNoAttributes​(java.util.Set lfns)
      Retrieves multiple entries for a given logical directory, up to the complete catalog.
      java.util.Map lookupNoAttributes​(java.util.Set lfns, java.lang.String handle)
      Retrieves multiple entries for a given logical directory, up to the complete catalog.
      static void main​(java.lang.String[] args)
      The main program
      private boolean matchMe​(ReplicaCatalogEntry full, ReplicaCatalogEntry part)
      Attempts to see, if all keys in the partial replica catalog entry are contained in the full replica catalog entry.
      int remove​(java.lang.String lfn)
      Removes all mappings for an LFN from the replica catalog.
      int remove​(java.util.Set lfns)
      Removes all mappings for a set of LFNs.
      int removeByAttribute​(java.lang.String handle)
      Removes all entries associated with a particular resource handle.
      int removeByAttribute​(java.lang.String name, java.lang.Object value)
      Removes all entries from the replica catalog where the PFN attribute is found, and matches exactly the object value.
      private void traverse​(java.io.File directory, java.lang.String prefix)
      Traverses a directory and populates the mappings in memory
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Field Detail

      • DEFAULT_SITE_HANDLE

        public static final java.lang.String DEFAULT_SITE_HANDLE
        The default site handle to use.
        See Also:
        Constant Field Values
      • DEFAULT_URL_PREFIX

        public static final java.lang.String DEFAULT_URL_PREFIX
        The default URL Prefix to use.
        See Also:
        Constant Field Values
      • DIRECTORY_PROPERTY_KEY

        public static final java.lang.String DIRECTORY_PROPERTY_KEY
        The name of the key that specifies the path to directory.
        See Also:
        Constant Field Values
      • SITE_PROPERTY_KEY

        public static final java.lang.String SITE_PROPERTY_KEY
        The name of the key that specifies the site attribute to be associated
        See Also:
        Constant Field Values
      • URL_PRFIX_PROPERTY_KEY

        public static final java.lang.String URL_PRFIX_PROPERTY_KEY
        The name of the key that specifies the url prefix to be associated with the PFN's
        See Also:
        Constant Field Values
      • FLAT_LFN_PROPERTY_KEY

        public static final java.lang.String FLAT_LFN_PROPERTY_KEY
        the name of the key that specifies whether we want flat lfns or not. By default it is false, i.e we construct deep lfn's when traversing through the directory hierarchy
        See Also:
        Constant Field Values
      • mDirectory

        protected java.lang.String mDirectory
        Records the name of the on-disk representation.
      • mLFNMap

        protected java.util.Map mLFNMap
        Maintains a memory slurp of the file representation.
      • mReadOnly

        boolean mReadOnly
        A boolean indicating whether the catalog is read only or not.
      • mConstructFlatLFN

        boolean mConstructFlatLFN
        A boolean indicating whether the catalog is to construct flat lfns or not
      • mSiteHandle

        protected java.lang.String mSiteHandle
        The site handle to use.
      • mURLPrefix

        protected java.lang.String mURLPrefix
        The URL prefix to use
    • Constructor Detail

      • Directory

        public Directory()
        Default empty constructor creates an object that is not yet connected to any database. You must use support methods to connect before this instance becomes usable.
        See Also:
        connect( Properties )
    • Method Detail

      • connect

        public boolean connect​(java.util.Properties props)
        Establishes a connection to the database from the properties. You will need to specify a "file" property to point to the location of the on-disk instance. If the property "quote" is set to a true value, LFNs and PFNs are always quoted. By default, and if false, LFNs and PFNs are only quoted as necessary.
        Specified by:
        connect in interface Catalog
        Parameters:
        props - is the property table with sufficient settings to establish a link with the database.
        Returns:
        true if connected, false if failed to connect.
        Throws:
        java.lang.Error - subclasses for runtime errors in the class loader.
      • connect

        public boolean connect​(java.lang.String directory)
        Does the file listing on the directory to create the mappings in memory.
        Parameters:
        directory - is the name of the file to read.
        Returns:
        true, if the in-memory data structures appear sound.
      • traverse

        private void traverse​(java.io.File directory,
                              java.lang.String prefix)
        Traverses a directory and populates the mappings in memory
        Parameters:
        directory - the directory to traverse.
        prefix - the LFN prefix to be applied
      • close

        public void close()
        This operation will dump the in-memory representation back onto disk. The store operation is strict in what it produces. The LFN and PFN records are only quoted, if they require quotes, because they contain special characters. The attributes are always quoted and thus quote-escaped.
        Specified by:
        close in interface Catalog
      • isClosed

        public boolean isClosed()
        Predicate to check, if the connection with the catalog's implementation is still active. This helps determining, if it makes sense to call close().
        Specified by:
        isClosed in interface Catalog
        Returns:
        true, if the implementation is disassociated, false otherwise.
        See Also:
        close()
      • lookup

        public java.lang.String lookup​(java.lang.String lfn,
                                       java.lang.String handle)
        Retrieves the entry for a given directory and site handle from the replica catalog.
        Specified by:
        lookup in interface ReplicaCatalog
        Parameters:
        lfn - is the logical directory to obtain information for.
        handle - is the resource handle to obtain entries for.
        Returns:
        the (first) matching physical directory, or null if no match was found.
      • lookup

        public java.util.Collection lookup​(java.lang.String lfn)
        Retrieves all entries for a given LFN from the replica catalog. Each entry in the result set is a tuple of a PFN and all its attributes.
        Specified by:
        lookup in interface ReplicaCatalog
        Parameters:
        lfn - is the logical directory to obtain information for.
        Returns:
        a collection of replica catalog entries
        See Also:
        ReplicaCatalogEntry
      • lookupNoAttributes

        public java.util.Set lookupNoAttributes​(java.lang.String lfn)
        Retrieves all entries for a given LFN from the replica catalog. Each entry in the result set is just a PFN string. Duplicates are reduced through the set paradigm.
        Specified by:
        lookupNoAttributes in interface ReplicaCatalog
        Parameters:
        lfn - is the logical directory to obtain information for.
        Returns:
        a set of PFN strings
      • lookup

        public java.util.Map lookup​(java.util.Set lfns)
        Retrieves multiple entries for a given logical directory, up to the complete catalog. Retrieving full catalogs should be harmful, but may be helpful in an online display or portal.
        Specified by:
        lookup in interface ReplicaCatalog
        Parameters:
        lfns - is a set of logical directory strings to look up.
        Returns:
        a map indexed by the LFN. Each value is a collection of replica catalog entries for the LFN.
        See Also:
        ReplicaCatalogEntry
      • lookupNoAttributes

        public java.util.Map lookupNoAttributes​(java.util.Set lfns)
        Retrieves multiple entries for a given logical directory, up to the complete catalog. Retrieving full catalogs should be harmful, but may be helpful in an online display or portal.
        Specified by:
        lookupNoAttributes in interface ReplicaCatalog
        Parameters:
        lfns - is a set of logical directory strings to look up.
        Returns:
        a map indexed by the LFN. Each value is a set of PFN strings.
      • lookup

        public java.util.Map lookup​(java.util.Set lfns,
                                    java.lang.String handle)
        Retrieves multiple entries for a given logical directory, up to the complete catalog. Retrieving full catalogs should be harmful, but may be helpful in online display or portal.

        Specified by:
        lookup in interface ReplicaCatalog
        Parameters:
        lfns - is a set of logical directory strings to look up.
        handle - is the resource handle, restricting the LFNs.
        Returns:
        a map indexed by the LFN. Each value is a collection of replica catalog entries (all attributes).
        See Also:
        ReplicaCatalogEntry
      • lookupNoAttributes

        public java.util.Map lookupNoAttributes​(java.util.Set lfns,
                                                java.lang.String handle)
        Retrieves multiple entries for a given logical directory, up to the complete catalog. Retrieving full catalogs should be harmful, but may be helpful in online display or portal.

        Specified by:
        lookupNoAttributes in interface ReplicaCatalog
        Parameters:
        lfns - is a set of logical directory strings to look up.
        handle - is the resource handle, restricting the LFNs.
        Returns:
        a map indexed by the LFN. Each value is a set of physical filenames.
      • lookup

        public java.util.Map lookup​(java.util.Map constraints)
        Retrieves multiple entries for a given logical directory, up to the complete catalog. Retrieving full catalogs should be harmful, but may be helpful in online display or portal.
        Specified by:
        lookup in interface ReplicaCatalog
        Parameters:
        constraints - is mapping of keys 'lfn', 'pfn', or any attribute name, e.g. the resource handle 'pool', to a string that has some meaning to the implementing system. This can be a SQL wildcard for queries, or a regular expression for Java-based memory collections. Unknown keys are ignored. Using an empty map requests the complete catalog.
        Returns:
        a map indexed by the LFN. Each value is a collection of replica catalog entries.
        See Also:
        ReplicaCatalogEntry
      • list

        public java.util.Set list()
        Lists all logical filenames in the catalog.
        Specified by:
        list in interface ReplicaCatalog
        Returns:
        A set of all logical filenames known to the catalog.
      • list

        public java.util.Set list​(java.lang.String constraint)
        Lists a subset of all logical filenames in the catalog.
        Specified by:
        list in interface ReplicaCatalog
        Parameters:
        constraint - is a constraint for the logical directory only. It is a string that has some meaning to the implementing system. This can be a SQL wildcard for queries, or a regular expression for Java-based memory collections.
        Returns:
        A set of logical filenames that match. The set may be empty
      • insert

        public int insert​(java.lang.String lfn,
                          ReplicaCatalogEntry tuple)
        Inserts a new mapping into the replica catalog. Any existing mapping of the same LFN, PFN, and HANDLE will be replaced, including all of its attributes.
        Specified by:
        insert in interface ReplicaCatalog
        Parameters:
        lfn - is the logical directory under which to book the entry.
        tuple - is the physical directory and associated PFN attributes.
        Returns:
        number of insertions, should always be 1. On failure, throw an exception, don't use zero.
      • insert

        public int insert​(java.lang.String lfn,
                          java.lang.String pfn,
                          java.lang.String handle)
        Inserts a new mapping into the replica catalog. This is a convenience function exposing the resource handle. Internally, the ReplicaCatalogEntry element will be contructed, and passed to the appropriate insert function.
        Specified by:
        insert in interface ReplicaCatalog
        Parameters:
        lfn - is the logical directory under which to book the entry.
        pfn - is the physical directory associated with it.
        handle - is a resource handle where the PFN resides.
        Returns:
        number of insertions, should always be 1. On failure, throw an exception, don't use zero.
        See Also:
        insert( String, ReplicaCatalogEntry ), ReplicaCatalogEntry
      • insert

        public int insert​(java.util.Map x)
        Inserts multiple mappings into the replica catalog. The input is a map indexed by the LFN. The value for each LFN key is a collection of replica catalog entries. Note that this operation will replace existing entries.
        Specified by:
        insert in interface ReplicaCatalog
        Parameters:
        x - is a map from logical directory string to list of replica catalog entries.
        Returns:
        the number of insertions.
        See Also:
        ReplicaCatalogEntry
      • delete

        public int delete​(java.lang.String lfn,
                          java.lang.String pfn)
        Deletes a specific mapping from the replica catalog. We don't care about the resource handle. More than one entry could theoretically be removed. Upon removal of an entry, all attributes associated with the PFN also evaporate (cascading deletion).
        Specified by:
        delete in interface ReplicaCatalog
        Parameters:
        lfn - is the logical directory in the tuple.
        pfn - is the physical directory in the tuple.
        Returns:
        the number of removed entries.
      • delete

        public int delete​(java.util.Map x,
                          boolean matchAttributes)
        Deletes multiple mappings into the replica catalog. The input is a map indexed by the LFN. The value for each LFN key is a collection of replica catalog entries. On setting matchAttributes to false, all entries having matching lfn pfn mapping to an entry in the Map are deleted. However, upon removal of an entry, all attributes associated with the pfn also evaporate (cascaded deletion).
        Specified by:
        delete in interface ReplicaCatalog
        Parameters:
        x - is a map from logical directory string to list of replica catalog entries.
        matchAttributes - whether mapping should be deleted only if all attributes match.
        Returns:
        the number of deletions.
        See Also:
        ReplicaCatalogEntry
      • matchMe

        private boolean matchMe​(ReplicaCatalogEntry full,
                                ReplicaCatalogEntry part)
        Attempts to see, if all keys in the partial replica catalog entry are contained in the full replica catalog entry.
        Parameters:
        full - is the full entry to check against.
        part - is the partial entry to check with.
        Returns:
        true, if contained, false if not contained.
      • delete

        public int delete​(java.lang.String lfn,
                          ReplicaCatalogEntry tuple)
        Deletes a very specific mapping from the replica catalog. The LFN must be matches, the PFN, and all PFN attributes specified in the replica catalog entry. More than one entry could theoretically be removed. Upon removal of an entry, all attributes associated with the PFN also evaporate (cascading deletion).
        Specified by:
        delete in interface ReplicaCatalog
        Parameters:
        lfn - is the logical directory in the tuple.
        tuple - is a description of the PFN and its attributes.
        Returns:
        the number of removed entries, either 0 or 1.
      • hasMatchingAttr

        private boolean hasMatchingAttr​(ReplicaCatalogEntry rce,
                                        java.lang.String name,
                                        java.lang.Object value)
        Looks for a match of an attribute value in a replica catalog entry.
        Parameters:
        rce - is the replica catalog entry
        name - is the attribute key to match
        value - is the value to match against
        Returns:
        true, if a match was found.
      • delete

        public int delete​(java.lang.String lfn,
                          java.lang.String name,
                          java.lang.Object value)
        Deletes all PFN entries for a given LFN from the replica catalog where the PFN attribute is found, and matches exactly the object value. This method may be useful to remove all replica entries that have a certain MD5 sum associated with them. It may also be harmful overkill.
        Specified by:
        delete in interface ReplicaCatalog
        Parameters:
        lfn - is the logical directory to look for.
        name - is the PFN attribute name to look for.
        value - is an exact match of the attribute value to match.
        Returns:
        the number of removed entries.
      • deleteByResource

        public int deleteByResource​(java.lang.String lfn,
                                    java.lang.String handle)
        Deletes all PFN entries for a given LFN from the replica catalog where the resource handle is found. Karan requested this convenience method, which can be coded like
          delete( lfn, RESOURCE_HANDLE, handle )
         
        Specified by:
        deleteByResource in interface ReplicaCatalog
        Parameters:
        lfn - is the logical directory to look for.
        handle - is the resource handle
        Returns:
        the number of entries removed.
      • remove

        public int remove​(java.lang.String lfn)
        Removes all mappings for an LFN from the replica catalog.
        Specified by:
        remove in interface ReplicaCatalog
        Parameters:
        lfn - is the logical directory to remove all mappings for.
        Returns:
        the number of removed entries.
      • remove

        public int remove​(java.util.Set lfns)
        Removes all mappings for a set of LFNs.
        Specified by:
        remove in interface ReplicaCatalog
        Parameters:
        lfns - is a set of logical directory to remove all mappings for.
        Returns:
        the number of removed entries.
        See Also:
        remove( String )
      • removeByAttribute

        public int removeByAttribute​(java.lang.String name,
                                     java.lang.Object value)
        Removes all entries from the replica catalog where the PFN attribute is found, and matches exactly the object value.
        Specified by:
        removeByAttribute in interface ReplicaCatalog
        Parameters:
        name - is the PFN attribute key to look for.
        value - is an exact match of the attribute value to match.
        Returns:
        the number of removed entries.
      • removeByAttribute

        public int removeByAttribute​(java.lang.String handle)
        Removes all entries associated with a particular resource handle. This is useful, if a site goes offline. It is a convenience method, which calls the generic removeByAttribute method.
        Specified by:
        removeByAttribute in interface ReplicaCatalog
        Parameters:
        handle - is the site handle to remove all entries for.
        Returns:
        the number of removed entries.
        See Also:
        removeByAttribute( String, Object )
      • clear

        public int clear()
        Removes everything. Use with caution!
        Specified by:
        clear in interface ReplicaCatalog
        Returns:
        the number of removed entries.
      • main

        public static void main​(java.lang.String[] args)
        The main program
        Parameters:
        args - the arguments