Field Detail

• LOG

  private static final org.apache.commons.logging.Log LOG

• REGION_INFO_FILE

  public static final String REGION_INFO_FILE

  Name of the region info file that resides just under the region directory.

  See Also:

  Constant Field Values

• REGION_MERGES_DIR

  public static final String REGION_MERGES_DIR

  Temporary subdirectory of the region directory used for merges.

  See Also:

  Constant Field Values

• REGION_SPLITS_DIR

  public static final String REGION_SPLITS_DIR

  Temporary subdirectory of the region directory used for splits.

  See Also:

  Constant Field Values

• REGION_TEMP_DIR

  private static final String REGION_TEMP_DIR

  Temporary subdirectory of the region directory used for compaction output.

  See Also:

  Constant Field Values

• regionInfo

  private final HRegionInfo regionInfo

• regionInfoForFs

  private final HRegionInfo regionInfoForFs

• conf

  private final org.apache.hadoop.conf.Configuration conf

• tableDir

  private final org.apache.hadoop.fs.Path tableDir

• fs

  private final org.apache.hadoop.fs.FileSystem fs

• hdfsClientRetriesNumber

  private final int hdfsClientRetriesNumber

  In order to handle NN connectivity hiccups, one need to retry non-idempotent operation at the client level.

• baseSleepBeforeRetries

  private final int baseSleepBeforeRetries

• DEFAULT_HDFS_CLIENT_RETRIES_NUMBER

  private static final int DEFAULT_HDFS_CLIENT_RETRIES_NUMBER

  See Also:

  Constant Field Values

• DEFAULT_BASE_SLEEP_BEFORE_RETRIES

  private static final int DEFAULT_BASE_SLEEP_BEFORE_RETRIES

  See Also:

  Constant Field Values

Constructor Detail

• HRegionFileSystem

  HRegionFileSystem(org.apache.hadoop.conf.Configuration conf,
                   org.apache.hadoop.fs.FileSystem fs,
                   org.apache.hadoop.fs.Path tableDir,
                   HRegionInfo regionInfo)

  Create a view to the on-disk region

  Parameters:

  conf - the Configuration to use

  fs - FileSystem that contains the region

  tableDir - Path to where the table is being stored

  regionInfo - HRegionInfo for region

Method Detail

• getFileSystem

  public org.apache.hadoop.fs.FileSystem getFileSystem()

  Returns:

  the underlying FileSystem

• getRegionInfo

  public HRegionInfo getRegionInfo()

  Returns:

  the HRegionInfo that describe this on-disk region view

• getRegionInfoForFS

  public HRegionInfo getRegionInfoForFS()

• getTableDir

  public org.apache.hadoop.fs.Path getTableDir()

  Returns:

  Path to the region's root directory.

• getRegionDir

  public org.apache.hadoop.fs.Path getRegionDir()

  Returns:

  Path to the region directory.

• getTempDir

  org.apache.hadoop.fs.Path getTempDir()

  Returns:

  Path to the region's temp directory, used for file creations

• cleanupTempDir

  void cleanupTempDir()
                throws IOException

  Clean up any temp detritus that may have been left around from previous operation attempts.

  Throws:

  IOException

• getStoreDir

  public org.apache.hadoop.fs.Path getStoreDir(String familyName)

  Returns the directory path of the specified family

  Parameters:

  familyName - Column Family Name

  Returns:

  Path to the directory of the specified family

• createStoreDir

  org.apache.hadoop.fs.Path createStoreDir(String familyName)
                                     throws IOException

  Create the store directory for the specified family name

  Parameters:

  familyName - Column Family Name

  Returns:

  Path to the directory of the specified family

  Throws:

  IOException - if the directory creation fails.

• getStoreFiles

  public Collection<StoreFileInfo> getStoreFiles(byte[] familyName)
                                          throws IOException

  Returns the store files available for the family. This methods performs the filtering based on the valid store files.

  Parameters:

  familyName - Column Family Name

  Returns:

  a set of StoreFileInfo for the specified family.

  Throws:

  IOException

• getStoreFiles

  public Collection<StoreFileInfo> getStoreFiles(String familyName)
                                          throws IOException

  Throws:

  IOException

• getStoreFiles

  public Collection<StoreFileInfo> getStoreFiles(String familyName,
                                        boolean validate)
                                          throws IOException

  Returns the store files available for the family. This methods performs the filtering based on the valid store files.

  Parameters:

  familyName - Column Family Name

  Returns:

  a set of StoreFileInfo for the specified family.

  Throws:

  IOException

• getStoreFilePath

  org.apache.hadoop.fs.Path getStoreFilePath(String familyName,
                                           String fileName)

  Return Qualified Path of the specified family/file

  Parameters:

  familyName - Column Family Name

  fileName - File Name

  Returns:

  The qualified Path for the specified family/file

• getStoreFileInfo

  StoreFileInfo getStoreFileInfo(String familyName,
                               String fileName)
                           throws IOException

  Return the store file information of the specified family/file.

  Parameters:

  familyName - Column Family Name

  fileName - File Name

  Returns:

  The StoreFileInfo for the specified family/file

  Throws:

  IOException

• hasReferences

  public boolean hasReferences(String familyName)
                        throws IOException

  Returns true if the specified family has reference files

  Parameters:

  familyName - Column Family Name

  Returns:

  true if family contains reference files

  Throws:

  IOException

• hasReferences

  public boolean hasReferences(HTableDescriptor htd)
                        throws IOException

  Check whether region has Reference file

  Parameters:

  htd - table desciptor of the region

  Returns:

  true if region has reference file

  Throws:

  IOException

• getFamilies

  public Collection<String> getFamilies()
                                 throws IOException

  Returns:

  the set of families present on disk

  Throws:

  IOException

• deleteFamily

  public void deleteFamily(String familyName)
                    throws IOException

  Remove the region family from disk, archiving the store files.

  Parameters:

  familyName - Column Family Name

  Throws:

  IOException - if an error occours during the archiving

• generateUniqueName

  private static String generateUniqueName(String suffix)

  Generate a unique file name, used by createTempName() and commitStoreFile()

  Parameters:

  suffix - extra information to append to the generated name

  Returns:

  Unique file name

• createTempName

  public org.apache.hadoop.fs.Path createTempName()

  Generate a unique temporary Path. Used in conjuction with commitStoreFile() to get a safer file creation. Path file = fs.createTempName(); ...StoreFile.Writer(file)... fs.commitStoreFile("family", file);

  Returns:

  Unique Path of the temporary file

• createTempName

  public org.apache.hadoop.fs.Path createTempName(String suffix)

  Generate a unique temporary Path. Used in conjuction with commitStoreFile() to get a safer file creation. Path file = fs.createTempName(); ...StoreFile.Writer(file)... fs.commitStoreFile("family", file);

  Parameters:

  suffix - extra information to append to the generated name

  Returns:

  Unique Path of the temporary file

• commitStoreFile

  public org.apache.hadoop.fs.Path commitStoreFile(String familyName,
                                          org.apache.hadoop.fs.Path buildPath)
                                            throws IOException

  Move the file from a build/temp location to the main family store directory.

  Parameters:

  familyName - Family that will gain the file

  buildPath - Path to the file to commit.

  Returns:

  The new Path of the committed file

  Throws:

  IOException

• commitStoreFile

  private org.apache.hadoop.fs.Path commitStoreFile(String familyName,
                                          org.apache.hadoop.fs.Path buildPath,
                                          long seqNum,
                                          boolean generateNewName)
                                             throws IOException

  Move the file from a build/temp location to the main family store directory.

  Parameters:

  familyName - Family that will gain the file

  buildPath - Path to the file to commit.

  seqNum - Sequence Number to append to the file name (less then 0 if no sequence number)

  generateNewName - False if you want to keep the buildPath name

  Returns:

  The new Path of the committed file

  Throws:

  IOException

• commitStoreFiles

  void commitStoreFiles(Map<byte[],List<StoreFile>> storeFiles)
                  throws IOException

  Moves multiple store files to the relative region's family store directory.

  Parameters:

  storeFiles - list of store files divided by family

  Throws:

  IOException

• removeStoreFile

  public void removeStoreFile(String familyName,
                     org.apache.hadoop.fs.Path filePath)
                       throws IOException

  Archives the specified store file from the specified family.

  Parameters:

  familyName - Family that contains the store files

  filePath - Path to the store file to remove

  Throws:

  IOException - if the archiving fails

• removeStoreFiles

  public void removeStoreFiles(String familyName,
                      Collection<StoreFile> storeFiles)
                        throws IOException

  Closes and archives the specified store files from the specified family.

  Parameters:

  familyName - Family that contains the store files

  storeFiles - set of store files to remove

  Throws:

  IOException - if the archiving fails

• bulkLoadStoreFile

  org.apache.hadoop.fs.Path bulkLoadStoreFile(String familyName,
                                            org.apache.hadoop.fs.Path srcPath,
                                            long seqNum)
                                        throws IOException

  Bulk load: Add a specified store file to the specified family. If the source file is on the same different file-system is moved from the source location to the destination location, otherwise is copied over.

  Parameters:

  familyName - Family that will gain the file

  srcPath - Path to the file to import

  seqNum - Bulk Load sequence number

  Returns:

  The destination Path of the bulk loaded file

  Throws:

  IOException

• getSplitsDir

  org.apache.hadoop.fs.Path getSplitsDir()

  Returns:

  Path to the temp directory used during split operations

• getSplitsDir

  org.apache.hadoop.fs.Path getSplitsDir(HRegionInfo hri)

• cleanupSplitsDir

  void cleanupSplitsDir()
                  throws IOException

  Clean up any split detritus that may have been left around from previous split attempts.

  Throws:

  IOException

• cleanupAnySplitDetritus

  void cleanupAnySplitDetritus()
                         throws IOException

  Clean up any split detritus that may have been left around from previous split attempts. Call this method on initial region deploy.

  Throws:

  IOException

• cleanupDaughterRegion

  void cleanupDaughterRegion(HRegionInfo regionInfo)
                       throws IOException

  Remove daughter region

  Parameters:

  regionInfo - daughter HRegionInfo

  Throws:

  IOException

• commitDaughterRegion

  org.apache.hadoop.fs.Path commitDaughterRegion(HRegionInfo regionInfo)
                                           throws IOException

  Commit a daughter region, moving it from the split temporary directory to the proper location in the filesystem.

  Parameters:

  regionInfo - daughter HRegionInfo

  Throws:

  IOException

• createSplitsDir

  void createSplitsDir()
                 throws IOException

  Create the region splits directory.

  Throws:

  IOException

• splitStoreFile

  org.apache.hadoop.fs.Path splitStoreFile(HRegionInfo hri,
                                         String familyName,
                                         StoreFile f,
                                         byte[] splitRow,
                                         boolean top,
                                         RegionSplitPolicy splitPolicy)
                                     throws IOException

  Write out a split reference. Package local so it doesnt leak out of regionserver.

  Parameters:

  hri - HRegionInfo of the destination

  familyName - Column Family Name

  f - File to split.

  splitRow - Split Row

  top - True if we are referring to the top half of the hfile.

  splitPolicy -

  Returns:

  Path to created reference.

  Throws:

  IOException

• getMergesDir

  org.apache.hadoop.fs.Path getMergesDir()

  Returns:

  Path to the temp directory used during merge operations

• getMergesDir

  org.apache.hadoop.fs.Path getMergesDir(HRegionInfo hri)

• cleanupMergesDir

  void cleanupMergesDir()
                  throws IOException

  Clean up any merge detritus that may have been left around from previous merge attempts.

  Throws:

  IOException

• cleanupMergedRegion

  void cleanupMergedRegion(HRegionInfo mergedRegion)
                     throws IOException

  Remove merged region

  Parameters:

  mergedRegion - HRegionInfo

  Throws:

  IOException

• createMergesDir

  void createMergesDir()
                 throws IOException

  Create the region merges directory.

  Throws:

  IOException - If merges dir already exists or we fail to create it.

  See Also:

  cleanupMergesDir()

• mergeStoreFile

  org.apache.hadoop.fs.Path mergeStoreFile(HRegionInfo mergedRegion,
                                         String familyName,
                                         StoreFile f,
                                         org.apache.hadoop.fs.Path mergedDir)
                                     throws IOException

  Write out a merge reference under the given merges directory. Package local so it doesnt leak out of regionserver.

  Parameters:

  mergedRegion - HRegionInfo of the merged region

  familyName - Column Family Name

  f - File to create reference.

  mergedDir -

  Returns:

  Path to created reference.

  Throws:

  IOException

• commitMergedRegion

  void commitMergedRegion(HRegionInfo mergedRegionInfo)
                    throws IOException

  Commit a merged region, moving it from the merges temporary directory to the proper location in the filesystem.

  Parameters:

  mergedRegionInfo - merged region HRegionInfo

  Throws:

  IOException

• logFileSystemState

  void logFileSystemState(org.apache.commons.logging.Log LOG)
                    throws IOException

  Log the current state of the region

  Parameters:

  LOG - log to output information

  Throws:

  IOException - if an unexpected exception occurs

• getRegionInfoFileContent

  private static byte[] getRegionInfoFileContent(HRegionInfo hri)
                                          throws IOException

  Parameters:

  hri -

  Returns:

  Content of the file we write out to the filesystem under a region

  Throws:

  IOException

• loadRegionInfoFileContent

  public static HRegionInfo loadRegionInfoFileContent(org.apache.hadoop.fs.FileSystem fs,
                                      org.apache.hadoop.fs.Path regionDir)
                                               throws IOException

  Create a HRegionInfo from the serialized version on-disk.

  Parameters:

  fs - FileSystem that contains the Region Info file

  regionDir - Path to the Region Directory that contains the Info file

  Returns:

  An HRegionInfo instance gotten from the Region Info file.

  Throws:

  IOException - if an error occurred during file open/read operation.

• writeRegionInfoFileContent

  private static void writeRegionInfoFileContent(org.apache.hadoop.conf.Configuration conf,
                                org.apache.hadoop.fs.FileSystem fs,
                                org.apache.hadoop.fs.Path regionInfoFile,
                                byte[] content)
                                          throws IOException

  Write the .regioninfo file on-disk.

  Throws:

  IOException

• checkRegionInfoOnFilesystem

  void checkRegionInfoOnFilesystem()
                             throws IOException

  Write out an info file under the stored region directory. Useful recovering mangled regions. If the regionInfo already exists on-disk, then we fast exit.

  Throws:

  IOException

• writeRegionInfoOnFilesystem

  private void writeRegionInfoOnFilesystem(boolean useTempDir)
                                    throws IOException

  Write out an info file under the region directory. Useful recovering mangled regions.

  Parameters:

  useTempDir - indicate whether or not using the region .tmp dir for a safer file creation.

  Throws:

  IOException

• writeRegionInfoOnFilesystem

  private void writeRegionInfoOnFilesystem(byte[] regionInfoContent,
                                 boolean useTempDir)
                                    throws IOException

  Write out an info file under the region directory. Useful recovering mangled regions.

  Parameters:

  regionInfoContent - serialized version of the HRegionInfo

  useTempDir - indicate whether or not using the region .tmp dir for a safer file creation.

  Throws:

  IOException

• createRegionOnFileSystem

  public static HRegionFileSystem createRegionOnFileSystem(org.apache.hadoop.conf.Configuration conf,
                                           org.apache.hadoop.fs.FileSystem fs,
                                           org.apache.hadoop.fs.Path tableDir,
                                           HRegionInfo regionInfo)
                                                    throws IOException

  Create a new Region on file-system.

  Parameters:

  conf - the Configuration to use

  fs - FileSystem from which to add the region

  tableDir - Path to where the table is being stored

  regionInfo - HRegionInfo for region to be added

  Throws:

  IOException - if the region creation fails due to a FileSystem exception.

• openRegionFromFileSystem

  public static HRegionFileSystem openRegionFromFileSystem(org.apache.hadoop.conf.Configuration conf,
                                           org.apache.hadoop.fs.FileSystem fs,
                                           org.apache.hadoop.fs.Path tableDir,
                                           HRegionInfo regionInfo,
                                           boolean readOnly)
                                                    throws IOException

  Open Region from file-system.

  Parameters:

  conf - the Configuration to use

  fs - FileSystem from which to add the region

  tableDir - Path to where the table is being stored

  regionInfo - HRegionInfo for region to be added

  readOnly - True if you don't want to edit the region data

  Throws:

  IOException - if the region creation fails due to a FileSystem exception.

• deleteRegionFromFileSystem

  public static void deleteRegionFromFileSystem(org.apache.hadoop.conf.Configuration conf,
                                org.apache.hadoop.fs.FileSystem fs,
                                org.apache.hadoop.fs.Path tableDir,
                                HRegionInfo regionInfo)
                                         throws IOException

  Remove the region from the table directory, archiving the region's hfiles.

  Parameters:

  conf - the Configuration to use

  fs - FileSystem from which to remove the region

  tableDir - Path to where the table is being stored

  regionInfo - HRegionInfo for region to be deleted

  Throws:

  IOException - if the request cannot be completed

• createDir

  boolean createDir(org.apache.hadoop.fs.Path dir)
              throws IOException

  Creates a directory. Assumes the user has already checked for this directory existence.

  Parameters:

  dir -

  Returns:

  the result of fs.mkdirs(). In case underlying fs throws an IOException, it checks whether the directory exists or not, and returns true if it exists.

  Throws:

  IOException

• rename

  boolean rename(org.apache.hadoop.fs.Path srcpath,
               org.apache.hadoop.fs.Path dstPath)
           throws IOException

  Renames a directory. Assumes the user has already checked for this directory existence.

  Parameters:

  srcpath -

  dstPath -

  Returns:

  true if rename is successful.

  Throws:

  IOException

• deleteDir

  boolean deleteDir(org.apache.hadoop.fs.Path dir)
              throws IOException

  Deletes a directory. Assumes the user has already checked for this directory existence.

  Parameters:

  dir -

  Returns:

  true if the directory is deleted.

  Throws:

  IOException

• sleepBeforeRetry

  private void sleepBeforeRetry(String msg,
                      int sleepMultiplier)
                         throws InterruptedException

  sleeping logic; handles the interrupt exception.

  Throws:

  InterruptedException

• createDirOnFileSystem

  private static boolean createDirOnFileSystem(org.apache.hadoop.fs.FileSystem fs,
                              org.apache.hadoop.conf.Configuration conf,
                              org.apache.hadoop.fs.Path dir)
                                        throws IOException

  Creates a directory for a filesystem and configuration object. Assumes the user has already checked for this directory existence.

  Parameters:

  fs -

  conf -

  dir -

  Returns:

  the result of fs.mkdirs(). In case underlying fs throws an IOException, it checks whether the directory exists or not, and returns true if it exists.

  Throws:

  IOException

• sleepBeforeRetry

  private static void sleepBeforeRetry(String msg,
                      int sleepMultiplier,
                      int baseSleepBeforeRetries,
                      int hdfsClientRetriesNumber)
                                throws InterruptedException

  sleeping logic for static methods; handles the interrupt exception. Keeping a static version for this to avoid re-looking for the integer values.

  Throws:

  InterruptedException