org.apache.hadoop.hbase.master

Class MasterWalManager

java.lang.Object

org.apache.hadoop.hbase.master.MasterWalManager

@InterfaceAudience.Private
public class MasterWalManager
extends Object

This class abstracts a bunch of operations the HMaster needs when splitting log files e.g. finding log files, dirs etc.

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.conf.Configuration	conf
private org.apache.hadoop.fs.FileSystem	fs
private boolean	fsOk
private static org.slf4j.Logger	LOG
(package private) static org.apache.hadoop.fs.PathFilter	META_FILTER
private MetricsMasterFileSystem	metricsMasterFilesystem
(package private) static org.apache.hadoop.fs.PathFilter	NON_META_FILTER
private org.apache.hadoop.fs.Path	oldLogDir
private org.apache.hadoop.fs.Path	rootDir This is the hbase rootdir.
private MasterServices	services
private Lock	splitLogLock
private SplitLogManager	splitLogManager

Constructor Summary

Constructors

Constructor and Description
MasterWalManager(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, MasterServices services)
MasterWalManager(MasterServices services)

Method Summary

Modifier and Type	Method and Description
void	archiveMetaLog(ServerName serverName) For meta region open and closed normally on a server, it may leave some meta WAL in the server's wal dir.
private boolean	checkFileSystem() Checks to see if the file system is still accessible.
(package private) List<org.apache.hadoop.fs.Path>	createAndGetLogDirs(Set<ServerName> serverNames) Finds WAL dirs for serverNames and renames them with '-splitting' suffix.
Set<ServerName>	getFailedServersFromLogFolders() Deprecated. With proc-v2, we can record the crash server with procedure store, so do not need to scan the wal directory to find out the splitting wal directory any more. Leave it here only because RecoverMetaProcedure(which is also deprecated) uses it.
org.apache.hadoop.fs.FileSystem	getFileSystem()
Set<ServerName>	getLiveServersFromWALDir() Get Servernames that COULD BE 'alive'; excludes those that have a '-splitting' suffix as these are already being split -- they cannot be 'alive'.
(package private) org.apache.hadoop.fs.Path	getOldLogDir() Get the directory where old logs go
Set<ServerName>	getServerNamesFromWALDirPath(org.apache.hadoop.fs.PathFilter filter)
(package private) SplitLogManager	getSplitLogManager()
Set<ServerName>	getSplittingServersFromWALDir() Get Servernames which are currently splitting; paths have a '-splitting' suffix.
(package private) org.apache.hadoop.fs.Path	getWALDirectoryName(ServerName serverName) Depends on current FS Layout!
(package private) org.apache.hadoop.fs.Path	getWALDirPath()
org.apache.hadoop.fs.FileStatus[]	getWALDirPaths(org.apache.hadoop.fs.PathFilter filter)
(package private) boolean	isWALDirectoryNameWithWALs(ServerName serverName)
void	splitLog(ServerName serverName)
void	splitLog(Set<ServerName> serverNames)
void	splitLog(Set<ServerName> serverNames, org.apache.hadoop.fs.PathFilter filter) This method is the base split method that splits WAL files matching a filter.
void	splitMetaLog(ServerName serverName) Specialized method to handle the splitting for meta WAL
void	splitMetaLog(Set<ServerName> serverNames) Specialized method to handle the splitting for meta WAL
void	stop()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

META_FILTER

static final org.apache.hadoop.fs.PathFilter META_FILTER

NON_META_FILTER

static final org.apache.hadoop.fs.PathFilter NON_META_FILTER

metricsMasterFilesystem

private final MetricsMasterFileSystem metricsMasterFilesystem

services

private final MasterServices services

conf

private final org.apache.hadoop.conf.Configuration conf

fs

private final org.apache.hadoop.fs.FileSystem fs

oldLogDir

private final org.apache.hadoop.fs.Path oldLogDir

rootDir

private final org.apache.hadoop.fs.Path rootDir

This is the hbase rootdir. We'll put the WALs under this dir.

splitLogLock

private final Lock splitLogLock

splitLogManager

private final SplitLogManager splitLogManager

fsOk

private volatile boolean fsOk

Constructor Detail

MasterWalManager

public MasterWalManager(MasterServices services)
                 throws IOException

Throws:

IOException

MasterWalManager

public MasterWalManager(org.apache.hadoop.conf.Configuration conf,
                        org.apache.hadoop.fs.FileSystem fs,
                        MasterServices services)
                 throws IOException

Throws:

IOException

Method Detail

stop

public void stop()

getSplitLogManager

SplitLogManager getSplitLogManager()

getOldLogDir

org.apache.hadoop.fs.Path getOldLogDir()

Get the directory where old logs go

Returns:

the dir

getFileSystem

public org.apache.hadoop.fs.FileSystem getFileSystem()

checkFileSystem

private boolean checkFileSystem()

Checks to see if the file system is still accessible. If not, sets closed

Returns:

false if file system is not available

getSplittingServersFromWALDir

public Set<ServerName> getSplittingServersFromWALDir()
                                              throws IOException

Get Servernames which are currently splitting; paths have a '-splitting' suffix.

Returns:

ServerName

Throws:

IOException - IOException

getLiveServersFromWALDir

public Set<ServerName> getLiveServersFromWALDir()
                                         throws IOException

Get Servernames that COULD BE 'alive'; excludes those that have a '-splitting' suffix as these are already being split -- they cannot be 'alive'.

Returns:

ServerName

Throws:

IOException - IOException

getServerNamesFromWALDirPath

public Set<ServerName> getServerNamesFromWALDirPath(org.apache.hadoop.fs.PathFilter filter)
                                             throws IOException

Returns:

listing of ServerNames found by parsing WAL directory paths in FS.

Throws:

IOException

getWALDirPath

org.apache.hadoop.fs.Path getWALDirPath()
                                 throws IOException

Returns:

Returns the WALs dir under rootDir

Throws:

IOException

getWALDirPaths

public org.apache.hadoop.fs.FileStatus[] getWALDirPaths(org.apache.hadoop.fs.PathFilter filter)
                                                 throws IOException

Returns:

List of all RegionServer WAL dirs; i.e. this.rootDir/HConstants.HREGION_LOGDIR_NAME.

Throws:

IOException

getFailedServersFromLogFolders

@Deprecated
public Set<ServerName> getFailedServersFromLogFolders()
                                                           throws IOException

Deprecated. With proc-v2, we can record the crash server with procedure store, so do not need to scan the wal directory to find out the splitting wal directory any more. Leave it here only because RecoverMetaProcedure(which is also deprecated) uses it.

Inspect the log directory to find dead servers which need recovery work

Returns:

A set of ServerNames which aren't running but still have WAL files left in file system

Throws:

IOException

splitLog

public void splitLog(ServerName serverName)
              throws IOException

Throws:

IOException

splitMetaLog

public void splitMetaLog(ServerName serverName)
                  throws IOException

Specialized method to handle the splitting for meta WAL

Parameters:

serverName - logs belonging to this server will be split

Throws:

IOException

splitMetaLog

public void splitMetaLog(Set<ServerName> serverNames)
                  throws IOException

Specialized method to handle the splitting for meta WAL

Parameters:

serverNames - logs belonging to these servers will be split

Throws:

IOException

isWALDirectoryNameWithWALs

boolean isWALDirectoryNameWithWALs(ServerName serverName)

Returns:

True if a WAL directory exists (will return true also if WALs found in servername'-splitting' too).

getWALDirectoryName

org.apache.hadoop.fs.Path getWALDirectoryName(ServerName serverName)

Depends on current FS Layout!

Returns:

The Path to the WAL directory for serverName

createAndGetLogDirs

List<org.apache.hadoop.fs.Path> createAndGetLogDirs(Set<ServerName> serverNames)
                                             throws IOException

Finds WAL dirs for serverNames and renames them with '-splitting' suffix.

Returns:

List of '-splitting' directories that pertain to serverNames

Throws:

IOException

splitLog

public void splitLog(Set<ServerName> serverNames)
              throws IOException

Throws:

IOException

splitLog

public void splitLog(Set<ServerName> serverNames,
                     org.apache.hadoop.fs.PathFilter filter)
              throws IOException

This method is the base split method that splits WAL files matching a filter. Callers should pass the appropriate filter for meta and non-meta WALs.

Parameters:

serverNames - logs belonging to these servers will be split; this will rename the log directory out from under a soft-failed server

Throws:

IOException

archiveMetaLog

public void archiveMetaLog(ServerName serverName)

For meta region open and closed normally on a server, it may leave some meta WAL in the server's wal dir. Since meta region is no long on this server, The SCP won't split those meta wals, just leaving them there. So deleting the wal dir will fail since the dir is not empty. Actually We can safely achive those meta log and Archiving the meta log and delete the dir.

Parameters:

serverName - the server to archive meta log