org.apache.hadoop.hbase.master

Class MasterWalManager

java.lang.Object

org.apache.hadoop.hbase.master.MasterWalManager

@InterfaceAudience.Private
public class MasterWalManager
extends Object

This class abstracts a bunch of operations the HMaster needs when splitting log files e.g. finding log files, dirs etc.

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.conf.Configuration	conf
private org.apache.hadoop.fs.FileSystem	fs
private boolean	fsOk
private static org.slf4j.Logger	LOG
(package private) static org.apache.hadoop.fs.PathFilter	META_FILTER Filter *in* WAL files that are for the hbase:meta Region.
private MetricsMasterFileSystem	metricsMasterFilesystem
static org.apache.hadoop.fs.PathFilter	NON_META_FILTER Filter *out* WAL files that are for the hbase:meta Region; i.e.
private org.apache.hadoop.fs.Path	oldLogDir
private org.apache.hadoop.fs.Path	rootDir
private MasterServices	services
private Lock	splitLogLock
private SplitLogManager	splitLogManager Deprecated. since 2.3.0 and 3.0.0 to be removed in 4.0.0; replaced by SplitWALManager.

Constructor Summary

Constructors

Constructor and Description
MasterWalManager(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, MasterServices services)
MasterWalManager(MasterServices services)

Method Summary

Modifier and Type	Method and Description
void	archiveMetaLog(ServerName serverName) The hbase:meta region may OPEN and CLOSE without issue on a server and then move elsewhere.
private boolean	checkFileSystem() Checks to see if the file system is still accessible.
Set<ServerName>	getFailedServersFromLogFolders() Deprecated. With proc-v2, we can record the crash server with procedure store, so do not need to scan the wal directory to find out the splitting wal directory any more. Leave it here only because RecoverMetaProcedure(which is also deprecated) uses it.
org.apache.hadoop.fs.FileSystem	getFileSystem()
Set<ServerName>	getLiveServersFromWALDir() Get Servernames that COULD BE 'alive'; excludes those that have a '-splitting' suffix as these are already being split -- they cannot be 'alive'.
(package private) List<org.apache.hadoop.fs.Path>	getLogDirs(Set<ServerName> serverNames)
(package private) org.apache.hadoop.fs.Path	getOldLogDir() Get the directory where old logs go
Set<ServerName>	getServerNamesFromWALDirPath(org.apache.hadoop.fs.PathFilter filter)
(package private) SplitLogManager	getSplitLogManager()
Set<ServerName>	getSplittingServersFromWALDir() Get Servernames which are currently splitting; paths have a '-splitting' suffix.
org.apache.hadoop.fs.FileStatus[]	getWALDirPaths(org.apache.hadoop.fs.PathFilter filter)
void	splitLog(ServerName serverName)
void	splitLog(Set<ServerName> serverNames)
void	splitLog(Set<ServerName> serverNames, org.apache.hadoop.fs.PathFilter filter) This method is the base split method that splits WAL files matching a filter.
void	splitMetaLog(ServerName serverName) Specialized method to handle the splitting for meta WAL
void	splitMetaLog(Set<ServerName> serverNames) Specialized method to handle the splitting for meta WAL
void	stop()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

META_FILTER

static final org.apache.hadoop.fs.PathFilter META_FILTER

Filter *in* WAL files that are for the hbase:meta Region.

NON_META_FILTER

public static final org.apache.hadoop.fs.PathFilter NON_META_FILTER

Filter *out* WAL files that are for the hbase:meta Region; i.e. return user-space WALs only.

metricsMasterFilesystem

private final MetricsMasterFileSystem metricsMasterFilesystem

services

private final MasterServices services

conf

private final org.apache.hadoop.conf.Configuration conf

fs

private final org.apache.hadoop.fs.FileSystem fs

oldLogDir

private final org.apache.hadoop.fs.Path oldLogDir

rootDir

private final org.apache.hadoop.fs.Path rootDir

splitLogLock

private final Lock splitLogLock

splitLogManager

@Deprecated
private final SplitLogManager splitLogManager

Deprecated. since 2.3.0 and 3.0.0 to be removed in 4.0.0; replaced by SplitWALManager.

Superceded by SplitWALManager; i.e. procedure-based WAL splitting rather than 'classic' zk-coordinated WAL splitting.

See Also:

SplitWALManager

fsOk

private volatile boolean fsOk

Constructor Detail

MasterWalManager

public MasterWalManager(MasterServices services)
                 throws IOException

Throws:

IOException

MasterWalManager

public MasterWalManager(org.apache.hadoop.conf.Configuration conf,
                        org.apache.hadoop.fs.FileSystem fs,
                        MasterServices services)
                 throws IOException

Throws:

IOException

Method Detail

stop

public void stop()

getSplitLogManager

SplitLogManager getSplitLogManager()

getOldLogDir

org.apache.hadoop.fs.Path getOldLogDir()

Get the directory where old logs go

Returns:

the dir

getFileSystem

public org.apache.hadoop.fs.FileSystem getFileSystem()

checkFileSystem

private boolean checkFileSystem()

Checks to see if the file system is still accessible. If not, sets closed

Returns:

false if file system is not available

getSplittingServersFromWALDir

public Set<ServerName> getSplittingServersFromWALDir()
                                              throws IOException

Get Servernames which are currently splitting; paths have a '-splitting' suffix.

Returns:

ServerName

Throws:

IOException - IOException

getLiveServersFromWALDir

public Set<ServerName> getLiveServersFromWALDir()
                                         throws IOException

Get Servernames that COULD BE 'alive'; excludes those that have a '-splitting' suffix as these are already being split -- they cannot be 'alive'.

Returns:

ServerName

Throws:

IOException - IOException

getServerNamesFromWALDirPath

public Set<ServerName> getServerNamesFromWALDirPath(org.apache.hadoop.fs.PathFilter filter)
                                             throws IOException

Returns:

listing of ServerNames found by parsing WAL directory paths in FS.

Throws:

IOException

getWALDirPaths

public org.apache.hadoop.fs.FileStatus[] getWALDirPaths(org.apache.hadoop.fs.PathFilter filter)
                                                 throws IOException

Returns:

List of all RegionServer WAL dirs; i.e. this.rootDir/HConstants.HREGION_LOGDIR_NAME.

Throws:

IOException

getFailedServersFromLogFolders

@Deprecated
public Set<ServerName> getFailedServersFromLogFolders()
                                                           throws IOException

Deprecated. With proc-v2, we can record the crash server with procedure store, so do not need to scan the wal directory to find out the splitting wal directory any more. Leave it here only because RecoverMetaProcedure(which is also deprecated) uses it.

Inspect the log directory to find dead servers which need recovery work

Returns:

A set of ServerNames which aren't running but still have WAL files left in file system

Throws:

IOException

splitLog

public void splitLog(ServerName serverName)
              throws IOException

Throws:

IOException

splitMetaLog

public void splitMetaLog(ServerName serverName)
                  throws IOException

Specialized method to handle the splitting for meta WAL

Parameters:

serverName - logs belonging to this server will be split

Throws:

IOException

splitMetaLog

public void splitMetaLog(Set<ServerName> serverNames)
                  throws IOException

Specialized method to handle the splitting for meta WAL

Parameters:

serverNames - logs belonging to these servers will be split

Throws:

IOException

getLogDirs

List<org.apache.hadoop.fs.Path> getLogDirs(Set<ServerName> serverNames)
                                    throws IOException

Throws:

IOException

splitLog

public void splitLog(Set<ServerName> serverNames)
              throws IOException

Throws:

IOException

splitLog

public void splitLog(Set<ServerName> serverNames,
                     org.apache.hadoop.fs.PathFilter filter)
              throws IOException

This method is the base split method that splits WAL files matching a filter. Callers should pass the appropriate filter for meta and non-meta WALs.

Parameters:

serverNames - logs belonging to these servers will be split; this will rename the log directory out from under a soft-failed server

Throws:

IOException

archiveMetaLog

public void archiveMetaLog(ServerName serverName)

The hbase:meta region may OPEN and CLOSE without issue on a server and then move elsewhere. On CLOSE, the WAL for the hbase:meta table may not be archived yet (The WAL is only needed if hbase:meta did not close cleanaly). Since meta region is no long on this server, the ServerCrashProcedure won't split these leftover hbase:meta WALs, just leaving them in the WAL splitting dir. If we try to delete the WAL splitting for the server, it fail since the dir is not totally empty. We can safely archive these hbase:meta log; then the WAL dir can be deleted.

Parameters:

serverName - the server to archive meta log