Field Detail

• SPLIT_SKIP_ERRORS_DEFAULT

  public static final boolean SPLIT_SKIP_ERRORS_DEFAULT

  By default we retry errors in splitting, rather than skipping.

  See Also:

  Constant Field Values

• walDir

  protected final org.apache.hadoop.fs.Path walDir

• walFS

  protected final org.apache.hadoop.fs.FileSystem walFS

• conf

  protected final org.apache.hadoop.conf.Configuration conf

• sequenceIdChecker

  protected final LastSequenceId sequenceIdChecker

• distributedLogReplay

  protected boolean distributedLogReplay

• lastFlushedSequenceIds

  protected Map<String,Long> lastFlushedSequenceIds

• regionMaxSeqIdInStores

  protected Map<String,Map<byte[],Long>> regionMaxSeqIdInStores

• failedServerName

  protected String failedServerName

• SPLIT_WRITER_CREATION_BOUNDED

  public static final String SPLIT_WRITER_CREATION_BOUNDED

  See Also:

  Constant Field Values

Method Detail

• splitLogFile

  public static boolean splitLogFile(org.apache.hadoop.fs.Path walDir,
                     org.apache.hadoop.fs.FileStatus logfile,
                     org.apache.hadoop.fs.FileSystem walFS,
                     org.apache.hadoop.conf.Configuration conf,
                     CancelableProgressable reporter,
                     LastSequenceId idChecker,
                     CoordinatedStateManager cp,
                     org.apache.hadoop.hbase.protobuf.generated.ZooKeeperProtos.SplitLogTask.RecoveryMode mode,
                     WALFactory factory)
                              throws IOException

  Splits a WAL file into region's recovered-edits directory. This is the main entry point for distributed log splitting from SplitLogWorker.

  If the log file has N regions then N recovered.edits files will be produced.

  Parameters:

  rootDir -

  logfile -

  walFS - FileSystem to use for WAL reading and splitting

  conf -

  reporter -

  idChecker -

  cp - coordination state manager

  Returns:

  false if it is interrupted by the progress-able.

  Throws:

  IOException

• split

  public static List<org.apache.hadoop.fs.Path> split(org.apache.hadoop.fs.Path walRootDir,
                                      org.apache.hadoop.fs.Path logDir,
                                      org.apache.hadoop.fs.Path oldLogDir,
                                      org.apache.hadoop.fs.FileSystem walFs,
                                      org.apache.hadoop.conf.Configuration conf,
                                      WALFactory factory)
                                               throws IOException

  Throws:

  IOException

• finishSplitLogFile

  public static void finishSplitLogFile(String logfile,
                        org.apache.hadoop.conf.Configuration conf)
                                 throws IOException

  Completes the work done by splitLogFile by archiving logs

  It is invoked by SplitLogManager once it knows that one of the SplitLogWorkers have completed the splitLogFile() part. If the master crashes then this function might get called multiple times.

  Parameters:

  logfile -

  conf -

  Throws:

  IOException

• getRegionDirRecoveredEditsDir

  public static org.apache.hadoop.fs.Path getRegionDirRecoveredEditsDir(org.apache.hadoop.fs.Path regionDir)

  Parameters:

  regionDir - This regions directory in the filesystem.

  Returns:

  The directory that holds recovered edits files for the region regionDir

• getSplitEditFilesSorted

  public static NavigableSet<org.apache.hadoop.fs.Path> getSplitEditFilesSorted(org.apache.hadoop.fs.FileSystem walFS,
                                                                org.apache.hadoop.fs.Path regionDir)
                                                                         throws IOException

  Returns sorted set of edit files made by splitter, excluding files with '.temp' suffix.

  Parameters:

  walFS - FileSystem to use for reading Recovered edits files

  regionDir - Directory where Recovered edits should reside

  Returns:

  Files in passed regionDir as a sorted set.

  Throws:

  IOException

• moveAsideBadEditsFile

  public static org.apache.hadoop.fs.Path moveAsideBadEditsFile(org.apache.hadoop.fs.FileSystem walFS,
                                                org.apache.hadoop.fs.Path edits)
                                                         throws IOException

  Move aside a bad edits file.

  Parameters:

  walFS - FileSystem to use for WAL operations

  edits - Edits file to move aside.

  Returns:

  The name of the moved aside file.

  Throws:

  IOException

• isSequenceIdFile

  public static boolean isSequenceIdFile(org.apache.hadoop.fs.Path file)

  Is the given file a region open sequence id file.

• writeRegionSequenceIdFile

  public static long writeRegionSequenceIdFile(org.apache.hadoop.fs.FileSystem walFS,
                               org.apache.hadoop.fs.Path regionDir,
                               long newSeqId,
                               long saftyBumper)
                                        throws IOException

  Create a file with name as region open sequence id

  Parameters:

  walFS - FileSystem to write Sequence file to

  regionDir - WALRegionDir used to determine where to write edits files

  newSeqId -

  saftyBumper -

  Returns:

  long new sequence Id value

  Throws:

  IOException

• getReader

  protected WAL.Reader getReader(org.apache.hadoop.fs.FileStatus file,
                     boolean skipErrors,
                     CancelableProgressable reporter)
                          throws IOException,
                                 org.apache.hadoop.hbase.wal.WALSplitter.CorruptedLogFileException

  Create a new WAL.Reader for reading logs to split.

  Parameters:

  file -

  Returns:

  A new Reader instance, caller should close

  Throws:

  IOException

  CorruptedLogFileException

  org.apache.hadoop.hbase.wal.WALSplitter.CorruptedLogFileException

• createWriter

  protected WALProvider.Writer createWriter(org.apache.hadoop.fs.Path logfile)
                                     throws IOException

  Create a new WALProvider.Writer for writing log splits.

  Returns:

  a new Writer instance, caller should close

  Throws:

  IOException

• getReader

  protected WAL.Reader getReader(org.apache.hadoop.fs.Path curLogFile,
                     CancelableProgressable reporter)
                          throws IOException

  Create a new WAL.Reader for reading logs to split.

  Returns:

  new Reader instance, caller should close

  Throws:

  IOException

• getMutationsFromWALEntry

  public static List<WALSplitter.MutationReplay> getMutationsFromWALEntry(org.apache.hadoop.hbase.protobuf.generated.AdminProtos.WALEntry entry,
                                                          CellScanner cells,
                                                          Pair<WALKey,WALEdit> logEntry,
                                                          Durability durability)
                                                                   throws IOException

  This function is used to construct mutations from a WALEntry. It also reconstructs WALKey & WALEdit from the passed in WALEntry

  Parameters:

  entry -

  cells -

  logEntry - pair of WALKey and WALEdit instance stores WALKey and WALEdit instances extracted from the passed in WALEntry.

  durability -

  Returns:

  list of Pair<MutationType, Mutation> to be replayed

  Throws:

  IOException