13.7. NameNode

For more information on the NameNode, see Section 9.9, “HDFS”.

13.7.1. HDFS Utilization of Tables and Regions

To determine how much space HBase is using on HDFS use the hadoop shell commands from the NameNode. For example...

hadoop fs -dus /hbase/

...returns the summarized disk utilization for all HBase objects.

hadoop fs -dus /hbase/myTable

...returns the summarized disk utilization for the HBase table 'myTable'.

hadoop fs -du /hbase/myTable

...returns a list of the regions under the HBase table 'myTable' and their disk utilization.

For more information on HDFS shell commands, see the HDFS FileSystem Shell documentation.

13.7.2. Browsing HDFS for HBase Objects

Somtimes it will be necessary to explore the HBase objects that exist on HDFS. These objects could include the WALs (Write Ahead Logs), tables, regions, StoreFiles, etc. The easiest way to do this is with the NameNode web application that runs on port 50070. The NameNode web application will provide links to the all the DataNodes in the cluster so that they can be browsed seamlessly.

The HDFS directory structure of HBase tables in the cluster is...

/hbase
     /<Table>             (Tables in the cluster)
          /<Region>           (Regions for the table)
               /<ColumnFamiy>      (ColumnFamilies for the Region for the table)
                    /<StoreFile>        (StoreFiles for the ColumnFamily for the Regions for the table)
            

The HDFS directory structure of HBase WAL is..

/hbase
     /.logs
          /<RegionServer>    (RegionServers)
               /<HLog>           (WAL HLog files for the RegionServer)
            

See the HDFS User Guide for other non-shell diagnostic utilities like fsck.

13.7.2.1. Zero size HLogs with data in them

Problem: when getting a listing of all the files in a region server's .logs directory, one file has a size of 0 but it contains data.

Answer: It's an HDFS quirk. A file that's currently being to will appear to have a size of 0 but once it's closed it will show its true size

13.7.2.2. Use Cases

Two common use-cases for querying HDFS for HBase objects is research the degree of uncompaction of a table. If there are a large number of StoreFiles for each ColumnFamily it could indicate the need for a major compaction. Additionally, after a major compaction if the resulting StoreFile is "small" it could indicate the need for a reduction of ColumnFamilies for the table.

comments powered by Disqus