HBase 0.96.x will run on hadoop 1.x or hadoop 2.x. HBase 0.98 will run on both also (but HBase 0.98 deprecates use of hadoop 1). HBase 1.x will NOT run on hadoop 1. In what follows, we make a distinction between HBase 1.x builds and the awkward process involved building HBase 0.96/0.98 for either hadoop 1 or hadoop 2 targets.
Building 0.98 and 0.96, you must choose which hadoop to build against; we cannot make a single HBase binary that can run against both hadoop1 and hadoop2. Since we include the Hadoop we were built against -- so we can do standalone mode -- the set of modules included in the tarball changes dependent on whether the hadoop1 or hadoop2 target is chosen. You can tell which HBase you have -- whether it is for hadoop1 or hadoop2 by looking at the version; the HBase for hadoop1 bundle will include 'hadoop1' in its version. Ditto for hadoop2.
Maven, our build system, natively will not let you have a single product
built against different dependencies. It is understandable. But neither could
we convince maven to change the set of included modules and write out
the correct poms w/ appropriate dependencies even though we have two
build targets; one for hadoop1 and another for hadoop2. So, there is a prestep
required. This prestep takes as input the current pom.xmls and it generates hadoop1 or
hadoop2 versions using a script in
generate-hadoopX-poms.sh. You then reference these generated
poms when you build. For now, just be aware of the difference between HBase 1.x
builds and those of HBase 0.96-0.98. Below we will come back to this difference
when we list out build instructions.
Publishing to maven requires you sign the artifacts you want to upload. To have the
build do this for you, you need to make sure you have a properly configured
settings.xml in your local repository under
Here is my
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"> <servers> <!- To publish a snapshot of some part of Maven --> <server> <id>apache.snapshots.https</id> <username>YOUR_APACHE_ID </username> <password>YOUR_APACHE_PASSWORD </password> </server> <!-- To publish a website using Maven --> <!-- To stage a release of some part of Maven --> <server> <id>apache.releases.https</id> <username>YOUR_APACHE_ID </username> <password>YOUR_APACHE_PASSWORD </password> </server> </servers> <profiles> <profile> <id>apache-release</id> <properties> <gpg.keyname>YOUR_KEYNAME</gpg.keyname> <!--Keyname is something like this ... 00A5F21E... do gpg --list-keys to find it--> <gpg.passphrase>YOUR_KEY_PASSWORD </gpg.passphrase> </properties> </profile> </profiles> </settings>
You must use maven 3.0.x (Check by running mvn -version).
I'll explain by running through the process. See later in this section for more detail on particular steps. These instructions are for building HBase 1.0.x. For building earlier versions, the process is different. See this section under the respective release documentation folders.
If you are making a point release (for example to quickly address a critical incompatability or security problem) off of a release branch instead of a development branch the tagging instructions are slightly different. I'll prefix those special steps with Point Release Only.
I would advise before you go about making a release candidate, do a practise run by deploying a SNAPSHOT. Also, make sure builds have been passing recently for the branch from where you are going to take your release. You should also have tried recent branch tips out on a cluster under load running for instance our hbase-it integration test suite for a few hours to 'burn in' the near-candidate bits.
At this point you should tag the previous release branch (ex: 0.96.1) with the new point release tag (e.g. 0.96.1.1 tag). Any commits with changes or mentioned below for the point release should be appled to the new tag.
The Hadoop How To Release wiki page informs much of the below and may have more detail on particular sections so it is worth review.
Update CHANGES.txt with the changes since the last release. Make sure the URL to the JIRA points to the properly location listing fixes for this release. Adjust the version in all the poms appropriately. If you are making a release candidate, you must remove the -SNAPSHOT from all versions. If you are running this receipe to publish a SNAPSHOT, you must keep the -SNAPSHOT suffix on the hbase version. The Versions Maven Plugin can be of use here. To set a version in all the many poms of the hbase multi-module project, do something like this:
$ mvn clean org.codehaus.mojo:versions-maven-plugin:1.3.1:set -DnewVersion=0.96.0
CHANGES.txt and any version changes.
Update the documentation under
src/main/docbkx. This usually involves copying the
latest from trunk making version-particular adjustments to suit this release candidate version.
Now, build the src tarball. This tarball is hadoop version independent. It is just the pure src code and documentation without a particular hadoop taint, etc.
-Prelease profile when building; it checks files for licenses and will fail the build if unlicensed files present.
$ MAVEN_OPTS="-Xmx2g" mvn clean install -DskipTests assembly:single -Dassembly.file=hbase-assembly/src/main/assembly/src.xml -Prelease
Undo the tarball and make sure it looks good. A good test for the src tarball being 'complete' is to see if
you can build new tarballs from this source bundle.
If the source tarball is good, save it off to a version directory, i.e a directory somewhere where you are collecting
all of the tarballs you will publish as part of the release candidate. For example if we were building a
hbase-0.96.0 release candidate, we might call the directory
we will publish this directory as our release candidate up on people.apache.org/~YOU.
Now lets build the binary tarball.
-Prelease profile when building; it checks files for licenses and will fail the build if unlicensed files present.
Do it in two steps. First install into the local repository and then generate documentation and assemble the tarball
(Otherwise build complains that hbase modules are not in maven repo when we try to do it all in the one go especially on fresh repo).
It seems that you need the install goal in both steps.
$ MAVEN_OPTS="-Xmx3g" mvn clean install -DskipTests -Prelease $ MAVEN_OPTS="-Xmx3g" mvn install -DskipTests site assembly:single -Prelease
Undo the generated tarball and check it out. Look at doc. and see if it runs, etc. If good, copy the tarball to the above mentioned version directory.
The following step that creates a new tag can be skipped since you've already created the point release tag
I'll tag the release at this point since its looking good. If we find an issue later, we can delete the tag and start over. Release needs to be tagged when we do next step.
Now deploy hbase to the apache maven repository.
This time we use the
apache-release profile instead of just
release profile when doing mvn deploy;
it will invoke the apache pom referenced by our poms. It will also sign your artifacts published to mvn as long as your settings.xml in your local
repository is configured correctly (your
settings.xml adds your gpg password property to the apache profile).
$ MAVEN_OPTS="-Xmx3g" mvn deploy -DskipTests -Papache-release
The last command above copies all artifacts up to a temporary staging apache mvn repo in an 'open' state. We'll need to do more work on these maven artifacts to make them generally available.
dev-support/make_rc.sh automates alot of the above listed release steps.
It does not do the modification of the CHANGES.txt for the release, the close of the
staging repository up in apache maven (human intervention is needed here), the checking of
the produced artifacts to ensure they are 'good' -- e.g. undoing the produced tarballs, eyeballing them to make
sure they look right then starting and checking all is running properly -- and then the signing and pushing of
the tarballs to people.apache.org but it does the other stuff; it can come in handy.
Now lets get back to what is up in maven. Our artifacts should be up in maven repository in the staging area in the 'open' state. While in this 'open' state you can check out what you've published to make sure all is good. To do this, login at repository.apache.org using your apache id. Find your artifacts in the staging repository. Browse the content. Make sure all artifacts made it up and that the poms look generally good. If it checks out, 'close' the repo. This will make the artifacts publically available. You will receive an email with the URL to give out for the temporary staging repository for others to use trying out this new release candidate. Include it in the email that announces the release candidate. Folks will need to add this repo URL to their local poms or to their local settings.xml file to pull the published release candidate artifacts. If the published artifacts are incomplete or borked, just delete the 'open' staged artifacts.
See the hbase-downstreamer test for a simple example of a project that is downstream of hbase an depends on it. Check it out and run its simple test to make sure maven artifacts are properly deployed to the maven repository. Be sure to edit the pom to point at the proper staging repo. Make sure you are pulling from the repo when tests run and that you are not getting from your local repo (pass -U or delete your local repo content and check maven is pulling from remote out of the staging repo).
See Publishing Maven Artifacts for some pointers on this maven staging process.
We no longer publish using the maven release plugin. Instead we do mvn deploy. It seems to give us a backdoor to maven release publishing. If no -SNAPSHOT on the version string, then we are 'deployed' to the apache maven repository staging directory from which we can publish URLs for candidates and later, if they pass, publish as release (if a -SNAPSHOT on the version string, deploy will put the artifacts up into apache snapshot repos).
If the hbase version ends in
-SNAPSHOT, the artifacts go elsewhere. They are put into the apache snapshots repository
directly and are immediately available. Making a SNAPSHOT release, this is what you want to happen.
At this stage we have two tarballs in our 'version directory' and a set of artifacts up in maven in staging area in the
'closed' state publically available in a temporary staging repository whose URL you should have gotten in an email.
The above mentioned script,
make_rc.sh does all of the above for you minus the check of the artifacts built,
the closing of the staging repository up in maven, and the tagging of the release. If you run the script, do your checks at this
stage verifying the src and bin tarballs and checking what is up in staging using hbase-downstreamer project. Tag before you start
the build. You can always delete it if the build goes haywire.
If all checks out, next put the version directory up on people.apache.org. You will need to sign and fingerprint them before you push them up. In the version directory do this:
$ for i in *.tar.gz; do echo $i; gpg --print-mds $i > $i.mds ; done $ for i in *.tar.gz; do echo $i; gpg --armor --output $i.asc --detach-sig $i ; done $ cd .. # Presuming our 'version directory' is named 0.96.0RC0, now copy it up to people.apache.org. $ rsync -av 0.96.0RC0 people.apache.org:public_html
Make sure the people.apache.org directory is showing and that the mvn repo urls are good. Announce the release candidate on the mailing list and call a vote.
Make sure your
settings.xml is set up properly (see above for how).
Make sure the hbase version includes
-SNAPSHOT as a suffix. Here is how I published SNAPSHOTS of
a release that had an hbase version of 0.96.0 in its poms.
$ MAVEN_OPTS="-Xmx3g" mvn clean install -DskipTests javadoc:aggregate site assembly:single -Prelease $ MAVEN_OPTS="-Xmx3g" mvn -DskipTests deploy -Papache-release
make_rc.sh script mentioned above in the
(see Section 18.4.2, “Making a Release Candidate”) can help you publish
Make sure your hbase.version has a
-SNAPSHOT suffix and then run
the script. It will put a snapshot up into the apache snapshot repository for you.