As of Nuxeo 5.4, you need to install the full Java 6 SDK (not just the Runtime Environment, but the full Development Kit); Java 5 is no longer adequate.
Nuxeo supports OpenJDK as well. There are still some unresolved issues related to OpenJDK, but you're welcome to use OpenJDK and report any problem.
For more detailed Java-related requirements, you can for now read step-by-step instructions of how to install Java at one of the product installation pages, like here.
See the Requirements page in the Installation and Administration Guide.