JBoss Domain - Could not connect to master - Unable to connect due to authentication failure

Recently I ran in a problem related to a JBoss Domain which spans across two different physical hosts.

First, I configured a JBoss Domain in just one machine with a Domain Controller (DC) and two Host Controllers (HC). Everything was working fine, so I decided to go one step further and to configure two more HCs in another machine.
The procedure seemed quite obvious, and I cloned the machine one and just changed networks interfaces in this new machine. The rest of the configurations remained the same, i.e, host.xml in HC, where the secret key was configured in order to connect to DC.

But this didn't work. When I started the HC in machine 2 I got the following error:

[root@localhost bin]# ./domain.sh
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/jboss-eap-6.2-hc1

  JAVA: java

  JAVA_OPTS: -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true

=========================================================================

09:23:02,219 INFO  [org.jboss.modules] (main) JBoss Modules version 1.3.0.Final-redhat-2
09:23:02,432 INFO  [org.jboss.as.process.Host Controller.status] (main) JBAS012017: Starting process 'Host Controller'
[Host Controller] 09:23:03,382 INFO  [org.jboss.modules] (main) JBoss Modules version 1.3.0.Final-redhat-2
[Host Controller] 09:23:03,560 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA-redhat-1
[Host Controller] 09:23:03,667 INFO  [org.jboss.as] (MSC service thread 1-1) JBAS015899: JBoss EAP 6.2.0.GA (AS 7.3.0.Final-redhat-14) starting
[Host Controller] 09:23:04,778 INFO  [org.xnio] (MSC service thread 1-2) XNIO Version 3.0.7.GA-redhat-1
[Host Controller] 09:23:04,791 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.0.7.GA-redhat-1
[Host Controller] 09:23:04,830 INFO  [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 3.2.18.GA-redhat-1
[Host Controller] 09:23:04,940 INFO  [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on 192.168.56.202:9999
[Host Controller] 09:23:05,646 ERROR [org.jboss.remoting.remote.connection] (Remoting "localhost.localdomain:MANAGEMENT" read-1) JBREM000200: Remote connection failed: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed
[Host Controller] 09:23:05,653 ERROR [org.jboss.as.host.controller] (Controller Boot Thread) JBAS010901: Could not connect to master. Aborting. Error was: java.lang.IllegalStateException: JBAS010942: Unable to connect due to authentication failure.
[Host Controller] 09:23:05,665 INFO  [org.jboss.as.controller] (MSC service thread 1-2) JBAS014774: Service status report
[Host Controller] JBAS014775:    New missing/unsatisfied dependencies:
[Host Controller]       service jboss.server.controller.management.security_realm.ApplicationRealm.properties_authentication (missing) dependents: [service jboss.server.controller.management.security_realm.ApplicationRealm]
[Host Controller]
[Host Controller] 09:23:05,692 INFO  [org.jboss.as.controller] (MSC service thread 1-2) JBAS014774: Service status report
[Host Controller] JBAS014776:    Newly corrected services:
[Host Controller]       service jboss.server.controller.management.security_realm.ApplicationRealm.properties_authentication (no longer required)
[Host Controller]
[Host Controller] 09:23:05,695 INFO  [org.jboss.as] (MSC service thread 1-2) JBAS015950: JBoss EAP 6.2.0.GA (AS 7.3.0.Final-redhat-14) stopped in 27ms
09:23:06,045 INFO  [org.jboss.as.process.Host Controller.status] (reaper for Host Controller) JBAS012010: Process 'Host Controller' finished with an exit status of 99
09:23:06,051 INFO  [org.jboss.as.process] (Thread-8) JBAS012016: Shutting down process controller
09:23:06,052 INFO  [org.jboss.as.process] (Thread-8) JBAS012015: All processes finished; exiting

I don´t know why, but when DC and HC are in the same machine, in the host.xml of HC, you can put any slave name you want. But when DC and HC are in different machines, the slave name must be a management user in DC,

So what I did was:
1) Create a management user in HC, for example "adminHostController003".
    You have to launch add-user.sh script y HC server.
    This user will have the following characteristics:

• Management user (in ManagementRealm)
• No roles
• Allow remoting connections

2) Copy the secret key and edit host.xml in Host Controller JBoss. 

   In this file you have to configure:

• Host Name. The name of the user created in step 1
• Server identities. The key asociated to this user.
• Domain controller location. IP and port of DC

<host name="adminHostController003" xmlns="urn:jboss:domain:1.5">

 <management>
  <security-realms>
   <security-realm name="ManagementRealm">
    <server-identities>
     <secret value="Y2l4dGVjLjIwMTU="/>
    </server-identities>
    <authentication>                  
     <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
    </authentication>
    <authorization map-groups-to-roles="false">
     <properties path="mgmt-groups.properties" relative-to="jboss.domain.config.dir"/>
    </authorization>
   </security-realm>
   <security-realm name="ApplicationRealm">
    <authentication>
     <local default-user="$local" allowed-users="*"/>
     <properties path="application-users.properties" relative-to="jboss.domain.config.dir"/>
    </authentication>
    <authorization>
     <properties path="application-roles.properties" relative-to="jboss.domain.config.dir"/>
    </authorization>
   </security-realm>
  </security-realms>
  <audit-log>
   <formatters>
    <json-formatter name="json-formatter"/>
   </formatters>
   <handlers>
    <file-handler name="host-file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.domain.data.dir"/>
    <file-handler name="server-file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>
   </handlers>
   <logger log-boot="true" log-read-only="false" enabled="false">
    <handlers>
     <handler name="host-file"/>
    </handlers>
   </logger>
   <server-logger log-boot="true" log-read-only="false" enabled="false">
    <handlers>
     <handler name="server-file"/>
    </handlers>
   </server-logger>
  </audit-log>
  <management-interfaces>
   <native-interface security-realm="ManagementRealm">
    <socket interface="management" port="${jboss.management.native.port:9999}"/>
   </native-interface>
  </management-interfaces>
 </management>

 <domain-controller>
  <remote host="192.168.56.101" port="9999" security-realm="ManagementRealm"/>
 </domain-controller>

 <interfaces>
  <interface name="management">
   <inet-address value="${jboss.bind.address.management:192.168.56.202}"/>
  </interface>
  <interface name="public">
   <inet-address value="${jboss.bind.address:192.168.56.202}"/>
  </interface>
  <interface name="unsecure">
   <inet-address value="${jboss.bind.address.unsecure:192.168.56.202}"/>
  </interface>
 </interfaces>

 <jvms>
  <jvm name="default">
   <heap size="64m" max-size="256m"/>
   <permgen size="256m" max-size="256m"/>
   <jvm-options>
    <option value="-server"/>
   </jvm-options>
  </jvm>
 </jvms>

 <servers>    
 </servers>

</host>

Note.
After a while I realized that you can not copy all JBoss installation directory to the other machine (remember, I cloned the machines). This leads to problems with HornetQ because you have several servers in the same network with the same ID.
This ID is created the first time a node starts and it is stored in some internal dir, so if you copied the whole directory you are also copying the ID.
In these cases you will see this WARN message:

12:32:54,418 WARN  [org.hornetq.core.client] (hornetq-discovery-group-thread-dg-group1) HQ212034: There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=374ca1fe-61d7-11e5-b4c6-75baa50d332e

Remote Desktop with SSH Tunnel

In this post I will show you how to access to a PC with Windows Remote Desktop. Well, this would be quite easy, so to make it more funny, the target Machine is not accessible from the Source Machine, but do there is a third machine which is accessible from both of them.

The topology of the problem is depicted in the following image:

We have these elements:

• Target PC (windows). This is the machine I want to connect to.
• Source PC (windows). This is the machine from where I want to connect to the Target PC.
• Gateway (windows/Unix). This is the intermediary between Source and Target.

You have to keep in mind these constraints:

• Source cannot reach Target
• Source can reach Gateway
• Target can reach Gateway
• Gateway cannot reach Target

So, the idea is to use Gateway as a bridge between Source and Target. To make this happen we will use SSH Tunnels.

Previous preparation

In my concrete scenery, Gateway is a Windows Server PC, so I will have to configure SSH Server in it, but if you have a Unix Machine this step is already taken for granted.

You can use FreeSSHD as SSH Server. It is free (obvious) and very easy to configure. For example you can check this page.

The basic steps to configure it are:

1. Install with default options
2. Run as a service
3. Configure SSH port, only if 22 were already in use
4. Configure authentication to require password
5. Add some user

Besides, I'll add another step. Enable Tunneling options

Once you are done, you can check that all works properly by connecting with Putty to this machine. You should see one active user connected:

Tunneling

There will be two SSH tunnels: 

• From Target to Gateway
• From Source to Gateway

The topology of the solution is depicted in the following image:

Target will open a tunnel in which all connections in Gateway to port 9999 will be redirected to itself in port 3389. Port 3389 is the default Windows Remote Desktop port.

Source will open a tunnel in which all connections in Source to port 13389 will be redirected to Gateway in port 9999.

Now you should have noted that the bridge is Gateway´s port 9999.

Tunnel from Target to Gateway

Working on Target PC, with putty you have to set the following configuration.

In Session category:

• Host name and port of the SSH Server of Gateway

In Connection category:

• Set the parametter "Seconds beween keepalives" to 180

In Connection > SSH > Tunnels category:

• Check "Local ports accept connections from other hosts" option
• Check "Remote ports do the seame (SSH-2 only)" option
• Set source port 9999, destination localhost:3389 and check the option Remote. Then push "Add"

Tunnel from Source to Gateway

Working on Source PC, with putty you have to set the following configuration.

In Session category:

• Host name and port of the SSH Server of Gateway

In Connection category:

• Set the parametter "Seconds beween keepalives" to 180

In Connection > SSH > Tunnels category:

• Check "Local ports accept connections from other hosts" option
• Check "Remote ports do the seame (SSH-2 only)" option
• Set source port 13389, destination 192.168.65.133:9999 and check the option Local. Then push "Add"

Testing the tunnel

From Source, you only need to try to connect with remote desktop to localhost:13389. The connection will travel throught the first tunnel to Gateway, and from Gateway to Target throught the other tunnel.

Thanks to Mr V (alias coffee man) for helping me with this.

Bonus

In Target, you can configure a script to use Putty Command Line options to connect automatically.
In my example I named my session as "Tunel a Gateway", but for command line is better to set a one-word name, for example gatewayTunnel.

putty.exe -load gatewayTunnel -l userName -pw password

wsdl2java - Use java.util.Date instead of XMLGregorianCalendar

If you have a web service definition, WSDL, and you want to generate the equivalent java code (either for the client or for the server side) one of the simplest way is by using Apache CXF wsdl2java Maven plugin. Check here: http://cxf.apache.org/docs/maven-cxf-codegen-plugin-wsdl-to-java.html

Problems with XMLGregorianCalendar

The conversion between wsdl dataTypes and java dataTypes is held automatically by this plugin, so in most cases, you don´t need to worry about anything more.
But one annoying thing most people notes is the conversión between date/dateTime and XMLGregorianCalendar.

For example, by having this detifintion:

<xs:complexType name="operacionesPagoFiltroDTO">
 <xs:sequence>
  <xs:element minOccurs="0" name="entidad" type="xs:long" />
  <xs:element minOccurs="0" name="fechaDesde" type="xs:dateTime" />
  <xs:element minOccurs="0" name="fechaHasta" type="xs:dateTime" />
  <xs:element minOccurs="0" name="nifUsuario" type="xs:string" />
  <xs:element minOccurs="0" name="seccion" type="xs:long" />
 </xs:sequence>
</xs:complexType>

You will get this POJO:

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "operacionesPagoFiltroDTO", propOrder = {
    "entidad",
    "fechaDesde",
    "fechaHasta",
    "nifUsuario",
    "seccion"
})
public class OperacionesPagoFiltroDTO {

    protected Long entidad;
    @XmlSchemaType(name = "dateTime")
    protected XMLGregorianCalendar fechaDesde;
    @XmlSchemaType(name = "dateTime")
    protected XMLGregorianCalendar fechaHasta;
    protected String nifUsuario;
    protected Long seccion;

    (...)

}

So, you have to manually make the conversion in order to obtain java.util.Date.

Solution

The solution is to use a custom binding file and a dateAdapter. When CXF plugin is executed, it will check the binding configuration and will generate the java.util.Date automatically.

In pom.xml you have to specify the custom binding file:

<plugin>
 <groupId>org.apache.cxf</groupId>
 <artifactId>cxf-codegen-plugin</artifactId>
 <version>${cxf.version}</version>
 <executions>
  <execution>
   <id>generate-sources</id>
   <phase>generate-sources</phase>
   <configuration>
    <sourceRoot>${project.build.directory}/generated/cxf</sourceRoot>
    <wsdlOptions>        
     <wsdlOption>
      <wsdl>${basedir}/src/main/resources/wsdl/gestionEconomica/GestionEconomica.wsdl</wsdl>       
      <bindingFiles>
       <bindingFile>${basedir}/src/main/resources/wsdl/gestionEconomica/GestionEconomicaBinding.xml</bindingFile>
      </bindingFiles>
     </wsdlOption>
    </wsdlOptions>
   </configuration>
   <goals>
    <goal>wsdl2java</goal>
   </goals>
  </execution>
 </executions>
</plugin>

And this would be the binding file:

<jaxws:bindings wsdlLocation="GestionEconomica.xsd.wsdl"
 xmlns:jaxws="http://java.sun.com/xml/ns/jaxws" xmlns:xs="http://www.w3.org/2001/XMLSchema"
 xmlns:jxb="http://java.sun.com/xml/ns/jaxb" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/">

 <jaxws:bindings node="wsdl:definitions/wsdl:types/xs:schema[@targetNamespace='http://mytargetspace.es/']">
  
  <jxb:globalBindings xmlns:jxb="http://java.sun.com/xml/ns/jaxb" xmlns:xs="http://www.w3.org/2001/XMLSchema">
   <jxb:javaType 
       name="java.util.Date" 
       xmlType="xs:dateTime"
    parseMethod="org.apache.cxf.xjc.runtime.DataTypeAdapter.parseDateTime"
    printMethod="org.apache.cxf.xjc.runtime.DataTypeAdapter.printDateTime" />
  </jxb:globalBindings>
 </jaxws:bindings>
</jaxws:bindings>

By doing so, you will get this POJO:

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "operacionesPagoFiltroDTO", propOrder = {
    "entidad",
    "fechaDesde",
    "fechaHasta",
    "nifUsuario",
    "seccion"
})
public class OperacionesPagoFiltroDTO {

    protected Long entidad;
    @XmlElement(type = String.class)
    @XmlJavaTypeAdapter(Adapter1 .class)
    @XmlSchemaType(name = "dateTime")
    protected Date fechaDesde;
    @XmlElement(type = String.class)
    @XmlJavaTypeAdapter(Adapter1 .class)
    @XmlSchemaType(name = "dateTime")
    protected Date fechaHasta;
    protected String nifUsuario;
    protected Long seccion;

    (...)
}

You can specify any adapter you want, but CXF also provides you with a basic implementation: org.apache.cxf.xjc.runtime.DataTypeAdapter. In order to use it, you need to add this extra dependency to your pom.xml:

<dependency>
 <groupId>org.apache.cxf.xjc-utils</groupId>
 <artifactId>cxf-xjc-runtime</artifactId>
 <version>${cxf-xjc-runtime.version}</version>
</dependency>

Note: You can check more about wsdl2java and eclipse in one of my previous posts: http://trabajosdesisifo.blogspot.com.es/2015/04/eclipse-maven-configure-wsdl2java.html

JBoss CLI - Domain monitoring

These are some commands to get insight of the health of your deployments.
The idea was to have one script and run it periodically to get some KPI. This script invokes jboss-cli by passing the command as a parameter.
You can ask why don´t have a jboss-cli script file and to pass it as a parameter (./jboss-cli --file ). By doing so, you can not print custom messages, because we were using JBoss EAP 6.2. In 6.4 you do have an echo command in CLI.

echo "--------------------------------------------------"
echo "$(date)"

echo "Session info"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=web:read-attribute(name=expired-sessions)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=web:read-attribute(name=rejected-sessions)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=web:read-attribute(name=max-active-sessions)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=web:read-attribute(name=session-max-alive-time)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=web:read-attribute(name=sessions-created)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=web:read-attribute(name=session-avg-alive-time)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=web:read-attribute(name=context-root)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=web:read-attribute(name=virtual-host)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=web:read-attribute(name=active-sessions)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=web:read-attribute(name=duplicated-session-ids)"


echo "EJB info"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=methods)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=timers)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=pool-available-count)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=peak-concurrent-invocations)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=invocations)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=pool-create-count)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=declared-roles)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=security-domain)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=pool-name)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=run-as-role)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=component-class-name)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=pool-max-size)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=wait-time)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=execution-time)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=pool-remove-count)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/deployment=<deployment>/subsystem=ejb3/stateless-session-bean=EjbInvoker:read-attribute(name=pool-current-size)"


echo "Memory info"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/core-service=platform-mbean/type=memory : read-attribute(name=heap-memory-usage)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/core-service=platform-mbean/type=memory : read-attribute(name=non-heap-memory-usage)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/core-service=platform-mbean/type=memory : read-attribute(name=verbose)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/core-service=platform-mbean/type=memory : read-attribute(name=object-name)"
./jboss-cli.sh -c --user=<user> --password=<pass> --command="/host=<host>/server=<server>/core-service=platform-mbean/type=memory : read-attribute(name= object-pending-finalization-count)"

JBoss Domain JConsole connection

JBoss EAP is shipped with its own JConsole, so you can get insight about your server via JMX.
In this post I´ll talk about domain mode in JBoss EAP 6.2.

JConsole is in ${JBoss.home}/bin/jconsole.sh

Topology

This is my topology:

• Domain Controller in 192.168.56.101
• Host Controller 1 in 192.168.56.102
• Host Controller 2 in 192.168.56.103

I Have a server-group called jbossEAP6-sg1 with two servers, each one in its own HC:

• server-full-ha-1 in HC 1, bound to 192.168.56.130
• server-full-ha-2 in HC 2, bound to 192.168.56.140

This domain is in a CentOS remote PC and I Will connect from my Windows installation.

Configuration

I want to monitor Domain Controller in general, and my servers in particular. So I will make two different connections.

You have to configure jmx subsystem in domain.xml (this file is placed inside de Domain Controller). Once there you have to set the following config:

<subsystem xmlns="urn:jboss:domain:jmx:1.3">
 <expose-resolved-model/>
 <expose-expression-model/>
 <remoting-connector use-management-endpoint="false"/>
</subsystem>

Then, you need an application user in all the nodes (asuming you already have a management user in the domain). Simply use ./add-user.sh in DC, add a user, and copy the application-roles.properties and  application-users.properties files in the others HCs.

Connection to DC

You can start your connection with the followning parameters:

• Remote Process: service:jmx:remoting-jmx://192.168.56.101:9999
• Username/Password: domain user

Note that there is a CLI tab due to this is a DC.

Connection to server-full-ha-1 in HC 1

You can starts your connection with the followning parameters:

• Remote Process: service:jmx:remoting-jmx://192.168.56.130:4447
• Username/Password: application user

Note that in this case you need to set de 4447 port. I'm using a custom binding address for each server, so I don´t need to specify a port offset, otherwise you need to calculate the real port of your server.

Aditional info:

• https://developer.jboss.org/wiki/UsingJconsoleToConnectToJMXOnAS7