Abstract
One question which arouse from time to time is how transactions work and what kind of configuration you need to set in your application in order to make it run, and to keep your databases clean (no lost writes or incosistent states).
The question about how to make your application run is usually solved by developers with the classic try-and-error approximation. When you face an error, you could begin to change settings until the error disappear. Recently one of these problems was related with the access to two different databases through two JTA Datasources, and in the same transaction. The magic solution found was to change one of these Datasources to non JTA.
The question about how to keep your databases clean depends on how you solved the error. As you will see later, that solution was not much safe, but probably they didn't know.
Glosary
These are the main terms used here:
- Local transaction: A transaction which involves only one transactional resource.
- Global transaction: A transaction which involves multiple transactional resources.
- Distribuited transaction: A global transaction which spans over multiple hosts (in this case, servers).
- XA: eXtended Architecture
- XA Resource: Transactional resource which allows 2PC for global transactions.
- 1PC: One Phase Commit
- 2PC: Two Phase Commit
Test structure
The test application will be running in JBoss EAP 6.4. It will use Spring 3.X and Hibernate 4.X (sorry, I have in my to-do list to migrate all these stuff )
For these test there will a set of datasources provided for JBoss. Two of them will be datasources with JTA flag to false, two more with JTA to true (default value) and finally two XA-Datasources.
In each test the application will use different combinations of these datasources, but there will be always two different databases in the pitch.
Database access will be configured through Hibernate SessionFactory and Spring Transaction Manager:
<bean id="dataSource1" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:jboss/datasources/ExampleDSNoJTA"/>
</bean>
<bean id="sessionFactory1" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean" lazy-init="false">
<property name="dataSource" ref="dataSource1"/>
<property name="packagesToScan" value="es.sisifo.jboss.datasources.entity.one"/>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.HSQLDialect</prop>
<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.format_sql">true</prop>
<prop key="hibernate.use_sql_comments">true</prop>
<prop key="hibernate.hbm2ddl.auto">create</prop>
<prop key="hibernate.validator.apply_to_ddl">false</prop>
<prop key="hibernate.validator.autoregister_listeners">false</prop>
<prop key="hibernate.transaction.factory_class">org.hibernate.engine.transaction.internal.jta.CMTTransactionFactory</prop>
<prop key="hibernate.transaction.jta.platform">org.hibernate.service.jta.platform.internal.JBossAppServerJtaPlatform</prop>
</props>
</property>
</bean>
<!-- The same to sessionFactory2 -->
<!-- TX config -->
<tx:annotation-driven/>
<tx:jta-transaction-manager/>
During startup Spring will try to find the server transaction manager by checking the following JNDI entries:
- java:comp/TransactionManager
- java:appserver/TransactionManager
- java:pm/TransactionManager
- java:/TransactionManager. Bingo, it will use com.arjuna.ats.jbossatx.jta.TransactionManagerDelegate
In each database there will be one table/entity: EntityOne and EntityTwo, with the same dummy structure:
@Entity
@Table(name = "tableOne")
public class EntityOne {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private String code;
private Date time;
(...)
}
Each test will consists of calling three transactional methods in a service:
@Override
@Transactional
public void testInsertOK() {
final EntityOne entityOne = new EntityOne();
entityOne.setTime(new Date());
entityOne.setCode("Insert OK");
entityOneDao.save(entityOne);
final EntityTwo entityTwo = new EntityTwo();
entityTwo.setTime(new Date());
entityTwo.setCode("Insert OK");
entityTwoDao.save(entityTwo);
}
@Override
@Transactional(rollbackFor=Exception.class)
public void testInsertMiddleError() {
final EntityOne entityOne = new EntityOne();
entityOne.setTime(new Date());
entityOne.setCode("Insert with Middle Error");
entityOneDao.save(entityOne);
if (System.currentTimeMillis() > 0) {
throw new RuntimeException("Kaboom in the middle");
}
final EntityTwo entityTwo = new EntityTwo();
entityTwo.setTime(new Date());
entityTwo.setCode("Insert with Middle Error");
entityTwoDao.save(entityTwo);
}
@Override
@Transactional(rollbackFor=Exception.class)
public void testInsertFinalError() {
final EntityOne entityOne = new EntityOne();
entityOne.setTime(new Date());
entityOne.setCode("Insert with Final Error");
entityOneDao.save(entityOne);
final EntityTwo entityTwo = new EntityTwo();
entityTwo.setTime(new Date());
entityTwo.setCode("Insert with Final Error");
entityTwoDao.save(entityTwo);
throw new RuntimeException("Kaboom at the end");
}
At the end of the tests, both tables will be listed in order to check which writes persisted and which were rolled back.
Summary
Tests results
Non JTA Datasource + Non JTA Datasource
Result:
EntityOne [id=1, code=Insert OK, time=2018-03-25 20:03:22.577]
EntityOne [id=2, code=Insert with Middle Error, time=2018-03-25 20:03:22.683]
EntityOne [id=3, code=Insert with Final Error, time=2018-03-25 20:03:22.698]
EntityTwo [id=1, code=Insert OK, time=2018-03-25 20:03:22.663]
EntityTwo [id=2, code=Insert with Final Error, time=2018-03-25 20:03:22.701]
Database writes are auto-commited inmeditely in both resources.
You can use both resources inside the same transaction, but they are not enlisted.
Each resource has its own "transaction boundary". If there is an error inside the transaction, as there is no rollback, you can get data inconsistency.
JTA Datasource + Non JTA Datasource
Result:
EntityOne [id=1, code=Insert OK, time=2018-03-24 19:16:51.645]
EntityTwo [id=1, code=Insert OK, time=2018-03-24 19:16:51.709]
EntityTwo [id=2, code=Insert with Final Error, time=2018-03-24 19:16:51.733]
JTA resource is managed inside a JTA Transaction, so transaction boundary is demarcated by the transaction manager.
Database writes are committed on transaction completion for JTA resource, and auto-committed inmediately for non JTA resource .
Thus, non JTA resource is not managed by transaction manager, if there is an error during the transaction, JTA resource is rolled back and non JTA resource is autocommitted.
You can get database inconsistencies.
Non JTA Datasource + JTA Datasource
Result:
EntityOne [id=1, code=Insert OK, time=2018-03-24 19:25:33.252]
EntityOne [id=2, code=Insert with Middle Error, time=2018-03-24 19:25:33.325]
EntityOne [id=3, code=Insert with Final Error, time=2018-03-24 19:25:33.335]
EntityTwo [id=1, code=Insert OK, time=2018-03-24 19:25:33.312]
Similar to previous scenario. In this case, the difference is due to the insertions order.
During an exception inside the transaction, only JTA resource is rolled back.
JTA Datasource + JTA Datasource
You get the following error:
19:37:52,308 WARN [com.arjuna.ats.arjuna] (http-localhost/127.0.0.1:8080-2) ARJUNA012140: Adding multiple last resources is disallowed. Trying to add LastResourceRecord(XAOnePhaseResource(LocalXAResourceImpl@434c39c8[connectionListener=27aa95d connectionManager=5209886e warned=false currentXid=< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffc0a80133:17004d06:5ab68ccc:26e, node_name=1, branch_uid=0:ffffc0a80133:17004d06:5ab68ccc:275, subordinatenodename=null, eis_name=java:jboss/datasources/ExampleDS2 > productName=H2 productVersion=@PROJECT_VERSION@ (2012-07-13) jndiName=java:jboss/datasources/ExampleDS2])), but already have LastResourceRecord(XAOnePhaseResource(LocalXAResourceImpl@414aa356[connectionListener=501a920b connectionManager=1519056b warned=false currentXid=< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffc0a80133:17004d06:5ab68ccc:26e, node_name=1, branch_uid=0:ffffc0a80133:17004d06:5ab68ccc:272, subordinatenodename=null, eis_name=java:jboss/datasources/ExampleDS > productName=H2 productVersion=@PROJECT_VERSION@ (2012-07-13) jndiName=java:jboss/datasources/ExampleDS]))
19:37:52,309 INFO [stdout] (http-localhost/127.0.0.1:8080-2) [2018-03-24 19:37:52,309] (SqlExceptionHelper.java:145) WARN http-localhost/127.0.0.1:8080-2 org.hibernate.engine.jdbc.spi.SqlExceptionHelper SQL Error: 0, SQLState: null
19:37:52,309 INFO [stdout] (http-localhost/127.0.0.1:8080-2) [2018-03-24 19:37:52,309] (SqlExceptionHelper.java:147) ERROR http-localhost/127.0.0.1:8080-2 org.hibernate.engine.jdbc.spi.SqlExceptionHelper javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@27aa95d[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1a5a5cf8 connection handles=0 lastUse=1521916672308 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@11371f69 pool internal context=SemaphoreArrayListManagedConnectionPool@6f62844[pool=ExampleDS2] xaResource=LocalXAResourceImpl@434c39c8[connectionListener=27aa95d connectionManager=5209886e warned=false currentXid=null productName=H2 productVersion=@PROJECT_VERSION@ (2012-07-13) jndiName=java:jboss/datasources/ExampleDS2] txSync=null]
You are trying to enlist two transactional resources in the same transaction, i.e., you are running in a global transaction. This scenario is only allowed with XA resources.
The error is quite clear:
ARJUNA012140: Adding multiple last resources is disallowed.
Trying to add LastResourceRecord(XAOnePhaseResource(... jndiName=java:jboss/datasources/ExampleDS2])), but already have LastResourceRecord(XAOnePhaseResource(... jndiName=java:jboss/datasources/ExampleDS]))
What it's saying is that you have a 1P resource (ExampleDS) in an active transaction, and you are trying to enlist a second 1P Resource (ExampleDS) in the same transaction. You can only do that with XA resources.
JTA Datasource + JTA XA Datasource
Response:
EntityOne [id=1, code=Insert OK, time=2018-03-25 12:29:42.584]
EntityTwo [id=1, code=Insert OK, time=2018-03-25 12:29:42.695]
OK, this can sound weird. You have one JTA resource (which only supports 1PC) enlisted in a global transaction with other XA resource (which do supports 2PC). This should be an error, because you can not ensure the 2PC, i.e, the transaction coordinator can not ask the JTA resource if it is prepared.
Here is where
ARJUNA came to scene. JBoss uses ARJUNA as transaction core, and it uses a
LRCO - Last Resource Commit Optimization:
The doc is surprisingly clear. You do can have a single non-XA resource inside a global transaction, but only if all the other resources do are XA. In order to perform the 2PC, in the first phase the non XA resource is processed last, and only if all the XA resources responded affirmatively to the prepare request, the non-XA Resources is committed immediately. During second phase the rest of XA Resources are committed normally.
The operations are as follow:
- Prepare 2PC (The XA Resources)
- Commit LRCO (The non-XA Resource)
- Write tx log
- Commit 2PC (The XA Resources)
But there is still an error opportunity if something crash in step 3. So, this is not a fully reliable scenario.
In order to improve LRCO there is the
CMR - Commit Markable Resource, but it´s only valid for datasource (a transactional resource can be, for example, a JMS Queue, but you can not use CMR here). You need some extra configuration for this:
- Set the non-XA datasource as connectable
- Create a new table in database (the one for de non-XA datasource) to store XIDs
- Link Jboss transaction subsystem with this datasource and this table
JTA Datasource + JTA XA Datasource
Same result as previous test.
XA Datasource + XA Datasource
Same result as previous test.
This is a full global transaction scenario with XA Resources.
JTA XA Datasource + Non JTA Datasource
Same result as "JTA Datasource + Non JTA Datasource"
Non JTA Datasource + JTA XA Datasource
Same result as "Non JTA Datasource + JTA Datasource"
