How to configure a proxy JCR cluster

This tutorial will show you how to configure and initialize a JCR cluster in proxy mode for eXo ECM. The tutorial uses 4 tomcat nodes and 1 oracle database schema.

tutorial-jcr-proxy.png

1 Tutorial environment

1.1 Database

We will use an Oracle database. The JDBC URL is : 

jdbc:oracle:thin:@tornado.exoua-int:1523:orcl
In proxy mode a single user schema is required. We will use exocluster

Learn more about Database Configuration.

1.2 JCR settings

This tutorial was written and tested using JCR 1.10 but should also work properly for version 1.9.3

On windows we used a shared drive : \\192.168.0.20\shared_value_storage
On unix system you would most likely use a NFS or SAN mount point

1.3 Cluster shape

The cluster has 4 participant nodes each is bound to its own IP address:

  • cluster_node1 : 192.168.0.15
  • cluster_node2 : 192.168.0.135
  • cluster_node3 : 192.168.0.5
  • cluster_node4 : 192.168.0.3
Warning: Ensure that all participants are time-synchronized with the same NTP-server for example. This is a common requirement for most clustering solutions

1.4 Get an eXo tomcat bundle

The instructions of this tutorial should be compatible with eXo Portal 2.2.
Get and install eXo ECM 2.2 tomcat bundle from the download area.

Alternatively, you can always build the product you want from sources using eXoBuild tool. For exemple : 

exobuild --product=portal --version=2.2 --build --deploy

2 Configuring and starting 1st node

2.1 Add replication configuration

On node cluster_node1, create a file replication-config.xml at D:\java\exo-working\exo-tomcat\webapps\portal.war\WEB-INF\conf\jcr\: 

<?xml version="1.0" encoding="ISO-8859-1"?>
<configuration>
  <component>
  <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
  <init-params>
    <value-param>
        <name>force-xml-configuration</name>
        <value>true</value>      
    </value-param>
    <values-param>
      <name>repositories</name>
      <value>repository</value>
    </values-param>
    <properties-param>
      <name>replication-properties</name>
      <property name="enabled" value="false"/>
      <property name="mode" value="proxy"/>
      <property name="bind-ip-address" value="192.168.0.15"/>
      <property name="channel-config" value="TCP(oob_thread_pool.queue_max_size=100;thread_naming_pattern=cl;use_concurrent_stack=true;oob_thread_pool.rejection_policy=Run;discard_incompatible_packets=true;thread_pool.max_threads=40;oob_thread_pool.enabled=false;oob_thread_pool.max_threads=20;loopback=false;oob_thread_pool.keep_alive_time=5000;thread_pool.queue_enabled=false;oob_thread_pool.queue_enabled=false;max_bundle_size=64000;thread_pool.queue_max_size=100;thread_pool.enabled=false;enable_diagnostics=true;max_bundle_timeout=30;oob_thread_pool.min_threads=8;use_incoming_packet_handler=true;thread_pool.rejection_policy=Run;bind_addr=$bind-ip-address;thread_pool.min_threads=8;thread_pool.keep_alive_time=5000;enable_bundling=true):MPING(timeout=2000;num_initial_members=8;mcast_port=34526;mcast_addr=224.0.0.1):FD(timeout=2000;max_tries=5;shun=true):FD_SOCK:VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(max_xmit_size=60000;print_stability_history_on_failed_xmit=true;use_mcast_xmit=false;gc_lag=0;discard_delivered_msgs=true;retransmit_timeout=300,600,1200,2400,4800):pbcast.STABLE(stability_delay=1000;desired_avg_gossip=50000;max_bytes=8000000):pbcast.GMS(print_local_addr=true;join_timeout=3000;view_bundling=true;join_retry_timeout=2000;shun=true;merge_leader=true;reject_join_from_existing_member=true)"/>
      <property name="recovery-dir" value="../temp/replication/recovery"/>
      <property name="node-name" value="cluster_node1"/>
      <property name="other-participants" value="cluster_node2;cluster_node3;cluster_node4"/>
      <property name="wait-confirmation" value="2000"/>
    </properties-param>
    <properties-param>
      <name>replication-priority-properties</name>
      <property name="priority-type" value ="dynamic"/> <!-- {static, dynamic} -->
      <property name="node-priority" value="100"/> <!-- max == 100 -->
    </properties-param>
  </init-params>
</component>
</configuration>

JCR replication relies on JGroups for networking. Although the configuration should be done on a single line in xml file, we copied it here in a more readable format for easier understanding : 

TCP(
oob_thread_pool.queue_max_size=100
thread_naming_pattern=cl
use_concurrent_stack=true
oob_thread_pool.rejection_policy=Run
discard_incompatible_packets=true
thread_pool.max_threads=40
oob_thread_pool.enabled=false
oob_thread_pool.max_threads=20
loopback=false
oob_thread_pool.keep_alive_time=5000
thread_pool.queue_enabled=false
oob_thread_pool.queue_enabled=false
max_bundle_size=64000
thread_pool.queue_max_size=100
thread_pool.enabled=false
enable_diagnostics=true
max_bundle_timeout=30
oob_thread_pool.min_threads=8
use_incoming_packet_handler=true
thread_pool.rejection_policy=Run
bind_addr=$bind-ip-address
thread_pool.min_threads=8
thread_pool.keep_alive_time=5000
enable_bundling=true)

:MPING(
timeout=2000
num_initial_members=8
mcast_port=34526
mcast_addr=224.0.0.1)

:FD(
timeout=2000
max_tries=5
shun=true)

:FD_SOCK

:VERIFY_SUSPECT(timeout=1500)

:pbcast.NAKACK(
max_xmit_size=60000
print_stability_history_on_failed_xmit=true
use_mcast_xmit=false
gc_lag=0
discard_delivered_msgs=true
retransmit_timeout=300,600,1200,2400,4800)

:pbcast.STABLE(
stability_delay=1000
desired_avg_gossip=50000
max_bytes=8000000
)

:pbcast.GMS(
print_local_addr=true
join_timeout=3000
view_bundling=true
join_retry_timeout=2000
shun=true
merge_leader=true
reject_join_from_existing_member=true)

2.2 Verify the repository name

It must match repository name in repository-configuration.xml :

replication-config.xml 

...
<values-param>
<name>repositories</name>
<value>repository</value>
</values-param>
...

2.3 Import replication configuration

portal.war/WEB-INF/conf/configuration.xml 

...
  <import>war:/conf/ecm/cms-configuration.xml</import>
  <import>war:/conf/ecm/ecm-templates-configuration.xml</import>
  <import>war:/conf/ecm/ecm-scripts-configuration.xml</import>
  <import>war:/conf/ecm/ecm-actions-configuration.xml</import>
  <import>war:/conf/ecm/ecm-categories-configuration.xml</import>
  <import>war:/conf/ecm/ecm-views-configuration.xml</import>
  <import>war:/conf/ecm/ecm-drives-configuration.xml</import>
  <import>war:/conf/ecm/ecm-metadata-configuration.xml</import>
  <import>war:/conf/ecm/ecm-folksonomy-configuration.xml</import>
  <import>war:/conf/ecm/ecm-records-configuration.xml</import>
  <import>war:/conf/ecm/ecm-queries-configuration.xml</import>
  <import>war:/conf/ecm/organization-component-plugins-configuration.xml</import>
  <import>war:/conf/jcr/replication-config.xml</import>
</configuration>

2.4 Configure database

  • Configure datasource :
portal.war/WEB-INF/conf/database/database-configuration.xml 
<external-component-plugins>
  <target-component>org.exoplatform.services.naming.InitialContextInitializer</target-component>
  <component-plugin>
  ...
    <properties-param>
      <name>ref-addresses</name>
      <description>ref-addresses</description>
      <property name="driverClassName" value="oracle.jdbc.OracleDriver" />
      <property name="url" value="jdbc:oracle:thin:@tornado.exoua-int:1523:orcl" />
      <property name="username" value="exocluster" />
      <property name="password" value="exo12321" />
      <property name="maxActive" value="100" />
      <property name="maxIdle" value="2" />
      <property name="initialSize" value="2" />
    </properties-param>
  ...
  </component-plugin>
</external-component-plugins>

  • Set the JCR database dialect oracle :
repository-configuration.xml and jcr-configuration.xml 
<property name="dialect" value="oracle"/>

2.5 Configure value storage

Set the shared value storage for all workspaces in repository configuration.

repository-configuration.xml 

...
     <workspaces>
       <workspace name="system">
...
            <value-storages>
              <value-storage id="system" class="org.exoplatform.services.jcr.impl.storage.value.fs.TreeFileValueStorage">
                <properties>
                  <property name="path" value="\\192.168.0.20\shared_value_storage\temp\values\system"/>
                </properties>
...
              </value-storage>
            </value-storages>
          </container>
...
       </workspace>
       <workspace name="collaboration">
...
            <value-storages>
              <value-storage id="collaboration" class="org.exoplatform.services.jcr.impl.storage.value.fs.TreeFileValueStorage">
                <properties>
                  <property name="path" value="\\192.168.0.20\shared_value_storage\temp\values\collaboration"/>
                </properties>
...
              </value-storage>
            </value-storages>
          </container>
...
       </workspace>
       <workspace name="backup">
...
            <value-storages>
              <value-storage id="backup" class="org.exoplatform.services.jcr.impl.storage.value.fs.TreeFileValueStorage">
                <properties>
                  <property name="path" value="\\192.168.0.20\shared_value_storage\temp\values\backup"/>
                </properties>
...
              </value-storage>
            </value-storages>
          </container>
...
       </workspace>      
      </workspaces>
...

2.6 Copy the modified tomcat to other nodes

Copy the contents of the folder D:\java\exo-working\exo-tomcat in three folders:

  • cluster_node2 : D:\java\exo-working\exo-tomcat_2
  • cluster_node3 : D:\java\exo-working\exo-tomcat_3
  • cluster_node4 : D:\java\exo-working\exo-tomcat_4

2.7 Start tomcat

You can now start tomcat on cluser_node1. It will initialize JCR and leave replication inactive.

eXo.bat run

2.8 Stop tomcat

Once tomcat has started fully without errors hit CTRL-C to shut down tomcat on cluser_node1. The JCR is now initialized properly and we are ready to activate the replication.

2.9 Activate replication

Set the following parameters in replication config replication-config.xml 

<property name="enabled" value="true"/>

2.10 Start tomcat again

You can now start tomcat on cluser_node1. 

eXo.bat run

The first node of your JCR cluster is now properly initialized.

3 Configuring and starting 2nd node

3.1 Configure replication

Set the following parameters in replication config :

  • ip-address = 192.168.0.135
  • node-name = cluster_node2
  • other-participants = cluster_node1;cluster_node3;cluster_node4
  • node-priority = 50
replication-config.xml 
<configuration>
  <component>
  <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
  <init-params>
  ...
    <property name="enabled" value="false"/>
    ...
    <property name="bind-ip-address" value="192.168.0.135"/>
    ...
    <property name="node-name" value="cluster_node2"/>
    <property name="other-participants" value="cluster_node1;cluster_node3;cluster_node4"/>
    ...
    <property name="priority-type" value ="dynamic"/> <!-- {static, dynamic} -->
    <property name="node-priority" value="50"/> <!-- max == 100 -->
  ...
  </init-params>
</component>

notice that replication is not yet activated

3.2 Start tomcat

You can now start tomcat on cluser_node2. 

eXo.bat run
It will initialize JCR on the node and leave replication inactive.

3.3 Stop tomcat

Once tomcat has started fully without errors hit CTRL-C to shut down tomcat on cluser_node2. The JCR is now initialized properly and we are ready to activate the replication.

3.4 Activate replication

Set the following parameters in replication config replication-config.xml 

<property name="enabled" value="true"/>

3.5 Start tomcat again

You can now start tomcat on cluser_node2. 

eXo.bat run

The second node of your JCR cluster has now joined the cluster!

4 Configuring and starting 3rd and 4th node

Proceed similarly as for second cluster node.

4.1 Settings for 3rd cluster node

  • ip-address = 192.168.0.5
  • node-name = cluster_node3
  • other-participants = cluster_node1;cluster_node2;cluster_node4
  • priority-type = dynamic
  • node-priority = 30
  • path to tomcat for third cluster node = D:\java\exo-working\exo-tomcat_3

4.2 Settings for 4th cluster node

  • ip-address = 192.168.0.3
  • node-name = cluster_node4
  • other-participants = cluster_node1;cluster_node2;cluster_node3
  • priority-type = dynamic
  • node-priority = 20
  • path to tomcat for fourth cluster node = D:\java\exo-working\exo-tomcat_4

5 How to add a new node to the running cluster?

Let's see how to add a 5th node to the already established cluster

  • ip-address = 192.168.0.77
  • node-name = cluster_node5
  • other-participants = cluster_node1;cluster_node2;cluster_node3;cluster_node4
  • priority-type = dynamic
  • node-priority = 10
  • user name for database = exocluster
  • password for database = exo12321
  • URL to database server = jdbc:oracle:thin:@tornado.exoua-int:1523:orcl
First, we need to register the new participant in the configuration of all other nodes.

replication-config.xml (cluster_node1) 

<component>
  <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
    ...
    <property name="other-participants" value="cluster_node1;cluster_node3;cluster_node4;cluster_node5"/>
    ...
</component>

replication-config.xml (cluster_node2): 

<component>
  <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
    ...
    <property name="other-participants" value="cluster_node2;cluster_node3;cluster_node4;cluster_node5"/>
    ...
</component>

replication-config.xml (cluster_node3) 

<component>
  <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
    ...
    <property name="other-participants" value="cluster_node1;cluster_node2;cluster_node4;cluster_node5"/>
    ...
</component>

replication-config.xml (cluster_node4): 

<component>
  <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
    ...
    <property name="other-participants" value="cluster_node1;cluster_node2;cluster_node3;cluster_node5"/>
    ...
</component>

Now you can restart tomcat on the running nodes one by one. The cluster is now reconfigured and working without the 5th node.

5.1 Prepare tomcat on cluster_node5

Copy folder D:\java\exo-working\exo-tomcat from cluster_node1 to cluster_node5. Copy all files and subfolders, except logs, temp and work and extracted war directories in webapps.

5.2 Configure Replication

replication-config.xml 

<configuration>
  <component>
  <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
  <init-params>
    ...
    <property name="enabled" value="false"/>
    ...
    <property name="bind-ip-address" value="192.168.0.77"/>
    ...
    <property name="node-name" value="cluster_node5"/>
    <property name="other-participants" value="cluster_node1;cluster_node3;cluster_node4;cluster_node5"/>
    ...
    <property name="priority-type" value ="dynamic"/> <!-- {static, dynamic} -->
    <property name="node-priority" value="10"/> <!-- max == 100 -->
    ...
  </init-params>
</component>

notice that replication is not yet activated

5.3 Configure Database

Set following parameters in database configuration for cluster_node5 :

  • URL to database server = jdbc:oracle:thin:@tornado.exoua-int:1523:orcl;
  • user name for database = exocluster;
  • password for database = exo12321.
database-configuration.xml: 
<external-component-plugins>
  <target-component>org.exoplatform.services.naming.InitialContextInitializer</target-component>
  <component-plugin>
  ...
    <properties-param>
      <name>ref-addresses</name>
      <description>ref-addresses</description>
      <property name="driverClassName" value="oracle.jdbc.OracleDriver" />
      <property name="url" value="jdbc:oracle:thin:@tornado.exoua-int:1523:orcl" />
      <property name="username" value="exocluster" />
      <property name="password" value="exo12321" />
      <property name="maxActive" value="100" />
      <property name="maxIdle" value="2" />
      <property name="initialSize" value="2" />
    </properties-param>
   </init-params>
  </component-plugin>
</external-component-plugins>

5.4 Start tomcat

You can now start tomcat on cluser_node5. 

eXo.bat run
It will initialize JCR on the node and leave replication inactive.

5.5 Stop tomcat

Once tomcat has started fully without errors hit CTRL-C to shut down tomcat on cluser_node5. The JCR is now initialized properly and we are ready to activate the replication.

5.6 Activate replication

Set the following parameters in replication config replication-config.xml 

<property name="enabled" value="true"/>

5.7 Start tomcat again

You can now start tomcat on cluser_node5. 

eXo.bat run

The 5th node has now joined the cluster!

 
Products
generated on Tue Mar 16 06:07:13 UTC 2010
eXo Optional Modules
eXo Core Foundations
Recently Modified
Twilight3eclipsee | Twilight3eclipsex | Saga3Eclipsey | Mypeelite | Moitubbipscit
Creator: Alex Reshetnyak on 11/05/2008
Copyright (c) 2000-2009. Allright reserved - eXo platform SAS
1.6.13286