What is the purpose of server clusters?
Server clusters ensure high availability.
Diffusion uses a datagrid to share session and topic information between Diffusion servers within a cluster, providing high availability for clients connecting to load-balanced servers.
If you have two or more Diffussion servers, you can form a Diffusion cluster. Please refer <> on how to setup multiple Diffusion servers on your local machine.
Do you have a license which allows replication?
Session replication is a paid feature of Diffusion. You cannot form a cluster or replicate topics and/or sessions if you don't have a Diffusion Distribution or Diffusion Enterprise license. To form a Diffusion cluster, you need to enable replication. This can be done by changing the replication flag from false to true in the Replication.xml file in the etc folder.
<sessionReplication enabled="true" />
<topicReplication enabled="true">
<topics>
<!--
Use includes elements to specify replicated branches, ignoring all others:
<includes>path1</includes>
<includes>path2</includes>
Use excludes elements to exclude branches from replication:
<excludes>path1</excludes>
<excludes>path2</excludes>
Mixing <includes> and <excludes> elements is not allowed.
-->
</topics>
</topicReplication>
Ensure that you set this flag as true for all the servers which you want to be part of the cluster.
- Save the changes you've made on the .xml files.
If you don't have a license and you start Diffusion servers, you'll most likely end up seeing the below:
C:\Work\Diffusion 6.7.2\bin>diffusion.bat
2021-11-12 15:22:42.493|INFO|main|PUSH-000578|||Diffusion 6.7.2 "Russell" - Build #4fbb8bff, 9724@PTLT72, Oracle Corporation Java HotSpot(TM) Client VM 1.8.0_311.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:22:42.603|INFO|main|PUSH-000166|||Diffusion server 'PTLT72' starting.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:22:42.603|INFO|main|PUSH-000159|||The maximum message size is 32768 bytes.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:22:42.930|WARN|main|PUSH-000676|||This system has 16 available cores which exceeds the license limit of 8 cores. The server thread pools have been constrained to degrade performance.|com.pushtechnology.diffusion.licence.agent.c
2021-11-12 15:22:43.207|WARN|main||||
*******************************************************************************
Thank you for choosing Diffusion. This server is using a restricted default
license. It is limited to 1000 topics, 5 user sessions, and 8 CPU cores.
Production use is explicitly prohibited.
Get a more capable license for this server by joining the Push Technology
Developer Community at https://community.pushtechnology.com. Registration is
free and allows you to generate a community license that supports single
server production deployments and up to 1000 user sessions. You can also
generate an evaluation license that grants trial access to premium features
available with a commercial license including multi-server clusters for
high-availability and scale, topic persistence, and inter-server connections
across wide area networks.
For a commercial license that supports larger numbers of topics and sessions
and production use of the premium features, please contact our sales team.
https://www.pushtechnology.com/ [email protected] +44 20 3588 0900
*******************************************************************************
|com.pushtechnology.diffusion.licence.agent.LicenceLoaderImpl
2021-11-12 15:22:43.574|INFO|main|PUSH-000795|||Configuration replication is enabled.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:22:43.743|ERROR|main|PUSH-000696|||Feature 'SESSION_REPLICATION' is not licensed.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:22:43.744|ERROR|main|PUSH-000164|||Diffusion server not started.|com.pushtechnology.diffusion.api.server.DiffusionServer
com.pushtechnology.diffusion.licence.DiffusionLicenceException: Feature 'SESSION_REPLICATION' is not licensed.
at com.pushtechnology.diffusion.DiffusionController.start(DiffusionController.java:558) ~[diffusion.jar:6.7.2-4fbb8bff]
at com.pushtechnology.diffusion.api.server.DiffusionServer.start(DiffusionServer.java:167) [diffusion.jar:6.7.2-4fbb8bff]
at com.pushtechnology.diffusion.api.server.DiffusionServer.main(DiffusionServer.java:290) [diffusion.jar:6.7.2-4fbb8bff]
at com.pushtechnology.diffusion.Diffusion.main(Diffusion.java:22) [diffusion.jar:6.7.2-4fbb8bff]
2021-11-12 15:22:43.750|WARN|main|PUSH-000168|||Diffusion stopping, reason='DiffusionServer.stop() called' by administrator='anonymous'.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:22:43.753|INFO|main|PUSH-000167|||Diffusion stopped.|com.pushtechnology.diffusion.DiffusionController
Diffusion stops. The log message is straightforward that the Feature 'SESSION_REPLICATION' is not licensed. However, if you have a license, Diffusion will start.
Each server in a cluster must have a unique name. If you don't assign unique names for the servers, your Diffusion instance will fail to start. The exception in the below log is Server names in a cluster must be unique but Member [192.168.1.156]:5701 - 161a5352-8481-4001-994f-b39c874d353a has an identical server name [PTLT72]
C:\Work\Diffusion 6.7.2_2\bin>diffusion.bat
2021-11-12 15:25:02.326|INFO|main|PUSH-000578|||Diffusion 6.7.2 "Russell" - Build #4fbb8bff, 15420@PTLT72, Oracle Corporation Java HotSpot(TM) Client VM 1.8.0_311.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:25:02.503|INFO|main|PUSH-000166|||Diffusion server 'PTLT72' starting.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:25:02.504|INFO|main|PUSH-000159|||The maximum message size is 32768 bytes.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:25:03.195|INFO|main|PUSH-000202|||Product license expires in 341 day(s).|com.pushtechnology.diffusion.licence.agent.c
2021-11-12 15:25:03.198|INFO|main|PUSH-000767|||This server is licensed to asekhar. License ID 5dc588f1-5158-47c1-90b4-6395d415434d.|com.pushtechnology.diffusion.licence.agent.LicenceLoaderImpl
2021-11-12 15:25:03.597|INFO|main|PUSH-000795|||Configuration replication is enabled.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:25:03.801|INFO|main|PUSH-000791|||Session replication is enabled.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:25:03.806|INFO|main|PUSH-000796|||Topic replication is enabled.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:25:05.480|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] Hazelcast 4.2 (20210324 - 405cfd1) starting at [192.168.1.156]:5702|com.hazelcast.system
2021-11-12 15:25:06.174|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] No discovery strategy is applicable for auto-detection|com.hazelcast.spi.discovery.integration.DiscoveryService
2021-11-12 15:25:06.933|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] Using Multicast discovery|com.hazelcast.instance.impl.Node
2021-11-12 15:25:06.942|WARN|main||||[192.168.1.156]:5702 [dev] [4.2] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.|com.hazelcast.cp.CPSubsystem
2021-11-12 15:25:07.336|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.|com.hazelcast.internal.diagnostics.Diagnostics
2021-11-12 15:25:07.348|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] [192.168.1.156]:5702 is STARTING|com.hazelcast.core.LifecycleService
2021-11-12 15:25:07.543|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] Trying to join to discovered node: [192.168.1.156]:5701|com.hazelcast.internal.cluster.impl.MulticastJoiner
2021-11-12 15:25:07.586|INFO|hz.PTLT72.IO.thread-in-0||||[192.168.1.156]:5702 [dev] [4.2] Initialized new cluster connection between /192.168.1.156:61946 and /192.168.1.156:5701|com.hazelcast.internal.server.tcp.TcpServerConnection
2021-11-12 15:25:13.668|INFO|hz.PTLT72.priority-generic-operation.thread-0||||[192.168.1.156]:5702 [dev] [4.2]
Members {size:2, ver:2} [
Member [192.168.1.156]:5701 - 161a5352-8481-4001-994f-b39c874d353a
Member [192.168.1.156]:5702 - 30a4cbd2-6096-43ed-880f-c148eb5af33a this
]
|com.hazelcast.internal.cluster.ClusterService
.
.
.
2021-11-12 15:25:14.907|ERROR|main|PUSH-000164|||Diffusion server not started.|com.pushtechnology.diffusion.api.server.DiffusionServer
com.pushtechnology.diffusion.api.config.ConfigException: Server names in a cluster must be unique but Member [192.168.1.156]:5701 - 161a5352-8481-4001-994f-b39c874d353a has an identical server name [PTLT72]
at com.pushtechnology.diffusion.replication.hazelcast.HazelcastInstanceProvider.provide(HazelcastInstanceProvider.java:99) ~[diffusion.jar:6.7.2-4fbb8bff]
.
.
.
at com.pushtechnology.diffusion.DiffusionController.configureContainer(DiffusionController.java:607) ~[diffusion.jar:6.7.2-4fbb8bff]
at com.pushtechnology.diffusion.DiffusionController.start(DiffusionController.java:552) ~[diffusion.jar:6.7.2-4fbb8bff]
at com.pushtechnology.diffusion.api.server.DiffusionServer.start(DiffusionServer.java:167) [diffusion.jar:6.7.2-4fbb8bff]
at com.pushtechnology.diffusion.api.server.DiffusionServer.main(DiffusionServer.java:290) [diffusion.jar:6.7.2-4fbb8bff]
at com.pushtechnology.diffusion.Diffusion.main(Diffusion.java:22) [diffusion.jar:6.7.2-4fbb8bff]
2021-11-12 15:25:14.920|WARN|main|PUSH-000168|||Diffusion stopping, reason='DiffusionServer.stop() called' by administrator='anonymous'.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:25:14.926|INFO|main|PUSH-000167|||Diffusion stopped.|com.pushtechnology.diffusion.DiffusionController
How to assign names to your Diffusion servers?
You can assign names to your servers in the Server.xml file in the etc folder.
- To name the second server as say, server2, replace <server-name /> in the file with the below.
<server-name>server2</server-name>
- Save the changes and start your Diffusion server.
C:\Work\Diffusion 6.7.2_2\bin>diffusion.bat
2021-11-12 15:28:44.827|INFO|main|PUSH-000578|||Diffusion 6.7.2 "Russell" - Build #4fbb8bff, 16412@PTLT72, Oracle Corporation Java HotSpot(TM) Client VM 1.8.0_311.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:28:44.932|INFO|main|PUSH-000166|||Diffusion server 'server2' starting.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:28:44.932|INFO|main|PUSH-000159|||The maximum message size is 32768 bytes.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:28:45.509|INFO|main|PUSH-000202|||Product license expires in 341 day(s).|com.pushtechnology.diffusion.licence.agent.c
2021-11-12 15:28:45.511|INFO|main|PUSH-000767|||This server is licensed to asekhar. License ID 5dc588f1-5158-47c1-90b4-6395d415434d.|com.pushtechnology.diffusion.licence.agent.LicenceLoaderImpl
2021-11-12 15:28:45.886|INFO|main|PUSH-000795|||Configuration replication is enabled.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:28:46.067|INFO|main|PUSH-000791|||Session replication is enabled.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:28:46.069|INFO|main|PUSH-000796|||Topic replication is enabled.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:28:47.481|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] Hazelcast 4.2 (20210324 - 405cfd1) starting at [192.168.1.156]:5702|com.hazelcast.system
2021-11-12 15:28:48.079|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] No discovery strategy is applicable for auto-detection|com.hazelcast.spi.discovery.integration.DiscoveryService
2021-11-12 15:28:48.788|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] Using Multicast discovery|com.hazelcast.instance.impl.Node
2021-11-12 15:28:48.796|WARN|main||||[192.168.1.156]:5702 [dev] [4.2] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.|com.hazelcast.cp.CPSubsystem
2021-11-12 15:28:49.161|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.|com.hazelcast.internal.diagnostics.Diagnostics
2021-11-12 15:28:49.171|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] [192.168.1.156]:5702 is STARTING|com.hazelcast.core.LifecycleService
2021-11-12 15:28:49.333|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] Trying to join to discovered node: [192.168.1.156]:5701|com.hazelcast.internal.cluster.impl.MulticastJoiner
2021-11-12 15:28:49.371|INFO|hz.server2.IO.thread-in-0||||[192.168.1.156]:5702 [dev] [4.2] Initialized new cluster connection between /192.168.1.156:61990 and /192.168.1.156:5701|com.hazelcast.internal.server.tcp.TcpServerConnection
2021-11-12 15:28:55.458|INFO|hz.server2.priority-generic-operation.thread-0||||[192.168.1.156]:5702 [dev] [4.2]
Members {size:2, ver:4} [
Member [192.168.1.156]:5701 - 161a5352-8481-4001-994f-b39c874d353a
Member [192.168.1.156]:5702 - e52a435a-f3bd-45bc-bef9-36f8335f714a this
]
|com.hazelcast.internal.cluster.ClusterService
2021-11-12 15:28:56.421|INFO|main||||[192.168.1.156]:5702 [dev] [4.2] [192.168.1.156]:5702 is STARTED|com.hazelcast.core.LifecycleService
2021-11-12 15:28:57.624|INFO|main|PUSH-000808|||JMX has loaded a TLS keystore from 'C:\Work\Diffusion 6.7.2_2\etc\sample.keystore'.|com.pushtechnology.diffusion.management.RemoteJMXService
2021-11-12 15:28:57.641|WARN|main|PUSH-000656|||Unable to start connector 'Management' on port 1099, address already in use.|com.pushtechnology.diffusion.management.ManagementServiceImpl
2021-11-12 15:28:57.647|INFO|main|PUSH-000651|||Cluster member HazelcastClusterMember[PTLT72, 192.168.1.156:8090, 8b7e511a-15a9-4d7b-b4a4-b46e42c3e3bb] joined the cluster.|com.pushtechnology.diffusion.replication.hazelcast.HzClusterService
2021-11-12 15:28:57.647|INFO|main|PUSH-000651|||Cluster member HazelcastClusterMember[server2, 192.168.1.156:8092, c5d3c113-7d34-4a61-848a-611fd7c39aa4] joined the cluster.|com.pushtechnology.diffusion.replication.hazelcast.HzClusterService
2021-11-12 15:28:57.688|INFO|main|PUSH-000193|||Connector 'Client Connector' has loaded a TLS keystore from 'C:\Work\Diffusion 6.7.2_2\etc\sample.keystore'.|com.pushtechnology.diffusion.comms.connector.ConnectorImpl
2021-11-12 15:28:57.759|INFO|main|PUSH-000362|||Started HTTP service for 'default-prometheus-service'.|com.pushtechnology.diffusion.webserver.services.HTTPService
2021-11-12 15:28:57.780|INFO|main|PUSH-000337|||Added file service virtual host 'default' to serve files from 'html' to requests where the host header matches '.*'.|com.pushtechnology.diffusion.webserver.services.fileservice.FileService
2021-11-12 15:28:57.787|INFO|main|PUSH-000072|||Connector 'Client Connector' started, listening on 0.0.0.0/0.0.0.0:8081.|com.pushtechnology.diffusion.comms.connector.ConnectorImpl
2021-11-12 15:28:57.791|INFO|main|PUSH-000712|||Connector 'Client Connector' has scheduled pings every 90000ms.|com.pushtechnology.diffusion.comms.connector.PingScheduler
2021-11-12 15:28:58.092|INFO|main|PUSH-000806|||The file service is serving files at https://192.168.1.156:8081 .|com.pushtechnology.diffusion.comms.connector.ConnectorImpl
2021-11-12 15:28:58.092|INFO|main|PUSH-000806|||The file service is serving files at https://192.168.56.214:8081 .|com.pushtechnology.diffusion.comms.connector.ConnectorImpl
2021-11-12 15:28:58.094|INFO|main|PUSH-000193|||Connector 'High Volume Connector' has loaded a TLS keystore from 'C:\Work\Diffusion 6.7.2_2\etc\sample.keystore'.|com.pushtechnology.diffusion.comms.connector.ConnectorImpl
2021-11-12 15:28:58.094|INFO|main|PUSH-000072|||Connector 'High Volume Connector' started, listening on 0.0.0.0/0.0.0.0:8092.|com.pushtechnology.diffusion.comms.connector.ConnectorImpl
2021-11-12 15:28:58.095|INFO|main|PUSH-000712|||Connector 'High Volume Connector' has scheduled pings every 90000ms.|com.pushtechnology.diffusion.comms.connector.PingScheduler
2021-11-12 15:28:58.390|INFO|main|PUSH-000806|||The file service is serving files at https://192.168.1.156:8092 .|com.pushtechnology.diffusion.comms.connector.ConnectorImpl
2021-11-12 15:28:58.390|INFO|main|PUSH-000806|||The file service is serving files at https://192.168.56.214:8092 .|com.pushtechnology.diffusion.comms.connector.ConnectorImpl
2021-11-12 15:28:58.471|INFO|main|PUSH-000165|||Diffusion server started.|com.pushtechnology.diffusion.DiffusionController
2021-11-12 15:28:58.569|INFO|ForkJoinPool.commonPool-worker-8|PUSH-000834|||The server has restored shared configuration from the cluster.|com.pushtechnology.diffusion.comms.connector.ReadinessConditionsImpl
2021-11-12 15:28:58.629|INFO|ForkJoinPool.commonPool-worker-8|PUSH-000648|||Cluster member HazelcastClusterMember[PTLT72, 192.168.1.156:8090, 8b7e511a-15a9-4d7b-b4a4-b46e42c3e3bb] is connected to this server.|com.pushtechnology.diffusion.clustering.peers.InboundPeerConnection
2021-11-12 15:28:58.746|INFO|InboundThreadPool Thread_5|PUSH-000835|||The server has restored replicated topics from the cluster.|com.pushtechnology.diffusion.comms.connector.ReadinessConditionsImpl
You've now got your second server started. If you have more than two servers to be part of the cluster, repeat the above steps. Thus, you will have a cluster of Diffusion servers.
Where can you see the cluster?
Open Management Console of all servers (in this example, http://localhost:8080/ and http://localhost:8081/)
Go to Network from the menu on the left pane and that shows the cluster.
Server 1:
Server 2:
You can also see the name of the server (as you named (server2) in the Server.xml) on the top right.
Are you wondering how the Diffusion servers are auto detected to form a cluster?
Diffusion uses a datagrid to share session and topic information between Diffusion servers within a cluster, providing high availability for clients connecting to load-balanced servers. Diffusion uses Hazelcast™ as its datagrid. Hazelcast is a third-party product that is included in the Diffusion server installation and runs within the Diffusion server process. The datagrid is responsible for the formation of clusters and the exchange of replicated data. These clusters operate on a peer-to-peer basis and by default there is no hierarchy of servers within the cluster. More details on this can be read in the Diffusion User Manual (Design Guide > Designing your solution > Server clusters for high availability).
The Hazelcast documentation on Setting Up Clusters describes Hazelcast clusters, the discovery mechanisms such as Auto detection, TCP, Multicast used by cluster members and native clients to form a Hazelcast cluster.
In Diffusion, you can find the hazelcast.xml file in the etc folder. For instance, if you want to discover your cluster members by TCP, you
- set the tcp-ip enabled attribute to true
- add member elements within the tcp-ip element
For example:
<network>
<join>
<multicast enabled="false" />
<tcp-ip enabled="true">
<member>127.0.0.1</member>
</tcp-ip>
</join>
</network>