Administration and log analysis commands for TIBCO EMS 10.4 with FTL quorum storage: three FTL servers, a fault-tolerant pair and a standby server. It is quite different from classic EMS (it is not started with -config and the logs go to the FTL server), so here you have what changes and the day-to-day commands, checked against the official user guide for version 10.4. Everything is also in the README.md on the left.
What you’ll learn
- How EMS 10.4 works with FTL storage: three servers, a fault-tolerant pair and a standby server.
- Loading the configuration into the FTL cluster and starting the servers.
- The
tibemsadmincommands to check the server, queues, consumers and stores. - Where the logs are with FTL storage, how to adjust the traces and how to analyse them.
- Checking health with the monitoring endpoints and stopping the cluster safely.
Prerequisites
- TIBCO EMS 10.4 and TIBCO FTL 7.0.1 or later, on Linux (FTL storage is only available on Linux).
- Three machines for the three FTL servers, with connectivity between them.
- An EMS administrator user and terminal access (bash) to the three machines.
- Familiarity with the basics of EMS: queues, topics, durable subscriptions and fault tolerance.
Step by step
1. Understand the quorum architecture
| Component | What it does |
|---|---|
| Three FTL servers | They form the cluster that replicates the data. With FTL storage and fault tolerance, exactly three are needed; with only one, EMS is not fault tolerant. |
| One EMS inside each FTL | Each EMS server (tibemsd) runs as a service inside its FTL server, which starts it and restarts it if it goes down. |
| Fault-tolerant pair | Two of the three EMS servers form the pair: the cluster decides which one is active. With -preferred_active you specify which one you prefer to be active. |
Standby server (-standby_only) | The third one never becomes active: it provides the data replica so that there is a majority. |
| Data | Each persistent message is replicated to at least a majority of the servers before it is confirmed. |
| Configuration | Lives in the FTL cluster, in JSON format. EMS is not started with -config. |
Clients connect only to the URLs of the two servers in the pair, never to the standby server’s.
2. Review the FTL cluster YAML
The three FTL servers share a YAML file that describes the cluster and the services of each one. Its main keys:
servers: one block per FTL server (ftl1, ftl2, ftl3), each with its services.tibemsd: the EMS of that server, withexepath(path totibemsd),-listens(listen URL),-license,-config_wait,-store(data directory) and-monitor_listen(health and metrics). One of the three also has-standby_only.realm: the realm service, with itsdatadirectory.ftlserver.properties:logfile,max.log.sizeandmax.logs, that is, where the log goes (including the EMS log) and how it rotates.
The full example is in the Fault Tolerance with FTL Stores chapter of the EMS 10.4 user guide.
3. Load the EMS configuration into the cluster
The EMS configuration (queues, topics, users, stores…) is stored in the FTL cluster as JSON. If you are coming from an installation with .conf files, convert them and upload them:
# Convert the classic configuration (.conf) to JSON
tibemsconf2json -conf tibemsd.conf -json tibemsd.json
# Upload it to the FTL cluster (the tool needs the FTL libraries)
export LD_LIBRARY_PATH=$FTL_HOME/lib
tibemsjson2ftl -url http://ftl1:8080 -json tibemsd.json
# Download the current configuration from the cluster (to review or version it)
tibemsjson2ftl -url http://ftl1:8080 -json tibemsd-current.json -download
In stores.conf, the stores are of type ftl, with mode sync or async (default: async):
[$sys.failsafe]
type=ftl
mode=sync
[$sys.nonfailsafe]
type=ftl
mode=async
With FTL storage, all the ft_* parameters in tibemsd.conf are ignored except ft_reconnect_timeout, and so is listen: listening is defined in the YAML.
4. Start the FTL servers
Each FTL server starts its EMS; you don’t launch tibemsd by hand:
# On each of the three machines, its FTL server with the cluster YAML
tibftlserver -n ftl1 -c /opt/ems/ftl-ems.yaml
tibftlserver -n ftl2 -c /opt/ems/ftl-ems.yaml
tibftlserver -n ftl3 -c /opt/ems/ftl-ems.yaml
5. Connect with tibemsadmin
Use a password file (-pwdfile) instead of typing the password on the command line, so that it doesn’t end up in the history:
# Connect to the fault-tolerant pair (the two URLs of the pair, not the standby_only one)
tibemsadmin -server "tcp://ems1:7222,tcp://ems2:7222" -user admin -pwdfile /opt/ems/.admin.pwd
# In script mode, to automate checks
tibemsadmin -server "tcp://ems1:7222,tcp://ems2:7222" -user admin -pwdfile /opt/ems/.admin.pwd -script checks.txt
If the active server changes, tibemsadmin disconnects and you have to connect again.
6. Check the server state
show server # version, state, connections, memory and storage (alias: info)
show state # server state: active or standby
show config # configuration in use
show stores # stores and their usage
show store $sys.failsafe
time on # adds the time to each output
On a server that is not active, only some commands work: show state, show connections, show server, show config, rotatelog and shutdown.
7. Check queues, topics and consumers
The first thing to look at in a messaging incident: are messages piling up? Do they have consumers? Who is connected?
show queues # all queues: pending messages, receivers…
show queues orders.* static # by pattern, static ones only
show queue orders.inbound # details of one queue
show topics
show connections full # connected clients, host and user
show consumers queue=orders.inbound full
show durables # durable subscriptions
8. Make changes to destinations
With autocommit on, each change is applied immediately; otherwise, you have to confirm them with commit. Be careful with purge: it deletes the messages and cannot be undone.
autocommit on
create queue orders.retries maxbytes=100MB
purge queue orders.test # empties the queue (irreversible)
delete queue orders.old
set server console_trace=DEFAULT,+CONNECT_ERROR,+AUTH
commit # if autocommit is off
9. Adjust the traces
With FTL storage, the classic EMS log parameters (logfile, log_trace, logfile_max_size, logfile_max_count) and the rotatelog command are not used. EMS output goes to the log of the FTL server that hosts it, and what appears in it is controlled with console_trace:
show config # see the current traces
set server console_trace=DEFAULT # replaces the set
set server console_trace=DEFAULT,+CONNECT,+ADMIN # adds categories
set server console_trace=DEFAULT,-ADMIN # removes a category
Some useful categories: CONNECT and CONNECT_ERROR (client connections), AUTH and ACL (authentication and permissions), ADMIN (administration operations), DEST (destinations), FLOW and LIMITS (flow control and limits), FTL, SSL and WARNING. Enable the detailed ones only while you investigate: they generate a lot of volume.
10. Analyse the logs
The log path and rotation are set by ftlserver.properties in the YAML (logfile, max.log.size, max.logs). A working method:
# The EMS log is the log of the FTL server that hosts it ("logfile" path in ftlserver.properties)
tail -f /var/log/ems/ftl1.log
# First sweep: errors and warnings from the last few hours
grep -i -E "error|warn|fail" /var/log/ems/ftl1.log | tail -n 100
# Compare the same time window on the three servers (active, pair and standby)
for s in ftl1 ftl2 ftl3; do echo "== $s"; grep -i -E "error|warn" /var/log/ems/$s.log | tail -n 20; done
- Pin down the time of the incident and check the same time window on all three servers: the problem may be on the active server, on its partner or on the replica.
- Cross-check what you see with
show stateandshow connectionsat that moment. - If the log does not give enough detail, add the related trace category (step 9), reproduce the problem and remove it again.
11. Check health and metrics
With -monitor_listen, each EMS exposes health endpoints (useful for the load balancer or Kubernetes) and metrics for Prometheus, which in 10.4 include the status of disaster recovery and of the disk:
# Endpoints on the port set in -monitor_listen of each EMS server
curl -s http://ems1:7220/isLive
curl -s http://ems1:7220/isReady
curl -s http://ems1:7220/metrics/server
curl -s http://ems1:7220/metrics/queues
12. Stop and restart safely
Killing the tibemsd process does not work: the FTL server starts it again. To stop it, use shutdown from tibemsadmin (it stops the EMS and its FTL server) or tibftladmin:
# One FTL server (and the EMS it hosts)
tibftladmin --ftlserver http://ftl1:8080 --shutdown
# The whole cluster
tibftladmin --ftlserver http://ftl1:8080 --shutdown_cluster
Common errors
EMS does not start with -config or -ft_active
With FTL storage, -config, -secondary, -ft_active and -ft_activation are not supported: the configuration is in the FTL cluster and the cluster decides the active server.
Fault tolerance changes in tibemsd.conf have no effect
All ft_* parameters are ignored except ft_reconnect_timeout, and so is listen. What used to go there now goes in the cluster YAML.
The EMS log does not appear where logfile says
With FTL storage, logfile, log_trace and rotatelog are not used: look at the FTL server log and adjust the traces with console_trace.
Clients do not reconnect after the active server changes
The client’s server list must include the two URLs of the fault-tolerant pair (and never the URL of the -standby_only server, which does not serve clients).
tibemsadmin only accepts some commands
You are connected to a server that is not the active one. Check with show state and connect to the pair’s list to reach the active server.
The tibemsd process reappears after you kill it
This is expected: its FTL server restarts it. To stop it, use shutdown or tibftladmin (step 12).
tibemsjson2ftl fails to start
It needs the FTL libraries: export LD_LIBRARY_PATH=$FTL_HOME/lib before running it.
FTL storage cannot be configured on Windows
FTL storage only works on Linux. On Windows, EMS has to use file-based storage.
Next steps
- Send the
/metricsmetrics to Prometheus and Grafana, with alerts on backed-up queues and servers that are not ready. - Configure a disaster recovery site (
setup_dr_siteandactivate_dr_site). - Secure connections with TLS and OAuth 2.0, both supported in EMS 10.4.
- Version the cluster’s JSON configuration and apply it with
tibemsjson2ftlfrom a pipeline.