All guides
  1. Home
  2. Technical guides
  3. Integration

IntegrationAdvanced level

TIBCO EMS 10.4 with FTL quorum storage: administration and log analysis

7 min read

All guides

TIBCO EMS 10.4 with FTL quorum storage: administration and log analysis

Administration and log analysis commands for TIBCO EMS 10.4 with FTL quorum storage: three FTL servers, a fault-tolerant pair and a standby server. It is quite different from classic EMS (it is not started with -config and the logs go to the FTL server), so here you have what changes and the day-to-day commands, checked against the official user guide for version 10.4. Everything is also in the README.md on the left.

What you’ll learn

  • How EMS 10.4 works with FTL storage: three servers, a fault-tolerant pair and a standby server.
  • Loading the configuration into the FTL cluster and starting the servers.
  • The tibemsadmin commands to check the server, queues, consumers and stores.
  • Where the logs are with FTL storage, how to adjust the traces and how to analyse them.
  • Checking health with the monitoring endpoints and stopping the cluster safely.

Prerequisites

  • TIBCO EMS 10.4 and TIBCO FTL 7.0.1 or later, on Linux (FTL storage is only available on Linux).
  • Three machines for the three FTL servers, with connectivity between them.
  • An EMS administrator user and terminal access (bash) to the three machines.
  • Familiarity with the basics of EMS: queues, topics, durable subscriptions and fault tolerance.

Step by step

1. Understand the quorum architecture

ComponentWhat it does
Three FTL serversThey form the cluster that replicates the data. With FTL storage and fault tolerance, exactly three are needed; with only one, EMS is not fault tolerant.
One EMS inside each FTLEach EMS server (tibemsd) runs as a service inside its FTL server, which starts it and restarts it if it goes down.
Fault-tolerant pairTwo of the three EMS servers form the pair: the cluster decides which one is active. With -preferred_active you specify which one you prefer to be active.
Standby server (-standby_only)The third one never becomes active: it provides the data replica so that there is a majority.
DataEach persistent message is replicated to at least a majority of the servers before it is confirmed.
ConfigurationLives in the FTL cluster, in JSON format. EMS is not started with -config.

Clients connect only to the URLs of the two servers in the pair, never to the standby server’s.

2. Review the FTL cluster YAML

The three FTL servers share a YAML file that describes the cluster and the services of each one. Its main keys:

  • servers: one block per FTL server (ftl1, ftl2, ftl3), each with its services.
  • tibemsd: the EMS of that server, with exepath (path to tibemsd), -listens (listen URL), -license, -config_wait, -store (data directory) and -monitor_listen (health and metrics). One of the three also has -standby_only.
  • realm: the realm service, with its data directory.
  • ftlserver.properties: logfile, max.log.size and max.logs, that is, where the log goes (including the EMS log) and how it rotates.

The full example is in the Fault Tolerance with FTL Stores chapter of the EMS 10.4 user guide.

3. Load the EMS configuration into the cluster

The EMS configuration (queues, topics, users, stores…) is stored in the FTL cluster as JSON. If you are coming from an installation with .conf files, convert them and upload them:

# Convert the classic configuration (.conf) to JSON
tibemsconf2json -conf tibemsd.conf -json tibemsd.json

# Upload it to the FTL cluster (the tool needs the FTL libraries)
export LD_LIBRARY_PATH=$FTL_HOME/lib
tibemsjson2ftl -url http://ftl1:8080 -json tibemsd.json

# Download the current configuration from the cluster (to review or version it)
tibemsjson2ftl -url http://ftl1:8080 -json tibemsd-current.json -download

In stores.conf, the stores are of type ftl, with mode sync or async (default: async):

[$sys.failsafe]
  type=ftl
  mode=sync

[$sys.nonfailsafe]
  type=ftl
  mode=async

With FTL storage, all the ft_* parameters in tibemsd.conf are ignored except ft_reconnect_timeout, and so is listen: listening is defined in the YAML.

4. Start the FTL servers

Each FTL server starts its EMS; you don’t launch tibemsd by hand:

# On each of the three machines, its FTL server with the cluster YAML
tibftlserver -n ftl1 -c /opt/ems/ftl-ems.yaml
tibftlserver -n ftl2 -c /opt/ems/ftl-ems.yaml
tibftlserver -n ftl3 -c /opt/ems/ftl-ems.yaml

5. Connect with tibemsadmin

Use a password file (-pwdfile) instead of typing the password on the command line, so that it doesn’t end up in the history:

# Connect to the fault-tolerant pair (the two URLs of the pair, not the standby_only one)
tibemsadmin -server "tcp://ems1:7222,tcp://ems2:7222" -user admin -pwdfile /opt/ems/.admin.pwd

# In script mode, to automate checks
tibemsadmin -server "tcp://ems1:7222,tcp://ems2:7222" -user admin -pwdfile /opt/ems/.admin.pwd -script checks.txt

If the active server changes, tibemsadmin disconnects and you have to connect again.

6. Check the server state

show server            # version, state, connections, memory and storage (alias: info)
show state             # server state: active or standby
show config            # configuration in use
show stores            # stores and their usage
show store $sys.failsafe
time on                # adds the time to each output

On a server that is not active, only some commands work: show state, show connections, show server, show config, rotatelog and shutdown.

7. Check queues, topics and consumers

The first thing to look at in a messaging incident: are messages piling up? Do they have consumers? Who is connected?

show queues                              # all queues: pending messages, receivers…
show queues orders.* static              # by pattern, static ones only
show queue orders.inbound                # details of one queue
show topics
show connections full                    # connected clients, host and user
show consumers queue=orders.inbound full
show durables                            # durable subscriptions

8. Make changes to destinations

With autocommit on, each change is applied immediately; otherwise, you have to confirm them with commit. Be careful with purge: it deletes the messages and cannot be undone.

autocommit on
create queue orders.retries maxbytes=100MB
purge queue orders.test                  # empties the queue (irreversible)
delete queue orders.old
set server console_trace=DEFAULT,+CONNECT_ERROR,+AUTH
commit                                   # if autocommit is off

9. Adjust the traces

With FTL storage, the classic EMS log parameters (logfile, log_trace, logfile_max_size, logfile_max_count) and the rotatelog command are not used. EMS output goes to the log of the FTL server that hosts it, and what appears in it is controlled with console_trace:

show config                                          # see the current traces
set server console_trace=DEFAULT                     # replaces the set
set server console_trace=DEFAULT,+CONNECT,+ADMIN     # adds categories
set server console_trace=DEFAULT,-ADMIN              # removes a category

Some useful categories: CONNECT and CONNECT_ERROR (client connections), AUTH and ACL (authentication and permissions), ADMIN (administration operations), DEST (destinations), FLOW and LIMITS (flow control and limits), FTL, SSL and WARNING. Enable the detailed ones only while you investigate: they generate a lot of volume.

10. Analyse the logs

The log path and rotation are set by ftlserver.properties in the YAML (logfile, max.log.size, max.logs). A working method:

# The EMS log is the log of the FTL server that hosts it ("logfile" path in ftlserver.properties)
tail -f /var/log/ems/ftl1.log

# First sweep: errors and warnings from the last few hours
grep -i -E "error|warn|fail" /var/log/ems/ftl1.log | tail -n 100

# Compare the same time window on the three servers (active, pair and standby)
for s in ftl1 ftl2 ftl3; do echo "== $s"; grep -i -E "error|warn" /var/log/ems/$s.log | tail -n 20; done
  • Pin down the time of the incident and check the same time window on all three servers: the problem may be on the active server, on its partner or on the replica.
  • Cross-check what you see with show state and show connections at that moment.
  • If the log does not give enough detail, add the related trace category (step 9), reproduce the problem and remove it again.

11. Check health and metrics

With -monitor_listen, each EMS exposes health endpoints (useful for the load balancer or Kubernetes) and metrics for Prometheus, which in 10.4 include the status of disaster recovery and of the disk:

# Endpoints on the port set in -monitor_listen of each EMS server
curl -s http://ems1:7220/isLive
curl -s http://ems1:7220/isReady
curl -s http://ems1:7220/metrics/server
curl -s http://ems1:7220/metrics/queues

12. Stop and restart safely

Killing the tibemsd process does not work: the FTL server starts it again. To stop it, use shutdown from tibemsadmin (it stops the EMS and its FTL server) or tibftladmin:

# One FTL server (and the EMS it hosts)
tibftladmin --ftlserver http://ftl1:8080 --shutdown

# The whole cluster
tibftladmin --ftlserver http://ftl1:8080 --shutdown_cluster

Common errors

EMS does not start with -config or -ft_active

With FTL storage, -config, -secondary, -ft_active and -ft_activation are not supported: the configuration is in the FTL cluster and the cluster decides the active server.

Fault tolerance changes in tibemsd.conf have no effect

All ft_* parameters are ignored except ft_reconnect_timeout, and so is listen. What used to go there now goes in the cluster YAML.

The EMS log does not appear where logfile says

With FTL storage, logfile, log_trace and rotatelog are not used: look at the FTL server log and adjust the traces with console_trace.

Clients do not reconnect after the active server changes

The client’s server list must include the two URLs of the fault-tolerant pair (and never the URL of the -standby_only server, which does not serve clients).

tibemsadmin only accepts some commands

You are connected to a server that is not the active one. Check with show state and connect to the pair’s list to reach the active server.

The tibemsd process reappears after you kill it

This is expected: its FTL server restarts it. To stop it, use shutdown or tibftladmin (step 12).

tibemsjson2ftl fails to start

It needs the FTL libraries: export LD_LIBRARY_PATH=$FTL_HOME/lib before running it.

FTL storage cannot be configured on Windows

FTL storage only works on Linux. On Windows, EMS has to use file-based storage.

Next steps

  • Send the /metrics metrics to Prometheus and Grafana, with alerts on backed-up queues and servers that are not ready.
  • Configure a disaster recovery site (setup_dr_site and activate_dr_site).
  • Secure connections with TLS and OAuth 2.0, both supported in EMS 10.4.
  • Version the cluster’s JSON configuration and apply it with tibemsjson2ftl from a pipeline.

Share

More guides on

See all guides

Other technical guides

See all guides

IntegrationIntermediate

TIBCO ActiveSpaces 5.2: basic administration commands

The day-to-day commands in TIBCO ActiveSpaces 5.2: realm service on FTL servers, tibdg, grid definition, tables, starting processes, status, scaling with copysets, checkpoints and logs.

Read the guide 8 min read

APIsIntermediate

Basic structure of an API in IBM API Connect 10

How an API is structured in IBM API Connect 10: OpenAPI contract with x-ibm-configuration, Client ID and Client Secret, product with plans and limits, and validation and publishing with the apic CLI.

Read the guide 7 min read

DevOpsBasic

Basic CI/CD structure with GitHub Actions

A complete pipeline with GitHub Actions: tests on every push and pull request, a Docker image in GitHub Container Registry and deployment to a protected environment, with a workflow ready to copy.

Read the guide 6 min read

DevelopmentBasic

Java Spring Boot development best practices

The practices that make the biggest difference in a Spring Boot service: structure, constructor injection, DTOs, validation, errors, configuration, transactions, logs, comments, tests and automated quality.

Read the guide 7 min read

Contact

Build your next competitive advantage today

Let’s talk about how we can help you. We will reply with a clear technical proposal, with no obligation.

Request information

Fields marked with are required.