# TIBCO EMS 10.4 with FTL quorum storage: administration and log analysis

A Bitflow Solutions guide. Web version: https://bitflow-solutions.com/en/guias-tecnicas/tibco-ems-10-4-ftl-quorum-admin-and-logs/

## What you’ll learn

- How EMS 10.4 works with FTL storage: three servers, a fault-tolerant pair and a standby server.
- Loading the configuration into the FTL cluster and starting the servers.
- The `tibemsadmin` commands to check the server, queues, consumers and stores.
- Where the logs are with FTL storage, how to adjust the traces and how to analyse them.
- Checking health with the monitoring endpoints and stopping the cluster safely.

## Prerequisites

- **TIBCO EMS 10.4** and **TIBCO FTL 7.0.1 or later**, on **Linux** (FTL storage is only available on Linux).
- Three machines for the three FTL servers, with connectivity between them.
- An EMS administrator user and terminal access (bash) to the three machines.
- Familiarity with the basics of EMS: queues, topics, durable subscriptions and fault tolerance.

## Step by step

### Architecture

- 3 FTL servers (exactly) with one EMS inside each one
- Fault-tolerant pair (the cluster chooses the active one; -preferred_active)
- One -standby_only server that is never active
- Replication to the majority; configuration in JSON inside the FTL cluster (no -config)
- Clients: only the URLs of the pair

### Cluster YAML

`servers`: one block per FTL server (ftl1, ftl2, ftl3), each with its services.


`tibemsd`: the EMS of that server, with `exepath` (path to `tibemsd`), `-listens` (listen URL), `-license`, `-config_wait`, `-store` (data directory) and `-monitor_listen` (health and metrics). One of the three also has `-standby_only`.


`realm`: the realm service, with its `data` directory.


`ftlserver.properties`: `logfile`, `max.log.size` and `max.logs`, that is, where the log goes (including the EMS log) and how it rotates.

### Configuration in the cluster

```bash
# Convert the classic configuration (.conf) to JSON
tibemsconf2json -conf tibemsd.conf -json tibemsd.json

# Upload it to the FTL cluster (the tool needs the FTL libraries)
export LD_LIBRARY_PATH=$FTL_HOME/lib
tibemsjson2ftl -url http://ftl1:8080 -json tibemsd.json

# Download the current configuration from the cluster (to review or version it)
tibemsjson2ftl -url http://ftl1:8080 -json tibemsd-current.json -download
```

stores.conf:

```text
[$sys.failsafe]
  type=ftl
  mode=sync

[$sys.nonfailsafe]
  type=ftl
  mode=async
```

### Start the FTL servers

```bash
# On each of the three machines, its FTL server with the cluster YAML
tibftlserver -n ftl1 -c /opt/ems/ftl-ems.yaml
tibftlserver -n ftl2 -c /opt/ems/ftl-ems.yaml
tibftlserver -n ftl3 -c /opt/ems/ftl-ems.yaml
```

### Connect with tibemsadmin

```bash
# Connect to the fault-tolerant pair (the two URLs of the pair, not the standby_only one)
tibemsadmin -server "tcp://ems1:7222,tcp://ems2:7222" -user admin -pwdfile /opt/ems/.admin.pwd

# In script mode, to automate checks
tibemsadmin -server "tcp://ems1:7222,tcp://ems2:7222" -user admin -pwdfile /opt/ems/.admin.pwd -script checks.txt
```

### Server state (tibemsadmin)

```text
show server            # version, state, connections, memory and storage (alias: info)
show state             # server state: active or standby
show config            # configuration in use
show stores            # stores and their usage
show store $sys.failsafe
time on                # adds the time to each output
```

### Queues, topics and consumers (tibemsadmin)

```text
show queues                              # all queues: pending messages, receivers…
show queues orders.* static              # by pattern, static ones only
show queue orders.inbound                # details of one queue
show topics
show connections full                    # connected clients, host and user
show consumers queue=orders.inbound full
show durables                            # durable subscriptions
```

### Changes to destinations (tibemsadmin)

```text
autocommit on
create queue orders.retries maxbytes=100MB
purge queue orders.test                  # empties the queue (irreversible)
delete queue orders.old
set server console_trace=DEFAULT,+CONNECT_ERROR,+AUTH
commit                                   # if autocommit is off
```

### Traces (console_trace) (tibemsadmin)

```text
show config                                          # see the current traces
set server console_trace=DEFAULT                     # replaces the set
set server console_trace=DEFAULT,+CONNECT,+ADMIN     # adds categories
set server console_trace=DEFAULT,-ADMIN              # removes a category
```

### Analyse the logs

```bash
# The EMS log is the log of the FTL server that hosts it ("logfile" path in ftlserver.properties)
tail -f /var/log/ems/ftl1.log

# First sweep: errors and warnings from the last few hours
grep -i -E "error|warn|fail" /var/log/ems/ftl1.log | tail -n 100

# Compare the same time window on the three servers (active, pair and standby)
for s in ftl1 ftl2 ftl3; do echo "== $s"; grep -i -E "error|warn" /var/log/ems/$s.log | tail -n 20; done
```

### Health and metrics

```bash
# Endpoints on the port set in -monitor_listen of each EMS server
curl -s http://ems1:7220/isLive
curl -s http://ems1:7220/isReady
curl -s http://ems1:7220/metrics/server
curl -s http://ems1:7220/metrics/queues
```

### Stop safely

```bash
# One FTL server (and the EMS it hosts)
tibftladmin --ftlserver http://ftl1:8080 --shutdown

# The whole cluster
tibftladmin --ftlserver http://ftl1:8080 --shutdown_cluster
```

## Common errors

- **EMS does not start with -config or -ft_active**: With FTL storage, `-config`, `-secondary`, `-ft_active` and `-ft_activation` are not supported: the configuration is in the FTL cluster and the cluster decides the active server.
- **Fault tolerance changes in tibemsd.conf have no effect**: All `ft_*` parameters are ignored except `ft_reconnect_timeout`, and so is `listen`. What used to go there now goes in the cluster YAML.
- **The EMS log does not appear where logfile says**: With FTL storage, `logfile`, `log_trace` and `rotatelog` are not used: look at the FTL server log and adjust the traces with `console_trace`.
- **Clients do not reconnect after the active server changes**: The client’s server list must include the two URLs of the fault-tolerant pair (and never the URL of the `-standby_only` server, which does not serve clients).
- **tibemsadmin only accepts some commands**: You are connected to a server that is not the active one. Check with `show state` and connect to the pair’s list to reach the active server.
- **The tibemsd process reappears after you kill it**: This is expected: its FTL server restarts it. To stop it, use `shutdown` or `tibftladmin` (step 12).
- **tibemsjson2ftl fails to start**: It needs the FTL libraries: export `LD_LIBRARY_PATH=$FTL_HOME/lib` before running it.
- **FTL storage cannot be configured on Windows**: FTL storage only works on Linux. On Windows, EMS has to use file-based storage.

## Next steps

- Send the `/metrics` metrics to Prometheus and Grafana, with alerts on backed-up queues and servers that are not ready.
- Configure a disaster recovery site (`setup_dr_site` and `activate_dr_site`).
- Secure connections with TLS and OAuth 2.0, both supported in EMS 10.4.
- Version the cluster’s JSON configuration and apply it with `tibemsjson2ftl` from a pipeline.

Source: TIBCO Enterprise Message Service 10.4.0 User's Guide (docs.tibco.com).
