{"id":5178,"date":"2026-11-18T21:22:31","date_gmt":"2026-11-18T21:22:31","guid":{"rendered":"https:\/\/bitflow-solutions.com\/guias-tecnicas\/tibco-ems-10-4-ftl-quorum-admin-and-logs\/"},"modified":"2026-11-18T21:22:31","modified_gmt":"2026-11-18T21:22:31","slug":"tibco-ems-10-4-ftl-quorum-admin-and-logs","status":"publish","type":"bf_guia","link":"https:\/\/bitflow-solutions.com\/en\/guias-tecnicas\/tibco-ems-10-4-ftl-quorum-admin-and-logs\/","title":{"rendered":"TIBCO EMS 10.4 with FTL quorum storage: administration and log analysis"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><strong>Administration<\/strong> and <strong>log analysis<\/strong> commands for <strong>TIBCO EMS 10.4<\/strong> with <strong>FTL quorum storage<\/strong>: three FTL servers, a fault-tolerant pair and a standby server. It is quite different from classic EMS (it is not started with <code>-config<\/code> and the logs go to the FTL server), so here you have what changes and the day-to-day commands, checked against the official user guide for version 10.4. Everything is also in the <strong>README.md<\/strong> on the left.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What you\u2019ll learn<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>How EMS 10.4 works with FTL storage: three servers, a fault-tolerant pair and a standby server.<\/li>\n\n\n<li>Loading the configuration into the FTL cluster and starting the servers.<\/li>\n\n\n<li>The <code>tibemsadmin<\/code> commands to check the server, queues, consumers and stores.<\/li>\n\n\n<li>Where the logs are with FTL storage, how to adjust the traces and how to analyse them.<\/li>\n\n\n<li>Checking health with the monitoring endpoints and stopping the cluster safely.<\/li>\n\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Prerequisites<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>TIBCO EMS 10.4<\/strong> and <strong>TIBCO FTL 7.0.1 or later<\/strong>, on <strong>Linux<\/strong> (FTL storage is only available on Linux).<\/li>\n\n\n<li>Three machines for the three FTL servers, with connectivity between them.<\/li>\n\n\n<li>An EMS administrator user and terminal access (bash) to the three machines.<\/li>\n\n\n<li>Familiarity with the basics of EMS: queues, topics, durable subscriptions and fault tolerance.<\/li>\n\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Step by step<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">1. Understand the quorum architecture<\/h3>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Component<\/th><th>What it does<\/th><\/tr><\/thead><tbody><tr><td><strong>Three FTL servers<\/strong><\/td><td>They form the cluster that replicates the data. With FTL storage and fault tolerance, exactly three are needed; with only one, EMS is not fault tolerant.<\/td><\/tr><tr><td><strong>One EMS inside each FTL<\/strong><\/td><td>Each EMS server (<code>tibemsd<\/code>) runs as a service inside its FTL server, which starts it and restarts it if it goes down.<\/td><\/tr><tr><td><strong>Fault-tolerant pair<\/strong><\/td><td>Two of the three EMS servers form the pair: the cluster decides which one is active. With <code>-preferred_active<\/code> you specify which one you prefer to be active.<\/td><\/tr><tr><td><strong>Standby server<\/strong> (<code>-standby_only<\/code>)<\/td><td>The third one never becomes active: it provides the data replica so that there is a majority.<\/td><\/tr><tr><td><strong>Data<\/strong><\/td><td>Each persistent message is replicated to at least a majority of the servers before it is confirmed.<\/td><\/tr><tr><td><strong>Configuration<\/strong><\/td><td>Lives in the FTL cluster, in JSON format. EMS is not started with <code>-config<\/code>.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Clients connect only to the URLs of the two servers in the pair, never to the standby server\u2019s.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">2. Review the FTL cluster YAML<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The three FTL servers share a YAML file that describes the cluster and the services of each one. Its main keys:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>servers<\/code>: one block per FTL server (ftl1, ftl2, ftl3), each with its services.<\/li>\n\n\n<li><code>tibemsd<\/code>: the EMS of that server, with <code>exepath<\/code> (path to <code>tibemsd<\/code>), <code>-listens<\/code> (listen URL), <code>-license<\/code>, <code>-config_wait<\/code>, <code>-store<\/code> (data directory) and <code>-monitor_listen<\/code> (health and metrics). One of the three also has <code>-standby_only<\/code>.<\/li>\n\n\n<li><code>realm<\/code>: the realm service, with its <code>data<\/code> directory.<\/li>\n\n\n<li><code>ftlserver.properties<\/code>: <code>logfile<\/code>, <code>max.log.size<\/code> and <code>max.logs<\/code>, that is, where the log goes (including the EMS log) and how it rotates.<\/li>\n\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The full example is in the <em>Fault Tolerance with FTL Stores<\/em> chapter of the EMS 10.4 user guide.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">3. Load the EMS configuration into the cluster<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The EMS configuration (queues, topics, users, stores\u2026) is stored in the FTL cluster as JSON. If you are coming from an installation with <code>.conf<\/code> files, convert them and upload them:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Convert the classic configuration (.conf) to JSON\ntibemsconf2json -conf tibemsd.conf -json tibemsd.json\n\n# Upload it to the FTL cluster (the tool needs the FTL libraries)\nexport LD_LIBRARY_PATH=$FTL_HOME\/lib\ntibemsjson2ftl -url http:\/\/ftl1:8080 -json tibemsd.json\n\n# Download the current configuration from the cluster (to review or version it)\ntibemsjson2ftl -url http:\/\/ftl1:8080 -json tibemsd-current.json -download<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">In <code>stores.conf<\/code>, the stores are of type <code>ftl<\/code>, with mode <code>sync<\/code> or <code>async<\/code> (default: <code>async<\/code>):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>[$sys.failsafe]\n  type=ftl\n  mode=sync\n\n[$sys.nonfailsafe]\n  type=ftl\n  mode=async<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">With FTL storage, all the <code>ft_*<\/code> parameters in <code>tibemsd.conf<\/code> are ignored except <code>ft_reconnect_timeout<\/code>, and so is <code>listen<\/code>: listening is defined in the YAML.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">4. Start the FTL servers<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Each FTL server starts its EMS; you don\u2019t launch <code>tibemsd<\/code> by hand:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># On each of the three machines, its FTL server with the cluster YAML\ntibftlserver -n ftl1 -c \/opt\/ems\/ftl-ems.yaml\ntibftlserver -n ftl2 -c \/opt\/ems\/ftl-ems.yaml\ntibftlserver -n ftl3 -c \/opt\/ems\/ftl-ems.yaml<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">5. Connect with tibemsadmin<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Use a password file (<code>-pwdfile<\/code>) instead of typing the password on the command line, so that it doesn\u2019t end up in the history:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Connect to the fault-tolerant pair (the two URLs of the pair, not the standby_only one)\ntibemsadmin -server \"tcp:\/\/ems1:7222,tcp:\/\/ems2:7222\" -user admin -pwdfile \/opt\/ems\/.admin.pwd\n\n# In script mode, to automate checks\ntibemsadmin -server \"tcp:\/\/ems1:7222,tcp:\/\/ems2:7222\" -user admin -pwdfile \/opt\/ems\/.admin.pwd -script checks.txt<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If the active server changes, <code>tibemsadmin<\/code> disconnects and you have to connect again.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">6. Check the server state<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>show server            # version, state, connections, memory and storage (alias: info)\nshow state             # server state: active or standby\nshow config            # configuration in use\nshow stores            # stores and their usage\nshow store $sys.failsafe\ntime on                # adds the time to each output<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">On a server that is not active, only some commands work: <code>show state<\/code>, <code>show connections<\/code>, <code>show server<\/code>, <code>show config<\/code>, <code>rotatelog<\/code> and <code>shutdown<\/code>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">7. Check queues, topics and consumers<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The first thing to look at in a messaging incident: are messages piling up? Do they have consumers? Who is connected?<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>show queues                              # all queues: pending messages, receivers\u2026\nshow queues orders.* static              # by pattern, static ones only\nshow queue orders.inbound                # details of one queue\nshow topics\nshow connections full                    # connected clients, host and user\nshow consumers queue=orders.inbound full\nshow durables                            # durable subscriptions<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">8. Make changes to destinations<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">With <code>autocommit on<\/code>, each change is applied immediately; otherwise, you have to confirm them with <code>commit<\/code>. Be careful with <code>purge<\/code>: it deletes the messages and cannot be undone.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>autocommit on\ncreate queue orders.retries maxbytes=100MB\npurge queue orders.test                  # empties the queue (irreversible)\ndelete queue orders.old\nset server console_trace=DEFAULT,+CONNECT_ERROR,+AUTH\ncommit                                   # if autocommit is off<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">9. Adjust the traces<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">With FTL storage, the classic EMS log parameters (<code>logfile<\/code>, <code>log_trace<\/code>, <code>logfile_max_size<\/code>, <code>logfile_max_count<\/code>) and the <code>rotatelog<\/code> command <strong>are not used<\/strong>. EMS output goes to the log of the FTL server that hosts it, and what appears in it is controlled with <code>console_trace<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>show config                                          # see the current traces\nset server console_trace=DEFAULT                     # replaces the set\nset server console_trace=DEFAULT,+CONNECT,+ADMIN     # adds categories\nset server console_trace=DEFAULT,-ADMIN              # removes a category<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Some useful categories: <code>CONNECT<\/code> and <code>CONNECT_ERROR<\/code> (client connections), <code>AUTH<\/code> and <code>ACL<\/code> (authentication and permissions), <code>ADMIN<\/code> (administration operations), <code>DEST<\/code> (destinations), <code>FLOW<\/code> and <code>LIMITS<\/code> (flow control and limits), <code>FTL<\/code>, <code>SSL<\/code> and <code>WARNING<\/code>. Enable the detailed ones only while you investigate: they generate a lot of volume.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">10. Analyse the logs<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The log path and rotation are set by <code>ftlserver.properties<\/code> in the YAML (<code>logfile<\/code>, <code>max.log.size<\/code>, <code>max.logs<\/code>). A working method:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># The EMS log is the log of the FTL server that hosts it (\"logfile\" path in ftlserver.properties)\ntail -f \/var\/log\/ems\/ftl1.log\n\n# First sweep: errors and warnings from the last few hours\ngrep -i -E \"error|warn|fail\" \/var\/log\/ems\/ftl1.log | tail -n 100\n\n# Compare the same time window on the three servers (active, pair and standby)\nfor s in ftl1 ftl2 ftl3; do echo \"== $s\"; grep -i -E \"error|warn\" \/var\/log\/ems\/$s.log | tail -n 20; done<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pin down the time of the incident and check the same time window on all <strong>three<\/strong> servers: the problem may be on the active server, on its partner or on the replica.<\/li>\n\n\n<li>Cross-check what you see with <code>show state<\/code> and <code>show connections<\/code> at that moment.<\/li>\n\n\n<li>If the log does not give enough detail, add the related trace category (step 9), reproduce the problem and remove it again.<\/li>\n\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">11. Check health and metrics<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">With <code>-monitor_listen<\/code>, each EMS exposes health endpoints (useful for the load balancer or Kubernetes) and metrics for Prometheus, which in 10.4 include the status of disaster recovery and of the disk:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Endpoints on the port set in -monitor_listen of each EMS server\ncurl -s http:\/\/ems1:7220\/isLive\ncurl -s http:\/\/ems1:7220\/isReady\ncurl -s http:\/\/ems1:7220\/metrics\/server\ncurl -s http:\/\/ems1:7220\/metrics\/queues<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">12. Stop and restart safely<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Killing the <code>tibemsd<\/code> process does not work: the FTL server starts it again. To stop it, use <code>shutdown<\/code> from <code>tibemsadmin<\/code> (it stops the EMS and its FTL server) or <code>tibftladmin<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># One FTL server (and the EMS it hosts)\ntibftladmin --ftlserver http:\/\/ftl1:8080 --shutdown\n\n# The whole cluster\ntibftladmin --ftlserver http:\/\/ftl1:8080 --shutdown_cluster<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Common errors<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">EMS does not start with -config or -ft_active<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">With FTL storage, <code>-config<\/code>, <code>-secondary<\/code>, <code>-ft_active<\/code> and <code>-ft_activation<\/code> are not supported: the configuration is in the FTL cluster and the cluster decides the active server.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Fault tolerance changes in tibemsd.conf have no effect<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">All <code>ft_*<\/code> parameters are ignored except <code>ft_reconnect_timeout<\/code>, and so is <code>listen<\/code>. What used to go there now goes in the cluster YAML.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The EMS log does not appear where logfile says<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">With FTL storage, <code>logfile<\/code>, <code>log_trace<\/code> and <code>rotatelog<\/code> are not used: look at the FTL server log and adjust the traces with <code>console_trace<\/code>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Clients do not reconnect after the active server changes<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The client\u2019s server list must include the two URLs of the fault-tolerant pair (and never the URL of the <code>-standby_only<\/code> server, which does not serve clients).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">tibemsadmin only accepts some commands<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">You are connected to a server that is not the active one. Check with <code>show state<\/code> and connect to the pair\u2019s list to reach the active server.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The tibemsd process reappears after you kill it<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">This is expected: its FTL server restarts it. To stop it, use <code>shutdown<\/code> or <code>tibftladmin<\/code> (step 12).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">tibemsjson2ftl fails to start<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">It needs the FTL libraries: export <code>LD_LIBRARY_PATH=$FTL_HOME\/lib<\/code> before running it.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">FTL storage cannot be configured on Windows<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">FTL storage only works on Linux. On Windows, EMS has to use file-based storage.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Next steps<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Send the <code>\/metrics<\/code> metrics to Prometheus and Grafana, with alerts on backed-up queues and servers that are not ready.<\/li>\n\n\n<li>Configure a disaster recovery site (<code>setup_dr_site<\/code> and <code>activate_dr_site<\/code>).<\/li>\n\n\n<li>Secure connections with TLS and OAuth 2.0, both supported in EMS 10.4.<\/li>\n\n\n<li>Version the cluster\u2019s JSON configuration and apply it with <code>tibemsjson2ftl<\/code> from a pipeline.<\/li>\n\n<\/ul>\n\n\n","protected":false},"excerpt":{"rendered":"<p>How EMS 10.4 works with FTL quorum storage and the day-to-day commands: configuration in the cluster, tibemsadmin, queues and consumers, traces with console_trace, log analysis, health and safe shutdown.<\/p>\n","protected":false},"featured_media":0,"template":"","bf_tecnologia":[102,106],"bf_nivel":[110],"class_list":["post-5178","bf_guia","type-bf_guia","status-publish","hentry","bf_tecnologia-integration","bf_tecnologia-monitoring","bf_nivel-advanced"],"acf":[],"_links":{"self":[{"href":"https:\/\/bitflow-solutions.com\/en\/wp-json\/wp\/v2\/bf_guia\/5178","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bitflow-solutions.com\/en\/wp-json\/wp\/v2\/bf_guia"}],"about":[{"href":"https:\/\/bitflow-solutions.com\/en\/wp-json\/wp\/v2\/types\/bf_guia"}],"version-history":[{"count":0,"href":"https:\/\/bitflow-solutions.com\/en\/wp-json\/wp\/v2\/bf_guia\/5178\/revisions"}],"wp:attachment":[{"href":"https:\/\/bitflow-solutions.com\/en\/wp-json\/wp\/v2\/media?parent=5178"}],"wp:term":[{"taxonomy":"bf_tecnologia","embeddable":true,"href":"https:\/\/bitflow-solutions.com\/en\/wp-json\/wp\/v2\/bf_tecnologia?post=5178"},{"taxonomy":"bf_nivel","embeddable":true,"href":"https:\/\/bitflow-solutions.com\/en\/wp-json\/wp\/v2\/bf_nivel?post=5178"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}