A practical cheat sheet with the administration commands for TIBCO ActiveSpaces 5.2: the realm service on FTL servers, defining the grid with tibdg, starting the processes, status, scaling, checkpoints and logs. All the commands have been checked against the official administration manual for version 5.2, and you also have them in the README.md on the left.
What you’ll learn
- The components of ActiveSpaces 5.2 and what changed from 4.x: the realm service is now FTL servers.
- Connecting
tibdgto the realm and defining a complete grid with a script. - Creating tables, columns and indexes, and starting the processes in the right order.
- Checking the grid status, scaling it with copysets and distributing clients across proxies.
- Taking checkpoints, restoring from them and knowing where the logs and metrics are.
Prerequisites
- TIBCO ActiveSpaces 5.2 installed, with its licence or activation service.
- TIBCO FTL 7.2 or later for the realm service.
- Terminal access (bash) to the grid machines and to the FTL server URL.
- Familiarity with the basic concepts of a data grid: tables, keys, replicas and quorum.
Step by step
1. Understand the components
| Component | What it does |
|---|---|
| Realm service (FTL servers) | Stores the grid configuration and coordinates everything else. In 5.x it is a cluster of TIBCO FTL servers (tibftlserver): 1 in development, 3, 5 or 7 in production. There is no longer a separate realm server. |
State keepers (tibdgkeeper) | Keep track of the grid state (which node is primary in each copyset). In production, 3 on different machines: the grid works as long as there is quorum (2 out of 3). |
Nodes (tibdgnode) | Store the data in memory and on disk. They are grouped into copysets: each copyset stores a part of the data, replicated across its nodes (2 per copyset in production). |
Proxies (tibdgproxy) | Receive connections from applications and route them to the node that holds each piece of data. |
Administration service (tibdgadmind) | Required for SQL definition operations (for example, DROP TABLE). Running it as a service inside the FTL server is recommended. |
For production, the minimum recommended configuration is: 3 state keepers on different machines and 2 nodes per copyset.
2. Start the realm service
Start the FTL servers before any other component:
# A single FTL server (development)
tibftlserver -n ftl1@host1:8080 --license https://activation.company.local:7070
# Cluster of FTL servers (production: 3, 5 or 7), each one with the same YAML
tibftlserver -c ftl-cluster.yaml -n ftl1 --license https://activation.company.local:7070
3. Connect tibdg to the realm
tibdg is the administration tool. Its general syntax is tibdg [-r realm_url] [-g grid] [-s script] [-m message] command arguments. If you don’t specify -g, it works on the _default grid.
# The realm service URL is given with -r (default http://localhost:8080)
tibdg -r http://ftl1:8080 status
# With a cluster of FTL servers, the URLs separated by |
tibdg -r "http://ftl1:8080|http://ftl2:8080|http://ftl3:8080" status
# Or just once, in an environment variable
export TIBDG_FTL="http://ftl1:8080|http://ftl2:8080|http://ftl3:8080"
tibdg -g mygrid status
# Help for any command
tibdg help
tibdg help grid create
tibdg help table
With security enabled, the credentials go in TIBDG_FTL_USER, TIBDG_FTL_PASSWD and TIBDG_FTL_TRUSTFILE.
4. Define the grid with a script
Save the definition in a file (mygrid.tibdg) so you can repeat it in every environment: the grid, its copyset, the nodes, the state keepers and the proxies, with automatic checkpoints every 10 minutes and the last 5 kept.
grid create statekeeper_count=3 copyset_size=2 checkpoint_interval=600 checkpoint_retention_limit=5 mygrid
copyset create cs1
node create --copyset cs1 --dir ./n1_data n1
node create --copyset cs1 --dir ./n2_data n2
keeper create --dir ./k1_data k1
keeper create --dir ./k2_data k2
keeper create --dir ./k3_data k3
proxy create p1
proxy create p2
And apply it in one go, with a message that is kept in the deployment history:
tibdg -r http://ftl1:8080 -g mygrid -s mygrid.tibdg -m "Initial grid definition"
If you prefer to be prompted step by step, tibdg grid generate mygrid.tibdg generates the script interactively.
5. Create tables, columns and indexes
Each table is created with its key and then columns are added (types: long, double, string, datetime, opaque and, in 5.2, vector_float32[dim]). Indexes only support long or string columns.
tibdg -g mygrid table create default_ttl=0 orders id long
tibdg -g mygrid column create orders customer string
tibdg -g mygrid column create orders amount double
tibdg -g mygrid column create orders created datetime
tibdg -g mygrid index create orders idx_customer customer
tibdg -g mygrid table list
6. Start the processes in order
Always state keepers, then nodes, then proxies. The options -k (log file prefix), -q (maximum size in bytes) and -w (number of files kept) control log rotation.
# 1. State keepers (all three; the grid needs quorum: at least 2)
tibdgkeeper -n k1 -r http://ftl1:8080 -g mygrid -k /var/log/as/k1 -q 10485760 -w 5
tibdgkeeper -n k2 -r http://ftl1:8080 -g mygrid -k /var/log/as/k2 -q 10485760 -w 5
tibdgkeeper -n k3 -r http://ftl1:8080 -g mygrid -k /var/log/as/k3 -q 10485760 -w 5
# 2. Nodes, one at a time: wait for each one to be synchronised before the next
tibdgnode -n n1 -r http://ftl1:8080 -g mygrid -k /var/log/as/n1
tibdgnode -n n2 -r http://ftl1:8080 -g mygrid -k /var/log/as/n2
# 3. Proxies (with -e if clients reach them through another address)
tibdgproxy -n p1 -r http://ftl1:8080 -g mygrid -e proxy1.company.local:8888
tibdgproxy -n p2 -r http://ftl1:8080 -g mygrid -e proxy2.company.local:8888
7. Check the grid status
tibdg status is the first command whenever in doubt: it shows the processes and the estimated size of each node, plus a reindexing section when an index is being built.
tibdg -g mygrid status # processes (nodes, keepers, proxies) and estimated size
tibdg -g mygrid node status n1 # details of one process
tibdg -g mygrid table list # defined tables
tibdg -g mygrid copyset list # copysets
tibdg -g mygrid proxy clients p1 # clients connected to a proxy
8. Scale the grid with copysets
Capacity grows by adding copysets. After starting the new nodes, grid redistribute redistributes the data; to scale down, deactivate the copyset, redistribute and remove it:
# Add capacity: a new copyset with its two nodes
tibdg -g mygrid copyset create cs2
tibdg -g mygrid node create --copyset cs2 --dir ./n3_data n3
tibdg -g mygrid node create --copyset cs2 --dir ./n4_data n4
# ... start n3 and n4 with tibdgnode ...
tibdg -g mygrid grid redistribute
tibdg -g mygrid status # follow the redistribution of the data
# Remove capacity: empty the copyset, redistribute and remove it
tibdg -g mygrid copyset deactivate cs2
tibdg -g mygrid grid redistribute
tibdg -g mygrid copyset remove cs2
9. Distribute clients across proxies
If one proxy is more loaded than the others, or before stopping it for maintenance, move its clients to other proxies without interrupting the service:
tibdg -g mygrid proxy clients p1 # client connections of the proxy
tibdg -g mygrid proxy shed p1 clients 10 # move 10 clients to other proxies
tibdg -g mygrid proxy shed p1 connection 4711 # move a specific connection
10. Take checkpoints
A checkpoint is a consistent copy of the grid data. In addition to the automatic ones (step 4), take a manual one before any significant change:
tibdg -g mygrid checkpoint create before-the-upgrade
tibdg -g mygrid checkpoint list # ID, name, directory, date and status
tibdg -g mygrid checkpoint tables before-the-upgrade
tibdg -g mygrid checkpoint validate before-the-upgrade
tibdg -g mygrid checkpoint delete before-the-upgrade
11. Restore from a checkpoint
Restoring takes the whole grid back to the moment of the checkpoint:
# 1. Load the grid definition saved with the checkpoint
tibdg -g mygrid -r http://ftl1:8080 grid load -rollback realmserver-grid.json
# 2. Mark the checkpoint you are going back to
tibdg -g mygrid rollback create 12
# 3. Restore each node's data directory from checkpoints/<date>_..._chkpt and start it
# 4. When finished, remove the rollback mark
tibdg -g mygrid -r http://ftl1:8080 rollback delete
12. Check logs and metrics
- If the processes run as services inside the FTL server, each one can have its own
logfilein the YAML; otherwise, their output goes to the FTL server log, prefixed with the component name. - If they run separately, each one’s log is the one you specified with
-k. - The FTL server collects the metrics from the ActiveSpaces processes and exposes them for Prometheus to scrape.
And the administration service, if you run it separately, with its own log and trace level (severe, warn, info or debug):
tibdgadmind -r http://ftl1:8080 -l http://0.0.0.0:7171 --logfile /var/log/as/admind --max-log-size 10485760 --max-logs 5 --trace warn
Common errors
tibdg cannot find the grid or the realm
Without -r or TIBDG_FTL, tibdg connects to http://localhost:8080; without -g, it uses the _default grid. Always specify them, especially if you manage several environments from the same machine.
The grid stops responding when a state keeper goes down
With 3 state keepers, quorum is required (2 running). If only one is left, the grid cannot work: start another one and spread them across different machines so that a single failure cannot take down two.
Nodes are slow or fail to synchronise at startup
Start them one at a time and wait for each node to finish synchronising (tibdg status) before starting the next one.
“grid redistribute” does not move data to the new copyset
The nodes of the new copyset must be running before you redistribute. Check with tibdg status that they appear as running.
There is no tibdg command to drop a table
Tables are dropped with SQL (DROP TABLE) from the API, and for that the administration service must be running (tibdgadmind or the service inside the FTL server). tibdg table purge only cleans up tables that have already been dropped.
Error when creating an index
The columns of an index must be of type long or string.
Unexpected behaviour after upgrading from 4.x
ActiveSpaces 5.2 requires FTL 7.2 or later and no longer uses its own realm server. Check the version of the FTL servers and the release notes before upgrading.
Next steps
- Enable FTL 7 security: your own certificates, mTLS and OAuth2.
- Configure forwarding between 5.2 grids for disaster recovery.
- Send the FTL server metrics to Prometheus and Grafana, with alerts on quorum and node disk space.
- Version the
tibdgscripts and apply them from a pipeline in each environment.