Clock Sync with NTP

concept

Network Time Protocol (NTP) is used to synchronize time across multiple servers. The purpose of NTP is to keep an individual node’s clock accurate. You can do this by having the node periodically synchronize its clock with a reference server. You can specify multiple servers to provide redundancy in case 1 or more time servers are unavailable. Correct time synchronization is important in Enterprise Analytics for Expiry behavior and to maintain accurate synchronicity in timestamp-based conflict resolution.

In a multi-cluster environment, it’s important for all the nodes in the cluster to have the same time from a global perspective.

Time Sync Configuration

You should install local NTP Servers and also have more than 1 NTP server for fault tolerance. Each Enterprise Analytics node should have an NTP Client configured, which synchronizes time with the local NTP Server.

The NTP server on each cluster should ideally reference the same NTP server on the Internet to have the same time.

Basic Setup and Configuration

NTP Server Configuration

For information about how to set up, best practices, and advanced configuration about NTP, refer to http://www.ntp.org/.

Recommended NTP Server Configuration

Different NTP deployment models are available for NTP clients installed on Couchbase nodes to synchronize time such as:

Using an external NTP server to synchronize each Couchbase node independently.
Using a dedicated NTP server, or a core server, possibly running on a separate instance, within the Couchbase cluster. This server synchronizes with external NTP servers over the Internet, while the rest of the cluster synchronizes with this one.
Using a dedicated pool of NTP servers within the Couchbase cluster. These NTP servers within a pool synchronize with external NTP servers over the Internet, while the rest of the cluster synchronizes with this one.

In case of a bigger, scalable cluster, Couchbase recommends that you run your own NTP server pool that’s externally synchronized. You should have at least 2 NTP sources on the same local network as each Couchbase server group. See Manage Groups. This configuration provides perfect relative and absolute clock synchronization, high availability, and easy maintenance.
Couchbase recommends configuring NTP servers/pool as peers across clusters. This enables the NTP servers/pool on 1 cluster to have a relationship between the NTP servers/pool of the other cluster.

NTP Client Configuration

To synchronize clocks effectively on an Enterprise Analytics cluster, you must:

Start NTP on each node.

Start NTP by starting the NTP service, or daemon, on your system. On most systems, starting the NTP daemon happens automatically on startup. You do not need to perform this action manually. However, if you need to make adjustments to the NTP configuration, it’s useful to know how to stop and start the service.

For example, the following command starts the daemon:
```
$ service ntp start -x
```

Point each instance of NTP to the same set of reference servers.

Specify the time that a server or servers in the NTP configuration file, usually /etc/ntp.conf. You can specify multiple servers, 1 server per line.

Couchbase recommends setting up your own NTP server and sync NTP clients.

server ntp1.phx.xxx.com
server ntp2.phx.xxx.com

The configuration file is read when the NTP service starts. If you change the configuration file after NTP is running, you must stop and restart the service to have the new configuration options take effect.

Once ntp.conf is configured with correct settings, start the ntp daemon.

/etc/init.d/ntp start

You see the NTP starts to synchronize the time of your Linux machine with the NTP Server.

As noted in the NTP documentation, it’s important to remember that ntpdate with contrived cron scripts is no substitute for the NTP daemon, which uses sophisticated algorithms to maximize accuracy and reliability while minimizing resource use.

Verifying Your NTP Configuration

To verify that your NTP configuration is working properly, run the following:

Use the ntpstat command to view the status of the NTP service on the instance:
```
[ec2-user ~]$ ntpstat
```
If your output resembles the following output, then NTP is working properly on the instance:
```
synchronised to NTP server (12.34.56.78) at stratum 3
time correct to within 399 ms
polling server every 64 s
```
If your output states unsynchronised, wait for about a minute and try again. The first synchronization may take a minute to complete.

If your output states unsynchronised, wait for about a minute and try again. The first synchronization may take a minute to complete.

If your output states Unable to talk to NTP daemon. Is it running?, you probably need to start the NTP service and enable it to automatically start at boot time.

(Optional) You can use the ntpq -p command to see a list of peers known to the NTP server and a summary of their state.

[ec2-user ~]$ ntpq -p

remote           refid           st t  when poll reach   delay   offset  jitter
================================================================================
+lttleman.deekay 204.9.54.119     2 u   15  128  377     88.649   5.946   6.876
-bittorrent.tomh 91.189.94.4      3 u  133  128  377    182.673   8.001   1.278
*ntp3.junkemailf 216.218.254.202  2 u   68  128  377     29.377   4.726   11.887
+tesla.selinc.co 149.20.64.28     2 u   31  128  377     28.586   -1.215   1.435

If the output of this command shows no activity, check whether your security groups, network ACLs, or firewall rules block access to the NTP port.