How to Set Up Real Time Performance Monitoring with Netdata

Introduction#

Netdata provides accurate performance monitoring through extensible web dashboards that visualize the processes and services on your Linux systems. It monitors metrics about CPU, memory, disks, networks, processes, and more.

Netdata needs no extra configuration necessary once installed, but provides significant customization. The efficiency and speed of the application aims to be comparable to native console administration tools such as vmstat, iostat, and htop .

The steps in this tutorial cover everything you need to successfully set up one Ubuntu 16.04 server running Netdata using its built-in web server or, optionally, using Nginx.

Installing Netdata#

Before we install anything, make sure the system package index is up to date.

Next, install Netdata’s dependencies, which include the gcc (a C compiler), GNU Autoconf tools, GUID management, and compression libraries for Netdata’s internal web server.

The next set of packages are optional but recommended by Netdata, and include Python, some Python packages, and Node.JS. The stable version of Node.js bundled with the system package manager is fine for Netdata’s requirements. Install these next.

To install Netdata itself, we must use the project’s GitHub repository. Clone the Netdata repository into your home directory.

Move to the newly cloned directory.

Now build and install the application using the netdata-installer.sh shell script in this directory. Make sure to append sudo here, otherwise the way in which Netdata collects system data (through data collectors) would not function correctly.

The output you’ll see at first contains information on where Netdata will store all of it’s components. You can read through these so you are a bit more familiar with how the program is spread out over the filesystem after its installation.

Press ENTER to continue with the installation. After a moment, at the end of the output chain, you’ll see the following message:

Netdata is now installed and running successfully, and will run automatically upon rebooting the server as well.

Open port 19999 for Netdata’s internal web server.

At this point, you can view the default dashboard by visiting http://your_server_ip:19999/ in your favorite browser. You’ll see an overview of the system’s live metrics. This is a good check that everything’s working so far, but before we explore the GUI, let’s customize Netdata a little.

Configuring Netdata’s Memory Usage#

You can view Netdata’s current configuration in your browser by visiting http://your_server_ip:19999/netdata.conf.

You will see here that all (or most) of the configurable options are commented out (i.e. prefaced with a #). This is because Netdata’s configuration works by using a set of assumed defaults. Any disabled settings use Netdata’s default value; if a setting is uncommented, the value specified will override the default. This keeps the configuration file contained to only what you’ve modified.

Back on your server, the configuration file itself is /etc/netdata/netdata.conf . The option we’ll customize in this tutorial is the history parameter. This controls the maximum size of the memory database in use by Netdata.

Here is where you will have to decide how much RAM you want to dedicate to Netdata, or how much time you want to keep the recorded chart data for before it is lost:

• 3600 seconds (1 hour of chart data retention) uses 15 MB of RAM
• 7200 seconds (2 hours of chart data retention) uses 30 MB of RAM
• 14400 seconds (4 hours of chart data retention) uses 60 MB of RAM
• 28800 seconds (8 hours of chart data retention) uses 120 MB of RAM
• 43200 seconds (12 hours of chart data retention) uses 180 MB of RAM
• 86400 seconds (24 hours of chart data retention) uses 360 MB of RAM

Bear in mind that the above estimations are based upon the number of charts in use by the stock dashboard. Removing or adding custom charts in the future will influence these estimates.

Open up Netdata’s main configuration file with nano or your favorite text editor.

Locate the history option in the [global] section.

/etc/netdata/netdata.conf

Replace the 3600 value to the one you decided on before. Here, we’re using 14400, which gives us 4 hours of data retention. Make sure to also uncomment the line by deleting the # symbol so this option is no longer ignored by Netdata.

/etc/netdata/netdata.conf

Save and close the file after making this change.

While this modification likely increased the amount of RAM that Netdata will use, the next modification we’ll implement in the next step should reduce it significantly.

Enabling Kernel Same-page Merging#

Even though Netdata runs and works straight out of the box, there are more modifications we can make to the way Netdata uses system resources which will speed up and optimize its performance. We’ll start with enabling Kernel Same-page Merging, or KSM for short. The Netdata developers estimate this reduces Netdata’s memory usage by 40-60%.

Once enabled, the KSM daemon periodically scans memory, looking for pages of identical content that can be replaced by a single write-protected page. In this context, it allows the sharing of identical memory pages amongst different processes or programs running on the system. This reduces creation of unnecessary duplicate memory content.

To permanently enable this aspect of the Linux system kernel, open the /etc/rc.local file with a text editor.

This /etc/rc.local file or run control file is executed after all other normal Linux system services and processes are started. It is useful for specifying custom services — or, in our case, for enabling KSM at server run time.

Add the two commands highlighted below to the end of the file, before the last exit 0 line, as shown here:

/etc/rc.local

The first command adds a 1 into the run KSM kernel file, which enables the feature. The second command tells the the KSM daemon to run once per second and evaluate 100 pages for de-duplication.

Once you save and close the file, KSM will now be enabled the next time the server reboots, and persist through future reboots. To enable it for the rest of this server’s current uptime, you need to run the two commands you added to rc.local manually on the command line.

First, initiate and enter a new shell session as the server’s root user by using the -s flag with sudo. This is necessary because the two commands from earlier use output redirection (via the > operator) and need to be executed by a root shell to succeed.

Now enter the two commands from before:

Then, leave the root shell and return your normal shell session.

Finally, to apply all the changes we have made in this step and the step before this, we must restart Netdata.

You can verify that KSM is now enabled by looking for the newly added active KSM chart in the Netdata dashboard. This is found in the menu tree on the right, under Memory > Memory Deduper.

Now we know that the dashboard and KSM are working, it’s worth granting access to the dashboard through a web server like Nginx instead of the built-in web server. This isn’t completely necessary, but it is recommended, so follow the next step to set it up.

Hosting the Dashboard through Nginx#

Hosting Netdata via Nginx makes it easier to secure access to the dashboard and also allows you to point to other servers’ Netdata dashboards in the future, should you need to.

You’ll also need to install the apache2-utils package. This package contains the htpasswd program, which we’re going to need later on for securing access to the dashboard page.

Next, create a new Nginx host configuration file. Note that here, we’re using a file in the /conf.d directory, but you could use /sites-available and /sites-enabled if you’re using Nginx for things other than Netdata.

This configuration file will tell Nginx to proxy incoming requests for the dashboard page. We’ll also add a simple authentication prompt so only people with the correct username and password can access it.

Below is the entire configuration file for you to copy and paste. Change the two red highlighted items for the listen and server_name directives above to your server’s IP address and allocated domain name, respectively. Take care to include the terminating ; at the end of each. If you don’t have a domain name, you can leave the server_name directive as example.com .

/etc/nginx/conf.d/default.conf

Once you are certain your own file matches this code block, you can save and exit to return to the command prompt.

Let’s explain this file so you understand what it’s doing. This is the first section:

/etc/nginx/conf.d/default.conf

This upstream module named as netdata-backend targets the built-in Netdata web server by using the server’s loopback address, 127.0.0.1, and Netdata’s port, 19999. A keepalive directive sets the maximum number of upstream idle connections that can remain open at any given time (for each Nginx worker process). This upstream module definition is required later on by the proxy_pass directive.

Directly after this is the main server block.

/etc/nginx/conf.d/default.conf

The first two line in the block defines the external IP address Nginx should listen for when clients attempt to connect. The server_name directive tells Nginx to run this server block when a client uses the specified domain name.

The last two lines in this snippet set up the simple HTTP username and password authentication. It does this using the auth_basic module to display a username and password prompt. You can customize the message for the prompt; here, it’s set to Authentication Required. auth_basic_user_file designates a file name and location to store the login credentials of the authentication prompt, created later on in this step.

The last location block, nested inside the server block, handles the proxying and passing on of incoming requests to Nginx.

/etc/nginx/conf.d/default.conf

In summary here, this code is what passes client requests to the Netdata web server, and in turn the dashboard. The proxy_pass directive line refers back to the upstream module at the start of the file must point to it by name, netdata-backend.

If you haven’t already, you can save and close this configuration file. Here is where the htpasswd program from the apache2-utils package helps us create the user login credentials for the dashboard login prompt.

First, we’ll create a username and password. Run the following command, replacing sammy with whichever username you’d like to use in Netdata. When prompted, enter the password you’d like to use.

This creates a file at /etc/nginx/netdata-access that contains the username and password you provided. The generated password is encrypted within the file, so it cannot be seen as plain text.

Restart Nginx to make the new configuration we added active.

To test the Nginx configuration for the first time, visit http://your_server_ip in your favorite web browser. You’ll see an authentication prompt. Enter in the username plus password you provided earlier on in this step to get access to the dashboard.

Exploring the Dashboard#

Visit http://your_server_ip:19999/ in your favorite browser now to access the dashboard.

You’ll see the dashboard and an overview of the system’s live metrics. The HUD style indicators at the top of the page change when you mouse-over each time period in the subsequent graphs.

Conclusion#

To learn more about the Netdata project and how to further expand what’s been set up here, visit the Netdata GitHub project. Information on the Netdata registry, setting alarms, and installing custom charts/modules are covered in the Netdata wiki, which also discusses topics like generating HTML badges or handling Netdata’s log files.