HDFS monitoring with Netdata
The Hadoop Distributed File System (HDFS) is a distributed
file system designed to run on commodity hardware.
This module monitors one or more Hadoop Distributed File System nodes, depending on your configuration.
Netdata accesses HDFS metrics over Java Management Extensions (JMX) through the web interface of an HDFS daemon.
Requirements#
hdfsnode with accessible/jmxendpoint
Charts#
It produces the following charts for namenode:
- Heap Memory in
MiB - GC Events in
events/s - GC Time in
ms - Number of Times That the GC Threshold is Exceeded in
events/s - Number of Threads in
num - Number of Logs in
logs/s - RPC Bandwidth in
kilobits/s - RPC Calls in
calls/s - RPC Open Connections in
connections - RPC Call Queue Length in
num - RPC Avg Queue Time in
ms - RPC Avg Processing Time in
ms - Capacity Across All Datanodes in
KiB - Used Capacity Across All Datanodes in
KiB - Number of Concurrent File Accesses (read/write) Across All DataNodes in
load - Number of Volume Failures Across All Datanodes in
events/s - Number of Tracked Files in
num - Number of Allocated Blocks in the System in
num - Number of Problem Blocks (can point to an unhealthy cluster) in
num - Number of Data Nodes By Status in
num
For datanode:
- Heap Memory in
MiB - GC Events in
events/s - GC Time in
ms - Number of Times That the GC Threshold is Exceeded in
events/s - Number of Threads in
num - Number of Logs in
logs/s - RPC Bandwidth in
kilobits/s - RPC Calls in
calls/s - RPC Open Connections in
connections - RPC Call Queue Length in
num - RPC Avg Queue Time in
ms - RPC Avg Processing Time in
ms - Capacity in
KiB - Used Capacity in
KiB - Bandwidth in
KiB/s
Configuration#
Edit the go.d/hdfs.conf configuration file using edit-config from the
Netdata config directory, which is typically at /etc/netdata.
Needs only url to server's /jmx endpoint. Here is an example for 2 servers:
For all available options, please see the module configuration file.
Troubleshooting#
To troubleshoot issues with the hdfs collector, run the go.d.plugin with the debug option enabled. The output should
give you clues as to why the collector isn't working.
First, navigate to your plugins directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your
system, open netdata.conf and look for the setting plugins directory. Once you're in the plugin's directory, switch
to the netdata user.
You can now run the go.d.plugin to debug the collector: