HDFS transparency
IBM Spectrum Scale™ HDFS transparency offers a set of interfaces that allows applications to use HDFS Client to access IBM Spectrum Scale through HDFS RPC requests.
All data transmission and metadata operations in HDFS are through the RPC mechanism and processed by the NameNode and the DataNode services within HDFS. IBM Spectrum Scale HDFS transparency implementation integrates both the NameNode and the DataNode services and responds to the request as in HDFS. Advantages of HDFS transparency are as follows:
- HDFS compliant APIs or shell-interface command.
- Application client isolation from storage. Application Client may access data in IBM Spectrum Scale without GPFS™ client installed.
- Improved security management by Kerberos authentication and encryption for RPCs.
- Simplified file system monitor by Hadoop Metrics2 integration.
- Supported IBM Spectrum Scale storage mode
- Hadoop cluster planning
In an Hadoop cluster that runs the HDFS protocol, a node can take on the roles of DFS Client, a NameNode, or a DataNode or all of them. The Hadoop cluster might contain nodes that are all a part of an IBM Spectrum Scale cluster or it might contain some of the nodes in the IBM Spectrum Scale cluster. - Installation and configuration of HDFS transparency
This section describes the installation and configuration of HDFS transparency with IBM Spectrum Scale. - High availability configuration
- Short-circuit read configuration
In HDFS, read requests go through the DataNode. When the client asks the DataNode to read a file, the DataNode reads that file off the disk and sends the data to the client over a TCP socket. The short-circuit read obtains the file descriptor from the DataNode, allowing the client to read the file directly. 
Multiple Hadoop clusters over the same file system
By using HDFS transparency, you can configure multiple Hadoop clusters over the same IBM Spectrum Scale file system. For each Hadoop cluster, you need one HDFS transparency cluster to provide the filesystem service.- Docker support
HDFS transparency supports running the Hadoop Map/Reduce workload inside the virtual machine container, Docker. - The HDFS transparency federation
Federation has been introduced in HDFS to solve the HDFS NameNode scaling problem. This topic provides an overview of the HDFS Federation feature and the configuration and management of the federated cluster. - Hadoop distcp support
The hadoop distcp command is used for data migration from HDFS to the IBM Spectrum Scale file system and between two IBM Spectrum Scale file systems. - Automatic Configuration Refresh
The Automatic configuration refresh feature is supported in gpfs.hdfs-protocol 2.7.0-2 and later. - Application interaction with HDFS transparency
The Hadoop application interacts with the HDFS transparency similar to their interactions with native HDFS. They can access data in the IBM Spectrum Scale filesystem using Hadoop file system APIs and Distributed File System APIs. - Security
HDFS transparency supports full Kerberos and it is verified over IBM® BigInsights® IOP 4.1. - Removing and upgrading HDFS transparency cluster
Before upgrading the HDFS transparency, you need to remove the older IBM Spectrum Scale Hadoop connector. - Limitation and difference from HDFS
This section describes the limitations and the configurations in IBM Spectrum Scale that are different from HDFS. - HDFS transparency security
- Guide for security setup
- Security configuration in Hadoop
This topic describes the security configuration in Hadoop.
Parent topic: Hadoop support for IBM Spectrum Scale