Accessing Hadoop HDFS Data Using Node.js and the WebHDFS REST API

You don't need a JVM to talk to HDFS. Hadoop's WebHDFS exposes the file system over plain HTTP, so a Node.js service can read and write big-data files directly.

What WebHDFS gives you

A REST interface to standard HDFS operations — OPEN, CREATE, APPEND, LISTSTATUS — over HTTP, no native Hadoop client required.

Reading a file from Node.js

const WebHDFS = require("webhdfs");
const hdfs = WebHDFS.createClient({ user: "hadoop", host: "namenode", port: 50070 });

hdfs.createReadStream("/data/events.log").pipe(process.stdout);

Notes for production

• WebHDFS may redirect to a datanode — follow redirects.
• Stream large files rather than buffering them in memory.
• Secure clusters need Kerberos/SPNEGO on the requests.

Takeaway

WebHDFS is the pragmatic bridge between a Node.js stack and a Hadoop cluster — HTTP in, big-data files out.