Apache NiFi

What is it?

Apache NiFi is a data integration platform with a drag-and-drop visual interface: you define dataflows that move, transform, and route data between heterogeneous systems without writing code. The programming model is processors connected in a directed graph.

What is it for?

• Ingesting from diverse sources: databases (JDBC), REST APIs, S3, SFTP, Kafka, file systems — dozens of connectors included out of the box.
• Conditional routing and lightweight transformation: filter, split, convert formats (JSON ↔ Avro ↔ CSV), enrich records with external lookups — all within the same visual flow.
• Immediate observability: every message (FlowFile) is traceable; the UI shows bottlenecks, backpressure, and data provenance in real time.

When to use it / when not

Use it when you need to move data between systems with complex routing rules, non-technical teams need to operate or monitor the flow, or you want data provenance traceability (who touched what data and when) without instrumenting code by hand.

Think twice if your need is batch orchestration with task dependencies (Airflow is more natural there: DAGs with retry logic and per-step SLAs), or high-throughput messaging at millions of events per second (Kafka is the right piece there; NiFi can act as a producer/consumer, but not the broker). NiFi also requires a JVM and generous memory — overkill for simple pipelines that fit in a Python script.

Get started in 1 minute

The fastest path is Docker; the official image exposes the UI on port 8443 (HTTPS with a self-signed certificate in recent versions):

docker run --name nifi \
  -p 8443:8443 \
  -e SINGLE_USER_CREDENTIALS_USERNAME=admin \
  -e SINGLE_USER_CREDENTIALS_PASSWORD=adminpassword \
  apache/nifi:latest

Open https://localhost:8443/nifi — accept the self-signed certificate in your browser. From the UI, drag a GetFile processor (or GenerateFlowFile to test without real files), connect it to LogAttribute, and start the flow. Not a single line of code.

For production deployments, see the official administration guide.