Protegrity & StreamSets
overview
Experience data protection that runs inside your data pipelines, not alongside them. Protegrity’s Application Protector (AP) Java is purpose‑built to embed directly within the StreamSets environment, securing sensitive data continuously as it flows. Whether you’re orchestrating complex multi‑cloud movement or managing real‑time streaming ingestion, protection is applied directly within the active dataflow—before data ever reaches a downstream system.
The result: StreamSets users can scale modern data architectures with confidence, maintaining high‑speed ingestion and routing while meeting strict global privacy and compliance requirements.
Key Integration Feature
Modern data pipelines move fast—but security often lags behind. Protegrity’s integration with StreamSets solves this by embedding data-centric protection directly into the stream itself. By embedding Application Protector (AP) Java directly within StreamSets pipelines, sensitive data is secured in motion, not after the fact. This eliminates the need for external security services, network calls, or post-processing controls that introduce latency and risk. Teams can build, modify, and scale streaming pipelines knowing sensitive data is protected by default, without sacrificing throughput or architectural flexibility.
Features & Capabilities
See how Protegrity protects data at stream speed inside StreamSets—combining in-flight enforcement, centralized policy, and flexible pipeline integration for modern data movement.
01
Vaultless Tokenization at Stream Speed
Why It Matters
Sensitive fields need protection that preserves structure and analytical value as data moves. Vaultless tokenization keeps schemas intact and avoids lookup latency, so downstream systems can continue to join, validate, and analyze data without changes.
How It Works
A retailer streams customer transactions through StreamSets into a cloud data lake. AP Java tokenizes identifiers in flight, allowing analytics teams to work on production-scale data while real identities stay protected.
02
In-Flight Protection with Embedded AP Java
Why It Matters
Streaming security must keep pace with data velocity. By running AP Java inside StreamSets, Protegrity avoids external API calls and centralized bottlenecks, helping protection scale with pipeline throughput.
How It Works
A healthcare provider ingests continuous patient telemetry. Protection runs locally within StreamSets processors, securing sensitive fields before the data lands in downstream repositories.
03
Dynamic Policy Enforcement Within Live Pipelines
Why It Matters
Not every field or destination requires the same treatment. Dynamic policy enforcement lets a single pipeline apply different protection methods based on sensitivity, destination, or use case without duplicating flows.
How It Works
When an HR data stream is routed to multiple targets, Social Security Numbers are tokenized before analytics delivery while less-sensitive attributes continue through the pipeline unchanged.
04
Embedded Pipeline Integration Without External Services
Why It Matters
Data engineering teams need security that fits existing workflows. Embedding protection directly in StreamSets reduces operational risk by removing external services, hard-coded secrets, and extra network dependencies from the pipeline design.
How It Works
Teams implement AP Java within StreamSets processors or evaluators, where cryptographic operations and policy checks run locally on execution nodes and sensitive keys stay out of pipeline logs and configurations.
05
Centralized Policy Management Across StreamSets Flows
Why It Matters
Streaming environments become hard to govern when each pipeline is configured differently. Centralized policy management helps ensure the same rules are enforced consistently across flows, teams, and destinations without manual updates.
How It Works
A financial institution defines a global credit-card policy in ESA. That rule is then enforced across StreamSets pipelines automatically, regardless of destination system or cloud platform.
Architecture &
Sample Data Flow
Protegrity’s StreamSets architecture is designed for high‑velocity, in‑flight protection. By embedding Application Protector (AP) Java directly into the pipeline, security becomes a built‑in layer of the data flow—not a downstream control. This approach ensures sensitive data is consistently protected as it moves between sources, pipelines, and target systems, across on‑premise, cloud, and hybrid environments—before exposure risk ever exists.
The data journey
Visualizing the data journey
The data journey
The data journey explained
-
01
Protect data at ingestion
As data enters StreamSets from source systems, Protegrity can apply tokenization, masking, or encryption immediately within the active flow. This helps ensure sensitive values are protected before they move into downstream pipelines, lakes, or warehouses.
-
02
Enforce protection during pipeline transformation
Protegrity runs inside StreamSets through embedded AP Java, allowing protection to happen in flight as data is transformed, routed, enriched, or filtered. This keeps sensitive fields governed without introducing external calls, added latency, or separate post-processing steps.
-
03
Deliver governed data to downstream platforms
Protected data can be streamed safely into destinations such as Databricks, Cloudera, cloud storage, or analytics platforms. Because sensitive fields remain tokenized or masked in transit, downstream teams can work with governed data without direct exposure to clear-text values.
-
04
Enable controlled unprotection and auditability
When approved users, systems, or workflows require access to original values, Protegrity supports controlled unprotection at the destination or within authorized consuming applications. Protection and access events are logged to support audit readiness, compliance reporting, and centralized governance.
DEPLOYMENT
Deploy Protegrity inside StreamSets so protection runs in the active dataflow—across on-prem, cloud, edge, and Kubernetes environments—without introducing external bottlenecks or post-processing controls.
Embedded AP Java in pipeline execution
Custom processors and scripting evaluators
High-speed ingestion and streaming pipelines
Cloud, edge, and Kubernetes deployment patterns
RESOURCES
Provide links to comprehensive documentation, guides. Include information for both developers and non-developers.
Docs Center
Explore product documentation, policy guidance, and implementation patterns for tokenization, masking, encryption, and AP Java-based protection across streaming and pipeline environments.
READ MOREIBM StreamSets Documentation
Official product documentation for StreamSets Control Hub and Data Collector.
READ MOREFrequently
Asked Questions
Here are five common questions related to the integration, deployment, and features of the Streamsets and Protegrity solution, with their answers: