Confluent simplifies integration between Kafka stream processing and Iceberg storage

Confluent Inc. at this time introduced new options in its cloud service that make it simpler for customers of its Apache Kafka-based streaming engine to retailer knowledge within the Apache Iceberg desk format.

The brand new Confluent Tableflow permits customers to transform Kafka subjects, related schemas and metadata to Iceberg tables with one click on, in addition to supplies higher supporting analytic workloads in knowledge lakes and knowledge warehouses.

That compares with what had beforehand been a “painful” course of, mentioned Addison Huddy, vp of Kafka at Confluent. “At the moment, you need to take into consideration how one can partition knowledge, devour it and write it out to S3 in a cost-performant and secure manner,” he mentioned, referring to Amazon Internet Providers Inc.’s object storage format. “You find yourself with actually small recordsdata in S3 that must be compacted, and sometimes you lose the schema. You find yourself reorganizing, grouping and including a schema with an entire bunch of pipelines,” constructed with Apache Spark and Apache Flink.

“Tableflow takes all that complexity and makes it pushbutton easy,” he mentioned. “You take a look at a subject in Kafka. It already has a schema, so you understand its form. You push a button, and it flips that stream and turns it right into a desk. With the Iceberg metadata interface, we will expose it as a Kafka and an S3 endpoint. I consider it like getting Iceberg knowledge bottled on the supply.”

Defacto requirements

Kafka has a virtually 39% market share within the fragmented queueing, messaging and background processing market, in keeping with 6sense Insights Inc. It’s utilized by greater than 80% of Fortune 100 corporations.

Apache Iceberg is an open-source desk format well-liked in knowledge lakes for its flexibility and consistency. Iceberg helps schema evolution, hidden partitioning and snapshot isolation for reliability. It may possibly additionally scale to handle petabytes of knowledge throughout billions of rows.

Tableflow works with Confluent’s knowledge streaming platform’s current capabilities, together with stream governance options and stream processing with Apache Flink, an open-source, unified stream-processing and batch-processing framework for giant knowledge volumes.

Iceberg makes use of the open-source Parquet file format with columnar storage written as recordsdata. “Our job is to make sure the recordsdata are properly organized,” Huddy mentioned. “An entire ecosystem has developed round utilizing Iceberg maps to get all of your Parquet knowledge.” Tableflow handles the interpretation and makes updates obtainable in actual time. It’s presently obtainable as a part of an early entry program.

Confluent can be increasing the variety of connectors to different knowledge sources to greater than 80 and including assist for personal networks utilizing DNS forwarding and Egress Entry Level on Amazon and Microsoft Corp. Azure cloud platforms. Provisioning time has been decreased, and the information switch throughput worth has been decreased to 2.5 cents per gigabyte. “Now you can arrange a connection rather more shortly and know straight away that it’s working,” Huddy mentioned.

Confluent Cloud prospects may even now have the corporate’s Stream Governance platform routinely enabled of their environments, offering entry to a schema registry, a knowledge portal, real-time stream lineage and different options.

“Kafka is the primary time in a streaming pipeline that knowledge is written, so that you wish to ensure that knowledge is ruled the minute it’s written,” Huddy mentioned. “With knowledge masking insurance policies, you’ll be able to instantly apply governance the minute knowledge is created.”

A part of stream governance referred to as Schema Registry helps implement common knowledge requirements to make sure knowledge high quality and consistency. The enterprise-focused Stream Governance Superior now presents a 99.99% service-level settlement for Schema Registry.

Picture: Flickr CC