Learning Path: Apache Iceberg for Spark Data Engineers

Apache Iceberg is an open table format for large analytic datasets – it brings the reliability of database tables to data lakes. This learning plan will help a Spark-savvy data engineer understand Iceberg’s core concepts and how to use it with Spark. The journey is structured into clear stages with practical exercises, comparisons, and resources. By the end, you should be confident in listing Apache Iceberg as a skill on your resume.

1. Introduction to Apache Iceberg and Modern Data Lakes

What is Apache Iceberg? – Apache Iceberg is an open-source table format originally developed at Netflix to overcome limitations of Hive tables. It treats a table as a canonical list of files with rich metadata rather than relying on file folder hierarchies. This design enables database-like features on data lakes: ACID transactions, schema evolution, time travel, and concurrent multi-engine access. In short, Iceberg brings “the reliability and simplicity of SQL tables to big data”. It’s now used with engines like Spark, Trino, Flink, Presto, Hive, and Impala.

Why Iceberg for modern data lakes? – As data lakes evolve into “lakehouse” architectures, robust table management is crucial. Traditional Hive-based lakes often lacked ACID guarantees and efficient metadata management – leading to inconsistencies and performance issues. Iceberg tracks table state in a central metadata layer, avoiding expensive file listings and enabling safe, concurrent writes. Companies like Netflix, Apple, Adobe, and Airbnb use Iceberg to manage petabyte-scale data reliably. Essentially, Iceberg allows your data lake to function like a transactional database without sacrificing file-based storage flexibility.

Key Iceberg Concepts at a Glance

Table format vs. storage: Iceberg is not a storage system by itself—it’s an abstraction that manages metadata (schemas, partitions, snapshots) on top of files (e.g. Parquet, ORC, Avro).
Petabyte-scale design: Built for huge analytic tables (hundreds of millions or billions of rows), Iceberg is designed to handle “slow-moving petabyte-scale tables” efficiently.

2. Core Features and Architecture of Apache Iceberg

Iceberg’s architecture introduces a metadata layer on top of your data files to power advanced features:

Layered Architecture Overview: The Iceberg Catalog tracks the current table metadata (snapshot pointer). The metadata layer stores snapshots that reference manifest lists and manifest files. Manifest files list individual data files, and the data layer is composed of the actual files stored (Parquet/ORC/Avro). This design avoids costly directory scans and ensures efficient query planning.

ACID Transactions and Versioning: Each commit creates a new snapshot, ensuring atomicity and safe concurrent writes.
Schema Evolution: Add, drop, or rename columns without rewriting existing data. Iceberg assigns stable IDs to columns to prevent issues like “zombie data.”
Partitioning and Partition Evolution: Uses hidden partitioning defined in table metadata so that you don’t need to manage partitions manually. The scheme can evolve over time without breaking queries.
Time Travel (Data Versioning): Query previous snapshots (e.g., using a SQL query like SELECT * FROM my_table FOR TIMESTAMP AS OF '2023-01-01 00:00:00') for debugging, auditing, or recovery.
Performance Optimizations: By storing table statistics (e.g., row counts, file sizes), Iceberg minimizes file scanning and supports incremental data reading.
Deletes and Updates: Supports SQL-like DELETE, UPDATE, and MERGE commands, using either delete markers or file rewrites.

These features work together by writing small metadata files with each commit that reference the full set of data files. The Iceberg Catalog maintains the pointer to the latest metadata, ensuring that all engines see a consistent view.

3. Comparing Apache Iceberg to Hive, Delta Lake, and Hudi

It’s important to understand how Iceberg stacks up against other table formats:

Iceberg vs. Hive tables: Hive relies on directory pointers and suffers from scalability issues. Iceberg’s file-level metadata and transaction support provide a more robust solution.
Iceberg vs. Delta Lake: Both formats offer ACID guarantees, schema evolution, and time travel. Delta Lake uses a JSON-based log, while Iceberg uses manifest files and snapshots. Iceberg supports partition evolution and multi-engine use.
Iceberg vs. Hudi: Hudi is optimized for real-time ingestion and upserts. Iceberg is geared more toward large-scale batch analytics with complex schema and partition management.

4. Setting Up and Using Apache Iceberg with Spark

Iceberg integrates with Spark via the DataSource V2 API and Spark SQL extensions. Here’s a quick guide:

Environment Setup

Add the Iceberg library to Spark’s classpath. For example, when running spark-shell or spark-submit, use:

--packages org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:<version>

Alternatively, use the official Docker Compose setup (such as the tabulario/spark-iceberg image) to quickly get a Spark environment with Iceberg preconfigured.

Catalog Configuration

Configure an Iceberg catalog. For a Hadoop catalog, add these settings to your Spark configuration:

spark.sql.catalog.my_cat = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.my_cat.type = hadoop
spark.sql.catalog.my_cat.warehouse = /path/to/warehouse
spark.sql.defaultCatalog = my_cat

This instructs Spark to use the specified Hadoop catalog. For a Hive catalog, change type to hive and provide the metastore URI.

Creating Tables

Create an Iceberg table using Spark SQL:

CREATE TABLE my_cat.db.sample (
  id BIGINT,
  data STRING
)
USING iceberg;

You can also use the DataFrame API:
df.writeTo("my_cat.db.sample").create()

Writing, Reading, and SQL Extensions

Insert data with:

INSERT INTO my_cat.db.sample SELECT ...

Read data with:

spark.table("my_cat.db.sample")

Additional SQL capabilities include:

MERGE INTO for upsert operations.
DELETE FROM and UPDATE for modifying records.
TIME TRAVEL queries to query data as of a specific snapshot or timestamp. For example:
SELECT * FROM my_cat.db.sample FOR TIMESTAMP AS OF '2023-01-01 00:00:00'

Integration with Structured Streaming

Spark Structured Streaming can read an Iceberg table in tailing mode, processing only new records since the last checkpoint—ideal for near-real-time pipelines.

For more details, check out the Apache Iceberg Spark Quickstart and related AWS guides.

5. Hands-On Exercises and Project Ideas

Try this mini-project in a sandbox environment (using Docker or a local Spark installation with Iceberg jars):

Exercise: Building a Mini Lakehouse with Spark and Iceberg

Setup a Local Iceberg Catalog: Install Spark 3.x, add the Iceberg JAR (or use --packages), and configure a Hadoop catalog pointing to a local warehouse.
Create an Iceberg Table: Run in Spark SQL:
CREATE TABLE local.db.users (user_id INT, name STRING, country STRING) USING iceberg PARTITIONED BY (country);
Verify that the metadata folder appears in your warehouse.
Ingest Sample Data: Load data using a DataFrame:
df.writeTo("local.db.users").append()
Then, run a SELECT * to verify the data.
Perform Updates and Time Travel: Update data using:
UPDATE local.db.users SET name = 'Alice' WHERE user_id = 1;
Then, query a previous snapshot:
SELECT * FROM local.db.users.snapshot('<snapshot-id>');
Schema Evolution: Alter the table schema:
ALTER TABLE local.db.users ADD COLUMN email STRING;
Or rename a column:
ALTER TABLE local.db.users RENAME COLUMN name TO full_name;
Partition Evolution (optional): Experiment with changing the partitioning strategy (for example, adding bucketing) to observe Iceberg’s flexibility.
Query from Another Engine (optional): If available, query the same table from another engine (e.g., Trino or Flink) to see multi-engine support in action.

Keep the Iceberg documentation handy while you work through these exercises.

For guided videos, try “Hands-On with Apache Iceberg” by Dremio or the “Apache Iceberg 101” session on YouTube.

6. Best Practices for Production Use of Iceberg

Choose the right partition strategy: Use domain knowledge to partition data (e.g., by date or region). Iceberg supports transform partitions like bucketing.
Leverage schema evolution (but monitor changes): Evolve your schema while ensuring downstream systems remain compatible.
Avoid too many small files: Use Iceberg’s compaction features (such as CALL merge_files(...)) or schedule Spark jobs to merge files.
Manage metadata growth: Regularly expire old snapshots using REMOVE SNAPSHOT or expire_snapshots to keep metadata size manageable.
Remove orphan files: Use the remove_orphan_files procedure to clean up files no longer referenced.
Plan for table maintenance: Integrate steps like compaction and snapshot expiration into your data pipelines.
Use the latest Iceberg version: Stay updated to benefit from new features and improvements.
Monitor and tune: Continuously monitor query performance, file counts, and metadata size, then adjust settings as needed.

7. Additional Resources (Docs, Courses, Blogs, Repos)

Here’s a curated list of resources to supplement your learning:

Resource & Link	Format	Description
Apache Iceberg Official Docs – iceberg.apache.org (Intro, Spark Guide)	Documentation	Comprehensive guides on the table format, configuration, and usage examples.
Apache Iceberg GitHub Repo – apache/iceberg (GitHub)	Code Repository	Browse the open-source codebase, issues, and release notes. The README and Wiki provide additional context.
“Creating Your First Iceberg Table” (Dremio Blog) – Introduction tutorial (Dremio Blog)	Blog / Tutorial	A step-by-step guide to setting up Iceberg with Spark, including code samples.
Starburst Data: Iceberg vs. Delta Lake – Comparison article (Starburst Blog)	Blog (Comparison)	In-depth comparison of Apache Iceberg and Delta Lake features and architectures.
Apache Hudi vs Delta vs Iceberg – Conceptual overview (Medium Article)	Blog (Comparison)	A concise post comparing the key features and use cases of Hudi, Delta Lake, and Iceberg.
Netflix Iceberg Case Study (Upsolver) – Use cases (Upsolver Blog)	Blog (Case Studies)	Real-world examples of how companies like Netflix use Apache Iceberg in production.
Apache Iceberg Best Practices – Tips & tricks (Monte Carlo Data Blog)	Blog (Best Practices)	Best practices for partitioning, schema evolution, file sizing, and metadata management.
Iceberg Table Maintenance Guide – Technical deep-dive (BigData Boutique Blog)	Blog (Technical)	Guidance on keeping Iceberg tables healthy, including manifest merging, file compaction, and snapshot expiration.
“Apache Iceberg 101” by Ryan Blue – Video (40 min) (YouTube)	Video (Talk)	An introductory talk by an Iceberg co-creator explaining core concepts and use cases.
Iceberg + Spark Hands-On Video – Tutorial video (YouTube)	Video (Tutorial)	A comprehensive demo on setting up an Iceberg table with Spark, including advanced features like Project Nessie.
Project Nessie – Git-like catalog for Iceberg (Nessie on GitHub)	Code Repository	An optional tool that adds branching and tagging for version control of Iceberg tables.

Use these resources to dive deeper or troubleshoot specific questions. The Apache Iceberg community is active – join the official Slack channel or mailing list (links available on the Iceberg website) to ask questions.

With theory, hands-on practice, and these references, you now have a comprehensive learning path covering what Apache Iceberg is, why it’s important for modern data lakes, how to use it with Spark, and how to manage it in production. This practical knowledge will help you confidently add Apache Iceberg to your skill set.

Good luck, and enjoy building your next-gen data lakehouse with Apache Iceberg!

TechJournalist

Apache Iceberg for Spark Data Engineers