Unified Home for Transactions, Analytics, Data Lake, and ML: MySQL HeatWave Lakehouse

  • Published
  • Posted in DBA / MySQL

Most organizations run their data through a costly relay race. Transactions land in an operational database. To analyze them, that data is extracted, transformed, and loaded into a separate data warehouse. Raw files pile up in a data lake and need yet another engine to query them. Machine learning workloads demand a fourth platform altogether. Every hop between these systems adds latency, cost, and a new point of failure: data goes stale between syncs, security policies multiply across tools, and engineering teams end up spending more time moving data than using it. The result is a fragmented architecture that slows down the very decisions it was built to speed up.

MySQL HeatWave Lakehouse is Oracle’s answer to that fragmentation. It is a capability of MySQL HeatWave, Oracle’s fully managed cloud database service, that extends HeatWave’s in-memory, massively parallel query engine beyond the database to files sitting in Object Storage. Instead of loading lake data into MySQL first, HeatWave Lakehouse queries CSV, Parquet, Avro, JSON, and exported files directly where they reside, and lets that data be joined, in a single SQL statement, with live transactional tables. Paired with HeatWave’s existing OLTP and OLAP engines and its built-in AutoML capability, this turns one MySQL service into a unified home for transactions, analytics, data lake exploration, and machine learning, with no new query engine to learn and no separate infrastructure to manage.

Reference Architecture: HeatWave Lakehouse

┌─────────────────────────────────────────────────────────────────────────────┐
│                   MYSQL HEATWAVE LAKEHOUSE ARCHITECTURE                     │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌──────────┐    ┌──────────────────────────────────────────────────────┐   │
│  │  App /   │───▶│  MySQL DB System (Managed)                           │   │
│  │  BI Tool │    │  ┌────────────────────────────────────────────────┐  │   │
│  └──────────┘    │  │  MySQL InnoDB (OLTP Engine)                    │  │   │
│                  │  │  – Transactions, CRUD operations               │  │   │
│                  │  │  – Standard MySQL compatibility                │  │   │
│                  │  └────────────────────────────────────────────────┘  │   │
│                  │                        │                              │   │
│                  │            Auto-propagation (real-time)               │   │
│                  │                        ▼                              │   │
│                  │  ┌────────────────────────────────────────────────┐  │   │
│                  │  │  HeatWave Cluster (In-Memory OLAP Engine)     │  │   │
│                  │  │  ┌──────────────────────────────────────────┐  │  │   │
│                  │  │  │  Massively Parallel Processing (MPP)    │  │  │   │
│                  │  │  │  – In-memory columnar storage           │  │  │   │
│                  │  │  │  – 400x faster analytics vs MySQL       │  │  │   │
│                  │  │  │  – Auto-scales HeatWave nodes           │  │  │   │
│                  │  │  └──────────────────────────────────────────┘  │  │   │
│                  │  │                                                │  │   │
│                  │  │  ┌──────────────────────────────────────────┐  │  │   │
│                  │  │  │  HeatWave Lakehouse Engine              │  │  │   │
│                  │  │  │  – Query Object Storage directly        │  │  │   │
│                  │  │  │  – CSV, Parquet, Avro, JSON, Export     │  │  │   │
│                  │  │  │  – No ETL / No data loading             │  │  │   │
│                  │  │  └──────────────────────────────────────────┘  │  │   │
│                  │  │                                                │  │   │
│                  │  │  ┌──────────────────────────────────────────┐  │  │   │
│                  │  │  │  HeatWave AutoML                        │  │  │   │
│                  │  │  │  – Train ML models with SQL             │  │  │   │
│                  │  │  │  – Classification, regression, etc.     │  │  │   │
│                  │  │  └──────────────────────────────────────────┘  │  │   │
│                  │  └────────────────────────────────────────────────┘  │   │
│                  └──────────────────────────────────────────────────────┘   │
│                                          ▲                                  │
│                                          │ Lakehouse External Tables        │
│                                          │                                  │
│                              ┌───────────┴──────────────┐                   │
│                              │  OCI Object Storage      │                   │
│                              │  ┌──────┐ ┌──────┐      │                   │
│                              │  │.csv  │ │.parq │ …  │                   │
│                              │  └──────┘ └──────┘      │                   │
│                              └──────────────────────────┘                   │
│                                                                             │

Performance Benchmarks

Benchmark MySQL Only HeatWave Speedup
TPC-H 1TB (Analytical) Hours Seconds 400x
Lakehouse Query (Parquet) N/A Sub-second N/A
AutoML Training External tools Minutes (in-DB) 10x simpler
Mixed OLTP+OLAP Degraded Isolated (no impact) Zero contention

Key Features of MySQL HeatWave Lakehouse

Oracle documents the following core capabilities for MySQL HeatWave Lakehouse:

  • Unified query engine for Object Storage and MySQL: Query CSV, Parquet, and Avro files stored in Object Storage using standard SQL, join that data with transactional MySQL tables in the same query, and write results back to Object Storage for downstream sharing.
  • Native JSON and JavaScript support: Process semi-structured JSON data stored in the data lake, and use built-in JavaScript support to build richer, more dynamic application logic.
  • Vector Store for unstructured documents: Upload and query unstructured documents through the MySQL HeatWave Vector Store, enabling generative AI and retrieval-augmented use cases directly against lake content.
  • Massive scale-out architecture: Scale out to as many as 512 nodes, distributing scans, joins, and aggregations across nodes and cores, so performance stays close to linear as data volume grows.
  • HeatWave Autopilot: Machine-learning-driven automation handles schema inference from source files, adaptive sampling on large datasets, adaptive query optimization that can improve ad hoc query performance by up to 25 percent, and automatic compression that can cut memory costs by up to 20 percent all without manual tuning.
  • Built-in AutoML: Train, deploy, and explain machine learning models for anomaly detection, forecasting, classification, regression, and recommendations, using data from both Object Storage and MySQL tables, at no additional licensing cost.
  • Fully managed, highly available service: Patching, upgrades, backups, and automatic recovery from compute failures are handled by the service itself.
  • Granular, secure access control: Lakehouse access to Object Storage is governed through OCI resource principal authentication, pre-authenticated requests, and AWS Identity and Access Management roles, keeping lake access as tightly controlled as database access.
  • Multicloud availability — Beyond Oracle Cloud Infrastructure, MySQL HeatWave and its Lakehouse capabilities are also available on Amazon Web Services and Microsoft Azure.

High-Level Use Cases

Use Case 1: Unified Transactional & Analytical Workloads

A SaaS company runs its application on MySQL InnoDB for transactions. HeatWave automatically mirrors data for analytics. Business analysts run complex reports (joins, aggregations) on HeatWave without impacting production OLTP performance — no separate data warehouse needed.

Key Benefits:

  • Single database for OLTP and OLAP — eliminates ETL and data warehouse
  • 400x faster analytical queries vs. standard MySQL
  • Zero impact on transactional workload during analytics
  • Cost savings: one service replaces MySQL + Redshift/Snowflake

Use Case 2: Data Lake Analytics with SQL

A logistics company has 10TB of shipment data in Parquet files on Object Storage. HeatWave Lakehouse queries this data directly using standard MySQL SQL — no data loading, no ETL, no separate query engine. Analysts use their existing MySQL BI tools.

Key Benefits:

  • Query data lake files with standard MySQL — no new tools
  • No ETL pipeline to build or maintain
  • Join lake data with MySQL transactional data in a single query
  • Cost-effective storage on Object Storage with HeatWave compute on demand

Best Practices & Recommendations

  1. Use HeatWave for all analytical queries — let InnoDB handle OLTP only
  2. Store historical data in Object Storage (Parquet format) and query via Lakehouse
  3. Use HeatWave AutoML for in-database ML — no need for separate ML infrastructure
  4. Enable Auto Pilot for automatic query optimization and node scaling
  5. Use PAR (Pre-Authenticated Requests) for secure Lakehouse access to Object Storage
  6. Monitor HeatWave cluster utilization and scale nodes based on query complexity

Conclusion

MySQL HeatWave Lakehouse breaks the traditional boundaries between transactional databases, analytical warehouses, and data lakes. By unifying all three in a single MySQL service, it eliminates the ETL pipelines, data copies, and tool fragmentation that have defined analytics architecture for decades. For organizations using MySQL, HeatWave Lakehouse is the most efficient path to modern analytics — add analytical and lake capabilities without changing your database or your SQL.