Syllables to Scenes: Literary-Guided Free-Viewpoint 3D Scene Synthesis from Japanese Haiku

Author Name

Paper Supplementary Code Dataset
Demo Animation
Headline Image

Abstract

In the era of the metaverse, where immersive technologies redefine human experiences, translating abstract literary concepts into navigable 3D environments presents a fundamental challenge in preserving semantic and emotional fidelity. This research introduces HaikuVerse, a novel framework for transforming poetic abstraction into spatial representation, with Japanese Haiku serving as an ideal test case due to its sophisticated encapsulation of profound emotions and imagery within minimal text. While existing text-to-3D methods struggle with nuanced interpretations, we present a literary-guided approach that synergizes traditional poetry analysis with advanced generative technologies. Our framework centers on two key innovations: (1) Hierarchical Literary-Criticism Theory Grounded Parsing (H-LCTGP), which captures both explicit imagery and implicit emotional resonance through structured semantic decomposition, and (2) Progressive Dimensional Synthesis (PDS), a multi-stage pipeline that systematically transforms poetic elements into coherent 3D scenes through sequential diffusion processes, geometric optimization, and real-time enhancement.

Method

Method
Overall Architecture. Our objective is to transform classical Japanese Haiku into 3D scenes through two stages: H-LCTGP: (1) Haiku parsing using large language models (LLMs); PDS: (2) Text-to-Image Generation with Relay Diffusion; (3) Paranomic Image Generation with Panorama Diffusion; (5) Depth Map Generation with Depth Diffusion; (5) Geometric Optimization with 3D Gaussian Splatting; (6) Real-time Image Enhancement for immersive, navigable 3D scene visualization.

3D Scene Generation

Results

Result
Visualization Result: Generating high-quality 3D scenes.
Comparison
Comparison with baseline: Superior content consistency and scene continuity.

More Results

Additional Result
Additional visualization example 1.
Additional Result
Additional visualization example 2.
Additional Result
Additional visualization example 3.

Haiku Enhancement

Haiku Enhancement
Haiku Enhancement integrating traditional literary analysis principles. An example of the text enhancement process for Haiku using LLMs.