WebGraph: The Next Generation (Is in Rust)

Tommaso Fontana (1) , Sebastiano Vigna (2) , Stefano Zacchiroli (3, 4)

1 DGD-I - Direction générale déléguée à l'innovation
2 UNIMI - Università degli Studi di Milano = University of Milan
3 ACES - Autonomic and Critical Embedded Systems
4 INFRES - Département Informatique et Réseaux

We report the results of a yearlong effort to port the WebGraph framework from Java to Rust. For two decades WebGraph has been instrumental in the analysis and distribution of large graphs for the research community of TheWebConf, but the intrinsic limitations of the Java Virtual Machine had become a bottleneck for very large use cases, such as the Software Heritage Merkle graph with its half a trillion arcs. As part of this clean-slate implementation of WebGraph in Rust, we developed a few ancillary projects bringing to the Rust ecosystem some missing features of independent interest, such as easy, consistent and zero-cost memory mapping of data structures. WebGraph in Rust offers impressive performance improvements over the previous implementation, enabling open-source graph analytics on very large datasets on top of a modern system programming language.

Mots clés: big data; compression; web graphs; social networks; Java; Rust

Domaines: Informatique [cs]

Tommaso Fontana, Sebastiano Vigna, Stefano Zacchiroli. WebGraph: The Next Generation (Is in Rust). ACM Web Conference 2024, May 2024, Singapore, Singapore. ⟨10.1145/3589335.3651581⟩. ⟨hal-04494627⟩