SQBenchmarks: SPARQL Querying Benchmarks

Half Day Tutorial at ISWC2016
 Monday, Morning, 17th October, 2016

Benchmarking is indispensable when aiming to assess technologies with respect to their suitability for given tasks. The importance of SPARQL queries for Linked Data management has led to the development of several benchmarks that allow assessing the performance of SPARQL query processing systems. This tutorial will explore the different benchmarks used for SPARQL query processing over Linked Data. In particular, we will focus on the design, key features, evaluation setup, and pros and cons of the benchmarks used to evaluate the performance of triple stores and federated SPARQL query engines. State-of-the-art triple stores and federated SPARQL benchmarks will be practically demonstrated with examples along with hands-on experience and exercises to be carried out by the participants. By the end of the tutorial, participants will obtain hands-on knowledge of the triple stores and federated SPARQL querying benchmarks, understand the main differences between state-of-the-art systems and benchmarks, and be able to pick the right benchmark based on their pros and cons and the use-case scenario. 


  • Triple Store Benchmarks (9:00 AM--10:00 AM, by Muhammad Saleem)  
      • Why benchmarks?
      • Benchmark categories
      • Benchmark Components and design principles
      • Key Features and Choke Points
      • Real vs. Synthetic Benchmarks
      • Analysis of LUBM, BSBM, SP2Bench, DBpedia Benchmark, WatDiv, FEASIBLE  
      • Federated SPARQL query processing
      • Comparison of SPLODGE, FedBench, and LargeRDFBench
Slides PPT (View Download
Slides PDF (
View Download)

10:30 AM -- 11:00 AM     Coffee Break

  • Hands-on Experience (11:00 AM--12:15 PM, Ivan Ermilov)
      • How to generate data and/or queries using the LUBM, BSBM, DBpedia, WatDiv and FEASIBLE Benchmarks
      • How to run benchmarks using IGUANA benchmark execution framework
      • How to run federated queries on FedX, HiBISCuS  etc. (if time permits) 

In order to stick to the program, please clone this git repository and follow installation instructions from the README.md file (front page on the Github). To be able to follow, you will need a Linux system. For federated SPARQL query processing, please refer to our ISWC 2015 tutorial technical requirements and follow the setup instructions. 

  • Introduction to HOBBIT project, a Holistic Benchmarking of Big Linked Data (12:15 PM--12:30 PM, by Axel Ngonga Ngomo)