Project for Advanced Topics in Database Systems class ECE NTUA 2021-2022
In this project we experimented with the use of Apache Spark and Apache Hadoop.
-
1st part:
We had to implement queries using either RDD API, or Spark SQL. When using Spark SQL queries we had to work both with .csv files and .parquet files. -
2nd part:
We had to implement repartition and broadcast join on RDD API, based on the pseudocodes of the paper "A Comparison of Join Algorithms for Log Processing in MapReduce".
Our conclusions in more detail are discussed in the report inside this repository.
Team members:
This project was graded with 100/100.