Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Price Crawler - Tracking Price Inflation

MongoDB integrations for Apache Arrow. Export MongoDB documents to numpy array, parquet files, and pandas dataframes in one line of code.

Streaming kafka events using Spark in avro format and saving the events in parquet format

ETL job with AWS Glue

A command line tool for inspecting parquet files with PyArrow.

A light-weight command-line tool to browse and query CSV, Excel and Apache Parquet files, regardless of their size.

Load data from the Million Song Dataset into a final dimensional model stored in S3.

Merge Parquet Files on S3 with this AWS Lambda Function

Processing and exporting data from EPW files into other formats.

ETL pipeline that transforms JSON files from AWS S3 bucket to Parquet files also in S3 bucket

Easily convert JSON data into Parquet format for efficient storage and analysis. Simplify data processing and analysis pipelines by converting JSON objects into optimized Parquet files.

Processes S3 Inventory Manifests and generates a report about the folder size and object size average

ECE NTUA Assignment

Udacity Data Engeneering Nanodegree Program - My Submission of Project: Data Lake

PoC Minio Docker with parquet parser

create files which formats are like "orc", "parquet", "xlsx", "json" and so on with Python

This repository contains the NYC Taxi Data Engineering Pipeline project, which aims to build a comprehensive data engineering pipeline using NYC taxi data from the years 2022 and 2023. The pipeline involves extracting, transforming and loading (ETL) data into a Snowflake database, followed by creating a dashboard for visualisation.

Aplicação que captura mensagens de um grupo de Telegram e as armazena diariamente em arquivos, utilizando AWS S3 para armazenamento em nuvem. Em seguida, as mensagens são analisadas com foco em sentimento, menções a produtos da empresa e detecção de intenção de compra. O processamento é automatizado em batch usando funções Lambda da AWS.

Practice of Python skill