This project focuses on the integration and automation of data extraction from the Marvel API using Docker, InfluxDB, and Grafana. The primary goal is to create a real-time data visualization system, leveraging the flexibility and capabilities of these tools.
- Introduction
- Background
- Objectives
- Development and Analysis
- Visualization in Grafana
- Conclusions
- References
To achieve this project, our group utilized the following tools:
- APIs: For flexible and integrative data extraction.
- Docker: To create consistent and reproducible development environments.
- InfluxDB: A time-series database ideal for real-time data monitoring and analysis.
- Grafana: An open-source tool for interactive data visualization and analysis.
The objectives of this project are:
- Extract data from the Marvel API using a Python script.
- Store the obtained data in an InfluxDB database.
- Visualize the stored data using Grafana.
- Automate the entire process using Docker.
Docker ensures the portability and replicability of the environment by encapsulating the Python script, InfluxDB, and Grafana in independent containers, facilitating orchestration and deployment.
The Python script performs HTTP requests to the Marvel API, processes the responses, and formats the data for storage in InfluxDB. Error handling and pagination are also managed within the script.
InfluxDB is used to store the extracted data. The script connects to InfluxDB, creates the necessary database, and inserts the formatted data into it.
Grafana is configured to use InfluxDB as a data source, allowing the creation of interactive dashboards for real-time data analysis and monitoring.
Interactive dashboards are designed to display various graphs obtained from the Marvel API data.
This graph shows the number of new characters introduced in Marvel comics over the years.
This graph displays the number of comics published by various writers in 2024.
This graph illustrates the number of characters appearing in significant Marvel events.
This graph shows the number of comics available for a selected list of characters.
We have developed a system capable of extracting, storing, and visualizing data from the Marvel API in an automated manner using Docker, InfluxDB, and Grafana. Future improvements could include additional Python script functionalities and new data visualizations in Grafana.