Generate GeoJSONs as Part of ETL and Store in Public Folder for CDN

[agennadi]

Context

Currently, our app fetches large datasets dynamically upon each page load, which increases our data transfer usage and leads to performance issues. Since the data updates only once a week, we can optimize this by pre-generating GeoJSON files as part of the ETL process and storing them in the public/ folder. This will allow Vercel’s CDN to serve them efficiently, reducing unnecessary API calls and preventing timeouts.

Definition of Done

• GeoJSON files are generated as part of the ETL workflow.
• Generated files are stored in the public/ folder of the repository.
• The application fetches the GeoJSONs directly from the CDN instead of making API calls.
• Data updates seamlessly every week with new ETL runs.

Engineering Details

• Modify the ETL scripts in the backend/etl/ folder to generate GeoJSONs from the existing dataset.
• Store the generated .geojson files in the public/ folder.