Skip to content
/ theblop Public

The Big List of Protests - An AI-assisted Protest Flyer parser and event aggregator

theblop.org/

License

MIT license
0 stars 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

sayhiben/theblop

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

149 Commits
.github/workflows
.github/workflows
 
 
.vscode
.vscode
 
 
assets
assets
 
 
dist
dist
 
 
events
events
 
 
examples
examples
 
 
src
src
 
 
.babelrc
.babelrc
 
 
.gitignore
.gitignore
 
 
.python-version
.python-version
 
 
CNAME
CNAME
 
 
Dockerfile
Dockerfile
 
 
LICENSE
LICENSE
 
 
Makefile
Makefile
 
 
README.md
README.md
 
 
about.html
about.html
 
 
app.py
app.py
 
 
build.js
build.js
 
 
favicon.ico
favicon.ico
 
 
googleAppsScript.gs
googleAppsScript.gs
 
 
index.html
index.html
 
 
package-lock.json
package-lock.json
 
 
package.json
package.json
 
 
prompt.txt
prompt.txt
 
 
requirements-cuda.txt
requirements-cuda.txt
 
 
requirements.txt
requirements.txt
 
 
sample.env
sample.env
 
 
theblop-logo.png
theblop-logo.png
 
 
webpack.config.js
webpack.config.js
 
 

Repository files navigation

Event Flyer Parser

Note: This is mostly a README I've generated by guiding AI to examine the project. My intent is to create enough docs here to be able to redeploy from scratch if I don't come back to this in a while.

I doubt anyone other than myself will actually use this. It's intended to be "good enough to get started" - please ping me in some manner if you actually intend to deploy this, and I will help you get it running.

Good luck. Godspeed.

- Ben, Feb. 2025

Project Overview

Event Flyer Parser is an automation tool that extracts structured event details from images (flyers) and logs them into Google Sheets. It monitors a Gmail inbox for incoming emails with event flyer attachments, uses an AI model to parse each flyer’s content, and outputs key information like event title, date, time, location, etc., as a JSON entry in a spreadsheet (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). The project integrates Google Apps Script (for email handling and Google Sheets interaction) with a serverless GPU service on Runpod (for AI-powered image OCR and text extraction). This allows for end-to-end processing: from receiving an email to having a new row in a Google Sheet with all the event details. By automating flyer data entry, the project saves time and reduces errors in compiling event information.

Architecture & Workflow

Components:

Workflow:

  1. Email Intake (Google Apps Script): A time-driven trigger invokes processEmails() periodically (e.g., every 5 minutes). This function searches for any unread emails in the inbox (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). For each unread email, it generates a unique UUID for tracking (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub), then saves any image attachments to the specified Google Drive folder (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). The script collects the email’s date, subject, body text, any URLs found in the body, and the Drive URLs/IDs of saved images. It appends a new row to the RawData sheet with these details and a “processed” flag set to "false" (indicating this email’s flyer still needs processing) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). After logging the data, the email is marked as read and the email thread is moved to trash to prevent duplicate processing in the future (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub).

  2. Job Creation (Google Apps Script -> Runpod): Another trigger invokes launchRunPodJobs() (ideally shortly after processEmails() runs). This function scans the RawData sheet for any entries with Processed = false (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). Each such row represents a flyer that has not been parsed yet. The script compiles all these pending entries into a submissions list (each submission has the UUID as submissionId and an array of image file IDs) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). If there are one or more submissions, the script calls the Runpod endpoint via HTTP POST, sending a JSON payload containing the submissions array (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). (The structure is: {"input": {"submissions": [ { "submissionId": "...", "imageIds": ["...", "..."] }, ... ]}}.) The request includes the Runpod API key in the header for authorization (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). Upon success, Runpod immediately returns a Job ID that references this batch of submissions for processing (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). The Google script logs this job by adding a new row to the Jobs sheet with columns: Timestamp, JobID, Status (PENDING initially), PollAttempts (0), NextPollMins (initially 5), and the JSON string of submissions (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). After logging the job, launchRunPodJobs() schedules a follow-up trigger to call pollRunPodJobs() in a few minutes (using Apps Script’s time-based trigger) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). This kicks off the asynchronous polling loop.

  3. Asynchronous Processing & Polling: The Runpod service, upon receiving the job, queues it and then runs the app.py code to process the flyer images. Meanwhile, the Google Apps Script pollRunPodJobs() function will wake up at the scheduled time (e.g., 5 minutes later) to check on all active jobs. When pollRunPodJobs() runs, it performs the following:

  4. AI Processing (Runpod, behind the scenes): When Runpod executes the app.py for a submitted job, it invokes the handle_inference(event) function with the payload we sent. The Python code then:

  5. Result Integration (Google Apps Script): When finalizeJobResults() is invoked (after an AI job completes), it takes the returned job data (the list of {submissionId, answer}) and the original submissions list (as a JSON string) to map results back to the RawData entries (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). Its steps:

Overall, the architecture ensures that the potentially slow AI processing (which involves downloading images and running a large model) does not block the email retrieval or the Google Apps Script execution. The use of the Jobs sheet and polling mechanism allows the script to hand off work to Runpod and come back later for results, working around Apps Script execution time limits. Google Apps Script coordinates the workflow and data storage, while the Runpod-hosted Python app performs the intensive OCR and parsing task.

Setup Instructions

Follow these steps to install and configure the Event Flyer Parser in your environment:

1. Google Sheets and Drive Setup

  • Create the “Inbox” Spreadsheet: In your Google Drive, create a new Google Spreadsheet (e.g., named "Event Parser Inbox"). Inside this spreadsheet, add two sheets (tabs) named RawData and Jobs.
  • Create the “Processed” Spreadsheet: Create another Google Spreadsheet (e.g., "Event Parser Processed"). Add a sheet (tab) named Processed. Set up a header row with the columns: UUID, Date, Time, Title, Description, City, State, Address, Meeting Location, Links, Sponsors, Image, Source, Extracted Text. These correspond to the fields the script will output for each event (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). When events are parsed, each will be appended as a new row under these headers.
  • Create a Drive Folder for Images: In Google Drive, create a folder where flyer images will be stored (e.g., "Event Parser Images"). Note the Folder ID from the URL. (The folder ID is the long string in the URL when you open the folder in Drive. For example, in https://drive.google.com/drive/u/0/folders/ABC123XYZ456, the ID is ABC123XYZ456.)
  • Share the Folder with Service Account: (This step will connect to Runpod setup later, but it’s convenient to mention here.) You will create a Google Cloud service account in step 3. Once you have its email, share the Drive folder you just created with that service account’s email (give Viewer or Editor access). This allows the Python app running on Runpod to download images from your Drive. You can also set the folder’s access to “Anyone with link can view” as an alternative, but sharing with the service account is more secure.

2. Google Cloud Service Account (for Drive API)

Since the Runpod service needs to download images from your Google Drive, it requires credentials. We will use a Google Cloud service account for this:

  • Go to the Google Cloud Console, create a new project (or use an existing project) to house the service account.
  • Enable the Google Drive API for this project (APIs & Services > Library > enable Google Drive API).
  • Create a new Service Account (IAM & Admin > Service Accounts > Create Service Account). Give it a name like "event-parser-sa".
  • Assign roles: For Drive access, you can give it the "Viewer" role on Drive, or more specifically, enable the scope later. You might not need to assign a role in Cloud IAM for accessing a shared folder, but to be safe, you can skip assigning any specific GCP role here since we’ll use the API scope directly.
  • After creation, go to the service account’s Keys section and add a new key (JSON). Download the JSON key file.
  • Open the JSON file in a text editor. You will need its content for the Runpod container. Specifically, in the next step we will set an environment variable GOOGLE_SERVICE_ACCOUNT_KEY with this JSON content (event-flyer-parser/app.py at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/app.py at main · sayhiben/event-flyer-parser · GitHub).

(Reminder: As mentioned earlier, make sure the Drive folder from step 1 is shared with the service account’s email. The email is listed in the JSON under client_email.)

3. Deploying the Runpod AI Service

Sign up for a Runpod account if you haven’t already. Runpod provides “serverless GPU” endpoints which we will use to run the app.py for parsing flyers.

  • Create a new Endpoint: In Runpod’s dashboard, create a new Serverless Endpoint. You will be asked to choose a base container or provide one. There are two ways to deploy:
    • Option A: Use Pre-built Image – The project’s Docker image may be available at sayhiben/minicpm-o-2.6-events-parser:latest (as indicated in the Makefile (event-flyer-parser/Makefile at main · sayhiben/event-flyer-parser · GitHub)). If this image is published (check Docker Hub for sayhiben/minicpm-o-2.6-events-parser), you can use it directly. In Runpod, specify this image name and tag.
    • Option B: Build from Source – If you prefer or if the image is not available, you can build your own. Use the provided Dockerfile to build the image, or let Runpod build from the GitHub repository:
      • You might upload the repository code to your own GitHub (or use this one if public) and have Runpod connect to it. Ensure the Dockerfile is present.
      • Alternatively, build locally using the Makefile (run make docker which executes a build command as shown in the Makefile (event-flyer-parser/Makefile at main · sayhiben/event-flyer-parser · GitHub)) and push to a container registry (like Docker Hub or GitHub Packages), then use that image in Runpod.
  • Select GPU and Resources: Choose an appropriate GPU type and memory for the endpoint. The model is ~2.6B parameters and loaded in bfloat16; a GPU with at least ~10GB memory is recommended (for example, NVIDIA T4, RTX A10G, or better). Runpod will also ask for CPU/Memory allocation; the defaults should suffice (the heavy load is on GPU and memory for the model).
  • Set Environment Variables: In the endpoint configuration, add an environment variable named GOOGLE_SERVICE_ACCOUNT_KEY. Paste the entire JSON content of the service account key file as the value (you can compress it to one line or ensure the JSON formatting is preserved as needed – Runpod should handle the JSON string, but if issues arise, escape quotes or encode it). This environment variable is read in app.py to initialize the Drive API client (event-flyer-parser/app.py at main · sayhiben/event-flyer-parser · GitHub). No other env vars are strictly required; the model name and paths are hard-coded in the app (MODEL_NAME, SYSTEM_PROMPT_PATH, etc.). Optionally, if you want to override the model or provide a Hugging Face token (for private models), you would set those here, but by default it’s not needed.
  • Deploy: Deploy/start the endpoint. Wait for the endpoint to be ready – it may need to download the model weights on first startup. You can monitor the endpoint logs in Runpod; you should see messages once it’s up (or when it processes a request).
  • Note the Endpoint URL: Once deployed, Runpod will provide an endpoint URL for invoking the service. It typically looks like:
    https://api.runpod.ai/v2/<ENDPOINT_ID>/run
    (Runpod might also show the endpoint ID separately. The Apps Script needs the full URL that ends with /run to submit jobs, which it then transforms to /status/<jobId> for polling (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub).)
    Also note/generate a Runpod API Key (available under your account settings or the endpoint settings). This key is required for the Apps Script to authenticate to the Runpod API.

4. Google Apps Script Setup

Now configure the Google Apps Script that ties everything together:

  • Create a Script Project: You can create a standalone script by visiting script.google.com and creating a New Project. Alternatively, you can bind the script to the "Inbox" Google Sheet (open the sheet, go to Extensions > Apps Script). A standalone script is fine since we’ll use explicit IDs for Sheets.

  • Copy the Code: Open the googleAppsScript.gs file from this repository. Copy its entire content and paste it into the script editor (replace any default function that might be there). The script is written in JavaScript (Apps Script environment). It contains all the functions (processEmails, launchRunPodJobs, etc.) described in the Architecture section.

  • Set Script Properties: In the Apps Script project, click on Project Settings (the gear icon), then find the Variables / Properties section (in older editor it’s under File > Project Properties > Script Properties). Add the following keys and their values:

    Make sure there are no extra spaces and that the keys match exactly those names (they are case-sensitive in the script). The script will fetch these properties at runtime (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub).

  • Authorize Services: Since the script uses Gmail, Drive, and Sheets, the first time you run it you’ll be prompted to authorize these scopes. In the script editor, go to Run > Run Function > select processEmails (for instance). It will ask for permissions; review and allow access to your Gmail, Google Drive, and spreadsheets. (Alternatively, deploying as a web app or clicking the trigger might also prompt later. It’s good to do an initial test run manually to handle authorization.)

  • Set Triggers: Click on the clock icon (Triggers) in the left sidebar of the script editor (or go to Edit > Current Project’s Triggers). Add the following triggers:

    • Trigger processEmails to run periodically. For example, set it to Time-driven, “Every 5 minutes” or a schedule that suits how frequently you receive event emails. (Every minute is possible but may be overkill and could hit Gmail rate limits; every 5 minutes is a common choice.)
    • Trigger launchRunPodJobs to run periodically. This can also be every 5 minutes, but offset from processEmails. For instance, you could set processEmails at :00, :05, :10 minutes of the hour and launchRunPodJobs at :02, :07, :12, etc. If your script environment doesn’t allow fine-grained offsets, running both every 5 minutes is generally okay: processEmails will usually finish quickly, and launchRunPodJobs will find any new RawData entries almost immediately after. They are idempotent when no new data is present (they simply do nothing if there’s nothing to process).

    Note: The pollRunPodJobs function should not be manually scheduled. It is invoked by launchRunPodJobs and reschedules itself as needed (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). So you only need triggers for the initial two functions above.

  • Test the Setup: To verify, send an email to the Gmail account with a sample flyer image attached (or use one of the example images from the repository). Within the next trigger cycle:

    • The email should be moved to trash (after being processed).
    • A new row should appear in the RawData sheet with the email’s info and Processed = false.
    • A job entry should appear in the Jobs sheet when the script submits to Runpod (Status PENDING or RUNNING).
    • After a few minutes (depending on model runtime), the RawData entry’s Processed flag should turn "true" or "failed", and for "true", a new row should be added in the Processed sheet with the extracted event details.
    • If things don’t appear to work, check the script’s logs and the Runpod logs (see Debugging tips in a later section).

5. Configuration Notes & Additional Dependencies

  • Apps Script Dependencies: The script uses built-in Google services (GmailApp, DriveApp, SpreadsheetApp). You do not need to enable advanced services or import libraries for these – they are available by default in Apps Script. Just ensure the account running the script has access to the Gmail inbox, the Drive folder, and the spreadsheets (using the same Google account for all is simplest).
  • Runpod Python Dependencies: The Docker image is configured to include all necessary Python packages. Key dependencies (as seen in requirements-cuda.txt) include torch (PyTorch for the model), transformers (Hugging Face Transformers library), google-api-python-client (for Drive API), and Pillow (for image handling). The Dockerfile also sets up caching for model downloads. There is no direct need for you to install anything manually in Runpod; deploying the container will handle this. If running the Python app locally for development, you should install the requirements in a Python environment that has GPU access.
  • Email Filtering: The Gmail search query in processEmails is currently 'in:inbox is:unread' (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub), which means it will grab all unread emails in the inbox. If you want to limit to specific senders or subjects (e.g., if that inbox receives other emails), you can modify the query. For example, you might use a label or a keyword and update the search string accordingly (e.g., label:event-flyers is:unread). Make sure to also update how you handle those emails (e.g., if using labels instead of trash to archive processed emails).
  • Quotas and Limits: Free Gmail accounts have sending/reading quotas via Apps Script (around 100 emails/day for free consumer accounts) and reading a large number of emails can be slow. If you expect high volume, consider using a Google Workspace account which has higher quotas or splitting the work. Similarly, Google Drive API (via the service account) has download quotas – but since images are small and likely only downloaded a few at a time, this should not be an issue under normal use.
  • Runpod Costs: Be aware that running a GPU endpoint on Runpod will incur costs based on usage time. The script is designed to submit jobs on demand and the endpoint will only run when jobs are submitted (serverless model). Still, the model load time means each job might keep the GPU busy for a couple of minutes. Monitor your Runpod usage and consider shutting down the endpoint when not in use, or exploring an alternative like a local server if continuous usage is needed.

With the above steps completed, the system should be fully operational. New event flyer emails will be auto-processed and the data will accumulate in your Google Sheet without manual intervention.

API and Function Details

This section provides technical details on the API calls between the Apps Script and Runpod, as well as explanations of key functions and data structures.

Runpod API Integration

Job Submission (POST /run): The Google Apps Script sends an HTTP POST request to the Runpod endpoint URL (stored as RUNPOD_ENDPOINT_URL) to start a job (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). The request body is a JSON object with an input field containing our submissions array (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub):

{
  "input": {
    "submissions": [
      {
        "submissionId": "<UUID>",
        "imageIds": ["<ImageID1>", "<ImageID2>", ...]
      },
      ... (more submissions if multiple unprocessed entries)
    ]
  }
}

Each submission corresponds to one row in RawData (one email) and includes the unique UUID and an array of Google Drive file IDs for images from that email (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). (Even if there is only one image, it is wrapped in an array.)

The script sets an HTTP header Authorization: Bearer <RUNPOD_API_KEY> for authentication (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). Content type is application/json. The Runpod API expects this format for serverless endpoints.

Submission Response: On a successful request, the Runpod service responds with a JSON containing a job identifier. The code captures respData.id or respData.jobId from the response (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). For example, the response may look like:

{
  "id": "abcd-efgh-1234-5678", 
  "status": "IN_QUEUE"
}

or

{
  "jobId": "abcd-efgh-1234-5678"
}

The script logs the HTTP status and response (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). If the status is not 200/201, it logs an error and does not create a Jobs entry (meaning the job submission failed) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). On success, the extracted jobId is stored in the Jobs sheet with status PENDING (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub).

Job Status (GET /status/{jobId}): To check progress, the script calls the status endpoint. The URL is derived by replacing the .../run part of the endpoint URL with .../status/ and appending the Job ID (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). For example:
https://api.runpod.ai/v2/<ENDPOINT_ID>/status/abcd-efgh-1234-5678

The request is a GET with the same Authorization: Bearer <API_KEY> header (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub).

  • If the job is still running or queued, the response might look like: 
    { "status": "IN_PROGRESS" }
    or { "status": "RUNNING" }. In some cases, if the job is very quick, the first poll might already get a completed status.
  • If the job completed successfully, the response will include status: "COMPLETED" and an output field. The output is the return value from our handle_inference function – which in our case is an array of {submissionId, answer} objects for each submission. For example: 
    {
  "status": "COMPLETED",
  "output": [
     {"submissionId": "1234-uuid-5678", "answer": "{ \"date\": \"2025-08-30\", ... }"},
     ...
  ]
}
  • If the job failed (e.g., an unhandled exception, or the container crashed), the response may have status: "FAILED" and possibly an error message or no output.

The Apps Script’s checkRunPodJobStatus function interprets the response as follows (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub):

Output Data Structure: The output data (when status is COMPLETED) is expected to be an array of results. Each result corresponds to one submission from the input and has:

Example of answer string (as returned by AI) for a flyer might be:

"{ \"date\": \"August 30, 2025\", \"time\": \"6:00 PM\", \"title\": \"Community Rally\", \"description\": \"Gathering for neighborhood safety...\",
\"city\": \"Springfield\", \"state\": \"IL\", \"address\": \"123 Main St\", \"meeting_location\": \"Central Park\", \"links\": \"http://example.com/info\", \"sponsors\": \"ABC Org\", \"image\": \"IMG_2025_0830.jpg\", \"source\": \"Email Newsletter\", \"extracted_text\": \"Join us on August 30... (full text) ...\" }"

(Note: It’s a JSON string inside a string in the output JSON, which is why it has escaped quotes. The Apps Script will parse it as text and then extract the JSON content.)

Google Sheets Data Structure

RawData Sheet: Each row in RawData represents one email (one potential event submission). Columns (as mentioned) are:

  1. Date – The date/time the email was received (the script uses message.getDate() which is the timestamp of the email) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). This is inserted as a Date object in the sheet (which Google Sheets will format; you can format the column as desired).
  2. UUID – A unique identifier generated for that email using Utilities.getUuid() (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). This is used to track the submission through the pipeline and match results.
  3. Subject – Email subject line (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub).
  4. Body – The plain text body of the email (message.getPlainBody()) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). (The script does not include HTML content; if your flyers come with HTML emails, you might adjust to use getBody() and strip HTML as needed. Plain body is usually fine for extracting URLs.)
  5. Links – Any URLs found in the email body. The script uses a regex to find http:// or https:// links and joins them into a comma-separated string (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub).
  6. Image URLs – URLs to the images saved in Drive. After saving attachments, the script gets each file’s URL via file.getUrl() (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). These are the standard Drive file share URLs (which will only work for authorized users – mainly for reference).
  7. Image IDs – The Google Drive file IDs of the saved images (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). These are what we send to Runpod for downloading the images. If multiple images, they are stored as a comma-separated list in this cell.
  8. Processed – A flag indicating processing status. Initially the script writes "false" (as text) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). Later, upon completion, this will be set to "true" or "failed":

The RawData sheet allows you to see what was extracted from the email and serves as a log of all incoming flyers. If you re-run the script on an email, it would generate a new UUID and new row (the script doesn’t currently prevent duplicate processing except by marking them read and moved to trash).

Jobs Sheet: Each row in Jobs sheet logs a Runpod job submission. Columns:

  1. Timestamp – When the job was created (the script uses new Date() at job submission time) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). This gets updated to a new timestamp on each poll attempt to record last check time (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub).
  2. JobID – The Runpod job ID returned by the API (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). This is used by the script to poll for status.
  3. Status – Can be PENDING, RUNNING, COMPLETED, or FAILED.
  4. PollAttempts – Number of times we have polled the status for this job (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). Starts at 0 when job is created, then increments on each poll where the job is still running. This helps implement backoff and can be used to troubleshoot if a job took many polls.
  5. NextPollMins – The current waiting interval before next poll (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). Initialized to 5 minutes for a new job (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). After first poll, it might reduce to 1 (for quick check) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub), and then double each time up to 10 minutes max (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). This dynamic scheduling is to balance not overloading the API with too frequent requests vs. getting results promptly.
  6. Submissions – A JSON string of the submissions that were sent in this job (essentially the same as the payload we POSTed, but stringified) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). It includes all the submissionId and imageIds that were processed together. This is used by finalizeJobResults and markSubmissionsAsFailed to know which RawData entries correspond to the job results (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). Storing this also allows post-mortem debugging: you can see which items were in a job if something went wrong.

The Jobs sheet is mainly for internal tracking and debugging; end users might not need to look at it often, but it’s useful for understanding system behavior.

Processed Sheet: Each row is an extracted event. Columns (as noted in Setup) are:

  • UUID, Date, Time, Title, Description, City, State, Address, Meeting Location, Links, Sponsors, Image, Source, Extracted Text.

This sheet is the final output that one would use. For example, you might share this spreadsheet with others or use it as a data source for an event calendar. Because each event’s details are structured in separate columns, it’s easy to filter or search (e.g., filter by City or Date).

The script appends to this sheet in the same order for consistency (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). If any of the fields are missing in the flyer, they will just be blank in that cell. You might see blank City/State if the flyer didn’t include location, etc., or blank Links if none were found.

One additional note: The Image column in Processed is meant to hold some identifier of the source image. In the current implementation, the AI model is prompted to include an "image" field in its JSON output (maybe the file name or a reference). Since the model has access to the image, it could output something like the original file name (if it was in the prompt) or some label. However, in our current prompt, we don’t explicitly provide the file name to the model, so it might not populate the "image" field meaningfully (it could even echo a part of text mistakenly thinking it’s an image name). This field can be used or ignored depending on needs. The Source field is similarly optional; you might instruct the model to fill it with the email sender or subject if it adds value, but currently it might be blank or a static value from the prompt context.

Function Breakdown (Apps Script)

Here’s a closer look at key functions in googleAppsScript.gs and their roles:

AI Model and Prompt Details (Python App)

For those interested in the AI side:

Data Storage and Format Summary

  • Where emails go: They are not stored permanently in Gmail – the script trashes them after processing (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). The canonical storage of the info is the RawData sheet and the image files in Drive.
  • Where images go: Stored in the specified Drive folder. They remain there even after processing. Over time, you might clean up old images manually if they are large or not needed, but note that the Processed sheet only has the extracted text; if you ever need to refer back to the actual flyer, the image file will be in Drive (with the ID recorded in RawData). You could enhance the script to delete images after processing if storage is a concern.
  • Where parsed data goes: In the Processed spreadsheet. Consider protecting this sheet or using data validation if multiple people are collaborating, to avoid accidental edits to parsed data.

By understanding the above, developers can modify the behavior (e.g., change the JSON fields to extract, use a different ML model or service, adjust polling frequency, etc.) with knowledge of how parts connect.

Development & Contribution

If you wish to contribute to the project or modify it for your needs, here are some guidelines and tips:

Project Structure & Codebase:

  • The repository is structured to separate the Google Apps Script code and the Runpod service code. The key files are:
    • googleAppsScript.gs – the Google Apps Script code (JavaScript syntax, runs in Apps Script environment).
    • app.py – the Python server that runs on Runpod (or any similar platform).
    • prompt.txt – the system prompt text for the AI model.
    • examples/ – directory containing example images and an examples.json file with few-shot example data.
    • requirements.txt and requirements-cuda.txt – Python dependencies (the latter including CUDA-specific ones for GPU).
    • Dockerfile – to containerize the Python app for deployment.
    • Makefile – helper commands for building/pushing the Docker image.
    • LICENSE – MIT license file.
    • README.md – documentation (to be updated with any changes you make).

When working on the code, ensure changes to one side (Apps Script or Python) remain compatible with the other. For example, if you alter the JSON structure the model outputs, update finalizeJobResults accordingly.

Setting up a Dev Environment (Python): You can run app.py locally on a machine with a GPU for testing. Ensure you have Python 3.10+, PyTorch, and the other packages installed. Set the environment variable GOOGLE_SERVICE_ACCOUNT_KEY in your shell to the service account JSON content (or you can modify the code to load from a file in dev). Since the Runpod serverless expects handle_inference to be called via their system, you can simulate this by manually calling handle_inference:

import json
from app import handle_inference

# Load a sample payload
event = {"input": {"submissions": [
    {"submissionId": "test-uuid-1234", "imageIds": ["<your test image file ID>"]}
]}}
result = handle_inference(event)
print(json.dumps(result, indent=2))

Make sure to set up credentials and have a test image in the Drive folder accessible. This will let you iterate on the prompt or model and see results immediately, rather than going through the full Gmail->Apps Script loop.

Testing the Apps Script: You can simulate parts of it by creating dummy entries in RawData and calling launchRunPodJobs() from the script editor (with a valid Runpod endpoint configured). Also, you can use the Logger logs to debug. In the script editor, go to Executions to see logs of each run or use Logger.log statements (which we have plenty of) to trace values. For example, if a submission isn’t getting picked up, check the RawData flags; the logs will show how many were found (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub).

Debugging Common Issues:

Extending the Project:

  • Multiple Accounts or Sources: If you want to process flyers from multiple email accounts, you could set up multiple instances of the script (or have one script account with delegation to other inboxes, though GmailApp doesn’t directly support multi-account). Easiest would be separate Projects per account, or have all mails forwarded to one account.
  • Alternate OCR Approach: You could replace the Runpod LLM approach with an API like Google Vision for text extraction and then use a smaller local model (or even rule-based parsing) for pulling structured data from text. This might reduce complexity/cost. The current setup is a demonstration of an LLM doing end-to-end parsing, which is flexible (understands context) but resource-intensive.
  • Model Fine-tuning: If the extraction accuracy isn’t great, you could fine-tune a model specifically on event flyers, or use a different pre-trained model (like Donut by Naver for documents, or an instruction-tuned multimodal model). If you switch the model, update MODEL_NAME and possibly the prompt/examples to match it. Ensure the output JSON keys remain the same or adjust the Apps Script accordingly.
  • Error Notifications: For a production use, you might want the script to notify you (e.g., send an email or Slack message) if a job fails or if an email couldn’t be processed. Currently, it just marks failures in the sheet. Adding a notification in markSubmissionsAsFailed or when marking row failed could be useful if failure is rare and you want to know when to intervene.
  • UI or Manual Trigger: You could integrate this with a Google Sheets custom menu to allow manual triggers (e.g., a "Process Now" button that calls these functions). Given it’s automated, that’s optional.

Contributing Guidelines:

  • Before making a pull request, test your changes end-to-end with at least one sample email and flyer to ensure nothing breaks.
  • Keep code style consistent. The Apps Script code uses a lot of logging and clear variable names for readability; please maintain that for any new code. The Python code is structured to initialize everything each invocation (since it may not run as a persistent server between jobs on Runpod’s serverless). If you modify it to keep the model in memory between calls (e.g., using a global or caching mechanism), ensure that it still works in the serverless context (Runpod might reuse the container for a short time, but it’s not guaranteed for subsequent jobs).
  • Document any changes in this README. The documentation is crucial for others (and your future self) to understand the system.
  • If you find and fix a bug, describe the root cause and solution clearly in your commit and consider adding comments in code to prevent regressions.

We welcome contributions that improve accuracy, performance, or usability. For example, better prompt tuning, support for more fields, or integration with calendar APIs to automatically create events from the parsed data would all be interesting enhancements.

If you encounter issues, you can use the GitHub issue tracker of the repository to report them. When reporting, include relevant log excerpts (if possible) or a description of the scenario (perhaps anonymize any personal data).

Cloud Configuration Details

This project spans multiple cloud components. Below is a summary of configuration details and environment variables across Google Apps Script and Runpod, and how to properly set them up:

Google Apps Script (GAS) Project Properties:

Upon deploying the Apps Script, five script properties must be defined so the script knows how to connect to other services:

These can be added in the Apps Script editor under Project Settings. They are retrieved in code via scriptProperties.getProperty at runtime (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub) (event-flyer-parser/googleAppsScript.gs at main · sayhiben/event-flyer-parser · GitHub). Storing them as script properties keeps them out of your code (good for security, especially for the API key).

Google Apps Script Scopes/Authorization:

The script uses the following Google services which will prompt for OAuth scopes:

When you first run the script or set up triggers, Google will prompt you to allow these scopes. You might see warnings if the app isn’t verified since it’s a custom script; you’ll have to acknowledge and allow. In a G Suite domain, an admin might need to whitelist the scopes. The scopes are standard (Gmail, Drive, Sheets).

Runpod Endpoint Environment:

On the Runpod side, the container requires configuration to access Google Drive and load the model:

Google Apps Script Properties vs. Hardcoded values:

The design keeps all deploy-specific IDs/URLs in properties for flexibility. For example, if you duplicate this setup in another account, you only need to change the properties, not the code. Similarly, on the Python side, the service account and any model details are via environment or constants at the top for easy modification.

Security and Access:

  • The service account JSON is sensitive. In Runpod, treat it as a secret. Do not expose it in logs (the code doesn’t log it, and you shouldn’t print it).
  • The Runpod API key is also sensitive; keep it in Script Properties (which are not visible in the code). Google Apps Script project properties are not publicly visible unless someone has edit access to the script project.
  • The data in Google Sheets and images in Drive are as secure as your Google account. Since we move emails to trash, the source email isn’t easily accessible (until trash is emptied after 30 days), which might be considered a feature or a drawback. You might choose to archive instead of trash if you want to keep the original emails.
  • If multiple people need to use the system, consider using a shared Google account or a Google Workspace service account approach (Gmail API via service accounts is complicated for consumer Gmail, so likely a user-bound script is simplest).
  • Ensure the service account’s Drive API access is limited to only what it needs (readonly on that specific folder). By sharing just the one folder with the service account, you prevent it from accessing other files in your Drive.

Environment Variables Summary (for reference):

  • Google Apps Script: INBOX_SPREADSHEET_ID, PROCESSED_SPREADSHEET_ID, DRIVE_FOLDER_ID, RUNPOD_ENDPOINT_URL, RUNPOD_API_KEY.
  • Runpod/Python: GOOGLE_SERVICE_ACCOUNT_KEY (plus optionally things like MODEL_NAME if you wanted to override it, or HF API token if using a private model; none required for default use).

Each time you update something like the Runpod endpoint (e.g., deploying a new container with changes), you might get a new endpoint URL or ID. Update the script property RUNPOD_ENDPOINT_URL accordingly. The note in the initial README draft “When you get a new deployed SHA, update the runpod endpoint settings” likely refers to this – if you push a new Docker image and update the endpoint to that image, the endpoint ID/URL stays the same, but if you instead create a new endpoint, you’ll have a new URL that the script needs to know.

Logging and Monitoring:

  • The Apps Script logs (accessible in Script Dashboard or Stackdriver logging) will show events of the script. You might integrate Stackdriver alerts if this were a long-running project (e.g., alert if an error message appears frequently).
  • Runpod allows you to see logs per job or overall. It might be useful to monitor memory usage there. If memory is an issue, consider reducing MAX_DIM in the Python (currently 1280, which resizes images to at most 1280px in largest dimension to limit size) (event-flyer-parser/app.py at main · sayhiben/event-flyer-parser · GitHub).

Cleaning Up:

If you want to disable the system, remove the triggers in Apps Script (so it stops processing new emails) and possibly stop the Runpod endpoint (to avoid accidental charges). Data already in Sheets/Drive will remain until you delete it manually.

License: This project is released under the MIT License (GitHub - sayhiben/event-flyer-parser: Parses events described in images and posts the event details to a Google Sheet), which means you are free to use, modify, and distribute it. Please give credit in your project repo or mention if you build upon it, so others know where it originated. Happy parsing!

About

The Big List of Protests - An AI-assisted Protest Flyer parser and event aggregator

theblop.org/

Topics

react ocr ai computer-vision democracy activism google-apps-script protest protesting protest-events protests runpod-serverless

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Contributors 2

  •  
  •  

Languages