Around-backend

Geo-index based social network backend using Golang
Frontend implementation uses React, please see here.

Overview of project

• Web services in Golang to handle posts, seardh and user login, logout are deployed to Google App Engine(GAE flex).
• ElasticSearch in GCE provide storage and geo-location based search for user nearby posts within a distance.
• Use Google Dataflow to dump posts from BigTable to BigQuery for offline analysis
• Use Google Cloud Storage(GCS) to store post image.
• Use OAuth 2.0 to support token based authentication.
• Use Redis(lazy-loading) to improve read performance with a little data consistency sacrifice.

Services provided and API design

• /signup
  • save to elasticSearch.
• /login
  • check login credential in elasticSearch, if correct return token
• /search - search nearby posts.
  1. have token-based authentication first.
  2. search in redis cache, if not found then search elasticSearch(lazy-loading).
  3. use "type" : "geo_point" to map (lat, lon) to geo_point, ES will use geo-indexing to search(KD tree) nearby posts.
• /post
  1. save post image in GCS.
  2. save post info in ElasticSearch, bigTable(optional).

Storage

• ElasticSearch(save user and post infos)

  • user info example

   {
      "_index" : "around",
      "_type" : "user",
      "_id" : "jack",
      "_score" : 1.0,
      "_source" : {
        "username" : "jack",
        "password" : "jack"
      }
    }

  • post info example

  {
      "_index" : "around",
      "_type" : "post",
      "_id" : "b2c32515-c07d-4154-b2b1-6c7ab5e06d42",
      "_score" : 1.0,
      "_source" : {
        "user" : "jack",
        "message" : "Nice star!",
        "location" : {
          "lat" : 44.70415541365263,
          "lon" : -78.12385288120937
        },
        "url" : "https://www.googleapis.com/download/storage/v1/b/.../6c7ab5e06d42?generation=1522902512391320&alt=media"
      }
    }
  

• Google Cloud Storage

  • elasticserach saves image url, GCS store real image file.

• Redis

  • redis can be simply regarded as key-value store
    • key: lat:lon:range, range is redius based on (lat,lon) as circle center.
    • value: post info

• BigTable, BigQuery

  • we can save posts data to BigTable, use Dataflow to pass posts from BigTable to BigQuery for data analysis. See DataFlow code here.
  • Several cases can be done in BigQuery:
    • get number of posts per user id -- base to detect spam user.
    • find all messages in LA(lat range [33, 34], lon range [-118, -117]) -- base for geo-based filter.
    • find all message with spam words

Implimentation Details

• routing and auth
  • Here use gorilla/mux for routing and dgrijalva/jwt-go for JWT(JSON Web Token) token based authentication, useful doc.
• redis cache
  • adopt lazy-loading(load DB after a cache miss) pattern.
  • 30MB RAM, 30 connections for free.
  • Sample code

References

• QuickStart for Go in GAE flex.
• Example elastic search in go.
• Example writing object to GCS.
• Example client connection to GCS.
• Example writing to BigQuery
• Example Go parseMultipart form
• uuid : each post is unique
• encoding json