Skip to content

v0.0.2
Compare
Choose a tag to compare
@thtmnisamnstr thtmnisamnstr released this 09 Feb 21:37
· 108 commits to main since this release
v0.0.2
159eebe
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

V0.0.2 - Current status -- what works and what doesn't

What works? 🚀

rtdl is not full-featured yet, but it is currently functional. You can use the API on port 80 to
configure streams that ingest json from an rtdl endpoint on port 8080, process them into Parquet,
and save the files to a destination configured in your stream. rtdl can write files locally, to
AWS S3, and to GCP Cloud Storage, and you can query your data with Dremio on port 9047 (login with
Username: rtdl and Password rtdl1234).

What's new? 💥

  • Switched from Apache Hive Metastore + Presto to Dremio. Dremio works for all storage types. This was
    incorrectly noted as not functioning in the original release notes.
  • Added support for using a flattened JSON object as value for gcp_json_credentials field in the
    createStream API call. Previously, you had to double-quote everything and flatten.
  • Added CONTRIBUTING.md and decided to use a DCO over a CLA - tl;dr use -s when you commit, like
    git commit -s -m "..."

What doesn't work/what's next on the roadmap? 🚴🏼

  • Add support for Azure Blob Storage
  • Add support for Segment Webhooks as a source
  • Add support for more compressions - currently default Snappy compression is supported
Assets 2
Loading