Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Spark 1.6's new unhandledFilters API #128

Closed
wants to merge 1 commit into from

Conversation

JoshRosen
Copy link
Contributor

Spark 1.6 extends BaseRelation with a new API which allows data sources to tell Spark which filters they handle, allowing Spark to eliminate its own defensive filtering for filters that are handled by the data source (see apache/spark#9399 for more details).

This patch implements this new API in spark-redshift and adds tests.

@JoshRosen JoshRosen added this to the 0.5.3 milestone Nov 24, 2015
@codecov-io
Copy link

Current coverage is 95.08%

Merging #128 into master will increase coverage by +0.01% as of c72ee35

@@            master    #128   diff @@
======================================
  Files           12      12       
  Stmts          508     509     +1
  Branches       123     123       
  Methods          0       0       
======================================
+ Hit            483     484     +1
  Partial          0       0       
  Missed          25      25       

Review entire Coverage Diff as of c72ee35

Powered by Codecov. Updated on successful CI builds.

@rxin
Copy link
Contributor

rxin commented Nov 24, 2015

LGTM

@JoshRosen JoshRosen closed this in f6ff246 Nov 24, 2015
@JoshRosen JoshRosen deleted the support-filter-skipping-in-spark-1.6 branch November 24, 2015 23:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants