Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bug in overwrite when using schema-qualified table names #102

Closed

Conversation

JoshRosen
Copy link
Contributor

This patch fixes #97, an issue where using SaveMode.Overwrite with schema-qualified table names (such as my_schema.my_table) would lead to "Invalid operation: syntax error at or near ".";" messages from Redshift.

The problem is that the ALTER TABLE ... RENAME command only accepts unqualified table names as the new names, so we need to strip off the schema prior to performing the renaming.

In order to keep this code clean, this patch introduces a new TableName case class which manages all of the escaping and parsing logic for table names, then uses that class to fix the Overwrite issue.

@JoshRosen
Copy link
Contributor Author

/cc @cfeduke

if (s.startsWith("\"") && s.endsWith("\"")) s.drop(1).dropRight(1) else s
def unescapeQuotes(s: String) = s.replace("\"\"", "\"")
def unescape(s: String) = unescapeQuotes(dropOuterQuotes(s))
str.split('.').toSeq match {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Argh, it turns out that it's legal for . to appear inside of delimited identifiers, so I'll have to do a quote-aware splitting here.

@JoshRosen
Copy link
Contributor Author

My experience in working with a wrapper for TableName makes me want to use similar wrappers to automatically handle column name quoting in the other parts of spark-redshift. I'll leave that refactoring to future work, though.

@codecov-io
Copy link

Current coverage is 94.73%

Merging #102 into master will decrease coverage by -0.08% as of a9b41a1

@@            master    #102   diff @@
======================================
  Files           11      12     +1
  Stmts          444     475    +31
  Branches       109     116     +7
  Methods          0       0       
======================================
+ Hit            421     450    +29
  Partial          0       0       
- Missed          23      25     +2

Review entire Coverage Diff as of a9b41a1

Powered by Codecov. Updated on successful CI builds.

@marmbrus
Copy link
Contributor

marmbrus commented Oct 5, 2015

LGTM

@JoshRosen JoshRosen closed this in c631e1c Oct 5, 2015
@JoshRosen JoshRosen deleted the fix-overwrite-with-schema-qualified-table-names branch October 5, 2015 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Amazon](500310) Invalid operation: syntax error at or near ".";
3 participants