Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the Quick testing framework #1240

Merged
merged 5 commits into from
Dec 1, 2021
Merged

Conversation

lawrence-forooghian
Copy link
Collaborator

@lawrence-forooghian lawrence-forooghian commented Nov 29, 2021

What does this pull request do?

Removes the Quick testing framework.

Why are we doing this?

See #1201 for details. The main two reasons are:

  • to get the ability to run individual tests using Xcode’s UI;
  • using the Apple native test framework instead of a third-party library that will be unfamiliar to many Swift developers.

How do we do it?

I wrote an automated migrator tool to do the bulk of the work here. (The code for the migrator is still rather messy. We're thinking about tidying it up to make it publicly available, but it's not there yet.)

How is this pull request structured?

Each of the steps below is a commit which I recommend reviewing individually (else you'll struggle to make any sense of this PR, I think):

  1. (Migrator tool) Move all of the spec / describe / context level local variable and function declarations into private variables at the top of the file.
  2. (Migrator tool) Convert each QuickSpec subclass into a subclass of XCTestCase. We replace the spec method declaration with a set of reusable functions and func test… instance methods.
  3. (Manual) Step 2 has left the code quite messy (see my comment on this PR about indentation) so we run SwiftFormat (a code formatting tool) on it to sort this out.
  4. (Manual) We replace the use of Quick's configuration.beforeSuite hook.
  5. (Manual) We remove Quick as a dependency of the project.

Does this change the behaviour of the tests?

For the most part, no. One of my main aims was for the migrated tests to behave exactly the same as the originals running under Quick. This turned out to be important, because we have tests that are dependent upon things like order of variable initialization and order of test execution (see my diff-level comments on this PR, and #1241).

The only difference is in the behaviour of local variables in the reusableTests functions. See this comment.

How did I verify that the behaviour is unchanged?

The migrator has a --add-logging option, which adds logging statements to all of the before/afterEach and it calls. I used the ably-cocoa-tests-linter-log-comparison tool in the migrator repo to compare the logs emitted by the pre and post migration tests, and confirmed that everything runs in the same order.

(The tool doesn't verify the order of variable initialization; that I did in a more ad-hoc way, adding a few logging statements selectively by hand.)

Notes

I've left comments on the PR for things that I wanted to point out, or which I'd like opinions on. Please ignore the "Outdated" tag for now; nothing is outdated yet 🙂

Spec/Stats.swift Outdated
}

enum TestCase_ReusableTestsTestDirection {
case should_return_a_MessageTraffic_object
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(SwiftFormat will indent these cases, don't worry.)

Spec/Stats.swift Outdated
}

func test__002__Stats__all__should_return_value_for_message_counts() {
test__Stats__all__reusableTestsTestAttribute(testCase: .should_return_value_for_message_counts)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(SwiftFormat will indent these statements, don't worry.)

expect(data.count) == 32 //32 bytes digest
expect(data.count) == 32
afterEach()
//32 bytes digest
Copy link
Collaborator Author

@lawrence-forooghian lawrence-forooghian Nov 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Observation, doesn't block review)

The migrator has moved this comment to after the afterEach call for some reason. I'll see if I can figure out why, but if it's a huge amount of work then I'll just fix this manually.

Spec/Auth.swift Outdated
}

context("with TTL") {

// TE6
Copy link
Collaborator Author

@lawrence-forooghian lawrence-forooghian Nov 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Observation, doesn't block review)

The migrator has put this comment in the wrong place for some reason. It should be above the declaration of reusableTestsTestTokenRequestFromJson. I'll see if I can figure out why, but if it's a huge amount of work then I'll just fix this manually.

This is the result of running ably/quick-to-xctest-migrator@fe10916 with
--rewrite-locals-to-globals --no-rewrite-test-code.
This is the result of running SwiftFormat 0.48.18 with --swiftversion=5
on all of the files modified by 2499cd5 and 6eea1de.
@lawrence-forooghian
Copy link
Collaborator Author

lawrence-forooghian commented Dec 1, 2021

@lukasz-szyszkowski in the end I decided to implement your suggestion of using setUp / tearDown, in the manner that I described there (convert spec-level beforeEach to setUp, and then only make explicit calls to context-level hooks). It makes the output quite a bit more concise, and since we're going to have to live with this code for a long time, it seemed worth the time. (It also had the nice side effect of fixing this comment-related issue, since the call to afterEach has now vanished.)

@maratal I made the reusableTests calls more concise by making the beforeEach and afterEach params have a default value of nil. This clears up the call sites considerably.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Investigate how we might get rid of the Quick testing framework
3 participants