Threats.tex

\section{Threats to Validity}\label{threats}
As in any empirical study, there are threats to validity with our work.
We attempt to remove these threats where possible, and mitigate the effect when removal is not possible.

\paragraph{Construct Validity.}
Interview questions were open-ended and designed to elicit developer opinions about the experiences, difficulties, and perceptions of merge conflicts.
We determined particular factors and needs after concluding all interviews, and thus did not bias interview participants to only factors previously mentioned.
We created survey questions using factors found through card-based unitization.
This methodology allowed us to capture the common themes that developers experience when working with merge conflicts, but might have allowed themes specific to particular sub-groups to be unrepresented in our results.

\paragraph{Internal Validity.}
Confounding and extraneous factors can affect conclusions relating to cause and effect.
We lessen this effect by using multiple methods to triangulate our results, and compare against other datasets where appropriate.
Because we use these methods to highlight stronger answers, this also means that we may have missed subtle trends across our data that could have been visible otherwise.

\paragraph{External Validity.}
Interview results may not generalize to all developers due to a small sample size, but we reduce this effect by selecting interview participants from open- and closed-source projects, varying industries, and varying project sizes (see Table \ref{interview_demographics}).
To expand and confirm our interview results, we survey 264 developers on varying aspects to ensure our results match with trends in the larger software development community.
We do not report a response rate for our surveys, since social media and mailing lists do not allow accurate measurement of the number of individuals that read our recruitment message and did not choose to participate.