Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Social competence survey for bonus points #839

Merged
merged 9 commits into from
Mar 5, 2024
14 changes: 7 additions & 7 deletions general_rules/PenaltiesBonuses.tex
Original file line number Diff line number Diff line change
Expand Up @@ -42,18 +42,18 @@ \subsection{Bonus for outstanding performance}\label{rule:outstanding_performanc
\item It is the decision of the \iaterm{Technical Committee}{TC} if (and to which degree) the bonus score is granted.
\end{enumerate}

\subsection{Bonus for perceived social intelligence of the robot}\label{rule:perceived_intelligence}
\subsection{Bonus for perceived social intelligence}\label{rule:perceived_intelligence}
\begin{enumerate}
\item For the task \iterm{Receptionist} in \iterm{Stage~I} and, \iterm{Restaurant} in \iterm{Stage~II} tests. Teams are allowed to request an assessment of the robot's performance regarding its perceived social intelligence.
\item For the test \iterm{Receptionist} in \iterm{Stage~I} and, \iterm{Restaurant} in \iterm{Stage~II} tests. Teams are allowed to request an assessment of the robot's performance regarding its perceived social intelligence.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imho should be mandatory and integrated in the scoring of the tasks - to make sure league/referee prepares volunteers.
To be useable data we have to define a minimum number of respondents,
for testing the procedure (or getting an overview of the league) in local leagues we can always opt to skip parts of a task e.g. this survey.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it has to be mandatory, but I agree, that it should be integrated in the scoring of the tasks. Since you can't lose anything anyway, any rational team should always opt for social scoring.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we can just make it mandatory

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Teams are allowed Teams are assessed


\item This bonus, ranging from 0 to 50, depends on the robot's social performance. Referees and task volunteers will assess the task's social performance using specially designed scales in a survey.
\item This bonus, ranging from 0 to 50, depends on the robot's social performance which will be assessed by Referees using a specially designed scales in a survey.

\item Every team seeking a social assessment should inform the referee before the task. that they want to be evaluated using perceived social intelligence.

\item Teams requesting a social assessment are both required and responsible for bringing a minimum of two individuals from the general public to evaluate their performance. Evaluators from the same team re not allowed.

\item Every team seeking a social assessment should inform the referee before the task. that they want to be evaluated using perceived social intelligence.

\item After the task is completed, the evaluators will fill out the form accessible via the QR code in Figure \ref{fig:qr-survey}, or through this \href{https://forms.office.com/Pages/ResponsePage.aspx?id=6sSEXw03nkuDDHVvi_G1H7VNGCdGFtZJs0ryJVVWtCFUQVFSWDlYM0FHRVA2QllIT0tOQjI2QUcxQi4u}{link}.

\item The referee has the authority to skip the social assessment test if they believe the robot's performance is not suitable for measurement.

\item The score will be automatically recorded in this \href{https://urjc-my.sharepoint.com/:x:/r/personal/juan_pena_urjc_es/Documents/ROBOCUP%20@HOME%20PSI%20SCALE%20PROPOSAL.xlsx?d=wfdc816bee34742e1a9e5bea95677985d&csf=1&web=1&e=zRwl4u}{spreadsheet}.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to be hosted somewhere under control of the league. Fallback script maybe? Link or document how to calculate score atleast.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think we need to mention how the score is calculated. Average value from all 15 ratings collected times 10, so that a maximum of 50 points can be achieved.

$$\text{Social Score} = \text{scaling} \times \frac{1}{n} \sum_{i=1}^{n} \text{Rating}_i$$

$$\text{scaling} = \frac{\text{max Social Score}}{\text{max Rating}} = 10$$

$$\text{n} = | \text{{Rating}} |$$

\end{enumerate}

Expand Down