-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
【New feature】新增黑客松小助手 #37
Changes from 9 commits
3223813
e655be8
7bb4f21
c75d620
99df700
b9e39a1
583de9f
4f6a748
069f0d7
cf3978a
ea64705
b899fcf
e1653cd
a9ea5fd
f7b6737
f9dbfcc
2f13a30
cc812c5
3fb01cb
d3883c0
da23a93
7d11255
be6ddeb
2784c76
d8ebbc9
6b055da
e6aa11b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
__pycache__ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,112 @@ | ||
# 黑客松小助手RFC | ||
|
||
## 📄 背景 | ||
|
||
💡💡💡 `Hackathon5`开赛在即,本次大赛会有上百个赛题等着大家攻克,但统计这些赛题的报名信息和当前进度是一项繁重的工作 (感谢 @花花!🍻)。 | ||
|
||
如果有一个小助手💡可以帮助我们自动统计和展示赛题进度,那自然是极好的,所以`黑客松小助手`它带着期盼走来了!🎉🎉🎉 | ||
|
||
## 🚩 功能 | ||
|
||
`黑客松小助手`目前的主要功能如下,欢迎各位大佬前来补充: | ||
|
||
* 根据issue回复自动填写报名信息,完成任务认领。 | ||
* 根据提交的PR状态,自动更新issue中表单信息,完成状态变更。 | ||
* 汇总所有的赛题,形成总体看板。 | ||
|
||
## 🚩 实现方案 | ||
|
||
### 🚀 状态 | ||
|
||
在比赛期间,我们为参加比赛的大佬们设置了`五种状态`(如下),整体状态的变更顺序为:`报名状态` -> `提交RFC` -> `完成设计文档` -> `提交PR` -> `完成任务`。 | ||
|
||
| 状态名称 | 状态标志 | 状态说明 | | ||
| :--------------: | :----------------------------------------------------------: | :-------------------------: | | ||
| 报名状态 | <img src="https://img.shields.io/badge/状态-报名-2ECC71" /> | 表明通过issue评论区进行报名 | | ||
| 提交RFC状态 | <img src="https://img.shields.io/badge/状态-提交RFC-F1C40F" /> | 表明已经提交设计方案RFC | | ||
| 完成设计文档状态 | <img src="https://img.shields.io/badge/状态-完成设计文档-3498DB" /> | 表明已经完成设计文档 | | ||
| 提交PR状态 | <img src="https://img.shields.io/badge/状态-提交PR-F39C12" /> | 表明已经提交PR | | ||
| 完成任务状态 | <img src="https://img.shields.io/badge/状态-完成任务-9B59B6" /> | 表明已经完成任务 | | ||
|
||
### 🚀 榜单设计 | ||
|
||
为了更好地展示赛题进度,我们设计了榜单。整体的榜单设计如下,比赛进行期间,小助手会自动更新`Github ID/状态/PR`栏和`完成队伍`栏信息。 | ||
|
||
![image-20230729121046527](./images/1.png) | ||
|
||
`Github ID/状态/PR`栏内容的格式为`Github ID + 状态 + Pr`,其中`PR`可以有多个,`赛题参加人数`也可以有多个,以回车`<br>`分隔。 | ||
|
||
### 🚀 报名信息监控 | ||
|
||
考虑到报名赛题不会产生任何`PR`,所以通过`评论`来实现此功能,其他状态均可以通过监控`PR`实现。 | ||
|
||
#### 实现逻辑 | ||
|
||
通过监控`issue`下的评论完成报名信息监控,实现逻辑如下: | ||
|
||
* 获取`issue`下评论。 | ||
* 从评论中抽取报名信息,比如`Github ID`、`报名赛题编号`等信息。 | ||
* 更新`issue`中个人报名信息,将个人状态更新为`报名状态`。 | ||
|
||
#### 报名格式 | ||
|
||
为了自动填写报名信息,需要在`issue`下回复报名信息,格式如下: | ||
|
||
``` | ||
【状态】: 报名 | ||
【序号】:2、3 | ||
``` | ||
|
||
> 其中`【状态】: 报名`表示当前评论是赛题`报名`,`序号`表示报名的赛题序号,多个赛题之间需要用`中文顿号、`分隔。 | ||
|
||
### 🚀 其他状态变更监控 | ||
|
||
除`报名状态`外,剩下`四种状态`的变更可以通过监控`PR`的状态来完成, 具体的实现逻辑如下: | ||
|
||
* 获取指定仓库下黑客松`开始之后`标题中包含`Hackathon No.`字样的所有`PR`,具体是`bot.py`文件的`repo_urls`变量下的所有仓库。 | ||
* 如果`PR`是`PaddlePaddle/community`仓库下的,说明该PR与设计文档有关。继续判断是否`merge`,如果未`merge`,说明状态为`提交RFC`;如果已经`merge`,说明状态为`完成设计文档`。 | ||
* 如果`PR`不是`PaddlePaddle/community`仓库下的,说明该`PR`与提交代码有关。继续判断是否`merge`,如果未`merge`,说明状态为`提交PR`;如果已经`merge`,说明状态为`完成任务`。 | ||
* 每次更新后的issue内容将会保存在`logs`文件夹下,文件名是更新日期。 | ||
|
||
> 对于人工处理的赛题,可以将赛题题号加入`utils.py`文件的`un_handle_tasks`变量中,小助手不会处理这些任务。 | ||
|
||
#### PR格式 | ||
|
||
为了完成状态变更,只需要在`PR`的标题中以`【Hackathon No.xxx】`开头即可,程序会自动提取赛题编号并更新榜单。 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 这里需要考虑的特殊情况:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 所以这个定时任务每次跑都是:
行倒是也行,就是感觉有点暴力,如果榜单能做到增量更新最好,这样也留有人工修改issue的空间 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 目前的逻辑确实是这样的,每次都是在重写榜单,如果想要增量更新,可以更新后记录更新时间,下次更新时只改变上一次更新时间之后的回复和PR。 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 我觉得不是修改查询时间来解决的,这个时间肯定是需要有overlap的,或者是全量的去查也没有问题,但是需要记录上一次的状态,和这一次做对比,然后拿到diff,更新到表单上 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 根据讨论可以先这样: 假设我们每天定时任务更新2次,0-12h更新一次,12-24h更新一次 然后计算一下API的调用频次,@gouzil 表示一个token每天最多调用60次,可能需要估算一下需要多少token 以及记得保存日志和异常情况报警 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 是不是可以考虑一下仓库的 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
|
||
### 🚀 看板功能 | ||
看板功能是将HTML转为图片存放在`./image`文件夹下,所以issue中的图片链接是需要指向这个图片的(这里存在一个图片文件放在哪里的问题)。 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 有几种解决方法
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 1、直接用GitHub上传图片的按钮功能没找到对应的api,再就是定时任务会上传很多次图片,GitHub对用户上传空间是不是也有限制呢; 更倾向于第3种解决方案~ There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. emmm, 理论上来说一般是不会给你提供任务服务器的,是否有ip这就取决于你部署在哪里了。不过反正都要上传日志,直接用仓库的 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
直接用 raw 链接应该会更方便一些 |
||
|
||
示例效果: | ||
![image](./images/board%E7%A4%BA%E4%BE%8B.jpg) | ||
|
||
感谢 [@AndSonder](/~https://github.com/AndSonder) 提供看板样式的代码🍻 | ||
|
||
## 🚩 代码结构 | ||
|
||
整体的代码文件分为两个: | ||
|
||
* `utils.py`:负责拉取评论和PR、根据评论更新状态、根据PR更新状态。 | ||
* `bot.py`:小助手整体运行逻辑。 | ||
|
||
## 🚩 运行方式 | ||
1、替换`utils.py`文件中的`Github token`;替换`bot.py`文件中的`issue_url`和`repo_urls`,代表监控的issue链接和仓库链接; | ||
|
||
2、如果不开代理,请将`utils.py`文件中`proxies`置为`None`; | ||
|
||
3、初始化`utils.py`文件中的`task_types`和`task_types`,指明每个赛题所属的赛道; | ||
> 注:如果运行在`Linux`中,需要删除`utils.py`文件`377`行`con = imgkit.config(wkhtmltoimage='D:\\Software\\wkhtmltox\\bin\\wkhtmltoimage.exe')`这句代码 | ||
|
||
4、执行如下命令,代码会每两小时更新一次issue,每次更新后的issue内容会保存在`logs`文件夹下; | ||
```shell | ||
cd HackathonBot | ||
|
||
python bot.py | ||
``` | ||
|
||
## 🚩 未来规划 | ||
|
||
* **预警功能**:自动提醒每个赛题的状态,针对长时间没有变更状态的赛题进行预警。 | ||
huajiao-hjyp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,95 @@ | ||
import time | ||
import json | ||
import schedule | ||
|
||
import utils | ||
|
||
|
||
def update_issue_automatically(): | ||
# 1. 获取issue表格 | ||
response = utils.request_get_issue(issue_url) | ||
|
||
# 从issue中提取题目列表 | ||
task_list = utils.process_issue(response['body']) | ||
|
||
# 2. 根据评论更新表格 | ||
comment_url = issue_url + '/comments' | ||
comments = utils.request_get_multi(comment_url) | ||
for comment in comments: | ||
utils.update_status_by_comment(task_list, comment) | ||
|
||
|
||
# 3. 根据PR更新表格 | ||
# - 根据提出的PR更新状态为已提交 | ||
# - 根据close的PR更新状态为已完成 | ||
|
||
# TODO:这里可以改变监测的仓库 | ||
repo_urls = ['https://api.github.com/repos/Tomoko-hjf/paddleviz/pulls'] | ||
|
||
for repo_url in repo_urls: | ||
params = { | ||
"state": "open" | ||
} | ||
pulls = utils.request_get_multi(repo_url, params) | ||
for pull in pulls: | ||
utils.update_status_by_pull(task_list, pull) | ||
|
||
params = { | ||
"state": "closed" | ||
} | ||
pulls = utils.request_get_multi(repo_url, params) | ||
for pull in pulls: | ||
if pull['merged_at']: | ||
utils.update_status_by_pull(task_list, pull) | ||
|
||
|
||
# 4. 更新榜单内容 | ||
updated_issue = response['body'] | ||
for task in task_list: | ||
num = int(task['num'].strip(' ')) | ||
start = updated_issue.find('| {} |'.format(num)) | ||
end = start + 1 | ||
while end < len(updated_issue) and updated_issue[end] != '\r' and updated_issue[end] != '\n': | ||
end += 1 | ||
|
||
# TODO:这里后期需要定制化表头 | ||
row = '| {} | {} | {} | {} | {} |'.format(num, task['difficulty'], task['issue'], task['status'], task['team']) | ||
updated_issue = f'{updated_issue[:start]}{row}{updated_issue[end:]}' | ||
|
||
# 5. 更新看板信息 | ||
utils.update_board(task_list) | ||
|
||
|
||
# 处理换行符 | ||
updated_issue = updated_issue.replace('\r', '') | ||
file_name = time.strftime('%Y-%m-%dT%H-%M-%S', time.localtime()) | ||
with open('./logs/{}.md'.format(file_name), mode='w', encoding='utf-8') as f: | ||
f.write(updated_issue) | ||
|
||
# 5. 更新 issue | ||
data = {} | ||
data['body'] = updated_issue | ||
data['title'] = response['title'] | ||
data['assignee'] = response['assignee'] | ||
data['state'] = response['state'] | ||
data['state_reason'] = response['state_reason'] | ||
data['milestone'] = response['milestone'] | ||
data['labels'] = response['labels'] | ||
|
||
res = utils.request_update_issue(issue_url, json.dumps(data)) | ||
|
||
|
||
if __name__ == '__main__': | ||
|
||
# issue链接 | ||
issue_url = 'https://api.github.com/repos/Tomoko-hjf/paddleviz/issues/1' | ||
|
||
# 运行一次查看效果 | ||
update_issue_automatically() | ||
|
||
# 每两小时运行一次 | ||
schedule.every(2).hours.do(update_issue_automatically) | ||
|
||
while True: | ||
schedule.run_pending() | ||
time.sleep(10) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
| 序号 | 难度 | 任务 ISSUE | 队伍名称/状态/PR | 完成队伍 | | ||
| :--: | :--: |:--: | :--: |:--: | | ||
| 1 | ⭐ | [新增星星任务](/~https://github.com/Tomoko-hjf/paddleviz/issues/1) | @Tomoko-hjf <img src="https://img.shields.io/badge/状态-提交PR-F39C12" /> [#2](/~https://github.com/Tomoko-hjf/paddleviz/pull/2) <br> | | | ||
| 2 | ⭐⭐ | [保护星星](/~https://github.com/Tomoko-hjf/paddleviz/issues/1) | @Tomoko-hjf <img src="https://img.shields.io/badge/状态-报名-2ECC71" /> <br> | | | ||
| 3 | ⭐⭐⭐ | [收集星星](/~https://github.com/Tomoko-hjf/paddleviz/issues/1) | @Tomoko-hjf <img src="https://img.shields.io/badge/状态-报名-2ECC71" /> <br> | | | ||
| 4 | ⭐⭐⭐⭐ | [飞桨之星](/~https://github.com/Tomoko-hjf/paddleviz/issues/1) | | | |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
| 序号 | 难度 | 任务 ISSUE | 队伍名称/状态/PR | 完成队伍 | | ||
| :--: | :--: |:--: | :--: |:--: | | ||
| 1 | ⭐ | [新增星星任务](/~https://github.com/Tomoko-hjf/paddleviz/issues/1) | @Tomoko-hjf <img src="https://img.shields.io/badge/状态-提交PR-F39C12" /> [#2](/~https://github.com/Tomoko-hjf/paddleviz/pull/2)<br>@gouzil <img src="https://img.shields.io/badge/状态-报名-2ECC71" /> <br> | | | ||
| 2 | ⭐⭐ | [保护星星](/~https://github.com/Tomoko-hjf/paddleviz/issues/1) | @Tomoko-hjf <img src="https://img.shields.io/badge/状态-报名-2ECC71" /> <br>@gouzil <img src="https://img.shields.io/badge/状态-报名-2ECC71" /> <br> | | | ||
| 3 | ⭐⭐⭐ | [收集星星](/~https://github.com/Tomoko-hjf/paddleviz/issues/1) | @Tomoko-hjf <img src="https://img.shields.io/badge/状态-报名-2ECC71" /> <br> | | | ||
| 4 | ⭐⭐⭐⭐ | [飞桨之星](/~https://github.com/Tomoko-hjf/paddleviz/issues/1) | | | |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
| 序号 | 难度 | 任务 ISSUE | 队伍名称/状态/PR | 完成队伍 | | ||
| :--: | :--: |:--: | :--: |:--: | | ||
| 1 | ⭐ | [新增星星任务](/~https://github.com/Tomoko-hjf/paddleviz/issues/1) | | | | ||
| 2 | ⭐⭐ | [保护星星](/~https://github.com/Tomoko-hjf/paddleviz/issues/1) | | | | ||
| 3 | ⭐⭐⭐ | [收集星星](/~https://github.com/Tomoko-hjf/paddleviz/issues/1) | | | | ||
| 4 | ⭐⭐⭐⭐ | [飞桨之星](/~https://github.com/Tomoko-hjf/paddleviz/issues/1) | | | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
需要考虑对不符合格式规范的回复如何处理?提供两条思路:
:
:
(btw你提供的格式里也没有区分中英文的冒号哦There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这个确实会有风险,这里我加一个判断赛题题号的逻辑吧,如果数字大于赛题数,那么就在评论区提示用户
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
建议我们在实现的啥时候,支持非中文的[]:、,符号,以及对空格的兼容,这样可能就没有找第一个数字字符的问题存在。例如代码中可以通过以下方式实现
s = '【序号】: 2, 3'
s = s.replace(' ').replace(':',':').replace(',',‘、’).replace('[','【').replace(']','】')
这样可以获取到较为纯净的字符,格式可能也较为统一
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
定时任务更新表单和一些不规范指令识别的逻辑肯定是要分开的,定时任务一天跑两次,但是报名信息格式校验应该是实时的,或者至少一小时一次,自动格式化好像也不是不行,可以调研一下技术路径,飞桨项目有一个机器人paddle-bot用于自动回复,可以看看能不能接入
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
我们最开始的时候是不是考虑过这个机器人,但是可能漏包?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
漏包的问题现在有人在看,可以先假设这个问题修复了?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
漏包的话可以用定时兜底
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
确认了排期,漏包的问题9.8可以完成修复