feat: Autonomous mode #2435

SmartManoj · 2024-06-14T11:31:28Z

Solves #1798.

For Future PRs: continue the previous task

yufansong

yufansong · 2024-06-15T13:52:39Z

agenthub/codeact_agent/codeact_agent.py

+        if config.agent.is_autonomous:
+            messages.append(
+                {'role': 'agent', 'content': f'Iteration {state.iteration}'}
+            )


Why add this

Without this, the agent repeatedly gives step 1 for the given task. Renaming Iteration to step would be appropriate?

Also, it gives a hint about how many steps it has tried.

I don't understand it either. Doesn't look right to me...

I don't think role == 'agent' is a supported behavior of litellm, it should be user message? Did you test the code?

Litellm doesn't check that but openai does. Ran with a custom model.

agenthub/codeact_agent/action_parser.py

xingyaoww

I'm a bit hestitant to merge this as is, unless we figure out a way to add a checkbox on the frontend that allows the user to select [autonomous mode].

I think a lot of OpenDevin's helpfulness comes from the ability to request inputs from the user, and i think it is probably not a good idea to completely remove this ability just to implement autonomous mode.

xingyaoww · 2024-06-16T04:37:57Z

agenthub/codeact_agent/codeact_agent.py

+        if config.agent.is_autonomous:
+            messages.append(
+                {'role': 'agent', 'content': f'Iteration {state.iteration}'}
+            )


I don't think role == 'agent' is a supported behavior of litellm, it should be user message? Did you test the code?

tobitege · 2024-06-16T04:43:12Z

I'm a bit hestitant to merge this as is, unless we figure out a way to add a checkbox on the frontend that allows the user to select [autonomous mode].

I think a lot of OpenDevin's helpfulness comes from the ability to request inputs from the user, and i think it is probably not a good idea to completely remove this ability just to implement autonomous mode.

I'd rather have an instantly working Stop frontend button that can interrupt even a running task, without waiting for the current one to finish. Let the user control. This PR's behavior seems highly LLM dependent.

xingyaoww · 2024-06-16T04:51:26Z

I'd rather have an instantly working Stop frontend button that can interrupt even a running task, without waiting for the current one to finish. Let the user control.

OHHH yes! This is what i want too -- something like ctrl+C. This + Autonomous mode should be pretty good for most stuff.

But i still think when appropriate, the agent should still require input from the user. Otherwise, you'd need to educate the user first to "know there's a stop button that can do XYZ" which can increase the entry barrier.

tobitege · 2024-06-16T04:55:41Z

Is this PR just for a specific edge-case and does it warrant patching around the core that make it less maintainable?

SmartManoj · 2024-06-16T05:08:01Z

i think it is probably not a good idea to completely remove this ability

@xingyaoww, it is still a config.

xingyaoww · 2024-06-16T05:24:01Z

it is still a config.

If you don't teach the user to tweak the config, there's just no effect from a user's perspective -- we are writing code that's not actually useful to most users since most of them will start OpenDevin using docker (and by default, without autonomous mode).
If the user configures the autonomous mode -- We throw away the ability to await user input, which is one of our core features. Moreover, we currently rely on some hacks to implement the autonomous mode, which creates a future maintenance burden.

Therefore, I don't think it is good to merge unless we can solve (1) in a meaningful way - bc if (1) is not solved, user experience is not gonna change, and it basically means we are adding code that is not actively used by people.

SmartManoj · 2024-06-16T05:32:32Z

It's ~~currently for advanced users and~~ completely solves #1586

neubig · 2024-07-03T04:12:26Z

Because there has been no response to @mamoodi 's comments above, I am going to close this PR, but please feel free to explain and reopen if you think otherwise.

SmartManoj · 2024-07-03T07:14:28Z

I have already discussed this with @mamoodi in Slack as the PR has been waiting for the reviewers.

And @xingyaoww also requested a review from @rbren

Finally, there is a commit and a review request after the comment too.

SmartManoj · 2024-07-03T07:17:55Z

based on the comments by @xingyaoww seems like this might not be feasible as is.

@mamoodi There are some commits and a request review after that.

xingyaoww

Sorry it takes so long for review --Just saw it!

I think this version looks good to me, just some nits, and I'll need to locally test this before approve it.

@SmartManoj can you share a screenshot of the modified frontend so reviewer can see its effect?

frontend/src/components/chat/ChatInterface.tsx

SmartManoj · 2024-07-03T15:00:13Z

can you share a screenshot of the modified frontend so reviewer can see its effect?

Already added in the description.

frontend/src/components/chat/ChatInterface.tsx

Co-authored-by: Xingyao Wang <xingyao6@illinois.edu>

This reverts commit 134ce6f.

xingyaoww · 2024-07-03T15:46:07Z

The tooltip is a bit laggy in this PR.
I suggested a change by re-using the Tooltip component we are using for the pause and stop button: SmartManoj#1

xingyaoww · 2024-07-03T15:48:30Z

My input is: "Can you build a website for me?"

Then i hit with the following errors:

Why it seems the "autonomous" message being concatenated to the messages from the agent?

xingyaoww · 2024-07-03T15:50:11Z

agenthub/codeact_agent/codeact_agent.py

+            )
+            action.content += user_msg  # type: ignore
+            action.wait_for_response = False  # type: ignore
+        return action


I think you probably don't need to change these on the backend, since you are already dispatching user messages on the FE?

Not dispatched on the front end. Changed here and parsed in FE.

Can you remove changes from the backend so that it won't affect other things (e.g., eval), and move this message to dispatched by the frontend?

Ideally, the "auto mode" should just be a way for the user to say "i want to keep clicking the [continue] button when it comes up"

eval is not affected by this.

xingyaoww · 2024-07-03T15:51:54Z

opendevin/core/config.py

    """

    name: str = 'CodeActAgent'
    memory_enabled: bool = False
    memory_max_threads: int = 2
+    is_autonomous: bool = False


And why do we need this? I think is_autonoumous can be entirely handled by the FE - no?
e.g., when the autonomous mode is enabled, whenever the agent is requesting user inputs, the FE will always send back a pre-defined user message, asking the agent to "continue". We can probably re-use the continue button logic - the "checkbox" will just work by keep clicking the "continue" button for the user.

Removed the config and added it as an attribute to Agent. The Agent should behave differently when autonomous mode is enabled like when to detect the loop.

Why we need to change the backend AND the agent for the autonomous mode?

ref: #2435 (comment)

for different prompts?

what do you mean by "for different prompts"? can you elaborate?

for different system prompts.

why? I don't want us to add unnecessary additional complexity to the repo unless there are a few specific examples showing the necessity. The "continue" button on FE does not need to be modified on the backend; why does this need to modify BE?

Why did you add "⚠️ Use with caution!" in the tooltip?

SmartManoj · 2024-07-04T14:56:18Z

Implemented this in the frontend itself in #2782

Have any thoughts that the agent should be aware of the autonomous mode in the future?

xingyaoww · 2024-07-04T14:57:51Z

Close this in favor of #2782.

@SmartManoj I think we might need the agent to be aware of this, but i don't yet see a clear reason / motivating example of doing so.

SmartManoj added 5 commits June 14, 2024 16:29

Autonomous mode

be190ae

Prettified

bb8ca60

Merge branch 'main' into auto

1910a8b

linted

c35b62f

Merge branch 'auto' of /~https://github.com/SmartManoj/Kevin into auto

fdf7a7b

yufansong requested changes Jun 15, 2024

View reviewed changes

li-boxuan reviewed Jun 15, 2024

View reviewed changes

agenthub/codeact_agent/action_parser.py Outdated Show resolved Hide resolved

xingyaoww requested changes Jun 16, 2024

View reviewed changes

Merge branch 'main' into auto

793405d

xingyaoww requested a review from rbren June 16, 2024 06:52

SmartManoj added 3 commits June 16, 2024 12:32

reworded

40a5396

Added checkbox for Auto Mode

b789a9a

prettified

493ad9f

SmartManoj requested a review from xingyaoww June 16, 2024 12:25

Merge branch 'main' into auto

37d3867

neubig assigned SmartManoj Jun 16, 2024

SmartManoj requested a review from yufansong June 17, 2024 09:28

neubig assigned yufansong and xingyaoww and unassigned SmartManoj Jun 20, 2024

SmartManoj mentioned this pull request Jun 22, 2024

Added Sound Notification 🎵 #2203

Merged

Merge branch 'main' into auto

79efd2f

SmartManoj requested review from enyst and neubig June 24, 2024 03:05

neubig closed this Jul 3, 2024

SmartManoj removed their assignment Jul 3, 2024

xingyaoww reviewed Jul 3, 2024

View reviewed changes

frontend/src/components/chat/ChatInterface.tsx Outdated Show resolved Hide resolved

xingyaoww reopened this Jul 3, 2024

SmartManoj added 2 commits July 3, 2024 20:36

apply suggestion from review

42014f0

Merge branch 'main' into auto

ab12e9f

xingyaoww reviewed Jul 3, 2024

View reviewed changes

frontend/src/components/chat/ChatInterface.tsx Outdated Show resolved Hide resolved

SmartManoj and others added 4 commits July 3, 2024 20:58

Update frontend/src/components/chat/ChatInterface.tsx

134ce6f

Co-authored-by: Xingyao Wang <xingyao6@illinois.edu>

Revert "Update frontend/src/components/chat/ChatInterface.tsx"

f7cb780

This reverts commit 134ce6f.

Update ChatInterface.tsx

7f63176

improve tooltip latency

a3abf5d

check action type

d0bb5d7

xingyaoww reviewed Jul 3, 2024

View reviewed changes

SmartManoj added 4 commits July 3, 2024 21:27

remove is_autonomous config

71df703

Merge branch 'auto' into xw/improve-tooltip

1ea5520

Merge pull request #1 from xingyaoww/xw/improve-tooltip

f52fe36

Merge branch 'main' into auto

dbae7e0

SmartManoj marked this pull request as draft July 3, 2024 16:20

SmartManoj mentioned this pull request Jul 4, 2024

feat: Add Auto Mode #2782

Closed

xingyaoww closed this Jul 4, 2024

SmartManoj mentioned this pull request Aug 2, 2024

Feat: Add auto mode SmartManoj/Kevin#15

Merged

feat: Autonomous mode #2435

feat: Autonomous mode #2435

Conversation

SmartManoj commented Jun 14, 2024 • edited Loading

yufansong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SmartManoj Jun 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SmartManoj Jun 16, 2024 • edited Loading

Choose a reason for hiding this comment

xingyaoww left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tobitege commented Jun 16, 2024 • edited Loading

xingyaoww commented Jun 16, 2024 • edited Loading

tobitege commented Jun 16, 2024

SmartManoj commented Jun 16, 2024

xingyaoww commented Jun 16, 2024 • edited Loading

SmartManoj commented Jun 16, 2024 • edited Loading

neubig commented Jul 3, 2024

SmartManoj commented Jul 3, 2024 • edited Loading

SmartManoj commented Jul 3, 2024 • edited Loading

xingyaoww left a comment

Choose a reason for hiding this comment

SmartManoj commented Jul 3, 2024 • edited Loading

xingyaoww commented Jul 3, 2024

xingyaoww commented Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

SmartManoj Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xingyaoww Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SmartManoj commented Jul 4, 2024

xingyaoww commented Jul 4, 2024

SmartManoj commented Jun 14, 2024 •

edited

Loading

SmartManoj Jun 15, 2024 •

edited

Loading

SmartManoj Jun 16, 2024 •

edited

Loading

tobitege commented Jun 16, 2024 •

edited

Loading

xingyaoww commented Jun 16, 2024 •

edited

Loading

xingyaoww commented Jun 16, 2024 •

edited

Loading

SmartManoj commented Jun 16, 2024 •

edited

Loading

SmartManoj commented Jul 3, 2024 •

edited

Loading

SmartManoj commented Jul 3, 2024 •

edited

Loading

SmartManoj commented Jul 3, 2024 •

edited

Loading

xingyaoww commented Jul 3, 2024 •

edited

Loading

SmartManoj Jul 3, 2024 •

edited

Loading

xingyaoww Jul 3, 2024 •

edited

Loading