AutoDroid: LLM-powered Task Automation in Android

Mobile task automation is an attractive technique that aims to enable voice-based hands-free user interaction with smartphones. However, existing approaches suffer from poor scalability due to the...

https://arxiv.org/abs/2308.15272

Relevance

• Current commercial AI assistants use developer-driven approaches with NLU modules and developer-defined functions, but face scalability challenges when supporting new tasks.

• LLMs can now utilize tools like search engines, code interpreters, and APIs.

Purpose

• an autonomous agent that can complete user-specified tasks by interacting with the smartphone

Key Problem

1. GUI Representation: Convert GUI states and actions to text format to help LLMs understand and make decisions.

2. Knowledge Integration: LLMs need domain-specific knowledge to navigate and complete tasks in complex smartphone apps.

3. Cost Optimization: Optimize LLM query efficiency to provide a responsive task automation experience.

Method

Task-oriented UI Prompting

• a GUI parsing module to convert GUI to a simplified HTML representation:automatically scrolls and records the information

• Restricting the Action Space with Selections:requirement“- id=<id number> action=<tap/input> input text=<text or N/A> (in the event of task completion, id=-1)”

Exploration-based Memory Injection

Simulated Task Generation（LLM Generte）

• UTG——UI Transition Graph

• Simulated task——function of element

Augmenting Prompts with App Memory

1. embedding model: {S}
cosine similarity between the embeddings of the simulated task S and the current task T

2. Give hints about the UI elements in {S}

3. PromptGenertor(T,UI,History)

Multi-granularity Query Optimization

1. Pruning Tokens by Merging Functionally Equivalent Elements.
• two UI elements leads to the same interface
• UI leaf nodes sharing the same interactive ancestor (button, checkbox, text field, etc.)

2. Reducing Query Times by Shortcuts and GUI Merging.
• GUI merging is to include several GUI states into one prompt if LLMs need them all to make decisions.(scroll down)
• execute simple actions directly with the help of the app memory