86 lines
8.1 KiB
Django/Jinja
86 lines
8.1 KiB
Django/Jinja
Identify actions to help user progress towards the user goal using the DOM elements given in the list and the screenshot of the website.
|
|
Include only the elements that are relevant to the user goal, without altering or imagining new elements.
|
|
Accurately interpret and understand the functional significance of SVG elements based on their shapes and context within the webpage.
|
|
Use the user details to fill in necessary values. Always satisfy required fields if the field isn't already filled in. Don't return any action for the same field, if this field is already filled in and the value is the same as the one you would have filled in.
|
|
MAKE SURE YOU OUTPUT VALID JSON. No text before or after JSON, no trailing commas, no comments (//), no unnecessary quotes, etc.
|
|
Each interactable element is tagged with an ID. Avoid taking action on a disabled element when there is an alternative action available.
|
|
If you see any information in red in the page screenshot, this means a condition wasn't satisfied. prioritize actions with the red information.
|
|
If you see a popup in the page screenshot, prioritize actions on the popup.
|
|
|
|
Reply in JSON format with the following keys:
|
|
{
|
|
"user_goal_stage": str, // A string to describe the reasoning whether user goal has been achieved or not.
|
|
"user_goal_achieved": bool, // True if the user goal has been completed, otherwise False.
|
|
"action_plan": str, // A string that describes the plan of actions you're going to take. Be specific and to the point. Use this as a quick summary of the actions you're going to take, and what order you're going to take them in, and how that moves you towards your overall goal. Output "COMPLETE" action in the "actions" if user_goal_achieved is True. Output "TERMINATE" action in the "actions" if your plan is to terminate the process.
|
|
"actions": array // An array of actions. Here's the format of each action:
|
|
[{
|
|
"reasoning": str, // The reasoning behind the action. This reasoning must be user information agnostic. Mention why you chose the action type, and why you chose the element id. Keep the reasoning short and to the point.
|
|
"user_detail_query": str, // Think of this value as a Jeopardy question and the intention behind the action. Ask the user for the details you need for executing this action. Ask the question even if the details are disclosed in user goal or user details. If it's a text field, ask for the text. If it's a file upload, ask for the file. If it's a dropdown, ask for the relevant information. If you are clicking on something specific, ask about what the intention is behind the click and what to click on. If you're downloading a file and you have multiple options, ask the user which one to download. Examples are: "What product ID should I input into the search bar?", "What file should I upload?", "What is the previous insurance provider of the user?", "Which invoice should I download?", "Does the user have any pets?". If the action doesn't require any user details, describe the intention behind the action.
|
|
"user_detail_answer": str, // The answer to the `user_detail_query`. The source of this answer can be user goal or user details.
|
|
"confidence_float": float, // The confidence of the action. Pick a number between 0.0 and 1.0. 0.0 means no confidence, 1.0 means full confidence
|
|
"action_type": str, // It's a string enum: "CLICK", "INPUT_TEXT", "UPLOAD_FILE", "SELECT_OPTION", "WAIT", "SOLVE_CAPTCHA", "COMPLETE", "TERMINATE". "CLICK" is an element you'd like to click. "INPUT_TEXT" is an element you'd like to input text into. "UPLOAD_FILE" is an element you'd like to upload a file into. "SELECT_OPTION" is an element you'd like to select an option from. "WAIT" action should be used if there are no actions to take and there is some indication on screen that waiting could yield more actions. "WAIT" should not be used if there are actions to take. "SOLVE_CAPTCHA" should be used if there's a captcha to solve on the screen. "COMPLETE" is used when the {{"complete criterion has been met" if complete_criterion else "user goal has been achieved"}} AND if there's any data extraction goal, you should be able to get data from the page. Never return a COMPLETE action unless the {{ "complete criterion is met" if complete_criterion else "user goal is achieved" }}. "TERMINATE" is used to terminate the whole task with a failure when it doesn't seem like the user goal can be achieved. Do not use "TERMINATE" if waiting could lead the user towards the goal. Only return "TERMINATE" if you are on a page where the user goal cannot be achieved. All other actions are ignored when "TERMINATE" is returned.
|
|
"id": str, // The id of the element to take action on. The id has to be one from the elements list
|
|
"text": str, // Text for INPUT_TEXT action only
|
|
"file_url": str, // The url of the file to upload if applicable. This field must be present for UPLOAD_FILE but can also be present for CLICK only if the click is to upload the file. It should be null otherwise.
|
|
"download": bool, // Can only be true for CLICK actions. If true, the browser will trigger a download by clicking the element. If false, the browser will click the element without triggering a download.
|
|
"option": { // The option to select for SELECT_OPTION action only. null if not SELECT_OPTION action
|
|
"label": str, // the label of the option if any. MAKE SURE YOU USE THIS LABEL TO SELECT THE OPTION. DO NOT PUT ANYTHING OTHER THAN A VALID OPTION LABEL HERE
|
|
"index": int, // the index corresponding to the option index under the select element.
|
|
"value": str // the value of the option. MAKE SURE YOU USE THIS VALUE TO SELECT THE OPTION. DO NOT PUT ANYTHING OTHER THAN A VALID OPTION VALUE HERE
|
|
},
|
|
{% if error_code_mapping_str %}
|
|
"errors": array // A list of errors. This is used to surface any errors that matches the current situation for COMPLETE and TERMINATE actions. For other actions or if no error description suits the current situation on the screenshots, return an empty list. You are allowed to return multiple errors if there are multiple errors on the page.
|
|
[{
|
|
"error_code": str, // The error code from the user's error code list
|
|
"reasoning": str, // The reasoning behind the error. Be specific, referencing any user information and their fields in your reasoning. Keep the reasoning short and to the point.
|
|
"confidence_float": float // The confidence of the error. Pick a number between 0.0 and 1.0. 0.0 means no confidence, 1.0 means full confidence
|
|
}]
|
|
{% endif %}
|
|
}],{% if verification_code_check %}
|
|
"verification_code_reasoning": str, // Let's think step by step. Describe what you see and think if there is somewhere on the current page where you must enter the verification code now for login or any verification step. Explain why you believe a verification code needs to be entered somewhere or not. Do not imagine any place to enter the code if the code has not been sent yet.
|
|
"place_to_enter_verification_code": bool // Whether there is a place on the current page to enter the verification code now. {% endif %}
|
|
}
|
|
|
|
Consider the action history from the last step and the screenshot together, if actions from the last step don't yield positive impact, try other actions or other action combinations.
|
|
Action history from previous steps: (note: even if the action history suggests goal is achieved, check the screenshot and the DOM elements to make sure the goal is achieved)
|
|
```
|
|
{{ action_history }}
|
|
```
|
|
{% if complete_criterion %}
|
|
Complete criterion:
|
|
```
|
|
{{ complete_criterion }}
|
|
```{% endif %}
|
|
|
|
User goal:
|
|
```
|
|
{{ navigation_goal }}
|
|
```
|
|
{% if error_code_mapping_str %}
|
|
Use the error codes and their descriptions to surface user-defined errors. Do not return any error that's not defined by the user. User defined errors:
|
|
{{ error_code_mapping_str }}
|
|
{% endif %}
|
|
{% if data_extraction_goal %}
|
|
User Data Extraction Goal:
|
|
```
|
|
{{ data_extraction_goal }}
|
|
```
|
|
{% endif %}
|
|
|
|
User details:
|
|
```
|
|
{{ navigation_payload_str }}
|
|
```
|
|
|
|
Clickable elements from `{{ current_url }}`:
|
|
```
|
|
{{ elements }}
|
|
```
|
|
|
|
The URL of the page you're on right now is `{{ current_url }}`.
|
|
|
|
Current datetime, ISO format:
|
|
```
|
|
{{ local_datetime }}
|
|
```
|