From ae9f288ebf60cdfb75d43e45cdbaa42041c76082 Mon Sep 17 00:00:00 2001 From: kati3-design Date: Fri, 20 Jun 2025 15:42:11 -0400 Subject: [PATCH] Update prompting-guide.mdx (#2758) Co-authored-by: Shuchang Zheng --- fern/docs.yml | 2 +- fern/getting-started/prompting-guide.mdx | 76 ++++++++++++++++++++++-- 2 files changed, 72 insertions(+), 6 deletions(-) diff --git a/fern/docs.yml b/fern/docs.yml index 9d3acbc9..d02670c0 100644 --- a/fern/docs.yml +++ b/fern/docs.yml @@ -70,7 +70,7 @@ navigation: path: getting-started/quickstart.mdx - page: Skyvern In Action path: getting-started/skyvern-in-action.mdx - - page: Task Prompting Guide + - page: Prompting and Troubleshooting Guide path: getting-started/prompting-guide.mdx - section: Tasks contents: diff --git a/fern/getting-started/prompting-guide.mdx b/fern/getting-started/prompting-guide.mdx index 6e42af78..8d947c6c 100644 --- a/fern/getting-started/prompting-guide.mdx +++ b/fern/getting-started/prompting-guide.mdx @@ -1,5 +1,5 @@ --- -title: Task Prompting Guide +title: Prompting and Troubleshooting Guide slug: getting-started/prompting-guide --- @@ -8,11 +8,11 @@ slug: getting-started/prompting-guide ## Choose the Best Block Type -- Most deterministic: Action block - - single action like click, input text, etc. -- Happy medium: Navigation block +- **Most deterministic:** Action block + - single action like click, input text, etc +- **Happy medium:** Navigation block - single goal, like your goal is to fill out the form -- Most flexible: Navigation V2 block +- **Most flexible:** Navigation V2 block - multi-goal, like your goal is to login, then fill the form, etc. @@ -42,6 +42,8 @@ slug: getting-started/prompting-guide - **Make an action list unless absolutely necessary.** If needed as extra details, make sure it’s accompanied by a main goal - **Name the HTML element for items.** Some HTML element items are accessible, but not all. Wouldn’t recommend this as a first course of action! +
+ ### Examples **Example Navigation/Task Block prompt:** @@ -64,3 +66,67 @@ Now that you have the location, your goal is to go to google maps and extract th **Example Action Block prompt:** Click the continue button + +
+ +# Workflow Troubleshooting Guide + +## Process + +Sometimes we test things and they run perfectly. Huzzah! Other times, issues crop up. The process of overcoming these issues by identifying errors is **troubleshooting** + +There are a two main reasons why a workflow might fail + +- prompt issues/configuration issues +- bugs + +
+ +## When you get a Failure or Termination + +### When you need to make a prompt change + +Common signs that indicate a prompt change + +- **Completed too soon:** the task wasn't finished, but the agent thought it was finished and moved on to the next block too early +- **Didn't complete soon enough:** the task finished but the agent didn't realize it was finished and kept taking actions on that block +- **Payload issue:** The information provided may be incorrectly structured or incomplete. This could mean: + - The format needs to change: JSON parameter, string parameter, breaking a parameter into smaller parameters + - Content is missing or invalid: check the payload to see if the parameter is missing or mislabeled + - Parameters might not be configured properly: does each parameter have a double bracket {{sample_parameter}}? Does the label in the bracket match the label in the parameters tab? + +### When you’re unsure + +What do you do when you’re unsure: + +1. Identify where the problem is happening + - **Check the screenshots with every step:** is everything happening as you expect it to? + - **Check the recording:** is there something missing in the screenshots that can explain why it failed? +2. Identify the problem + - **Check the parameters of the problem task:** Are the inputs what you expected them to be? Do they match what’s in the workflow? + - **Check diagnostics for the problem task:** Is the navigation goal correct? Were elements mapped properly? Do they “exist” according to the bot? What actions in the action list did it say it took? + +### When you need to file a bug + +- Usually, you’ve tried making multiple prompt changes and the error persists +- Sometimes it’s clear from the gate that there is a bug + - Issues with common bugs like dropdowns, like checkboxes, etc. + - Action lists referencing the wrong element or bounding boxes not mapped on properly + +
+ +## When you get a Complete + +### If it ain’t broke don’t fix it… + +- First, celebrate +- Check **reliability**: run again and again, are any errors popping up on subsequent runs? Add guardrails, change payloads, refine completion criteria +- Check **accuracy**: is the content correct? Improve prompting and add validation blocks to check the content + +### …or maybe fix it + +- If you have a **content-sensitive workflow,** plan for edge cases ahead of time + - What happens if I end up on a different screen? Set termination criteria preemptively + - Add validation blocks to check a review page and make sure the content matches. Make sure to pass in the payload to check! +- **In ForLoops,** it is especially important to add guardrails since each block builds on the last + - For instance, if you are clicking into each item in a table, consider passing in termination criteria that stops the task in the case you’re not looking at the correct page. Pro Tip: set continue on failure and a first block so it continues even if it messes up one!