How to Choose a Conversion Rate Optimization Agency Devon Cox, President, ConversionTeam Length: 12:34 Transcript and on-screen text 00:00 How to choose a conversion rate optimization agency Choose a conversion rate optimization agency that understands how your business makes money, can explain the evidence behind its recommendations, and can get the work live. You also need a plan your own team can support. I'm Devon Cox, President of ConversionTeam. I'll explain the questions I'd use to compare agencies, what a useful answer sounds like, and the warning signs I'd pay attention to before signing. On screen: - How to choose a conversion rate optimization agency - Devon Cox, President, ConversionTeam 00:20 Which business result are we improving? Start by telling the agency what you need to improve. For an ecommerce business, that could be completed orders and the money left after fulfilling them. For a business with a sales team, more form submissions may create more work without producing more customers. Ask how the agency would connect website activity to the result your business needs. A useful answer asks about the buying process and how you measure success today. For a subscription business, that conversation may extend past signup to whether people start using the product and stay. Agree on the business outcome before comparing proposals built around different goals. On screen: - Which business result are we improving? - Ecommerce: orders and contribution - Sales-led businesses: qualified leads and customers - Subscriptions: signup, use and retention - Start with your business model 00:52 What experience carries over? Ask for an example with a similar buying decision. An identical industry can help, but ask what the agency learned that applies to you. Consider a hypothetical company selling configurable office furniture. An agency's experience helping shoppers choose complex equipment might be relevant: both buyers need to compare options and check whether the product fits. Shipping requirements and customer expectations may still differ. Have the agency explain those differences and how it would research them. A familiar logo in a proposal gives you less to evaluate than a clear explanation of the customer problem, the work involved, and what still needs to be learned. On screen: - What experience carries over? - Ask about a similar buying decision - Identify what transfers - Ask what still needs customer research - Hypothetical example: configurable office furniture 01:26 Can they work with your actual setup? Platform experience is a separate question. Tell the agency how your site is built, which systems it connects to, and who controls releases. A platform name alone leaves a lot unanswered. Ask about a comparable setup and the person who would build your tests. Have that person explain what they'd check before promising a launch date. For a sales-led site, include the form and its connection to your customer database. For a store, include product options and the purchase process. A useful answer names what they already know and what they'll need to inspect with your developers. On screen: - Can they work with your actual setup? - Discuss the setup behind the platform name - Meet the person who will build the tests - Include connected systems and release ownership 01:56 Ask them to explain an implementation Take a hypothetical Shopify store with a product-options app. You want to test a different way to present those options. Ask the agency what it would check before launch. The customer needs to select an available option, see the right price, and receive the right item in the cart. The experiment also needs to record the intended outcome. Ask how they'll check those paths on the devices your customers use, and who can turn the change off if something breaks. An agency that hasn't seen your setup can explain its inspection plan. A launch promise made before checking the dependencies gives you a reason to press for details. On screen: - Ask them to explain an implementation - Option selection, price and cart stay consistent - Check the customer paths and tracking - Name the person who can stop a broken test - Hypothetical Shopify store; no client result 02:27 What does the case study establish? Ask the agency to explain a case study beyond its headline. What changed on the site? Which outcome did they measure, and how did they reach the conclusion? If they show a conversion lift, ask for the starting rate and the resulting rate, so you can understand the size of the change. Ask whether visitors were compared in a controlled experiment or whether the result compares different time periods. Also ask who the change affected and how long the observation ran. They may need to remove client details. An anonymized explanation should still let you assess the method and the limits of the evidence. On screen: - What does the case study establish? - The change and the measured outcome - Starting conditions and comparison method - Audience, observation period and limits 02:56 Check what else changed Here's a hypothetical case-study review. An agency says orders rose after a new product page went live. During the same period, the business also ran a promotion and attracted a different mix of visitors. The result may be useful, but it leaves a question: how much of the change came from the page? Ask what evidence helps separate those explanations. If there was a controlled experiment, ask how the groups were compared and whether anything disrupted the test. If there wasn't, keep the conclusion within what the comparison can show. You can learn about the agency's reasoning even when the result has limits. On screen: - Check what else changed - Reported change - New product page - Orders rose afterward - Questions to ask - Did promotions or traffic change? - What comparison supports the conclusion? - Hypothetical evidence review; no measured result 03:27 How does that result apply to us? Then ask what the case study suggests for your business. A change that helped customers understand a complicated product may offer a useful research question for another company. It doesn't tell you the result you'll get. Ask the agency to distinguish the transferable idea from the outcome it observed for that client. Request an example that didn't produce a clear improvement, too. What did they decide afterward? That conversation shows how they handle uncertainty. You should leave knowing what the evidence supports, what remains unknown, and why the agency thinks the experience is relevant to your situation. On screen: - How does that result apply to us? - Separate the useful idea from the prior outcome - Ask for an unclear or unsuccessful result - Understand the next decision 03:59 How will they decide what to change? Before discussing a redesign or new copy, ask how the agency would investigate the problem. Analytics can help identify where customers leave or struggle. Customer research can help explain what they expected and what confused them. Ask to see the connection between a finding and a proposed change. If the proposal already prescribes the same solution for several different problems, ask what evidence could change that recommendation. A team can specialize in design or copy and still investigate carefully. The question is whether its recommendation follows the evidence and whether it can bring in the other skills the work requires. On screen: - How will they decide what to change? - Find the problem in the buying process - Connect customer evidence to the proposed change - Ask what could change the recommendation 04:32 Follow a finding to a proposed test Return to the hypothetical furniture store. Suppose customer sessions show people looking for dimensions and leaving without understanding whether a desk will fit. Support conversations raise the same question. The agency could check where that information appears, whether customers understand it, and how the issue affects different products. A proposed test might make the dimensions easier to find near the options customers are choosing. Ask why that change addresses the observed problem, what outcome it should affect, and what result would make the agency reconsider. If the actual problem is an unavailable size, rearranging the page won't create that inventory. The diagnosis changes the work. On screen: - Follow a finding to a proposed test - Evidence to inspect - Customers struggle to find dimensions - Support hears the same question - Decision to explain - Test clearer product information - Check whether availability is the real issue - Hypothetical research and test proposal 05:10 Can we trust the measurement? Ask how the agency will check tracking before relying on it. Have them explain how a completed order or successful form submission appears in the data, and how they'd investigate duplicates or missing records. For a sales-led business, agree on how website leads will be connected to lead quality and sales outcomes when that data is available. Then ask which result will guide the experiment decision, and what other outcomes could reveal a problem. A change could increase form submissions while sending the sales team more unsuitable leads. You need a measurement plan that can detect that trade-off. On screen: - Can we trust the measurement? - Check that recorded outcomes match real actions - Connect website activity to business outcomes - Watch for harmful trade-offs 05:40 What can our traffic support? Ask how your available traffic and conversion activity affect the proposed testing plan. There isn't a useful universal answer based only on total website visits. The agency needs to examine the people who can actually encounter the change, the outcome you're measuring, and the size of the difference you need to detect. Ask how it decides when a test can end and how it handles an inconclusive result. If a rarely visited page can't answer the proposed question in a practical period, discuss another research method or a different test. Necessary repairs can still be made. Their impact needs to be described according to the evidence available. On screen: - What can our traffic support? - Use traffic eligible to encounter the change - Discuss the outcome and detectable difference - Agree on stopping and inconclusive-result handling 06:12 Who owns each part of the work? Request a sample schedule with both teams' responsibilities. Ask who handles research, writing and design, and who builds and checks the changes. Then ask who puts a successful change into the permanent site. Running an experiment and maintaining the resulting experience can involve different people. In a hypothetical engagement, the agency might build the test while your developers own the permanent release. Your team may also need to approve product claims or supply information. Put those responsibilities next to the dates they affect. A proposal should make it possible to see when agency work is waiting on your company. On screen: - Who owns each part of the work? - Name the working team and client owners - Show approvals and dependencies in the schedule - Assign the permanent implementation - Responsibility example is hypothetical 06:46 What gets checked before and after launch? Ask what quality checks are included and who signs off. A useful answer describes checks relevant to your site: the customer can complete the task, the experience works on the required devices, and tracking records the intended actions. Ask what happens after launch, too. Who monitors problems? Who can stop the experiment? When a successful change becomes permanent, who checks that release and confirms the experiment has been removed? Request a sample checklist or handoff record. It should show what was checked, who checked it, and how an unresolved issue gets handled. On screen: - What gets checked before and after launch? - Agree on relevant customer and tracking checks - Name monitoring and stop responsibilities - Check the permanent release and close the experiment 07:15 How will we work together? Ask to see how the agency communicates during an ordinary week. Agree on where progress is recorded, when you'll review decisions together, and who responds when work is blocked. For example, if a test is waiting for approval of a product claim, you need to know who will request it, who on your team can answer, and what happens to the schedule while it's unresolved. Ask whether you'll work with the people you met during the sales process. Meet the person responsible for your account and discuss a current problem with them. You need to understand their answers well enough to make decisions together. On screen: - How will we work together? - Agree on updates and decision meetings - Name the owners of blockers - Meet the people responsible for the account 07:45 What should a test report tell us? Request a sample report before you sign. You should be able to identify the original question, what changed, and the outcome used to judge the test. The report should explain the result's uncertainty and any problems with the measurement or execution. Ask how losing and inconclusive tests appear alongside successful ones. Also ask for the next decision and the reason behind it. A report that's full of activity can still leave you unsure what to do. Have the person who would report to you explain one example in plain language, including what they would recommend keeping, changing or stopping. On screen: - What should a test report tell us? - Question, change and decision outcome - Uncertainty and execution problems - Successful, losing and inconclusive tests - Next decision with its reason 08:15 What happens when a test has no clear winner? Consider a hypothetical test of a shorter lead form. The report says submissions rose, but there isn't enough evidence to support the planned decision, and lead quality is still unknown. A useful follow-up would explain what can be concluded now, what data is missing, and whether more observation could answer the question under the agreed plan. It might recommend keeping the current form while the team investigates. An unsuccessful test can still inform the next decision, but the agency should be specific about what it learned. Ask how it avoids turning every unclear result into a success story or extending a test without a clear reason. On screen: - What happens when a test has no clear winner? - State what the evidence supports now - Explain missing data and the next decision - Keep unclear results visible - Hypothetical lead-form test; no client result 08:47 What will this cost both teams? Compare what the fee includes. Development, testing software and permanent implementation may be handled differently across proposals. Ask about work outside the agreed scope and how additional costs are approved. Include your team's time in the comparison. Someone may need to supply customer data, review claims, answer questions and approve releases. If your developers are already committed elsewhere, a plan that relies on them may be difficult to deliver. Ask the agency to describe the client work expected during setup and during a normal testing cycle. Then check that expectation with the people who would actually do it. On screen: - What will this cost both teams? - Compare included work and additional costs - Estimate the client time required - Check developer and approval capacity 09:20 Which assumptions drive the business case? Ask the agency to explain the economics with your inputs. For a hypothetical store, estimate how many additional orders a proposed improvement would represent over the chosen period. Multiply those orders by what the business keeps after the variable costs of fulfilling them. Then compare that additional contribution with agency fees, software, implementation costs and internal time. Show each assumption, including the improvement you haven't yet demonstrated. Revenue alone can make the opportunity look larger than the money available to pay for the work. For a sales-led business, the calculation also depends on how many additional qualified leads become customers and how long that takes. On screen: - Which assumptions drive the business case? - Potential contribution - Additional orders or customers - Contribution after variable costs - A stated time period - Total cost - Agency, software and implementation - Internal time - Release timing and delays - Hypothetical planning model; no price or result forecast 09:55 What if the assumptions change? Have the agency show a no-improvement scenario as well as the scenarios where the work helps. Also change the timing. For planning, assume a delayed permanent release postpones the expected benefit from full rollout. Some benefit may already occur among customers who see the test version. For the hypothetical furniture store, returns or delivery costs could change the contribution from additional orders. If the business case relies on repeat purchases, ask what customer history supports that assumption. You don't need a precise-looking forecast built on guesses. You need to see which assumptions could change the decision and how the team will check them. A projected return remains a planning estimate until there are results to evaluate. On screen: - What if the assumptions change? - Include a no-improvement case - Show the effect of a delayed release - Check margin and repeat-purchase assumptions - Hypothetical scenarios; no promised return 10:33 What will the agreement leave us with? Read the scope and terms alongside the plan. Confirm the contract length, cancellation process, and what happens to work in progress when the engagement ends. If there's a performance guarantee, ask exactly what it measures, which conditions apply, and what happens if the condition isn't met. A guarantee needs a definition you can evaluate. Also confirm ownership of research, copy, designs and code. Ask whether you'll keep access to the experiment records and reports, and how those materials will be handed over. Another team should be able to understand what was tried and why, without having to reconstruct the engagement from memory. On screen: - What will the agreement leave us with? - Scope, term, cancellation and work in progress - Any guarantee: measure, conditions and remedy - Ownership of work and access to experiment records - A usable handoff at the end 11:06 Which answers should make us pause? Pay attention to what happens when you ask for details. Warning signs include a result promise whose conditions the agency won't explain, a prescribed solution with no connection to customer evidence, and reports that omit unsuccessful tests. An unclear handoff is another concern: nobody can tell you who will put the change live or who will own the records afterward. Ask a direct follow-up and give the team a chance to answer. If the missing detail affects whether you can evaluate or deliver the work, resolve it before signing. Judge the proposed work and the people responsible for it, including how openly they discuss limits. On screen: - Which answers should make us pause? - Promises with unexplained conditions - Solutions disconnected from customer evidence - Reports that hide unsuccessful tests - Unclear delivery or record ownership 11:38 Use the same questions with each agency For each agency on your shortlist, write down the evidence behind its answers and the questions still open. Ask each team to discuss the same current problem on your site. Compare how they investigate it, what they'd need from you, and how they explain the next decision. You can record the evidence as a relevant example, a sample document, or an unanswered question. You don't need an invented score to make that comparison useful. Before choosing, check that the working team fits the problem and that your company can supply the time and access in the proposal. Resolve the remaining scope questions in writing. On screen: - Use the same questions with each agency - Discuss the same current business problem - Record evidence and unanswered questions - Confirm team fit and your own capacity - Resolve the remaining scope questions 12:07 Discuss your CRO program with ConversionTeam At ConversionTeam, we provide CRO research and strategy, design, development, QA, and analytics. If you're considering an agency, the description links to our written agency guide and case studies, along with our website and contact page. Use the guide during your agency conversations. For a discussion with us, bring the business result you want to improve, the platform you use, and the resources your team can provide. We can discuss the work involved and whether the proposed engagement fits your business. On screen: - Discuss your CRO program with ConversionTeam - Read the written agency guide - Review ConversionTeam case studies - Contact ConversionTeam - conversionteam.com