Dec 2024

https://maven.com/parlance-labs/gen-ai-mastery-for-busy-executives  Although below is a copy-paste from daily e-mails. https://hamel.kit.com/ai-for-execs. Greg Ceccarelli and Hamel Husain.

Day 1 - Welcome to AI Mastery for Executives!

There's a significant disconnect between those building AI products and the executives making decisions about them– leading to misguided strategies and wasted resources.

For non-practitioners, navigating the AI space is overwhelming. There's a sea of advice on how to create effective AI products, but much of it comes from sources that aren't hands-on with the technology. Without practical experience, it's challenging to separate fact from fiction and make informed decisions

On the flip side, AI practitioners who are in the trenches often struggle to communicate their knowledge in a way that resonates with executives. They tend to think and speak in technical terms, which can be off-putting or confusing for those without a deep technical background.

This is where we come in.

Who We Are

I'm Hamel Husain, an AI practitioner deeply involved in the technical aspects of artificial intelligence. My partner is Greg Ceccarelli, who brings extensive executive leadership experience as a Director and Chief Product Officer at large organizations like Pluralsight and GitHub.

With the combination of my hands-on AI experience and Greg's executive leadership background, you can be sure that everything you read from us is both technically accurate and accessible to executives like you.

Over the next week, you'll receive a series of emails designed to equip you with practical and battle-tested actionable insights on unlocking the true potential of AI to create competitive advantage.

What to Expect

Here's the breakdown of what to expect from this email course:

Why most AI advice is misleading

The counterintuitive approach to AI strategy

Communicating your AI strategy effectively

Mobilizing your organization for AI success

The number one mistake executives make with AI

Hiring the right talent at the right time

Uncovering critical AI flaws

Selecting the right tools

Each email will be concise, yet packed with insights you can put into action right away.

We're committed to delivering high-value content that respects your time. With our help, you'll be well on your way to:

Distinguish between industry hype and reality

Develop an AI roadmap aligned with your business goals

Navigate the complexities of AI implementation with confidence

Foster a culture of AI innovation within your organization

As a bonus, we'll also be providing you with cheat sheets, systems, and templates you can use to put these lessons into practice right away.

Your first lesson on misleading advice will arrive in your inbox tomorrow. Prepare to challenge misconceptions and gain a clearer view of AI's true potential for your business.

Best regards,

Hamel Husain & Greg Ceccarelli

PS: Don't let your emails go to spam! If you haven't already, please add hamel_husain@parlance-labs.com to your email contacts. If you use Gmail, here's how to do it:

Open Gmail on your Mac or PC.

Click the Google apps button at the top-right (it's next to your account icon and looks like 9 dots).

Click Contacts.

At the top-left of the screen, click Create contact.

Enter hamel_husain@parlance-labs.com, and click Save!

If you have any questions along the way, don't hesitate to reply to our emails– we're here to help.

Day 2 - 99% of Executives Are Being Misled by AI Advice

As an executive, you're bombarded with articles and advice on building AI products.

The problem is, a lot of this "advice" comes from other executives who rarely interact with the practitioners actually working with AI. This disconnect leads to misunderstandings, misconceptions, and wasted resources.

A Case Study in Misleading AI Advice

An example of this disconnect in action comes from an interview with Jay Keller, CEO of Casetext.

During the interview (clip linked above), Jay made a statement about AI testing that was widely shared:

One of the things we learned, is that after it passes 100 tests, the odds that it will pass a random distribution of 100k user inputs with 100% accuracy is very high. (emphasis added)

This claim was then amplified by influential figures like Jared Friedman and Garry Tan of Y Combinator, reaching countless founders and executives:

The morning after this advice was shared, I received numerous emails from founders asking if they should aim for 100% test pass rates.

If you're not hands-on with AI, this advice might sound reasonable. But any practitioner would know it's deeply flawed.

"Perfect" is Flawed

In AI, a perfect score is a red flag. This happens when a model has inadvertently been trained on data or prompts that are too similar to tests. Like giving a student the answers before an exam, they will look good on paper but be unlikely to perform well in the real world.

If you are sure your data is clean but you're still getting 100% accuracy, chances are your test is too weak or not measuring what matters. Tests that always pass don't help you improve; they're just giving you a false sense of security.

Most importantly, when your models have perfect scores, you lose the ability to differentiate between them. You won't be able to identify why one model is better than another, or strategize how to make further improvements.

The goal of evaluations isn't to pat yourself on the back for a perfect score.

It's to uncover areas for improvement and ensure your AI is truly solving the problems it's meant to address. By focusing on real-world performance and continuous improvement, you'll be much better positioned to create AI that delivers genuine value. Evals are a big topic, and we'll dive into them more in a future email.

Moving Forward

When you're not hands-on with AI, it's hard to separate hype from reality.

Here are some key takeaways to keep in mind:

Be skeptical of advice or metrics that sound too good to be true.

Focus on real-world performance and continuous improvement.

Seek advice from experienced AI practitioners who can communicate effectively with executives (you've come to the right place!).

We'll dive deeper into how to test AI, along with a data review toolkit in a future email.

In the next email, we'll look at the biggest mistake executives make when investing in AI.

Day 3 - The #1 Mistake Executives Make with AI

One of the first questions I ask tech leaders is how they plan to improve AI reliability, performance, or user satisfaction.

If the answer is "We just bought XYZ tool for that, so we're good," I know they're headed for trouble.

Focusing on tools over processes is a red flag, and the biggest mistake I see executives make when it comes to AI.

Improvement Requires Process

Assuming that buying a tool will solve your AI problems is like joining a gym but not actually going. You're not going to see improvement by just throwing money at the problem.

Tools are only the first step. The real work comes after.

For example, the metrics that come built-in to many tools rarely correlate with what you actually care about.

Instead, you need to design metrics that are specific to your business, along with tests to evaluate your AI's performance. The data you get from these tests should also be reviewed regularly to make sure you're on track.

No matter what area of AI you're working on– model evaluation, RAG, or prompting strategies– the process is what matters most.

Of course, there's more to making improvements than just relying on tools and metrics. You also need to develop and follow processes.

Rechat's Success Story

Rechat is a great example of how focusing on processes can lead to real improvements. The company decided to build an AI agent for Real Estate agents to help with a large variety of tasks related to the different aspects of the job.

However, they were struggling with consistency. When the agent worked, it was great, but when it didn't, it was a disaster.

The team would make a change to address a failure mode in one place, but end up causing issues in other areas. They were stuck in a cycle of whack-a-mole.

They didn't have visibility into their AI's performance beyond "vibe checks" and their prompts were becoming increasingly unwieldy.

When I came in to help, the first thing I did was apply a systematic approach that is illustrated here:

This is a virtuous cycle for systematically improving LLMs. The key insight is that you need both quantitative and qualitative feedback loops that are FAST. You start with LLM invocations (both synthetic and human-generated), then simultaneously:

1. Run unit tests to catch regressions and verify expected behaviors

2. Collect detailed logging traces to understand model behavior

These feed into evaluation and curation (which needs to be increasingly automated over time). The eval process combines:

• Human review

• Model-based evaluation

• A/B testing

The results then inform two parallel streams:

• Fine-tuning with carefully curated data

• Prompt engineering improvements

These both feed into model improvements, which starts the cycle again.

The dashed line around the edge emphasizes this as a continuous, iterative process - you keep cycling through faster and faster to drive continuous improvement.

By focusing on the processes outlined in this diagram, Rechat was able to reduce its error rate by over 50% without investing in new tools!

Check out this ~15-minute video on how we implemented this process-first approach at Rechat.

Avoid The Red Flags 🚩

Instead of asking which tools you should invest in, you should be asking your team:

• What are our failure rates for different features or use cases?

• What categories of errors are we seeing?

• Does the AI have the proper context to help users? How is this being measured?

• What is the impact of recent changes to the AI?

The answers to each of these questions should involve appropriate metrics, and a systematic process for measuring, reviewing, and improving them.

If your team struggles to answer these questions with data and metrics, you are in danger of going off the rails!

Avoiding Jargon Is Critical

We've talked about why focusing on processes is better than just buying tools. But there's one more thing that's just as important: how we talk about AI.

Using the wrong words can hide real problems and slow down progress. To focus on processes, we need to use clear language and ask good questions.

That's why I made an "AI Communication Cheat Sheet for Executives". This guide helps you:

• Understand what AI can and can't do

• Ask questions that lead to real improvements

• Ensure that everyone on your team can participate

Using this cheat sheet will help you talk about processes, not just tools. It's not about knowing every tech word. It's about asking the right questions to understand how well your AI is working and how to make it better.

In our next email, we'll reveal a counterintuitive approach to AI strategy that can save you time and resources in the long run. Don't miss it.

Until then, remember that when it comes to AI, process trumps tools every time. It's not as flashy, but it's the key to real, measurable progress.

Day 4 - The Counterintuitive Secret to AI Success

To help you decide on AI investments, we must first find your AI strategy. It is a crucial step. This is part one of our series on how to identify, communicate, and mobilize your Gen AI strategy.

The Trap of Hype-Driven Implementation

With all the buzz about Generative AI, it's tempting to rush into trendy apps like chatbots. But here's the hard truth: that's just chasing competitive parity, not creating advantage.

Remember, a business's strategy is to gain a competitive edge. It should make deliberate choices to do this. This advantage comes from having unique, delightful products that are hard to copy.

The Essence of Good Strategy

Good strategy involves three key elements:

1. A clear diagnosis of the challenges your organization faces

2. A coherent plan of action

3. Alignment of resources to address these challenges

Start with a Diagnosis

This is the step most organizations skip, but it's crucial. Take time to clearly define the challenges your organization faces. Be brutally honest - this transparency is the foundation of good strategy.

For example, when resetting our strategy in mid-2023, we examined:

• The competitive environment

• Impact of rising interest rates

• Areas where we'd been slow to innovate

• Gaps in our ecosystem strategy

Identify Your Sources of Power

What gives you competitive advantage? There are seven main sources:

• Brand

• Cornered resources

• Process power

• Network effects

• And others...

In our case, our brand and cornered resources stood out. But we found weaknesses. Our product was too easy to replace and we had no switching costs.

Develop a Guiding Policy

Based on your diagnosis and your edge over competitors, create a policy. It should guide how AI can boost your business strategy.

The Costly Mistake to Avoid

Implementing AI without this strategic foundation can be costly and ineffective. AI implementations are complex, involving:

• Rethinking customer experience for stochastic outputs

• New measurement and evaluation methods

• Potentially new infrastructure

If your AI app doesn't create new power for your business, it will likely fail.

The Payoff of Strategic AI Implementation

When you do this process right, the results can be incredible. First, diagnose your current state. Then, understand your sources of power. Finally, apply AI to enhance these, using a strategic approach.

In our next email, we'll discuss how to effectively communicate your AI strategy. We'll also give you a template with examples you can use to get started right away. Stay tuned!

Use This Proven System To Define Your Strategy

I've refined this toolkit over a decade of leading AI initiatives. The toolkit ensures you are maximizing business impact. It contains a step-by-step guide with examples.

Day 5 -Five Steps to Communicate Your AI Strategy

Last time, we talked about identifying your AI strategy– and if you did your homework, you should have developed your own document that identifies the challenges, strengths, and guiding policy for your AI initiatives.

With that strategy in hand, the next step is being able to communicate it effectively.

The "Business on a Page" Approach

Over the last 20 years, I've found the "Business on a Page" practice to be an incredibly useful tool for communicating strategy. This document distills your entire business strategy into a single page, forcing clarity and alignment around key priorities.

To get started, create a new document with these five key sections:

1. Big Picture Vision

Start by defining your company's long-term vision. This should go beyond the day-to-day and describe where you want to be in the future.

2. Why Your Organization Exists

Next, capture the "Why" behind your organization. This should include the fundamental reason you exist, and the impact you want to make. It should answer the question: "Why do we do what we do?"

3. What Your Organization Does

his section gets more specific about what your organization does, and the value it provides. Clearly state your unique value proposition and target market by answering these key questions:

• What problems do you solve for your customers? Who are they?

• What are the mechanics of your solutions?

• What is your positioning?

4. How Your Organization Will Succeed

Revisit your strategy document, and distill it to address how your organization will succeed:

• How are you different from your competitors?

• How do your products enable long-term transformation for your customers?

• How will your products be perceived?

5. Actions

The final, most important step is where you define the specific actions you will take for the next six to twelve months. These actions should be clear, specific, and measurable, and should align with your long-term vision and strategy.

They might be things you will do more of, less of, or things you will stop doing altogether. No matter what, they should be aligned with your long-term vision and strategy.

Clear Communication is Key

The "Business on a Page" exercise creates clarity and conciseness, making it an excellent basis for communicating strategy to internal teams, stakeholders, and customers alike.

For example, you might use it to communicate to employees how AI will affect their roles, or to investors how AI will drive growth and profitability, or to customers how AI will enhance their experience with your product.

Remember, communicating strategy isn't a one-time event. As you continue to implement and learn, you'll need to revisit this exercise and communicate updates as necessary.

With everyone aligned on your AI strategy, it's time to mobilize the team into action! We'll cover this in the next email.

Day 6 - My System For Nailing Execution on AI

At this point, you should have developed a strategy for your AI initiatives that can be communicated effectively. Now your organization needs to execute on it.

Getting people to take action can be challenging, especially in mature companies. These companies are optimized for their markets and struggle with adopting disruptive innovations like AI.

Fortunately, Geoffrey Moore's Zone to Win framework offers guidance. Over the past decade, I have adapted this framework for AI initiatives. It's allowed my leadership team to execute with consistency.

At the end of this email, I'll provide you with a step-by-step guide (with examples) for applying this framework.

The Four Zones

The Zone to Win framework divides your business into four zones, each with a specific purpose:

1. Performance

The Performance zone represents your core business that generates consistent revenue and profit. This zone focuses on meeting financial targets and maintaining operational excellence. Products and services in this zone are typically mature and well-established.

2. Productivity

The Productivity zone focuses on improving efficiency and effectiveness across the organization. AI initiatives naturally fit here. They can go beyond engineering and into HR, sales, and finance.

3. Incubation

The Incubation zone is for testing new ideas. It won't affect your core business right away. This is a safe space to develop and test new business models, especially those using generative AI.

4. Transformation

The Transformation zone acts as a bridge between the Incubation and Performance zones. This zone integrates your vetted AI initiatives into your core business.

Managing AI Initiatives

AI initiatives can apply to all four zones. However, they'll usually start in the Incubation Zone.

It will take some buy-in. But, you'll want to manage these initiatives differently than your core business. Focus on measuring like a startup, where speed and new customers are key. Allocate resources wisely. Give teams the freedom to experiment and test. Let them iterate quickly.

Ideally your Incubation projects will generate at least 10% of your revenue. This ensures you create real value, not just experiment.

Step-by-Step Guide

Now its time to learn the framework by applying it. Here is a step-by-step guide for applying this framework to your AI initiatives. It contains a case study to walk you through the process.

Day 7 - Hiring for AI Success

Another costly mistake that executives make is incorrectly hiring an AI team. This mistake can lead to wasted resources, team churn, and delayed AI efforts.

The AI Specialist Trap

Many executives rush to hire data scientists or machine learning engineers right away. While these roles are crucial, bringing them on too early is a classic "foot gun" scenario. Here's why:

Data scientists and MLEs need data to work with. If you're still in product discovery, there's often little for them to do.

The people who fulfill these roles likely aren't equipped to build full-stack products from scratch. Lacking a clear product direction, they may inadvertently steer you off course.

A Smarter Hiring Progression

Instead of immediately seeking AI specialists, follow this general progression when building an AI product:

1. Focus on building the product: Start by hiring Application Developers who can help iterate on the product concept quickly. At this stage, you're still figuring out what your product is and need to build a thesis.

2. Build the data foundations: Instrument your system to collect the right data and set up proper observability. This may require Platform or Data Engineers, depending on the type and scale of data you're working with.

3. Optimize your AI system: Once you have a working product and data, that's when you should hire MLEs or data scientists to optimize the system. At this point, they should be able to design metrics, build evaluation systems, run experiments, and debug stochastic systems.

The key is to maintain a strong domain expert presence throughout each stage.

Another great option is to bring in advisors instead of hiring full-time specialists. Bringing in outside help can help you accomplish the three-step progression above while avoiding the commitment of a full-time hire.

Hiring the Right Way

When you do reach the stage of hiring AI specialists, be strategic.

Test for data literacy by giving candidates messy datasets to work with. This crucial skill helps in designing effective metrics and evaluations. For a deeper dive, read Jason Liu's blog post 10 Ways to Be Data Illiterate (And How to Avoid Them)

When the time comes to hire MLEs, Eugene Yan has a great guide on interviewing.

Remember, the goal is to build a team– and the best way to do this is by hiring the right talent at the right time.

Day 8 - Our secret weapon for uncovering AI flaws

We've touched on the importance of having metrics that are specific to your business and a systematic process for measuring and reviewing them.

Today I wanted to share a recent experience that highlights a crucial lesson for AI projects:

There's no substitute for examining your data first-hand.

At the end of this email, I'll provide you with a toolkit that enables you to review data without getting lost.

I was recently working with a company that automates HR functions like recruiting and onboarding. Their engineering team had developed an evaluation suite with various metrics to measure the AI's performance.

One metric in particular caught my attention– the "edit distance" between the AI-generated email and the recruiter's final version.

In case you haven't heard of it, edit distance is a measurement of how similar two texts are.

This metric seemed like it would be a good one– it's business-specific, and it can be systematically measured.

The team had found that the average edit distance between the AI-generated emails and the recruiter's final version was 12%. This seemed like a good result, but the team was struggling with poor results.

It ends up that the metric was hiding a critical flaw.

Look at the Raw Data

In my experience, the best way to understand a metric is to look at the raw data. It might sound simple, but it's like a secret weapon that almost always uncovers something unexpected.

I asked to review some of these emails myself, and what I found was shocking:

For the most part, the AI-generated emails were perfectly reasonable. Instead, it was actually the human edits that were causing problems! The "improvements" that humans made often introduced grammatical errors, wordiness, and unclear messaging.

The assumption that human edits would always improve the emails was a fundamental flaw in the edit distance metric.

This discovery took me just a few hours. The team was stunned by how quickly I identified this issue that had eluded them for so long.

Key Takeaways for Executives

This story is a perfect example of why it's so important to regularly look at your raw data. By sanity-checking your data against metrics, you might spot any issues that might be hidden in aggregated reports.

Our System For Reviewing Data Without Getting Lost

In my consulting engagements, I always train executives on reviewing data. This approach is the main reason our clients see incredible results. This toolkit gives you the same training, but without the consulting fees!

Day 9 - Build it or buy it?

This age-old question applies just as much to AI as it does to traditional tech:

Should we build it, or should we buy it?

To help you make the right decision, let's walk through a simple framework you can use:

Ask yourself if the capability in question is part of your core skills or future products.

• If it is, build it.

• If not, buy.

When you've decided that buying is the right choice, finding the right tool is your next step.

Finding AI Tools

Recently, we were working with a client who wanted to create automated data analysis to speed up debugging. Because this was only going to be used by the team instead of customers, we suggested they look at AI-powered data analysis tools like Hex and Julius.

Of course, data analysis is just one area of emerging AI tools for your business.

To help you discover more, Greg developed the aptly-named Greg's List

This curated site has a list of AI tools for a wide variety of areas including HR, legal, and finance– just to name a few.

There are many tools out there that will benefit your business, but don't forget about looking into tools that will help you personally.

The Importance of Daily AI Use

As an executive, it's vital that you integrate AI into your daily workflow.

Regular use helps you grasp AI's real-world benefits and limitations, in turn informing your strategic decisions for applying it in your business. It's also a great way to inspire ideas for adding new AI features to your products.

You've hopefully been using ChatGPT and Claude already, but here are a few other AI tools that we've found particularly useful:

• CircleBack: Best-in-class meeting transcription and summarization.

• Hemmingway: A writing assistant we use for more effective business communications.

• Descript: Makes video editing as simple as editing text.

• Cursor: Versatile developer tool with surprising benefits for non-technical users that we'll cover in the future.

As an action step, choose one tool to integrate into your daily workflow this week. Set a goal to use AI in at least five different ways each day. This could be drafting emails, taking notes, or even personal tasks like meal planning.

Share your experiences with your team, and encourage a culture of AI adoption and experimentation.

The Big Takeaway

The big takeaway is this: If a capability isn't part of your core skills or future products, consider buying it instead of building it.

You don't need to become a tech expert, but by identifying opportunities for tools to save you time and improve your workflows, you'll be better positioned to lead your organization in the AI space.

Day 10 - Ready to master AI? Here's what's next

Two executives walk into a board meeting. Both are presenting their AI strategy.

One talks about chatbots, catching up to competitors, and buying the latest tools. The other talks about processes, data review, and amplifying their existing advantages.

Guess which one gets their budget approved? (Hint: it's not the chatbot person)

And here's what we've learned: most AI advice is backwards.

It either comes from technical people who can't speak "executive," or from executives who don't really understand AI. The results are predictable (and expensive):

• Wasted budgets

• Failed projects

• Unhappy teams

• Very happy consultants

It all came down to three ideas (that almost nobody talks about):

1. Stop chasing AI tools and focus on process first

2. Look at your raw data - don't trust metrics blindly

3. Build AI features that amplify your existing advantages

If you've been following these emails, you know these ideas work. Now it's time to put them into practice. Here are some resources you can use to learn more:

• ​Our free survey course on LLMs, taught by practitioners. This is more technical than this email series, but is a good resource for engineering teams to up-level on AI.

• The O’Reilly article "What We Learned From A Year Building With LLMs", written by Eugene Yan, Bryan Bischof, Charles Frye, Hamel Husain, Shreya Shankar, and Jason Liu.

• Hamel's twitter, where he shares information about AI.

If you would like more hands-on help, you can work with us directly through our paid consulting services (but our availability may be limited). Lastly, if you enjoyed this course, I would like to hear from you - just reply to this email. I reply to all emails personally!