How AI will open new possibilities for commerce — and how you can get started

Launching a new series on AI-enabled commerce, starting with an interview: Derek Fridman, design partner at Work & Co.

Artificial intelligence could profoundly change many aspects of life over the coming years, including how we buy things and how we interact with products, brands, and merchants. This is exactly the sort of collision of technology and the consumer experience that I set out to explore with The New Consumer.

So I’m excited to launch a new series focused on AI and commerce, where I’ll highlight what’s possible, what’s working, and what’s coming. When do chat-based interfaces actually make sense? Can anyone really rely on generative AI for media production yet? How should brands and teams get started? And — crucially — if these things work, then what?

My launch sponsor for this series is Work & Co, the Brooklyn-based design and technology company. 

Over the past decade, you may have seen their work for the likes of IKEA, Gatorade, Apple, and Aesop on a range of digital products and e-commerce experiences. And they just announced that they will be acquired by Accenture Song, where they’ll create those products — as well as more AI-focused experiences — on a bigger scale.

Anyway, the idea is to open this series — which I plan to publish monthly — to all. I’m grateful for their support.

As part of our partnership, I’m kicking things off with a conversation I recently had with Derek Fridman, design partner at Work & Co. Fridman has played a key role creating new products for Netflix, Lego, PGA Tour and The Home Depot, and is directly working with merchants and brands to explore how AI could move their commerce experiences forward in the near future.

We talked about: Why AI-powered chat is interesting as a shopping interface; why Apple’s Vision Pro and spatial computing are compelling for commerce; how generative AI will change the design process; what companies need to do now to get ready; and how they can start experimenting.

What follows is a lightly edited version of our conversation. Prefer to listen? Members: Visit this page to add The New Consumer Audio Edition to your podcast player.

Dan Frommer: Derek, thank you so much for joining me. I’m excited to talk about how AI is going to change the commerce experience for both consumers and merchants.

Let’s start with the big picture. What do commerce and brand leaders need to know about this broader state of AI today?

Derek Fridman: AI, right now, is raw, and a work in progress. As an industry, we still have a lot of work to be done across a number of different categories — in terms of copyright, starting to establish some rules, and how we leverage AI and how we use it.

I think it’s critically important for leaders to stay informed and up-to-date on all the developments that are going on in AI across chat, across image generation, video, and all the other areas that AI and models are impacting.

It’s important that everyone across industries, especially in retail, start to figure out ways to test and play with different ways that these new models might change their business. And how they might be able to engage with consumers and their customers in new and inventive ways.

And I think now is the best time for that experimentation, while we start to figure out some of the bigger questions and concerns around the practicality of launching AI on a grand scale.

In terms of actually integrating AI into commerce and into the shopping experience, it seems like a lot of the early ideation is around chat. Perhaps that’s because ChatGPT is also the first modern AI tool that’s really going mainstream that consumers are using?

Chatbots are highly powerful because, for years, we’ve had to teach ourselves as consumers to speak “search engine.”

We’ve had to take our thoughts and the things that we’re looking for and we’ve had to translate them in our minds into keywords, that we’ve gone to a search engine, or a retailer site, and entered, in hopes that we’re going to get a result that is the thing that we’re looking for. 

All of us have gone through scenarios where we have to go back and change what we’re searching for, getting a better result because we’re adding more words.

And we had to teach ourselves that because that was the common way that we communicated with screen rectangles and machines.

Now, what’s really interesting is we can start to communicate with machines in a way that feels much more natural. The fastest way that you and I can communicate is me sending you a text. What’s a text? It’s a bunch of words typed into a box in any way that I want — or emojis — and I hit submit.

Leveraging an AI model to have a conversation seems to be the first natural step into opening up something beyond the classic search field.

Do you think chat is a good interface for commerce? Is it something that will be a good shopping experience for consumers — and also help brands and help merchants sell things?

We need to start thinking about this as building a relationship with somebody. What makes a good relationship? It’s where we listen to one another — we remember things. And I think this is the critical piece.

If I walk into a retail store today — let’s say it’s a brand that I’ve spent tens of thousands of dollars with over the years. Let’s say it’s a big shoe brand, and I walk into their store today — they’re going to say, “How can I help you?” 

I want to be treated a little bit differently. I want to hear, “Derek, welcome. We’re glad you’re here. Are you ready to go ahead and commit to that thing that you were looking at?” Or, if I go to Japan and I walk into one of their stores, I want to hear, “Derek, you’re really far from home. We’re glad you’re here.”

That doesn’t exist today, and I think that’s what we want to have happen.

In the future, the way we’re used to doing things now is just going to feel like it was so cold and impersonal.

So we start to think about not designing for e-commerce, but designing for relationship commerce — it’s a phrase that we use a lot.

What makes a healthy relationship? What’s the dynamic between a customer and a business? And how do we operate on a level that’s a deeper, more meaningful kind of connection between the two?

And I think those are the brands that are going to stand the test of time. As we look forward in the future, it’s the ones that are going to have that deeper relationship.

Do you think it’s a specific type of purchase, or types of merchants, where these interfaces are going to really stand out and move the needle?

Let’s say, for example, I’m going to go backpacking in Yosemite for a couple of nights and I need a backpack.

“Well, when are you going? In September? Okay. Well I know the average temperature in Yosemite in September is between 84 during the day and 50 in the evening. This is going to be the right thing for you. There’s a 10% chance of rain in September. So let’s think about something that’s kind of waterproof for you.”

Keywords and search boxes and clicking on links don’t give us all that extra info. 

When we leverage something that’s more like a — “How can I get something done for you?” Or you talk to an associate that maybe has something in their ear that connects to the model. How do we add all that extra context to the conversation and remember that?

So when I come back, it already knows I have the backpack that I used in Yosemite, and knows it’s going to be great for Yellowstone in July.

Yes, there’s a level of privacy and a level of information that I’m willing to give up. It’s really my choice how verbose I want to be in those engagements. 

But if it comes with convenience — if it comes with a level of status, access, or power with the brand — I will give you anything. I will do anything, and I will continue — especially if I walk into that retailer and they go, “Derek, welcome back. I have this specially for you, because I know that’s what you’re coming in and looking for.”

It’s not just a matter of chatting and getting a set of results. It’s a matter of chatting and then building what we call purchase momentum: This idea that identifying or honing in on this one particular item or object means that these other ones that have direct relationships to them could benefit if brought into the cart equation. 

Any product where there’s an end goal, or a companion, or suite of products that make up a kit — that’s fertile ground to design against.

But it means you have to design differently, too. We can’t build classic grid square interfaces and structured interfaces when we have such an unstructured sort of concept as conversation.

Chat’s not going to replace all forms of interaction. It’s really going to depend on the person, the time, the place, and the device that they might be accessing with. 

As we look towards the future, things like spatial computing — typing on a keyboard in air isn’t that comfortable. I might do it to enter a password, because I don’t want to say it out loud, but I prefer to just say something.

Or, if we look at Apple’s Vision Pro headset and Vision OS, I could stare at something. I could just pinch my fingers lightly, and I can engage in ways that are beyond just a keyboard and mouse.

So, chat might be first. It might be weird. It’s going to become more popular. 

And then as we enter a three-dimensional space and spatial computing, it’s going to be more voice, gesture, and even reaction. Imagine you’re shopping. Something shows up and you just shake your head. “No, that’s not it.” So let me go find something else. 

All these different input methods are going to become more popular, even though they feel a little weird right now. I see that transitioning, and search to just feel very barbaric, in terms of just “keyword” and “result.”

You mentioned Vision OS, and Apple’s Vision Pro headset will be here very soon. How important do you think spatial computing is going to be? And how should brands and merchants think about it as a surface for commerce? 

I’ve owned every single VR headset that’s existed since the early days of the space. And I’ve watched us grow into this world where it was considered pretty much a game console that sat on your face. And we talked about the metaverse and games and things that you could play.

I loved, last summer, seeing Apple’s fresh take on it. It’s not VR or anything like that. It is a new way to compute. It’s a new type of computer. And I think that’s a really important statement to make. It’s not a game console. It’s not a metaverse machine. It is a new way to compute. It just happens to live on your face. 

And it happens to give you these new input methods, rather than a keyboard and mouse that were plugged in or connected via Bluetooth. You’re just using your hands, and you’re using the most natural thing ever, which are your eyeballs. 

And I think that’s the way we shop.

We walk into a store, we look at aisles, we look at things, we glance at them for a long time, we grab them off the shelf. We turn them around, we rotate them, we want to unbox them. 

Like, all of those natural ergonomics that exist — within the way that we want to engage with a physical product — are as close as we can be to being there, when presented in this Vision OS or spatial computing environment.

And I think that’s really interesting for retailers, in the sense that I can get you as close to the product as humanly possible without you being there — without me shipping you anything. So you might make better decisions. Rather than me sending you something, and paying to have it sent back, I can get in front of you and you can look at it.

Also, we’re done with rectangles. Like, right now we have a little rectangle, we have a big rectangle, we have a medium one. Now we have our full field of view. We can go from 2D to 3D. We can go all the way in or come out. 

And I think Tim Cook had said something, like, you just — until you use it, you don’t realize; there’s a click that happens. And that’s rare. Sort of like the world that we were in when we saw the iPhone, and we made the leap, and got excited.

I think this product, while a bit cost-prohibitive on day one, is going to be something that when you use it, you go, “Oh, I get where we’re going. Like, I get this.” So while we’re still maybe a few revs or versions out before everyone has one, or some flavor or version of what it can deliver, I really feel strongly that this is our natural evolution, and the next step in terms of how we’re going to compute with one another. 

And collaboration is massive. 

Like, you and I, right now, are staring at each other on rectangles set inside of other rectangles. And I think you and I could be sitting in an environment that feels like we’re in the same room very easily and naturally. And I could grab something, and I could bring it into the conversation.

You can say, “Oh, I bought this thing. Let me show it to you.” And we can look at it, and you can explain it to me.

None of that’s possible now, because you just message me photos and images and potentially videos. 

I could talk forever about it. I think it’s going to be kind of a magical period of time that we’re going to get into.

So between that and the AI models for chat interfaces, for brands and merchants, it sounds like more information, more data, more assets are needed than the very basic product descriptions that are on a product page on an e-commerce website.

Are there any things that companies can do now to start building those new sets of data? Or getting them ready for these new uses?

I think right now there is no one product I would point to — there’s no one platform I would point to. I would say, because we’re in such an early phase, to get your house in order. 

What I mean by that is: Classic e-com clients that we meet with every day are made up of tons of silos. That’s the person in that group over there. That’s store operations. That’s this person here. 

And rarely do you find that there’s an individual at the organization that’s looking over the whole customer experience. 

We had, like, Chief Customer Experience Officer, and things like that. These were more marketing titles.

But I feel as if there’s a title that’s missing today, which is this title for someone who’s responsible for the intelligent orchestration of all of the tools and services across the business as a whole. This doesn’t exist. I think this is a role that’s needed. 

It’s that game master role — that person that can look across and say: All right, we need to structure ourselves flat, break down some of these silo walls, and just start building data lakes across all of our product SKUs, all of our review data. 

Let’s get our best salespeople on the floor to start training a model on how they convert and get the sale done. To think about how AI will benefit associates and turn them into superheroes. 

So if I am on the floor and I want to know about, say, fly fishing in Jackson Hole — they may have never been to Jackson Hole; they may have never fly fished in their life. But they could leverage the model to better have a conversation with you to find the thing that you need.

And that’s where — we didn’t coin it, but we love it as a term — intelligent orchestration comes in. 

It’s the makeup of an operational team that’s able to see everything that’s coming and going and build these models out, so that they could collaboratively work together. You could have a model that’s great at recommending fishing gear, but it needs to talk to the rest of the business, and it needs to be integrated with the rest of the business as well.

You’ve talked a lot about retail associates, and of course many of the biggest e-commerce brands are also the biggest retail brands. They increasingly have omnichannel businesses, and they have physical stores that are as important to their customers as their websites.

How do you imagine those things connecting? Is the clerk holding a device with an app that’s connected to your purchase history? Is your entry into the store being announced somehow?

We’ve talked a lot to retailers around the idea of what we call “handshakes.” What are those ways that we can fist-bump, handshake, or just acknowledge that someone stepped in through the door? 

The most rudimentary and simple one is just through QR codes. Somehow we’ve now embraced QR codes, right? And one of the simplest ways is for someone to walk up to an associate and say: “This is me, this is my badge, this is my relationship with your brand.” And there’s a scan or that bump that happens. And then we can see all that data come forward for that moment of conversation. 

And we can add to it as well. So if you come in and you hang out with me in the aisle, we’re talking about a plumbing need that you have, I can append that note to that record that we have, and it goes up into the cloud somewhere. 

There are varying degrees of how much someone wants to access, but I think we need to have that moment where a single view of the customer is available to the associate at the right moment in time, and it’s open and available, and we’re feeding it info. 

It’s one thing to come in and have a conversation and then walk away. Nothing happens with that data. But if you come in, you’re acknowledged, that conversation’s kept in a record — later on, it could be a massive benefit. We don’t have to ask you what type of tile is on your roof or what kind of brick or marble you have on your countertop, or what color you painted your kid’s room. We already know because we captured that in one of those one-to-one moments.

And then, above and beyond that, the ability for them to tap into the model as well — to essentially have an admin version of the model that they can say, “Oh, customer asked for x, what’s the best response?” Or, “How do I guide them, or lead them, to a potential answer?”

Then you’re not relying on your superheroes, which are your best salespeople on the floor — the most knowledgeable. You’re giving everybody that superpower. You’re giving everyone access to those tools. 

And as long as you have feedback loops, then you continue to train it. When some weird scenario comes up in one city, and it gets added to the training, and that scenario comes up someplace else, it’s not weird anymore. We know how to handle and address this. 

So that’s where the orchestration comes in. And that’s where this fundamental change in how a business operates needs to begin happening.

And while AI is in this Renaissance initial phase, now is the perfect time to start thinking about that orchestration and how to start building the pools of data, and then breaking down those silos and walls that have held up e-commerce and retail for so many years. 

One of the more profound things about this newer wave of AI is its ability to generate really detailed images, almost on the fly.

Thinking about e-commerce interfaces as a whole, though, it seems like generative AI would be able to affect not just the product image on the page, but actually draw an entirely different storefront for each person. Is that intriguing as a designer?

Do you like that idea of being able to really have a one-of-one experience for each customer, and have it learn over time, and be different for everyone? Or is that going to be more trouble than it’s worth?

This is every designer’s dream. This is what we’ve hoped for for years — this idea of atomizing the design system and then allowing it to present itself at the right place, time, device, moment that feels very purpose-fit. 

The idea of moving away from breakpoints: “This is a tablet. This is a mobile phone.” They’re kind of ridiculous. That’s how we handled that right person, place, and time. 

The things that I need from a brand may not even have to happen on a URL or within a dedicated app. I should just be able to call my favorite store and ask them for something, and get a bunch of photos back.

I’m old — I’m old-school. I have an 11-year-old daughter, and she doesn’t want to talk to anybody in the store. She just wants to send links back and forth with her friends with pictures. So every retailer should be sending links and pictures back and forth, and just communicating on their level.

So I think we’re going to start moving away from traditional websites. I think the web will be an archive, a place that we go to if we want the full product catalog. But not really a place we’re going to spend a whole lot of time and hang out, in terms of browsing and decision making. YouTube — we spend a lot more time there browsing, looking at products and unboxings, than we do on a retailer site.

That’s kind of the final step. So as a designer, I’m super excited about dynamically generating and building interfaces on the fly. 

I think we’d be wasting time, money, and effort trying to solve that problem for the web, right? Website presentation, rectangle sort-of presentation.

The web does a job and it does it okay. And it can be somewhat personalized. So dynamically delivering websites has a big technical lift that — maybe time is better spent, depending on your customer base, in other places. Things like shopping through chat, and shopping through other channels, and third parties in new and inventive ways. 

So less “let’s make a dynamically drawn website for each specific person” — although that could be cool — but more: Let’s rethink what the idea of a store is, with all these new inputs and outputs that we have. I

think a lot of people have this idea that AI is going to replace humans, whether it’s replacing photographers, or designers, or editors, or writers, or whomever.

That perhaps is the lazy approach to this, but I’d love to hear — especially as you’re speaking with clients, and these products are not cheap; AI is not cheap to run on a cloud computer. When it comes to AI, what is the business case for growth that you’re making as you talk to these clients? How can merchants and brands think of this as a needle mover for, for revenue and not just for costs?

In terms of things like imagery and video, AI is a great time saver, but it’s not going to give you the answer or the final product. And I think it’s going to be a long time before it can just nail it right off the bat.

It’ll help you get there faster, but we’re not going to get rid of people anytime soon. There’s a lot of checks and balances that need to happen across the things that AI delivers. It might reduce shoot times, and sitting in front of Photoshop, and image generation — things like that. But you have to do the fine-tuning. 

And I think growth-opportunity-wise for individuals, it’s going to allow them to focus less on the manual tasks of their subject-matter expertise, and more so thinking strategically.

So I think the number-one thing we can do as designers is just be really good problem solvers and figure out: What’s the problem that we’re looking to address? And, then, how do we creatively solve it? I’m spending more time thinking about that than I am spending inside of Figma, or other tools, just designing all day.

We’re going to have a lot more volume created, but the designer, or the creator, is not going away any time soon. I don’t know that we’ll ever see that there’s something where the entire design organization within a company is gone, and some machine is just spitting everything out.

I think we’re a long way away from that. 

Zooming back out, it sounds like these technologies could have a major impact on how customers view brands, engage with brands, spend money, etc. So it is really almost mandatory for everyone up to the CEO to really understand these tools and the data they have — the data they’re using in order to not only build better products, but really maintain their brand.

Yeah. Maintain their brand. And then for those brands that are more like heritage or legacy brands, they’re sitting on a mountain of data and information. 

The ones that have been around for years and years and years have a massive pool of information and data that I would consider is highly valuable for other third parties and services to tap into.

So you might be, let’s say, a legacy motorcycle brand with years and years of history, heritage, and tons of information and subject-matter expertise. That data is not only valuable for you in terms of sales and getting people to the right bike, or what have you. 

But that’s valuable to any service that wants to provide information on things like road trips, travel, hospitality, anything that ties into your brand. You’re potentially sitting on a goldmine of information and data, that, if organized correctly — and you’ve built a small model for — you could potentially license out.

These are interesting conversations that we’ve been having with different brands — around topics like, “We are the experts in that particular category. Maybe we’ll have trouble competing with the aggregators of the world. But how are they going to answer really detailed questions around our products or our category?”

“Well, why don’t we give them access to that database, or that information?” So that’s been interesting, too, and having brands think about being — I don’t want to say software companies, but pivoting into a world where their models are licensable or accessible from other parties.

Interesting. 

Let’s have some practical advice: What is the best way for a brand to get started with AI?  Say I work at a brand or a merchant. We obviously are aware of AI, but we don’t know what to do with it yet. How do we get started? What should we start doing?

Playing and planning. It’s important to just start playing to get your feet wet and understand how things work. But also to start planning holistically.

We had the opportunity to do a workshop with one of our clients over the summer last year. They basically got their entire design operation together and said: What are ways that we could leverage AI in the future to deliver a next-generation customer experience?

And then at the same time, can you just educate our folks on what’s bleeding-edge and what’s coming up? So, moving into agents and some of the multipliers that are happening in AI.

We conducted those workshops and then worked with their team to start to develop ideas and concepts that essentially were mini product prototypes. “What-if” sort of ideas, both external and internal. And out of that project came their first project, which was: What if we start to create a tool for our associates that essentially allows them to tap into a model that’s an expert across our product suite? And that’s where they started: Let’s start experimenting there and let’s do a pilot in a few stores.

So that thought of coming together, getting up to speed on the current state of the union, what tools are available and accessible, and then building essentially a playground. We helped them build a walled garden of ChatGPT and then gave them access to a number of different tools and allowed them to experiment in this safe space. And they’ve gone on and are taking that forward. 

In terms of organizational structure, has this been a bottom-up-type thing where people are experimenting on their own and then it might turn into a really cool product for a company? Or do you think this needs to be a top-down, company-wide thing where there’s a Chief AI Officer who’s in charge of incorporating this through our whole company?

We’ve seen champions across all sorts of levels. 

In some cases, someone came up with a really great idea and built something, and it got some eyeballs, and they want to test it out and put some rigor around its presentation. 

And in other ones, at the highest levels of the organization, “What are we doing about AI?” That’s the question, and somebody has to go figure it out. And then the group is running.

But the thing I haven’t seen is that role that we talked about, which was that person responsible for the intelligent orchestration around the business right now. I think the number-one thing an organization can do is basically go do that.

What I said was, like: Get their house in order, and start to think about how to break the walls between the information that exists — because the model is only as good as what it’s been trained on. And if you want to deliver something that checks all the boxes, in terms of your overall customer experience, it needs to have access to everything. 

“I got it, I loved it, I kept it, I want the next thing.” “I got it, I hated it, I want to return it, and I want to do it all from one spot. And I want you to remember that.” 

We were passing around an image that was funny — we had somebody go out and buy, or they were just doing regular shopping, and they went and bought, like, a roll of tape, or something like that from a hardware store.

And then they got an email: How did that roll of tape work out? And it’s like, fine — it was great. But that’s the same email you would have gotten if you had bought a $1,000 tractor. “How did the tractor work out?” So just because you build it doesn’t mean it makes sense or it has the appropriate context.

And I think those are the things where someone needs to zoom out — see how the business is operating, and then start to make some really fundamental changes. 

So that you can be as hardware-, software-, and department-agnostic as possible, and just pivot as things evolve and change, and some of those bigger decisions get made.

Derek, thank you so much. I learned a lot and really enjoyed our conversation. Thank you. 

Yeah, for sure. It was great to chat with you.

My thanks again to Work & Co for partnering with me on this piece and series as its launch sponsor.

If you’d like to request or suggest a topic, please send me an email. To receive future installments by email, join The New Consumer’s free email list. Or to receive everything I publish as a newsletter and podcast — and directly support my work — become a member.

---

More

• AI
• AI & Commerce
• Apple Vision Pro
• Brands
• Derek Fridman
• Design
• E-Commerce
• Focus on AI
• Marketing
• Presented by Work & Co
• Retail
• Spatial Computing
• Sponsored
• Work & Co