[00:00:00] Nathan Wrigley: Welcome to the Jukebox podcast from WP Tavern. My name is Nathan Wrigley.

Jukebox is a podcast which is dedicated to all things WordPress. The people, the events, the plugins, the blocks, the themes, and in this case, how AI is shaping WordPress workflows.

If you’d like to subscribe to the podcast, you can do that by searching for WP Tavern in your podcast player of choice, or by going to wptavern.com forward slash feed forward slash podcast. And you can copy that URL into most podcast players.

If you have a topic that you’d like us to feature on the podcast, I’m keen to hear from you and hopefully get you, or your idea, featured on the show. Head over to wptavern.com forward slash contact forward slash jukebox and use the form there.

So on the podcast today, we have Paolo Belcastro. Paolo works at Automattic, where he’s focused on Jetpack and .blog products. He’s been working remotely since 1994, and managing distributed teams since 1998.

He’s on the podcast today to talk about AI. And he certainly brings a wealth of knowledge and experience in integrating AI with web development. .

Paolo brings detailed insights into how they’re making decisions about what to develop in Jetpack AI. This is to take it beyond simple tasks like typo corrections, and grammar adjustments, to more sophisticated functions, such as content translation, tone adjustment, and even complex texts summarizations. All done directly within the WordPress editor.

The focus isn’t just on the functionality AI brings, but also on the efficiencies it introduces for developers and content creators alike. Allowing them to shift their focus from tedious tasks to more creative and challenging aspects of web development.

We also tackle the broader implications of AI in the tech industry. Discussing potential risks, the ongoing concern about AI’s impact on jobs, and the ethical considerations of AI in creative roles.

We get into an exploration of the balance between AI’s utility, and the indispensable value of the human touch in crafting meaningful digital experiences.

With Paolo’s insights into the recent advancements in open source AI models and the collaborative efforts within the AI community to keep platforms accessible but innovative, you’ll gain a comprehensive view of where AI in web development stands today, and where it might head tomorrow.

If you’re a developer, a tech enthusiast, or anyone interested in the intersection of AI and creative processes, this episode is for you.

If you’re interested in finding out more, you can find all of the links in the show notes by heading to wptavern.com forward slash podcast, where you’ll find all the other episodes as well.

A quick note before we begin, this was recorded live at WordCamp Asia. There was quite a lot of background noise to contend with, and I’ve done my best to make the audio as easy to listen to as possible.

And so, without further delay, I bring you Paolo Belcastro.

I am joined on the podcast by Paolo Belcastro. Hello.

[00:03:48] Paolo Belcastro: Hi Nathan.

[00:04:59] Nathan Wrigley: Very, very nice to have you on the podcast today. We’re going to talk about AI, the topic which keeps on giving, throughout the year 2023, and probably throughout 2024 and 2025. Because we’re talking about AI, I guess, Paolo, it would be useful to know what your credentials are. To get some understanding of why what you say carries weight. So will you just tell us a little bit about you, what you do for a living, who you work for. Biography, really.

[00:04:15] Paolo Belcastro: Yes. So I work at Automattic, the company behind wordpress.com, Jetpack, WooCommerce, and I work particularly on Jetpack.

And my AI credentials, so to speak, are that, since last May, so about a year ago, we started working on Jetpack AI, that we launched in June 2023 at WordCamp Europe. And we have been developing this set of tools for people to use AI in the editor directly.

I’m also a very heavy user of AI tools. So for that you have to take my word, but I’ve been using them a lot. Yeah, and I made a presentation about AI yesterday, here at WordCamp Asia.

[00:04:58] Nathan Wrigley: Was it well attended? I imagine AI is popular at the moment.

[00:05:01] Paolo Belcastro: Yeah, it was actually as attended as the room allowed, like people were sitting on the floor, so I couldn’t. But yeah, it’s not a very big room, but I actually like a small room packed much more than a large room half empty, so I was pretty happy.

[00:05:14] Nathan Wrigley: How have you managed to integrate AI into the Jetpack suite of things? I’m guessing this is all still fairly new.

[00:05:22] Paolo Belcastro: Yeah it’s fairly new, still experimental. We are really trying to produce experiences, and listen to users who use them, and adapt them. Our first iteration was really about saying, let’s just bring the tools that everybody can now access in ChatGPT, or at the time was called Bard, it’s now I think Gemini. And basically bring them to the editor. Why would you have to go and copy paste with all the formatting problems that can arise?

So we extended the Gutenberg editor with a block, an AI block, that is a chat interface, where you can chat with the AI assistant. And we also extended a certain number of blocks, like the paragraph, the title, the headers, the table, and the forms, with an AI button that you can use to ask for a certain number of operations. And we started with what AI does best, which is text transformation.

So say you have written something, you can ask, of course, for typo and grammar correction, that’s the most used feature. You can ask for translation, which is really useful for people who, like a lot of people might want to publish in English, but that’s not their primary language. Of course, there’s other translation tools out there but, again, having it in the auditor is practical. And it does, of course, translation across many languages, not just to English.

We have also a tone transformation. If you, say you wrote something and you feel like it should be a little more formal, or more informal, maybe humorous, so you can choose one of the tones. You can of course summarise text, you can ask it to expand on something you wrote. So those are all tools to help you polish your text.

Now, I am a strong believer of the fact that AI is very valuable. Use it as a tool to improve what you are writing, as opposed to a tool that is going to write for yourself. So I don’t really believe we can get to any good results, at least today, by starting with a blank page and telling AI, hey, write me an article about Taipei, because I went to WordCamp Asia and I’m too lazy to blog. I don’t think that works. It’s going to write the same article for everyone, or roughly. And so that’s not interesting.

What is interesting is, I write a draft about Taipei because I’m here, and I write my experiences, and maybe it comes out a bit like a stream of consciousness. And then I can ask the AI, can you take all that text I wrote and make an outline for me?

So now I can organise my ideas that I put in that text. And maybe I feel that, okay, I should move that up, I should move that down, I should remove this. And once my outline has been polished, and has become a little more clear, I can actually tell the AI, can you take my original draft, this new outline, and reorganise my draft to match this new outline?

So now I have a new draft where I’ve been working with AI, like I would be working with an assistant, or with a colleague, or with someone helping me. I find that’s where the value really lies, because then it’s still me. It means that someone else, doing the exact same steps, will get to a completely different result, because it’s their mind, their content, their experience.

But we can accelerate the process, because it’s all about producing more, not for quantity, but because more writing leads to better writing. The quantity leads to quality. It’s not just about putting out more content. It’s that, whatever the craft you are in, whether it’s writing, photography, recording podcast, it doesn’t matter, the more you do it the better you get.

[00:08:53] Nathan Wrigley: So your key takeaway there, it feels like is that the human is at the center of all the things. You still need a human in order to produce something that humans would like to consume. So it’s not, you know, if you write the article about Taipei, and you don’t in any way inspect what’s been written, the chances are you’re going to end up with something that is, no human is going to wish to consume.

It will fill up the words on a page. It will look like 10,000 words, in the same way that everything else with 10,000 words would look. But you’ll get into it and you’ll immediately have an intuition that, actually, what is this? There was no care taken in this. Okay, that’s interesting.

[00:09:31] Paolo Belcastro: And what’s interesting is that, that happens of course. The tools are there available to everyone, and so definitely people started using them to produce tons of content. It’s not very different from what they were doing before. There were already content farms before, and people writing tons of articles about subjects that they know nothing about.

I don’t know if you like cooking, but if you have looked for a recipe online, you have all these sort of SEO content, that is completely irrelevant for cooking the cake, but it’s there. That existed before. AI allows you to do more of that, but it’s not more value than before. If you don’t share your experience, if you don’t share your feelings, if you haven’t lived through the things, then it’s not Interesting, interesting for humans.

[00:10:13] Nathan Wrigley: There was something that you said a moment ago, and I’ve written down a series of questions and I’ve shared them with you. But there was a couple of things that you said which interested me, and I’d like to pursue that for a moment.

You talked about the different things that Jetpack AI can do. And you mentioned you can do translation, you can change the tone. There was a bunch of other things. Are you only able to do those variety of things that the AI companies provide with their API? I don’t know if that’s how it’s working. So they offer the capacity to change the tone, they offer the capacity to change the, I don’t know, the language or what have you. Is there a limited subset of things that you can do because of what they provide to you?

[00:10:51] Paolo Belcastro: Well, we actually do a bit more because, yeah, I got on a tangent and we haven’t finished what Jetpack AI can do for you. So the way those API work, and this is the very interesting part is that those tools, those large language models as they’re called, manipulate text, generate text basically. The core abilities generate text, actually their core abilities figure out what the next most likely word is after a given text.

But text is not only content. Text can also be instructions, or text can be a configuration file, text can be a JSON object, it’s all text. So we can do more sophisticated things than just generating text. For example, the first very basic step is that, if you go to Jetpack AI and oh, can you give me the list of the 10 most visited monuments in the world, ranked by order of traffic, and put them in a table.

Now the AI is going to do couple of things. First is going to, of course, go and seek that data in the training data they have. Or online because now they can actually go online and check on other websites and get fresh data. Then it’s going to create content you asked, but then it can also format it, in this case for a table using markdown, so that in your editor you get a table block inserted, as opposed to a list of things.

That’s when the AI becomes interesting, is that, anyone who has used a table block in Gutenberg knows that, once the table is there, if you want to reorder it, for example, it’s a little bit of an annoying work, because you don’t really have a way to select all and say, sort by this field. The AI can do that, so you can say, reorder this table by alphabetical order, and it’s going to do that.

And we push the experiment a little further with Jetpack Forms. So Jetpack Forms allow you to create forms, a recipe form, a contact form, a subscription form, whatever. And the standard way we have been doing it for a while was that, when you insert a form block, we offer you seven templates. You choose one, or you choose an empty. And once you have your form, then you are manipulating manually. You can add blocks, remove blocks, all the sub elements of the form.

And now we have added the AI system to the form. So you can ask for a form that does exactly what you need. So for example, your RSVP form, let’s say it’s for a dinner, you can say, well, add menu to indicate allergies to food. It’s going populate it with the most common ones. Of course, you can ask for more if you need. Oh, actually I want to know where people come from, so can you add a menu with all the cities in Austria, because maybe I’m organising a dinner in vienna and I want to know that.

And that is very interesting because on the one hand it’s practical, because filling those menus with all the options is very long. But also now the AI is manipulating Gutenberg blocks. So we had to figure out how do we get, because typically, for example, something interesting is that the API will stream the result back to us. So we get that as a stream of data, but we cannot display it as a stream of data, because otherwise blocks would be invalid as long as they’re not closed.

So we had to learn how to buffer those elements, so that we only show blocks. But we don’t want to wait for the whole stream to be finished, because it’s nice to stream the result. And so now what we do is that we buffer the stream, so that we showed block by block, and so we stream in bigger chunks. This is all experimentation that is made possible by the fact that, yes, those API only manipulate language, but actually everything we do is language, because coding is language.

[00:14:32] Nathan Wrigley: It just occurred to me, sitting here now having this conversation, three years ago, if we were to sit here, the same conversation that we’re having now would’ve been absolutely unimaginable. There’s not one part of it where any of that was plausible. That was science fiction, what you just said, three years ago, that you write something in and a service which has seriously large amounts of data at its disposal, gives you information back, and then the website that you run puts it into a table, you can then tell it to be reordered, and we could go on. There’s a million different permutations.

Completely impossible. And yet now, well, not mundane, that’s the wrong word, but you get the point. It’s now trivial, and it’s what everybody expects. And that’s really remarkable.

[00:15:18] Paolo Belcastro: Yeah, that tells something about how progress doesn’t happen always in a continuous line, there’s steps. And we have seen those before. I mean, I was one of those people old enough to watch the presentation first iPhone. That keynote, that is almost ridiculous to think that this is just a consumer device, it’s like a piece of metal and silicon and everything.

But back then it made me happy, curious, excited to try a new gadget. Today, it almost makes me cry when I watch it because it’s a pivotal moment. Our lives have changed. Our lives were some way, and now are a different way.

As much as someone can disagree with, you cannot disagree really with that. You can be angry about it if you want, but the reality is that having internet in your pocket has changed the world. There were other moments like that, you know, your first dialup internet connection.

For me, that’s where it stops because I was born in 1970, so I had a computer in 1982, and that’s the first of those moments. I assume other people might have that with television, or the car in the past. So there’s those pivotal moments in our history that are really important. So I think this is one of those, that’s why it’s so critical.

And then though, it tells also something about our ability to adopt new technology fast. Because the first computer, I got one in 1982 in my house. It took years before there was a computer in every house. And then the iPhone took a lot less, but still a few years before everybody had an iPhone, or a Android equivalent phone in their pocket. And now because each one of those layers has enabled the next one, well, when the pivotal moment is an app, and everybody has a phone with internet in their pocket, the path is very short.

[00:17:08] Nathan Wrigley: To say that AI is available to a significant proportion of humanity, I think is true. And it was available right out of the gate, because you can do it from the phone, you can do it from your laptop. Incidentally, my moment there was the first time that I saw a flat LED color screen. I remember just staring at that thinking, how did they get color into that? It was remarkable.

Okay, so, absolutely true, there’s AI everywhere. Tell us what it can do at the moment to help WordPressers. Obviously you’ve been through that you can create forms, you can ask it to create content for you in the form of text, and what have you. Can it do other things? Is it possible to, for example, create entire pages, or entire websites, or images that can drop in?

[00:17:51] Paolo Belcastro: It can. So in terms of images, for example, we launched a feature recently that is actually not in Jetpack AI yet. We launched it on wordpress.com, which is a logo creator. And so what it does is that when you say, oh, I need a logo, because most WordPress themes have a space for a logo somewhere, generally on the top left corner. It takes whatever content is already in the site, maybe the tagline, the site name. There isn’t much generally, because that’s often something you do at the beginning, but it takes that and generates a logo for you.

And then of course you can alter it. You can chat with the assistant to say, I don’t like this, or I want it monochrome, or I want it more stylish, or I want it more modern. So that’s one example. We started with the logo because we figured that was one of those situations where you generally want to create an image out of nothing, as opposed to modify existing images.

Now, there’s other developments at Automattic, for example, on the WooCommerce side. I have to admit, I’m not up to date with the latest things they launch, because it’s a big company Automattic now, and it’s hard to follow everything in real time. But they are launching features centered around managing a store online.

There are a few tools you probably have heard out of WordPress, that allow you to create websites, or landing pages with AI. I cannot really go into detail because there are experiments running inside Automattic right now, they’re not yet ready for prime time. I only can say stay tuned. AI can do a lot more than just manipulating text.

[00:19:25] Nathan Wrigley: It feels like the goal for many people, the desire for many people, is to put a non-technical user in front of a computer, and within five minutes, something approximating a finished website will appear. So like you say, you can handle the logo right at the outset.

We install WordPress, vanilla version of WordPress, and we go through some kind of process. Tell me about your website, just give me a few sentences. Where are you based? Do you want a contact form? Yes. And then out the back, five minutes later comes a website.

I don’t know if that’s ever going to deliver the perfect website, but it feels like a lot of people would love something like that. And I think there are commercial products that do things like that, but I don’t really know.

[00:20:10] Paolo Belcastro: So there’s two interesting things here. I want to put a pin on the non-technical user that you mentioned, because I think we should come back to it later. It’s a very interesting point of discussion. But going to the creating the site, yes of course, everybody wants to be able to very quickly get to something that resembles what they need, or what they have in mind.

It’s not a new thing. For example, WordPress through themes has always allowed something like that, where you look at a theme repository, you’re like, oh, I really like this one and I can picture it with my images and this. And then, yes, once you install that theme, it took a little bit of knowledge to get it really where you want it. So definitely, we can have AI tools that accelerate this process.

And what’s interesting here is that , I go back to what the human puts in, it’s still going to need an understanding of how should a website, for example, you are building a restaurant website, the fact that you have tools that make it easier and faster to get the result you want, still require you to know exactly what is the best way to present a restaurant, so that it has more appeal, that people want to book it, that people understand the menu.

So what’s interesting here is that, again, we build tools that accelerate the process, but the value of the human remains the same. If someone has built sites for 35 restaurants, and someone else has never built one, the results that person that has experience will get with AI will be much better than the person who has never built one. Because, out of the box without the human input, the AI will have a tendency to, again, go average. That’s the whole point of figuring out the most likely token that comes out of a suite of tokens is converging to the middle. And so the presence of the human makes the whole difference.

And then there’s the question of, oh, is this a tool for an end user building one site, or is this a tool for a professional building sites for clients? I would argue for both, because the professional, probably the AI will save them less time per site, because they already had processes, they already had templates, they already a system to streamline launching a new site. But maybe they save less time per site, but because they do that hundreds of times a year, the benefit is huge.

Whereas the end user, they will save a lot more time on a given site. But maybe they build only one, so that’s going to be a one time gain. I think also about the fact that now we can customise the way AI behave a lot. There’s custom instruction, there’s system prompt, there’s a number of tools. I want this AI to consider this context, this data, this frame of reference. And we can imagine there that professional user that does that very often will invest more time in customising their AI, and making it suitable to their need, so that it’s more and more efficient for what their specialty is.

[00:23:09] Nathan Wrigley: Yeah. It’s interesting because one of the key things about your talk was, you really did want to make it clear that you are very bullish about the fact that AI is not going to be taking away jobs from web developers. I’d like to sort of develop that a little bit, because I, like everybody else, have been confronted with that possibility. And in my mind I’ve thought, if this gets really good at doing all the things that I can do, and it appears to be, that’s not a horizon which I can’t foresee us arriving at fairly soon. But you are much more sanguine. You think there’s always going to be space for the developer. There’s always going to be space for the human, and the technical things, the experience that they bring. So let’s just unpack that a little bit.

[00:23:48] Paolo Belcastro: Yeah. So let’s use a parallel with another technological revolution. Let’s think about the Industrial Revolution and Agra Revolution. There used to be a time in Europe where 9 out of 10 people needed to be farming the land or breeding animals. Before the Industrial Revolution, 9 people had to work in a farm to feed 10, and those 10 were themselves, and then one. So that gave one person the possibility of doing something else. And then we invented more tools, and mechanical tools, and we invented engines, and we automated all that. Today, 250 years later, maybe one person, every thousand, needs to be a farmer to feed everyone.

Now, there’s two things that are interesting here. First of all, it has taken a long time. It hasn’t happened overnight. Also, still today, we need farmers. So my point is to say AI, as an entity, is not going to take your job. Another developer using AI might be taking your job though, because they’re going to become a lot more efficient. But the reassuring part in that is that now becomes accessible to everyone. If I am afraid that AI is going to take my job, how can I compete with a machine? But if I’m looking at another person using a tool that I can also use, now it’s up to me to say, well, if I want to keep doing this job, I can use the tools, they’re available to me.

Now what’s going to happen, like in every previous technology revolution, is that it’s going to also create a ton of new opportunities. So there might be people that would’ve been developers without AI, and now will be something else. The same way as, there were 9 out of 10 people farming the land 250 years ago, now it’s 1 every 1000. We are way more people on Earth, and still almost everyone has a job.

[00:25:43] Nathan Wrigley: Yeah, those 999, they have things to do.

[00:25:46] Paolo Belcastro: They have things to do. Even just the last 50 years, like we look at the computers have removed a ton of jobs that are now useless, because they were so easy to automate that we didn’t need AI. I remember, for example, when I was a kid, a friend of my parents, I remember I was impressed because I visited the place and I was impressed. They were working in one of those central telephone station, where you would connect people.

[00:26:12] Nathan Wrigley: Oh yeah. Literally the wires plugged into the wall.

[00:26:15] Paolo Belcastro: That’s now completely automated. But those people, like they did something else. So AI is a tool. As a tool it’s going to accelerate the work we do in some areas. Create new opportunities in others, but it’s just a tool. And like any other tool, if it’s just laying down there, is not going to do anything.

[00:26:33] Nathan Wrigley: Yeah, I think the thing that maybe causes people concern is, a, the fact that it all seems to be happening so quickly. So, you know, the farming analogy, it probably took many hundreds of years for the, slowly the technology became better, incrementally. It wasn’t like we went from 1000 farmers to 1 farmer, because somebody invented the combine harvester, and the motorised tractor overnight. So I think we’re living through that, potentially quite scary bit, because it’s all happening so quickly.

But I do like your analogy of the developer armed with AI, is likely to be a better developer. Given two equal developers, who’ve got the same experience. You know, if you could literally clone a developer and have an identical one, but put AI with this one, and this one is not allowed to use AI, you can imagine that the AI version of that developer is going to get to a different result, possibly quicker, more efficient, and what have you. And I like that. That kind of makes the AI a partner in a way, feels nicer.

[00:27:34] Paolo Belcastro: Yeah, definitely. And I would add one thing that feels even more nice, which is that, if you could clone this developer just for the sake of the experiment, and one has AI and the other doesn’t. It’s not just that the one with AI is going to deliver faster. Fast forward six months, they’re not the same developer anymore, because the one with AI, in those six months, has been able to remove from their plate a lot of the tasks that are not really creative, where they don’t grow, and focus on the ones that are really hard.

So that’s when I was mentioning, I used the term tedious in my presentation a lot yesterday, because I was saying the AI can remove the tedious part of your work. Typically in the work of a developer, there is time spent thinking. By the way, most people overestimate the time a developer spends writing code on a keyboard. Most of the time you spend thinking. And then there’s the time spent writing on the keyboard. And a lot of that time is spent writing the same things over and over, because there are elements that are needed.

Whenever you start a new plugin for WordPress, for example, there’s a whole file structure that you have to put in place. Whenever you do this and these things, you will have to add all those sanitising functions, and you will have to add those unit tests, and that is not really creative. This is stuff you have done 20 times, 100 times, 1000 times. You just do them automatically.

If you can save all that time to focus on the pieces that you have never done before, that’s where you grow. You challenge yourself. So six months in, those two developers have a very different trajectory, that’s the more important point. It’s not just the acceleration, it’s that the learning experience is accelerated by removing those pieces.

[00:29:15] Nathan Wrigley: It feels like you are definitely reassuring me, because I think there is a part of me which sits on the worried side when it comes to AI, but every time you say a little bit more, I feel a worry going away, so this is good.

Let’s keep going down that track. Can you become better at using AI? Is it possible? In the same way that two people who can write PHP code, somebody is going to, you can just become better at it. There’s certain things that you can learn. Can you do the same with AI? Are there interesting novel ways to improve? Because I keep hearing in the media, that learning how to prompt AI is going to be a valuable skill of its own in the future.

[00:29:54] Paolo Belcastro: I think so. Definitely yes now. I would add that it’s imaginable that, as those tools become more and more sophisticated, the difference between abilities to prompt might fade. Because part of the skill today is in understanding the limitations, and leveraging, or working around those limitations to get the result you want. So it’s possible that as we go forward, those limitations disappear, and more sophisticated tools require less precision.

So what’s important today, I would say is training, testing, trying. I think that there is nothing like wanting a result, and trying in 20 different ways to get to that result, and see what works best. If you ask a question to, let’s use ChatGPT as an example, as the most popular. You ask a question to ChatGPT, it’s going to give an answer. There’s many ways to improve that answer. You can go back to your question, and you can say, my question might be a bit vague, maybe I have to specify the context.

Typically a lot of the early work was about saying, well, you have to tell ChatGPT who they represent. What is the role they’re playing? I remember, for example, reading, this was a blog post from the people who built Copilot at GitHub, and they were explaining a principle. It wasn’t exactly how they did it, but more the principle. They were saying, imagine you are configuring an assistant to be a support agent.

Now, you could tell the assistant, you are a support agent, and you are helping a customer fix their cable connection, whatever. The customer is angry, they want to watch their football game, you have to fix that fast. And maybe you can even say, oh, and here’s the documentation where you need to find information. But then what they were saying was, you can be more subtle and say, for example, you are an excellent support engineer, best of your class, and you work at this company where you support clients, and there’s these clients, there’s this case study of a client that had this problem.

And as it happens, you open your briefcase, and you look for the documentation of this case study, and the case study explains how you fix the problem. And this is the text that is on the page. And that’s where the AI takes over. And so that’s a way to say to the AI, you are an expert support agent, doesn’t mean much for the machine because the machine is not a being, sentient, these are just words.

So if you set the stage for the text they are supposed to be produced in the highest quality, which is, here’s an intervention where we help a customer, and it was so good that it’s a case study. And now you still give the documentation and everything, but you have set the stage for the AI assistant to actually continue that conversation properly.

This is one type of example. There’s another example of breaking down into step-by-step reasoning, chain of thought. You ask a question and you say, for example, you asked to write a function that does something. You get the output, and then you can ask the AI, can you explain me, step-by-step, how you got to this result? And now what’s interesting is that, by redoing the same work but step-by-step, the AI will develop each step and make corrections along the way. Because suddenly, where the first output was the result of this probabilistic analysis, the second output changes because, once the first point is laid down, oh, suddenly the second point that is more likely is not what was there before.

Because before it was seen as one thing. And now, step by step, the AI, there are people that say, this is not something I really have, I mean, I’ve tested, but just very briefly. There are people that say that you get better answers if you’re polite. And the idea behind that would be that, in the training data, conversations where people were polite were more likely to be constructive and helpful. Whereas conversation where people were rude, were probably less likely to be constructive and helpful, and therefore you direct the AI to use one area of the training data as a support.

But there’s other people that say that, actually it’s better to be direct and ask for the thing, as opposed to, can you do this and that? Because if you say, can you do this to a human, they’ll do it for you. But if you say, can you do this to a machine, they might sound like the sarcastic human that answers, yes I can, and then doesn’t do it. There’s people who have experimented with tipping, which you cannot actually do, but you can promise. You can say, hey, I’ll give you £100 if you do this and that for me. And apparently they say that the result would be better. Which, it’s hard to understand there what is the truth.

Recently something very interesting happened. There’s Anthropic launched Claude 3. It’s supposedly better than GPT 4. It has three levels of quality, based on how much you’re ready to pay for the responses. They started testing it. And now there’s a number of tests that are run on these LLMs to evaluate their performance.

The test that they ran was funny. It’s called the needle in a haystack test. So the idea is that you give a lot of content to the assistant, and then in that content you put one paragraph about a subject is completely unrelated to the rest of the content. And then you ask a question about that specific point. Previous versions of LLMs had a tendency to not be able to answer, because the fact that that content was very short, compared to the mass of other content, it was kind of overlooked.

On top of that, apparently LLMs had a tendency to over index on the beginning of the provided content, and the end, and less the middle, which by the way is very much similar to humans.

[00:35:50] Nathan Wrigley: Absolutely.

[00:35:51] Paolo Belcastro: But in that context, so there was, I can’t remember, the 200,000 tokens were about some subject, I can’t remember what. But the one paragraph that was the needle in the hay stack was about pizza toppings. And then they asked a question about pizza toppings. Claude 3 replied the right answer, so it got the reference.

But then added, it’s odd that there is content about pizza toppings in this text that is very long, and has no relationship with the subject. It’s almost as if this was a test for me to see if I’m paying attention.

Which is very funny because the first reaction is, oh my god, this thing is sentient. When the reality is very likely elsewhere. I think that the reality is much more that there might be in the training data mention of this type of text, because it’s hard now to isolate the data and say, oh, we shouldn’t train AI with anything that talks about AI. It’s almost impossible.

[00:36:45] Nathan Wrigley: Oh, so the AI may have picked up on a piece of text describing an equivalent AI test.

[00:36:50] Paolo Belcastro: I think that’s very likely, that it might have picked up on a paper, a research paper, or something like that. Now, is there a possibility that I am completely wrong, and these things have been sentient for a couple of years already, and they’re just telling us bit by bit? Yes. I don’t have a way of being 100% sure, but I would say I think that everything that, I think it’s, what is, Hanlon’s razor?

[00:37:14] Nathan Wrigley: Oh, Occam’s razor.

[00:37:16] Paolo Belcastro: Occam’s razor is that, the most simple explanation is very often the right one. And so I’d be tempted to still think they are tools. They use mathematics to figure out language, and so they are able to produce those answers that make a lot of sense.

[00:37:31] Nathan Wrigley: But I guess the thing that’s interesting there is it’s so surprising. And it presents us with things which are well within its own capabilities, but they’re beyond the bounds of a typical human. So it seems like there’s something else going on. But it’s more plausible that it’s just a function of the data that’s been given. But because we can’t hold onto that much data, it seems like a god-like quality, but it’s not.

[00:37:53] Paolo Belcastro: No. Yeah, and we have other examples of things like that, that we got used to over time. I mean. If you recall, you probably have seen, I mean, neither of us were alive back then, but we probably have seen the images. When the Lumiere brothers showed the first movie of the train arriving in the station, everybody ran away. It’s very similar. If you think today, like the quality of that black and white movie, how can you imagine that being realistic? But it is.

I grew up with small TVs and black and white images, and it felt surreal. And now, of course when I look back at those things, I’m like, how is that even possible? You know, we talk about impossible things for humans, but I came here on a plane. I try to fly by myself, doesn’t work.

[00:38:38] Nathan Wrigley: I never get over plane journeys. It’s always remarkable.

[00:38:41] Paolo Belcastro: How incredible is that? And so, yeah, this one is new and so we have this moment of reaction. At the beginning you said something, or we going to talk about AI 1, 2, 3, 4, 5 years from now. I think AI is going to stay. I don’t think we’re going talk about AI as much, as we don’t talk about electricity as much.

[00:39:01] Nathan Wrigley: Yeah, or the film, or the aeroplane.

[00:39:03] Paolo Belcastro: It becomes part of the landscape. It’s there, but the novelty is we have all those images. On top of that, you were mentioning before that the, at the moment of the Agricultural Revolution, people weren’t scared about what was happening, because it didn’t happen overnight. I would also argue, they didn’t have a ton of movies and TV shows showing them a future with no farmers.

We do have that, and so the moment we hear AI we’re like, oh my god, Sarah Connor, where is she? She has to save us. But these are movies.

[00:39:34] Nathan Wrigley: That’s a really good point. So the fact that we’re in this 24/7, plugged in, media fed ecosystem, you know, where you just pull out your iPhone and you’ve got data coming in about the news. Yeah, that’s a good point. I hadn’t really thought about that. And so if you were the farmer in the, 1500’s, nobody was telling you, this is a concern, this tractor. So you’re not going to be concerned. Yeah, it’s a really good point.

Do you trust AI with your code? Do you get it to generate code and find that it’s largely error free? I was saying to you before we hit record that, if you create an image with AI, and it’s slightly incorrect, it doesn’t really matter, because the fingers look strange, or the ear lobes look a bit weird, but we can forgive it.

And in the same way that if we create text for our blog post, we can modify that, or we may be happy with it at the beginning, and just accept that it’s got a few errors in it, and we’ll publish it anyway. But with code, I think we could be introducing security problems, or just simple errors. How confident are you writing code with AI, and how much do you have to go back and check every line?

[00:40:36] Paolo Belcastro: So this is a really good question. And I will use my specific context to answer it, because I haven’t been writing code that goes into production for a living since 1998. At Jetpack I am head of product, and I’ve been working with engineers for the last 26 years. So my code, I do still for fun, do stuff on my own, but is not critical at all. It’s little plugins that I’m the only user of. So it’s not really a concern for me.

But what I would say is, this is an area where it’s really interesting. And that’s where the dimension of AI as a tool really is important. If you are a really good developer, you won’t have that problem because the AI is going to make you save time. It is going to make you move faster, but you’ll be able to recognise the problem in an instant. The same as if I am writing either blog posts for my blog, or if I’m writing internally at Automattic for our P2, you know, about Jetpack, about the blog, about any of the things I work on.

I know the field I’m writing about. And so, if at some point in the process, and the back and forth I use AI for, it introduces something dumb, I see it instantly. My point basically, because it’s a tool, and it’s not sentient, and it’s not intelligent, use it to help you in area you have expertise. In the areas where you are an expert, you can use it to accelerate your work. In the areas you are just interested in, you can use it if it’s not critical work.

But please don’t go near it in areas where you have no knowledge, and or critical. As an example, I would say, a surgeon using AI to go through papers faster might make sense, because they can spot the mistake. Me using AI to diagnose symptoms that I have, maybe I found a rash on my arm, and I’m going to ask AI, very bad idea.

So I think in that scale, if you imagine that as a two by two matrix, am I the expert or am I a total newbie in this area? And then on the other axis, is this critical or is this completely safe? If we are on the safe side where there’s no consequences, it’s like a little plugin I’m writing for myself, or I’m asking AI about five things I could visit in Taipei, great. I can use it and just take the answer. Worst case scenario, it doesn’t work. If it’s critical, then I have to be the expert, and the AI is a tool that helps me be better at my job. But is not a tool that helps me do a job I’m not an expert of.

[00:43:17] Nathan Wrigley: Yeah, that’s a good answer. So moving on, I’ve got a few more questions. We’ll try to be as quick as we can. The first question was, there seem to be a few companies which are dominating the AI space. So you’ve mentioned already Google, I think we could throw Open AI into that, and Anthropic, and there’s probably some more, but they’re the only three that come into my mind.

Does that part of the jigsaw puzzle concern you? That we’re increasingly reliant on three or four, maybe there’s more five, six large language models. And I just wondered if, given that we’re WordPress and we love open source, if there’s a platform, a project, which is trying to do something, but more in the open. Even though it’s called Open AI, it doesn’t feel particularly open.

[00:43:58] Paolo Belcastro: Yeah. So the interesting thing in the last year or so is that there has been a lot of movement on the open source side. It’s a bit ironic that the biggest advances on the open source side come from Meta, which is yet another very large company. But it doesn’t really matter because that’s the power of open source is that, once it’s in the open. So one thing with AI, and LLM’s, and open source is that it’s really important to understand what is actually open. Because an LLM, in terms of code, there isn’t like that much code.

If I give you an LLM post-training, it’s great to have it, and to be able to read the code and everything, but you can’t really modify much on it. If I give you the training data, then it’s a lot more interesting, because then you can reproduce. Now, keep in mind I’m not at that level technically. I am very much out of my comfort zone. But so my point is that, what has happened is that the first few foray to open source, or kind of open, but only the tail end of it. And so, okay, you get a tool that you can use, but you can’t really modify it, or you can’t really understand how it works.

And now there are versions that are open source from A to Z, including the dataset. Now, they’re not as big dataset, they’re not as powerful as the ones from the big names you mentioned. But what it appears is that it’s actually evolving very fast, because of course, being open source, once you put that in the hands of thousands of developers, in conjunction with the fact that we live at a time where computers can actually run that stuff locally. And so developers on a MacBook Pro, M3 can actually work, which wasn’t the case not that long ago. I think that this is going to catch up very fast.

Now, I believe, frankly, that is going to resemble software, where you know, you have WordPress that is the leader, CMS, like 43% of the websites and everything. That doesn’t stop closed platforms from existing. The two systems are probably going to be side by side. What we see now, for the moment, but again, we are super early stage, is we’re still at a moment where the closed platforms are moving faster. Because the kind of means that Open AI, or Google, or Anthropic have, open source takes a bit of a longer time to set up.

Like it’s very fragmented. A lot of people are doing different things. There isn’t like a strong coordination. And also they have then the infrastructure to run those models at scale, and to provide answer very quickly at a cheap price. I don’t expect them to be shadowed by open source anytime soon. But I do think that we have definitely hope that, going forward, we’ll have open solutions.

[00:46:50] Nathan Wrigley: Yeah. It feels like that would be a nice outcome, if that did happen.

[00:46:54] Paolo Belcastro: One thing that is also important is that, for example, today the approach of Open AI, and Google, and Anthropic is very monolithic. It’s saying, we have this gigantic thing that knows everything and can help you. Whereas the open source approach is, because we cannot run such large models locally on laptops, it’s more to focus on simpler agents, that are more specialised at given tasks. And that then more complex task would be a collection of simpler agents, maybe orchestrated by another simple one.

So it’s interesting because we have seen the approach in the industry in general, right? You have monolithic industries that will build like a car. And every piece that goes in it, and every little part, they will have factories doing everything. And then you’ll have other type of industries where the same car is built by buying the tires here, the battery there, the engine there, the windshield there, and then assembling all that. And I think that both models can work. I don’t see a future where there’s going to be either only one, or only the other.

[00:48:00] Nathan Wrigley: I think that’s good. That makes me feel better about it all, because it does feel that, at the moment, it’s all about Open AI, and they really are sort of dominating. I would like to think of a future where there are different alternatives.

Okay, couple of questions. We’ll be very brief about these one, because I’ve used up lots of your time, but I’ve enjoyed it, I have to say.

[00:48:18] Paolo Belcastro: No worries.

[00:48:18] Nathan Wrigley: In your presentation, you make the point that AI offers you the possibility, and I’ll just quote here, AI removes all the tedious parts of our work so that we can hone our craft, and there’s more but that’ll do for now. I had this intuition when I read that, and I just thought, maybe tedium, tedious, boring is an important part of the human condition. And I don’t really have a question around that, but is life spoiled a little bit, if there’s no boredom anymore, if everything’s quick and rapid? From a business perspective, that’s a dumb question. But from a sort of psychological perspective, and just a human nature perspective, I wondered if you had any thoughts on that.

[00:48:58] Paolo Belcastro: Yeah, I actually don’t think it’s a dumb question, even from a business perspective. I just listen to a little snippet from Rory Sutherland, who talks about, of course he’s in the advertising market, and he’s talking about using AI to remove all those steps and create a visual for a campaign, for example. And he was saying, we have to be careful, because in some of those jobs, the end result is not that important. It’s everything we learn along the way that is important. It’s the questions we ask to our users, to our vendors, to our sales people to get to that result that are really important.

And I would say, I think that is very clear, that if we consider AI like a black box, in which I input an order, and I want a result on the other side, it’s not a good way to use it. It’s not progress. It’s not going to help us learn. So I go back to the example of me writing a blog post. If I tell the AI, write me a blog post about this subject. And even if I add, oh by the way, in Taipei, day one I did this, day two three, here’s five things I did, write me a blog post.

The blog post at the other side is not going to be great. I won’t have learned anything. I won’t have grown in any way. But if I use AI as a partner to write this post, and so what I do often is start by writing a draft that is really raw, really stream of consciousness, as it comes. Then I go to the AI and I say, can you, out of this draft, create an outline, and highlight the main ideas I’m talking about?

And that first step already allowed me to do something critical, which is that I have a natural tendency of holding too many ideas in my head, and then writing blog posts where I talk about too many things. And then I lose people, and then people don’t really know what it’s about. Or then there’s no conversation after with my readers, because there’s so many subjects they don’t know what to talk about.

But by giving the AI the draft, and asking for an outline, and the main ideas, now I have much more clarity about what I was writing. So now I can reorganise this outline and I can say, actually, okay these go in that order, and these actually go in this order but out of this post, this is another post. So let’s move that aside.

And so now I can go back to the AI, and have my simplified outline and I say, using my original draft, and using this modified outline, reorganise my original draft please, and write a new draft that fits this outline. And then I read that. And then I fix a lot of things. I correct, I remove any word that I wouldn’t use, and I put my way of speaking in it. But then I can go back to the AI and say, what is missing here? Am I missing an obvious point, or how am I wrong in this? And I sort of get the AI to challenge me a bit.

This is one of the things I mentioned in my presentation is that, it’s not just about accelerating, it’s also that the AI provides us with a partner that can give us feedback about our content. There’s the basic grammar, typos, and things like that, but also feedback about the quality, the clarity. But also a challenging partner that doesn’t belong to our own intellectual bubble.

Because when I ask feedback to my friends, I’m asking feedback to people that kind of think similar to me. I mean, we’re friends. What I’m saying basically is that, when I say removing the tedious part, I’m not saying shorten the path. It’s more about saying, remove the things I’ve done 20 times, 50 times, 1000 times in my life already, that don’t teach me anything anymore. And allow me to improve the quality, by challenging me, by giving me feedback, by doing those things that normally I wouldn’t have access to. Because, I mean, I’ve occasionally asked for feedback on a piece to friends or family, but I can do that once a year. I can’t bother people around me all the time like that.

[00:52:54] Nathan Wrigley: Yeah, the AI is good, it doesn’t get bored.

[00:52:56] Paolo Belcastro: I want to be very clear with that, that it’s not about shortening the journey itself, as much as it is increasing vastly the density of the enriching steps.

[00:53:06] Nathan Wrigley: Yeah, you said, at some point towards the beginning of your answer, you encapsulated it as, something along the lines of, it should allow you to grow, you’ll grow from it. That seems to be a nice place to sort of round it off.

So the idea of AI, if I’m getting you correct, is that it’s kind of a partnership. It’s not something you should use in isolation. It’s certainly not something that you should use if you don’t possess some capacity already, especially with code. And so use it as a partner. Enable it to speed things up. Enable it to teach you in the same way that you might go to the library and read a book. You’re going to learn something from it.

So it’s not something to be feared, it’s something to be used. But it’s also, I guess this conversation is, be mindful. Don’t fall into those pitfalls. Don’t just use it blindly to create those articles that the audience is going to be zero, because it’s going to be so anodyne and uninteresting.

That’s interesting. I was definitely more on the, not the terrified side, but the nervous side when we began this conversation. And you’ve definitely taken me off the ledge a little bit. So thank you so much, Paolo, for chatting to today. I really appreciate it.

[00:54:14] Paolo Belcastro: Oh, it’s my pleasure. It was really nice to have a conversation with you and nice to meet you.