The website that created an AI clone of its editor in chief

My fiancé works at Anthropic, whose models Every reviews and builds on, and which comes up below.

Earlier in our miniseries on productivity in the AI era, Replit's Amjad Masad described a "self-driving company" where most engineers don't look at the code anymore. For our penultimate episode, I wanted to visit someone running a version of that experiment in a somewhat unexpected place: the media business. Dan Shipper runs a website that reviews new technology alongside a product lab that’s building it — and for some time now I’ve wondered what it’s like to work in a place like that.

Shipper is the co-founder and CEO of Every, which he launched in 2020 with Nathan Baschez as a bundle of business newsletters. Today Every is a publication about AI that draws attention across the industry — for Shipper's column Chain of Thought, his podcast AI & I, and especially for the "vibe checks" in which he and his coworkers get early access to new frontier models and put them through their paces before their general release.

At the same time, Every is also a product studio. The roughly 30-person company offers Cora, an email assistant; Sparkle, a file organizer; Spiral, a writing tool; and Monologue, a dictation app — all of which are bundled with the journalism into a $20-a-month subscription. Shipper says AI now writes essentially all of the company's code, while humans still (mostly) write the essays.

But AI is changing the way that the company writes. Shipper told me Every has tried to clone the taste of its editor in chief, Kate Lee, by collecting a dataset of 30,000 of her historical edits, using it to build a copy-editing agent, and back-testing it against her past work. It’s an effort to capture the expertise of a single employee and distribute it more broadly throughout the enterprise — a preview, I think, of how more businesses will think about the relationship between AI and employees in the years to come.

Shipper was also candid about what it's like to publish a critical review of a frontier model from a lab the company depends on, arguing that Every's role as an arbiter may be one of its most durable assets. "No one trusts a model company to tell you where they objectively sit,” he said.

More controversially, Shipper told me that far more writers are integrating AI into their workflows than will admit it publicly. "I think there is a real dirty secret right now, which is that almost every writer is using it,” he said. “Just, most of them are not saying so."

I also had to ask Shipper a question at the heart of our miniseries: if AI automates the work, why does Every keep hiring? The company doubled from about 15 people to around 30 over the past year while loudly automating everything it can. His explanation — that AI is "trained on the residue of human expertise," but can’t see beyond it — may be good news for jobs in general, at least for as long as it holds true.

An excerpt of our conversation is below, edited for clarity and length. Listen to the entire conversation wherever you get your podcasts — just search for Platformer — or watch it on YouTube at youtube.com/caseynewton. And let us know what you think — we welcome your feedback at casey@platformer.news.

Casey Newton: Every does “vibe checks” of new models. I found when I worked at a site that reviewed gadgets, there was sometimes a tension between what the companies want from early reviews and what you give them as a critic. You published what I would say was a fairly critical review of Sonnet 5 — "a model pitched for everyone impresses no one." How did that affect your relationship with Anthropic, and how much did you think about that before you hit publish?

Dan Shipper: Obviously, we know a lot of people at OpenAI and Anthropic, and you never want to be totally mean to people you're friends with. But actually, even before we publish anything, they're asking, "What do you think?" Because they want to make the model better, and they know that if we don't like it, it means something — and they'd rather know beforehand, honestly, than find out from a ton of other people who use it.

They would probably also prefer that we didn't publish a big thing saying this model sucks. But they know we're not trying to be mean. We just have to say what we think, and if we think that, it's pretty likely a lot of other people are going to feel that way. My goal is never to shit on them; it is to help make better AI happen, and I think we do that in partnership with them. Sometimes it can get a little bit heated every once in a while — they're like, "I don't see how you could feel that way." But that's the exception to the rule.

Newton: The labs are enabling you to do these vibe checks, and you're using their models to build products. But it also seems like they are encroaching on your terrain, and everyone else's. Every ran a piece in June called "Built on Moving Ground," about the vertigo of building on models you don't control, where there's always a risk the labs will release as a feature something you spent the last year on. How do you think about that risk, and where do you see the durable value in a bundle like yours?

Shipper: It's a really good question, and I don't have an answer to it. There is just this dynamic where they're going to make their models better, and their models getting better does actually take a lot of the stuff that you build and make it less relevant. And they also have application layers, so there are different parts of the same company that are supporting you and also sort of competing with you. It's a messy dynamic.

I have a couple of feelings about this. One is that the thing we can do that no model company can do is tell you which models are good. No one trusts a model company to tell you where they objectively sit — how could they? So we have a good position as an arbiter between them, and that's not something that model progress will get rid of. And just because you make the model doesn't mean you know exactly how to use it well. I liken them a little bit to oven makers. You can make the oven, but it doesn't mean you know how to make a soufflé. Our job is to take an oven and say: what is the coolest thing that we could make that would be good? And they're like, "Cool, great, we'll make the oven better for that." But then sometimes they're like, "Well, maybe we'll make a soufflé, too."

The other part is that we live in this zone where things are moving really fast, and we can't rest in any one particular place. You have to both really want to make something awesome and high quality, and be willing to throw it out every three to six months as the capabilities change. That's a hard thing to do, but I think it's possible.

Newton: Where does AI make Every measurably more productive?

Shipper: We would never have been able to do almost all the things that we do without it. For a while we were maybe 12 or 15 people, and we were running six software products and a daily newsletter. That's insane. Even running a daily newsletter that grows, and that people like and read all the time, is hard. Then to add software products on top of that, without very much funding — we haven't raised very much money — it only became possible because we started to be able to get enough from a single engineer that you can have one person run an entire software product end to end. That was certainly not possible before at any real level of scale. 

Now that it is, once you have one person, you start to hire more people, so we have products with more than one person on them. But you can get signal, and really serve an actual customer base with a real product, with one person.

In a lot of ways you can think of, there are a lot of [structural] overlaps between The New York Times and Every. But the Times was only able to do the games bundle and Cooking and The Athletic after 150 years and a lot of scale, and we can start to do that much earlier and more quickly, with less money.

Newton: On the flip side, I'm curious if there's something you keep throwing models at that they're just terrible at, or that feels like a stubbornly human job.

Dan Shipper: Yes, all the time. Let me start simple. One thing we have been throwing models at for a long time only just started to work. We have an editor in chief, Kate Lee, who's fantastic, who I've been trying to automate out of a job for years in an extremely benevolent way. She does a ton of copy editing for us — she has the best copy-editing taste of anyone at the company. As the company has grown, she's no longer just copy editing articles; she's doing launch emails and landing pages, and making sure they all adhere to a standard. But her time is limited. She's an editor in chief; she has many other responsibilities.

Since GPT-3, I've been saying, I think we can make this better. And the answer has been "no, you can't" for a really long time. And it just started to work. Part of that is the models are good enough at instruction following that you can make a good enough prompt that it actually knows what to do in any given situation. Another thing is they're good enough at browser use, or computer use, that they can actually go into a Google Doc and make suggested changes, which is wild when you see it. 

Another thing is they're good enough now that I collected a dataset of 30,000 of her historical edits, used that to make a prompt, and then back-tested the prompt on all of the previous documents to hill-climb and make it better and better. Now we have an agent internally that we use, and anytime someone has a piece they're working on, or a landing page or whatever, they just @ the Every Agent — "do a Kate copy edit on it" — and it does it. It's not perfect, but it's much better than having her do everything. And it gets better automatically over time: as it makes edits, and then she goes in and makes more edits, it automatically learns "here are the things I missed." So that's one thing that just became — we call it “compounding” — compoundable. But even copy editing, which is rules-based, is super, super complicated and not fully automatable even now.

I see what we do less as "we're going to automate all copy editors" and actually more as: Kate has a specific set of skills as an expert inside of Every that she can only apply right now by spending her time. What we do with compounding is allow her to get some of that taste and viewpoint and set of skills into a little tool that lets her spread it throughout more of the org, where she doesn't have to spend her time to do more work. When you start seeing it that way, you're like, of course I want that — a tool I can teach my taste, so I can spend my time on higher-level, more interesting things.

Newton: My impression is that you aren't automating anyone out of a job. In fact, I think you went from about 15 people in the middle of last year to around 30 this spring. You've doubled while automating. I think you've called this the AI paradox, where the more things you automate, the more humans you need to do more things. Did you expect to double in size?

Dan Shipper: As a company, we try to automate everything we possibly can. So why did we double in size in terms of human employees? My ideal world is not one where I only hire agents and we have no humans — I'm not weird like that — but we don't have a ton of funding, and you would expect a company like ours to try to be efficient and not hire people unless we have to. And we've had to hire people. Part of that is that we're growing, so we can and we should. But I think there are also deeper structural reasons why automation weirdly creates more work for humans, especially for human experts.

The way that AI works is that it is trained on the residue of human expertise. It's trained on problems that have already been solved. One of the beauties of AI is that now you have this thing that knows how to solve every problem that's ever been solved, and you're trying to apply it to your problem. The interesting thing is that your problem is slightly different from any other problem that's ever been solved. What that creates is a situation where tons of people are just mashing on their keyboard — "solve my problem" — and it solves it, but only sort of. It's close, but not quite there. And that creates a ton of slop, and that's not really valuable. You have this glut of things that look impressive at first blush, but eventually you realize they're kind of worthless, and the market reprices. So what do you do now? You need an expert to come in and solve the problem for this particular situation — really think it through, and use AI to do that.

And who does that work? Because everybody can now do something that's sort of like what they do — everyone is a programmer, to some extent. [But] experts are involved in building systems to take the people who want to program and contribute, and make that actually productive. The same thing is true inside OpenAI: they have teams of people building infrastructure so that everyone can ask data-science questions and know that the answer is the approved thing that OpenAI would do — because if they just raw-Codexed or Clauded it, it would be something, but it wouldn't be right. 

That's one thing experts do. The other thing they do is moonshot things that wouldn't have been possible previously. So it both raises the floor and it raises the ceiling, and there's much more to do than ever before.

If you told me in 2020 that you could just send Fable off and vibe code an entire to-do app in a day, and asked what would happen to engineering, I'd have said, I don't know, that sounds nuts. And the reality is, we still have engineering. It's just moved up a level.

Newton: Let me ask about writing and AI. You're leaning very hard into having AI do as much as possible, but you're retaining some level of authorship. How do you think about how much of the writing — let's say of your vibe checks — should be a person typing words on a keyboard, and how much of it they can outsource?

Shipper: I will tell you, but first I want to ask: have you ever used an editor? Anyone that's going into your Google Doc and making suggested changes? Have you clicked accept on any of those changes? That is a similar dynamic — especially for public writing that has your name on it — to the appropriate use of AI. You want someone that understands what you think — and you have to know what you think, which sometimes you can get to with an AI — and is helping you create the best version of that. But it's yours, and whether or not you type the words does not really matter to me. But they have to be yours.

Newton: Well, how are they yours if you didn't type them?

Shipper: How are they yours if you just pressed accept on a change that your editor made?

Newton: It's a fair question. But people do have really strong feelings about AI. I don't think most people would get mad about AI suggesting a different phrase — but if it wrote the first version of an entire chunk of your vibe check and you published it as is, people would have feelings about that. How much theorizing have you had to do about where the lines are — and are they drawn in pen or in pencil?

Shipper: A lot of theorizing, a lot of trying different things. They're definitely in pencil, because things are changing. I also really want to separate out people's reactions from what I think the long-term norm is. There's a big difference between our audience and a mainstream audience, which would be much more sensitive to this. I'm thinking about what the right long-term norm is for people who are just used to this technology, for whom it feels like a part of everyday life, as opposed to this new big threatening thing.

There's a long history of this. When I started writing about writing and AI a couple of years ago, I researched the history of the typewriter. Mark Twain was the first American author who really loved the typewriter — he also, I think, spent a bunch of money and bankrupted himself trying to make typewriters a thing, so maybe not as good a businessman as he was a writer. At that time, it was a big deal for him to be into it, because people got offended if you sent them a typewritten letter. It wasn't your handwriting, so it looked like an advertisement. It looked impersonal. This is a very common thing in the history of technology. If I send my mom a text message, it doesn't feel as personal to her as a call — but a call didn't feel as personal to her mother as talking in person. So I'm trying to think about where the norm is going to go.

There's a whole range of different circumstances you have to consider, and we do different things in different circumstances. External writing that has your name on it and is from your perspective — that's different from, say, a long-form guide that is intended to be mostly informational rather than narrative-driven. There, we often include the AI as a co-writer, and I'm much more fine with having chunks of it be AI-written, because I think a lot of it's going to be AI-read. It's informational — the point is to put it in your agent and have it help you when you need it. For stuff that is from a person and feels like it's yours, that's different.

What is good writing, at its core? George Saunders says this, which I love: it is just applying your taste on every word, over and over and over again, until it is the most pure expression of what you've thought. I think you can use AI to help you with that — I do that all the time — but it requires a lot of your time. Writing — this is cliché to say — is about thinking. I often don't know what I think until I write something.

Newton: On your show AI & I, you've interviewed lots of creatives about their creative process. This is a fraught topic in the creative community, but for those who are curious: what has separated the writers who get better with AI from the ones who lose themselves in it?