Over the last few years, our team has experimented with AI at work and at home. We’ve built tools that help us render complex images of scientific equipment, pressure-test a product roadmap, and remember to give the dog his pill. There were plenty of flops along the way, but gradually we settled on ways of working that built a successful track record. We asked our co-founder Mike Swartz and CEO Joey Camire about their own AI habits. Here are a few of their takes on working effectively with it without sacrificing quality. (Maybe the most important takeaway is that most of this applies to human collaboration, too.)
Cognitive security, or don’t let Claude tell you your idea is great and then send it out into the world without running it past a human.
People naturally apply constraints to themselves, for better or for worse, like shame. That’s a powerful behavioral tool for us, but agents lack it. So we use different types of guardrails when working with agents. Some of it is common-sense technical security stuff, like having data security policies in place, but creative and cognitive security is just as important. That’s where our group chat is really useful—we’re finding the edges together, sensing where there are risks or quality issues. There haven’t been any mega fails yet, no production codebases deleted, but you have to hold both things in mind: it’s possible this thing could make our work so much better in all these ways, but it could also cause deep damage. You have to activate your inner Mike Burns [Upstatement’s CTO]. We’re constantly navigating that tension, and probably erring more on the side of conservative security for things exposed to PII or production databases. But at the same time, we really try not to have limits on experimentation itself. No one should ever feel like they shouldn’t try something. Try it; it’s the only way to know for sure. Failing is actually not a fail, it’s just another learning loop.
Cultivate caution. People who used these tools back when they were visibly bad learned to double-check. If you didn’t start early and learn it the hard way, you have to deliberately develop that skeptical instinct.
If you’re showing up now, when the output looks polished on the surface, you can easily miss the subtler hallucinations. But the people who were playing with tools when they were clearly not ready for the main stage developed a fluency with their capabilities, an ability to identify flaws, and a level of caution that you just don’t develop if you show up when things are already pretty capable. If you didn’t see runs one through four, where it was making massive, obvious mistakes—that’s a trap we’re seeing now. People pick it up and say, “This thing’s incredible,” and of course it is. And also it’s hallucinating like crazy, but more subtly and deceptively now. That internal mechanism of “I better double-check that, is that right, did it just make that up, can I verify that citation?”—those behaviors were established when you were using ChatGPT in a browser window, not working with an agent that’s ordering dog food while recompiling your kernel.
Performative asking is worthless. If you hand a model your work and just want to hear it’s fine, you’re not collaborating. The value shows up only when you’re open to being told it’s wrong.
Same goes for the colleague who reviews your work and won’t help you make it better. If they just say, “Yeah, it looks fine,” they’ve also abdicated their role as collaborator. This has applied to people since forever, but it really applies to interactions with models, and it goes back to engagement with the work. If you're like, “Write the whole thing, make no mistakes, just tell me when it’s done,” you’re not engaging deeply. Whereas if you say, “I want to explore this topic with you,” and you truly read the response and think about what it’s saying and push back, then you can get into a really interesting cycle.
Coming soon: an update from our AI Art Director three years later, and why your chatbot is annoying.
