To truly understand the impact and limitations of a piece of tech, you need to try it yourself. Over the Christmas break, I decided to build something using OpenAI’s infra. I figured it would help me understand the good from the bad, and value from hype.
This essay is a summary of my observations using AI in a product. If you are building in AI or thinking of incorporating it in your product, this is for you.
What did I build?
I wanted to build a tool with some degree of customisation. A product that simply made an API call to Open AI didn’t seem deep enough.
I built a custom Chatbot using my newsletter content. I’ve written 14 issues on open-source, SaaS and AI. On average, I write a 1,000 word essay every week, so that is 14K words.
The bot works as follows:
- User asks a question.
- It goes through all my content and finds relevant excerpts.
- Using context from (2), it answers the question.
You can play with the chat bot here: https://chat.ntkris.xyz
Here are my observations from building this:
Picking a use case
Whilst my intent was to discover the tech, I think it’s useful to start with some framing of use cases. The best way to pick a use case is to forget AI exists.
Find something that is inefficient. Inefficient in terms of time (e.g. responding to customer service requests) or cost (e.g. product photography). Validate it by talking to people and then see if AI can make it better.
There is another category that you could go after: things that are not possible today. For example, a car can’t drive itself today. These are moonshot problems and people will dedicate their life to solving them. It is high risk - high reward.
It goes without saying but my newsletter chatbot falls into neither. I wanted something I could work with in a matter of days :)
Be clear on the outcome
Before you begin, be very clear about the outcome that is sufficient. There are two thresholds you can aim for:
- 100% accurate
- Better than a human being
Generally speaking, I’ve seen very few (zero?) instances where AI is 100% accurate. On the flip side, if “better than human” is a sufficient outcome the possibilities in AI are very large.
A tangential consideration is thinking about the consequences of inaccuracy. If you assume that AI is wrong 1 out of 10 times, think about the consequences of that inaccuracy. If you are recommending a drug, you really don’t want to mess up. If you are providing copy for a website, something incorrect 10% of the time is okay.
Human + AI is where it’s at
AI is not going to replace us, it’s going to empower us.
There are few use cases I can think of where an AI algorithm can replace a human entirely. Even a simple use case like copy writing benefits from human oversight.
AI does run the show entirely in some cases, but these situations tend to be use cases where humans won’t be able to participate at all. Or, it’s been trained with sufficient human input already (e.g. self driving cars whenever we get there).
On the other hand, augmenting AI with human input is very powerful. If someone can do 10 tasks in a day, AI can increase that output by 10x or more. AI produces the first version, a human approves it and off you go. It also gets better over time because of reinforcement learning.
Sifting through lots of information
AI is incredible at sifting through large amounts of information.
This is easily forgotten because of the term “generative” AI. Aside from creating new content, it’s actually very good at finding the right content too. You can break this down into two steps: a) find the most relevant content, and b) summarise the relevant content.
With some degree of over simplification, it’s a bit like having Google’s search engine technology at your fingertips.
Response times are painful
AI takes its time.
The general benchmark for a user to receive a response to a request is 500 milliseconds, i.e. if you type in a search box you want to see something in half a second. For my Chatbot, it takes 6 - 8 seconds to get a response. I can probably shave off 50% with some optimisation but that is still 3 - 4 seconds. This will come down over time but keep this top of mind for the use case you are building for and plan your user experience accordingly.
It’s expensive
AI is more expensive than traditional SaaS. There are two types of cost:
- Training cost: this is a one-off cost. I was feeding sections of my newsletter content and generating embeddings (numerical representations of text data) and the cost was small. If you were training your model, expect this to be several magnitudes higher. It is falling very rapidly though.
- Inference cost: Every time someone asks a question. There’s an associated cost. On my Chatbot, these are simply the costs OpenAI has on it’s website (I used the Davinci model):
Inputs you can control
The main inputs you can control are the prompt, randomness (called temperature) and duplication in output.
I’m not going to go into the technical details of this. The point I want to make is that you can achieve a lot simply by playing around with these inputs. If you’re output doesn’t look right the first time, don’t give up. Play around with it and see where you get.
To close
If you haven’t done so already, I encourage you to play around with AI. You do need to be technical to do so. Start with ChatGPT if you haven’t already.
Once you’ve covered that, set up an account with OpenAI and play with their other models. It’s the best way to separate the value from the noise.