Dear friends,
Last week, I mentioned that one difference between traditional software and AI products is the problem of unclear technical feasibility. In short, it can be hard to tell whether it’s practical to build a particular AI system. That’s why it’s worthwhile to quickly assess technical feasibility before committing resources to build a full product.
If you have no data or only a handful of examples (enough to get a sense of the problem specification but too few to train an algorithm), consider the following principles:
- For problems that involve unstructured data (images, audio, text), if even humans can’t perform the task, it will be very hard for AI to do it.
- A literature review or analysis of what other teams (including competitors) have done may give you a sense of what’s feasible.
If you have a small amount of data, training on that data might give you some signals. At the proof-of-concept stage, often the training and test sets are drawn from the same distribution. In that case:
- If your system is unable to do well on the training set, that’s a strong sign that the input features x do not contain enough information to predict y. If you can’t improve the input features x, this problem will be hard to crack.
- If the system does well on the training set but not the test set, there’s still hope. Plotting a learning curve (to extrapolate how performance might look with a larger dataset) and benchmarking human-level performance (HLP) can give a better sense of feasibility.
- If the system does well on the test set, the question remains open whether it will generalize to real-world data.
If you’re building a product to serve multiple customers (say, a system to help different hospitals process medical records) and each customer will input data from a different distribution (say, each hospital has a different way of coding medical records), getting data from a few hospitals will also help you assess technical feasibility.
Given the heightened technical risk of building AI products, when AI Fund (Deeplearning.AI’s sister company that supports startups) looks at a company, it pays close attention to the team’s technical expertise. Teams with higher technical expertise are much more likely to get through whatever technical risk a business faces.
Keep learning!
Andrew