Developing AI Products Part 2: How To Assess Technical Feasibility

  • Share to Facebook
  • Share to Twitter
  • Share to LinkedIn
  • Share via Email

Dear friends,

Last week, I mentioned that one difference between traditional software and AI products is the problem of unclear technical feasibility. In short, it can be hard to tell whether it’s practical to build a particular AI system. That’s why it’s worthwhile to quickly assess technical feasibility before committing resources to build a full product.

If you have no data or only a handful of examples (enough to get a sense of the problem specification but too few to train an algorithm), consider the following principles:

  • For problems that involve unstructured data (images, audio, text), if even humans can’t perform the task, it will be very hard for AI to do it.
  • A literature review or analysis of what other teams (including competitors) have done may give you a sense of what’s feasible.

If you have a small amount of data, training on that data might give you some signals. At the proof-of-concept stage, often the training and test sets are drawn from the same distribution. In that case:

  • If your system is unable to do well on the training set, that’s a strong sign that the input features x do not contain enough information to predict y. If you can’t improve the input features x, this problem will be hard to crack.
  • If the system does well on the training set but not the test set, there’s still hope. Plotting a learning curve (to extrapolate how performance might look with a larger dataset) and benchmarking human-level performance (HLP) can give a better sense of feasibility.
  • If the system does well on the test set, the question remains open whether it will generalize to real-world data.

If you’re building a product to serve multiple customers (say, a system to help different hospitals process medical records) and each customer will input data from a different distribution (say, each hospital has a different way of coding medical records), getting data from a few hospitals will also help you assess technical feasibility.

Given the heightened technical risk of building AI products, when AI Fund (Deeplearning.AI’s sister company that supports startups) looks at a company, it pays close attention to the team’s technical expertise. Teams with higher technical expertise are much more likely to get through whatever technical risk a business faces.

Keep learning!

Andrew

  • Share to Facebook
  • Share to Twitter
  • Share to LinkedIn
  • Share via Email