Highlight from Only Three Kinds of AI Products Actually Work

This kind of product is usually called an “AI assistant”.

This doesn’t work well because savvy users can manipulate the chatbot into calling tools. So you can never give a support chatbot real support powers like “refund this customer”, because the moment you do, thousands of people will immediately find the right way to jailbreak your chatbot into giving them money. You can only give your chatbots tools that the user could do themselves - in which case, your chatbot is competing with the usability of your actual product, and will likely lose.