When I look at a lot of machine learning systems, I'm overwhelmed by the impression that they are very brittle, and require a massive amount of human effort to work properly... only for them to fail in unpredictable and spectacular ways. I find this troubling, because Earth is literally crawling with extremely robust, adaptable systems that can learn and problem-solve fully autonomously. We're talking not just humans and other apes, but rats, bugs, and even particularly motivated single-celled organisms! How have we not figured this out yet?
I think the basic answer is that we're actually too focused on intelligence, the ability to extract abstract patterns from examples that generalize to new problems. It's a great ability, but it's intrinsically fickle, the history of science is proof that no generalization survives forever. To survive in this world, you gotta ADAPT, and adaptation requires the ability to do things that are NEW.
In my thesis, I introduce a theoretical limit of adaptability to probe this question: hyperadaptability, the ability to solve any solvable problem, in finite time, without any prior knowledge or experience. Because the hyperadaptable agent might know nothing, there is practically no chance that it "know" what to do when it encounters a problem, it will, by definition, HAVE to do something completely new. The only way to be SURE it will solve the problem is to, as the saying goes, "leave no stone unturned", it will just have to roll up it's sleeves and try every potential solution.
But how is that possible? Strictly speaking, it's not: a potential solution is just a behavior, and the set of all possible behaviors is, technically speaking, uncountably infinite, it can't be "searched", which is what it would mean to "try every potential solution". However, there is a SUBSET of all behaviors, which CAN be searched exhaustively, and which APPROXIMATES to arbitrary precision EVERY behavior in the set of ALL possible behaviors.
Moreover, our agent can CHOOSE the ORDER in which it tries every behavior... and the order can CHANGE in real-time. What that means is that the agent can be clever, it can use the information about its successes and failures to prioritize and deprioritize different plans, without ever completely ignoring any plan forever... at least, until it finally finds a solution.
If you're curious for more details and some of the broader philosophical underpinnings of my work, why don't you take a look at my thesis, Hyperadaptability by Behavioral Search? If you read it, let me know what you think!