A scoring model for the importance of research problems

The other day I was reading the article “You and your research”, transcribed from a seminar by Richard Hamming. There is one paragraph about “choosing important problems” which I think is inspirational:

Let me warn you, `important problem’ must be phrased carefully. The three outstanding problems in physics, in a certain sense, were never worked on while I was at Bell Labs…. We didn’t work on (1) time travel, (2) teleportation, and (3) antigravity. They are not important problems because we do not have an attack. It’s not the consequence that makes a problem important, it is that you have a reasonable attack. That is what makes a problem important. When I say that most scientists don’t work on important problems, I mean it in that sense. The average scientist, so far as I can make out, spends almost all his time working on problems which he believes will not be important and he also doesn’t believe that they will lead to important problems.

I cannot agree more with this definition of “important problems”. After a discussion with others, I summarized and came up with the following scoring model for the importance of research problems.

$$ \begin{array}{l} S(X)\\ =P(\text{Solved}_X)R({\text{Solved}_X})\\ =\sum_{a} \underbrace{\pi(a|X,\mathcal{E})}_{\text{Problem-solving policy}}\ \ \underbrace{P(\text{Executed}_a)I(a\in\mathcal{A}_X)}_{\text{Success transition probability}}\ \ R(\text{Solved}_X) \end{array} $$

where $R(\text{Solved}_X)$ is the impact (reward) if the problem is solved, $a$ is a possible method/technique to solve the problem, $\mathcal{E}$ is one’s expertise or background knowledge, $\text{Executed}_a$ means successfully executing the method, and $\mathcal{A}_X$ is the groundtruth solution set of $X$.

Given an $X$, finding the optimal $\pi(a|X,\mathcal{E})$ corresponds to a multi-armed bandit problem: how to assign weights to methods one wants to try so that the expected success transition ($R(\text{Solved}_x)$ is a constant given $X$) is maximized? Of course like in many RL problems, one could randomly try out different methods and evaluate their transitions. However, because we only live one life and it usually takes some time to verify a method, our trial budget is actually limited.

Thus $P(\text{Solved}_X)$ reflects the problem-solving ability of a researcher of 1) finding the correct attacking method to the problem $X$, and 2) successfully executing the method. It can be used as a measure of how good a researcher is in terms of finding the correct solution(s) for a given problem. Note that one needs to widely read literatures because if a method $a'$ is never heard of, then its probability is automatically masked out.

However, there is another more important characteristic of a researcher, that is, the capability of building the transition and reward models, like in model-based RL. Given his/her past experience and literature reading, he/she extrapolates to estimates the success transition probability and the impact of solving the problem, so that he/she can simulate the trial of each particular method $a$ without actually carrying it out:

$$ \begin{array}{l} \hat{S}(X)\\ =\rho(\text{Solved}_X)r({\text{Solved}_X})\\ =\sum_{a} \underbrace{\pi(a|X,\mathcal{E})}_{\text{Problem-solving policy}}\ \ \underbrace{\rho(\text{Executed}_a)\rho(a\in\mathcal{A}_X)}_{\text{Success transition model}}\ \ \underbrace{r(\text{Solved}_X)}_{\text{Reward model}} \end{array} $$

where we use $\rho$ and $r$ to denote the researcher’s mental models. $\rho(\text{Executed}_a)$ is a self-evaluation of one’s engineering ability. $\rho(a\in\mathcal{A}_X)$ is one’s judgement about whether the method $a$ is a sensible attack. $r(\text{Solved}_X)$ is one’s belief of the problem’s value, probably depending on external feedback.

$\hat{S}(X)$ is very critical as it allows the researcher to ask himself/herself the question: how likely will I find the right attack and successfully execute it in one or two years, and how much impact will the solved problem bring? Given this, different research problems $X_1,X_2,\ldots,X_N$ can be ranked by their estimated scores $\hat{S}(X_1),\hat{S}(X_2),\ldots,\hat{S}(X_N)$, and those which get higher scores should be assigned with higher priorities to work on. This is basically a good way of setting one’s research agenda. It is widely believed that the ability of setting a right research agenda is more important than that of solving a given problem (i.e., asking the right questions is more important than finding the solutions).