Thank you so much. We'd love for you to join our Slack community (link on system.com).
Great question. There is no ground truth that we are modeling System after, i.e. there is no causal model of the world out there (to use Pearl's framing). So I'm not sure we can know how far along we are epistemologically. More practically, for the next few years we have plenty of work to just represent all the existing corpuses of scholarship! The truer and arguably more meaningful test of progress though is how decisions are improved — for users, for organizations — that use System.
Re completeness, as I wrote below, System Search results are not necessarily comprehensive — but they will be. System is in the early stages of its development as a public resource and you should expect that knowledge will be missing. The knowledge base will be constantly growing and improving and evolving as knowledge does. Our community will play an important role in relating what we expect or know should be related.
Great question. There is no ground truth that we are modeling System after, i.e. there is no causal model of the world out there (to use Pearl's framing). So I'm not sure we can know how far along we are epistemologically. More practically, for the next few years we have plenty of work to just represent all the existing corpuses of scholarship! The truer and arguably more meaningful test of progress though is how decisions are improved — for users, for organizations — that use System.
Quality is evaluated and presented using a variety of parameters like strength, significance, and reproducibility (full documentation here: https://docs.system.com/system/using-system/investigating-re...).
Re completeness, as I wrote below, System Search results are not necessarily comprehensive — but they will be. System is in the early stages of its development as a public resource and you should expect that knowledge will be missing. The knowledge base will be constantly growing and improving and evolving as knowledge does. Our community will play an important role in relating what we expect or know should be related.