At the heart of science is measurement, and the quality of measurements limits the quality of the resulting conclusions. In psychiatric research, the most common measurement has traditionally been through self-report, using scales that assess the degree and frequency of psychiatric symptoms. However, self-report has largely been eschewed within biological and computational psychiatry for lacking the ability to provide mechanistic insights into the disorders in question. Instead, researchers now focus primarily on task-based measures of behavior combined with model-based analyses. This approach is thought to allow a deeper insight into the underlying neural and computational mechanisms whose dysfunction ultimately gives rise to psychiatric symptoms and illness. Indeed, the RDoC framework is explicitly built around these underlying neurocognitive dimensions. The measures proposed in the framework are meant to assess the function of mechanisms instead of or in addition to the frequency and severity of symptoms. The subjective nature of self-report, compared to the seemingly objective nature of cognitive tasks in combination with sophisticated computational models, has led many researchers to move away from the former. Measurement quality issues have long been of concern within clinical psychology and psychiatry. Yet the focus has been on challenges posed by clinical conditions due to e.g. medication effects, comorbid cognitive deficits or differences in motor abilities. A growing body of work in cognitive psychology points towards more fundamental challenges to measurement quality from self-report and behavioral tasks regardless of clinical conditions. One often neglected consideration in the context of behavioral measures and model-based assessments is their psychometric characteristics. Whereas the publication of a novel self-report scale is necessarily accompanied by a demonstration of both its reliability (e.g. through test-retest and/or internal consistency measures) and its validity (e.g. convergent, divergent, and predictive validity), it is exceedingly rare to see psychometric assessments of cognitive measures and computational models. Here we discuss recent work that has raised pointed questions regarding the psychometric features of many task-based measures and model-based analyses that are the basis for biological and computational psychiatry. We hope that renewed interest in psychometrics will help improve the potential utility of these measures for understanding the foundations of mental illness. Although neuroimaging measures are central to these fields as well, we do not address those in the present paper; however, recent work has raised important questions about the reliability of fMRI results as well. Large-scale, computerized behavioral studies and consortia for task-based biological measures have enabled researchers to test psychometric properties of broad sets of both self-report and task measures. These efforts reveal concerns about the construct validity of various cognitive processes thought to underlie numerous clinical conditions, the poor psychometric features of cognitive task measures and limitations on the generalizability of model-based analyses. In this commentary we provide a summary of some of these findings, discuss their implications for psychiatry research and present suggestions to improve the field.