Innovation Overcomes A Longstanding “Speed Bump” In Behavioral Research and Treatment


Nonverbal and verbal communication forms the basis of our social connections and relationships.  During a conversation, partners respond to or reciprocate countless verbal and nonverbal cues, without thinking about it at all. For example, when one person smiles as they’re telling a story, the listener may smile or nod a split second later; if one person in the conversation looks off to the side, the other person is likely to follow their gaze; when one person crosses their legs, the other is likely to shift their position in response. The same goes for changes in tone of voice, volume, and countless other conversation markers. This conversational coordination is known as “synchrony”, and it helps to build the connection between individuals and facilitates the conversation by conveying interest, comprehension, empathy, and emotional response. 


Developmental and behavioral diagnoses like Autism spectrum disorder (ASD), anxiety, ADHD, and depression affect the way individuals communicate and interact socially, including eye contact, facial expressions, body language, the way the voice sounds and the way words are used. ASD, in particular, can also affect the way people interpret these social signals from others. Some individuals on the autism spectrum may also engage in repetitive behaviors that increase during social contact, such as hand flapping, rocking, or repetitive speech. 


The “Reliability” Gap

To understand the wide range of behavioral differences in autism and related conditions like anxiety, ADHD, and depression, researchers and clinicians have needed to find ways to objectively measure these social communication abilities. Traditionally, this has been done through expert clinician ratings during interviews and social interactions, or through parent feedback collected from questionnaires.


“Historically, to study behavior, we videotape study participants during interactions with a trained clinician. We then have human coders watch these videos meticulously, many times in a row, to rate and categorize the various behaviors that characterize autism,” says Ashley de Marchena PhD, a founding member of CAR’s Technology & Innovation Lab. “These traditional approaches are the current gold standard, and they have many strengths; however, they often lack precision and granularity, since there are only so many categories of behavior that can be reliably agreed on by human raters. Moreover, these evaluations are extremely time-consuming and depend on specially-trained clinicians who are in short supply nationally.  The result is that many families don’t have access to this level of clinical expertise and that behavioral research is limited to small sample sizes, adds de Marchena.


Small sample sizes translate to less-reliable study results that are more prone to failure when other labs attempt to replicate them- thus stalling the progress of research and clinical advancement, explains CAR Scientific Director, Robert Schultz, PhD. “The current methods for clinical behavioral evaluation are impossible to implement for large-scale studies, as they are too labor-intensive and would be cost-prohibitive. At the same time, large-scale studies are the only way we will advance some kinds of research.” Dr. Schultz cites an example: Genetic studies require thousands and even tens of thousands of participants to reach the statistical power needed to fuel discovery, but because these studies don’t collect much behavioral data, study conclusions are limited. “Researchers are handcuffed in their ability understand how genomic differences contribute to the wide variation of autism severity and the heterogeneity of clinical characteristics that pertain to ASD.”


Leveling Up in Behavior Analysis

Recognizing the significant limitations of traditional behavioral research methods, Keith Bartley, MS, a principal investigator and co-founder of CAR’s Technology and Innovation lab, saw an opportunity to capitalize on rapid advances in consumer technologies. He saw that computer vision, 3D motion capture, speech and language processing, and fitness tracking sensors could help CAR to fundamentally change behavioral research. Over the summer of 2014, Mr. Bartley and Dr. Schultz equipped an unused room at CAR with multiple synchronized 2D and 3D cameras. Suddenly, it became possible for CAR researchers to digitally record and measure each precise body and facial movement that occurs during a brief social interaction. Using “markerless motion capture” technology meant participants did not need to wear any special sensors or equipment- an especially helpful feature for participants with ASD who may have tactile sensitivities. 


In this lab, eight video cameras were mounted around the perimeter walls of the room to capture the movements of the body; meanwhile, a separate camera, nicknamed the “SensorTree”, was situated between participants during conversations to precisely record and collect data on their speech, language, and the smallest movements of the eyes and facial muscles. “We wanted to better define and quantify all of the tiny behaviors that take place during a social interaction,” says Mr. Bartley. “The hope is to find a way to dimensionalize the spectrum of behaviors, so can improve the accuracy of diagnosis and our ability to tell whether a treatment is working.”


By pairing advanced “machine learning” statistical methods with natural language processing and computer vision technology (face, limb and body motion-recognition), it became possible to develop a finely-grained representation of human behavior in raw data. This burgeoning new field has become known as “digital phenotyping”, where the phenotype is all observable expressions of ASD - or any other neurodevelopmental, behavioral, or psychiatric condition.


Psychiatric and developmental differences have so much overlap with one another- as well as with typical development. We can’t improve our understanding of autism and other psychiatric or developmental differences without measuring many different people, under many different circumstances,” says Elizabeth (Eli) Kim, PhD, a post-doctoral fellow and computer scientist on CAR’s Technology & Innovation team.


One of the persistent challenges in autism research is the heterogeneity of symptoms: one individual with ASD can possess a completely different set of clinical characteristics than another with the same diagnosis. “Our solution [to the challenge of heterogeneity] is to upend behavioral science as we currently know it,” says Dr. Schultz.  “Right now, behavioral science relies on judgements from expert clinicians and parents, gathered through interviews and questionnaires. We believed that if we could digitize all social behavior- vocalizations along with fine and gross movement- we could provide the raw data for behavioral science that has been lacking. We can actually measure behavior directly, in a much more reliable and fine grained manner. You might say that we are turbo charging the science in behavioral sciences.”


Painting Portraits and Landscapes with Data


Ideas and projects using the new SensorTree technology came at a furious pace, propelling CAR into a unique space in the field of autism research.  Within the year, the center began an international search to recruit a team of highly-trained computer and data science specialists who could use this raw data to paint a much more detailed and reliable portrait of each individual. Collecting these comprehensive data-portraits from many individuals will ultimately reveal patterns in the autism landscape and will clarify the genetic and developmental bases of ASD in a manner that is currently impossible.  “When we begin to precisely map symptoms and behaviors to specific differences in the brain or genome, we’ll be able to develop tailored clinical support programs that grow with each patient throughout their lives,” explains Dr. Schultz.  


“Because our approach collects lots of information in just a few minutes- as opposed to the hours required in current approaches- it is now possible to enroll large numbers of children and families into research, so we can begin to understand the multidimensional nature of  ASD,” continued Dr. Schultz. With this vision in mind, the goal, he says, is “to design large- scale studies with thousands of people, which can be carried out where they live their lives – their home, school, or community – not solely in the laboratory environment. This will enhance the validity of study findings, which in turn make the findings more meaningful for implementing new knowledge into clinical care.”


Find out where CAR’s Technology and Innovation program is headed next, and what it means for the future of ASD research and treatment.