From the energetic border collie to the friendly golden retriever, more than 350 dog breeds exist today, each with specific physical and behavioral traits. Although previous research on dog genomes has revealed the genetic basis of variations in body size and shape among breeds, the genetic underpinning of complex behaviors—hunting, herding, guarding, pointing, drafting, and more—has been a tough challenge to crack.

Now researchers at the National Human Genome Research Institute (NHGRI) have discovered that the ancestry relationships of eight or so major dog lineages map onto distinctive behaviors. As a result, the investigators could identify the underlying genetic basis of those behaviors. One major finding was that many genes involved in axon guidance, which determines how neurons connect to one another in the brain, had been changed in several lineages—and that was especially the case in the sheepdog lineage. Because the genes that influence brain development and behavior in dogs are likely to do the same in humans, identifying genetic variation in the locations of the human genome corresponding to those behavior-related regions in the dog genome could lead to new insights about the genetic basis of human behaviors and psychiatric conditions, the researchers say.

“This paper brings behavior back to the forefront when we think about where selection has acted in the evolution of dog breeds,” says Evan Maclean, director of the Arizona Canine Cognition Center at the University of Arizona, who was not involved in the study. “Although it’s true we have done a lot of breeding for aesthetic traits, this work makes clear that a lot of the genetic action is in pathways related to the brain (and so presumably connected to behavior and cognition).”

The Dog Genome Project, which began in the early 1990s, allowed researchers to map disease genes, as well as genes for traits such as body size, leg length and body shape. But “the truth is that the real motivation behind the Dog Genome Project was always to find genes for behavior,” says Elaine Ostrander, who leads the project at NHGRI and is senior author of the new study. Breeds have various stereotypical behaviors, but identifying the genes behind those behaviors proved to be extremely difficult until now, she explains.

To tackle the challenge, Emily Dutrow, a postdoctoral fellow in Ostrander’s lab at NHGRI and lead author of the new study, used methods inspired by single-cell sequencing—a now widely deployed technique to sequence RNA instructions, or transcripts, from individual cells to derive an understanding of the function and relatedness of various cell types in the body. Instead of applying the method to a set of cell transcripts, a transcriptome, she applied it to dog genomes to identify the ancestry relationships of dog breeds. Using genome data from more than 4,000 dogs (samples were shared with the Dog Genome Project by dog owners around the world), “I was able to draw out these really ancient relationships among all of the dog breeds that exist on Earth,” Dutrow says.

By first identifying these ancestry groups, or lineages, the researchers could then look for evolutionary change along each one. And because the lineages happened to cleanly fall into eight behavioral groups, it became possible to identify what parts of the genome make a herding dog different from a guarding dog, for example. To do this, the researchers linked their genomic analysis to behavioral data from a survey of more than 40,000 dogs that was conducted about 15 years ago at the University of Pennsylvania School of Veterinary Medicine.

“This paper uses a new framework to think about the relationships between breeds,” says Gitanjali Gnanadesikan, a Ph.D. candidate at the Arizona Canine Cognition Center, who was not involved in the study. “In evolutionary biology, we tend to think of relationships between species or lineages as branching trees. And while that has its place, dog breeds are particularly complicated because they are not separate species, and there’s a lot of interbreeding, especially at the time of breed formation. This new framework is able to take those complex relationships and both visualize and quantify them in a way that enables new analyses, which is exciting.”

Bridgett vonHoldt, a genomicist who studies canine ancestry at Princeton University and was also not involved in the work, concurs. This new approach, which she calls “incredibly elegant” and “impressive,” allowed the researchers to include not just purebred dogs but also mixed-breed ones in their investigation of the genetics underlying behaviors. She described the work as “a landmark study” for the field.

The researchers’ analysis revealed eight main trajectories of what vonHoldt calls “dogdom”: terriers, sled dogs, sheepdogs and cattle dogs, sight hounds, spitzes and “primitive-type” dogs, scent hounds, pointing dogs and retrievers.

The team found that whole suites of genes were involved in the evolution of specific behaviors in the different dogdom lineages. This contrasts with traits such as body size and coat color, which are regulated by only a few genes. “Behavior is just a lot more complex,” Ostrander explains. “So it’s going to involve more of the genome.” The researchers also determined that most of the genetic variants involved in behavior were in the regions with genes that turn other genes on or off rather than the coding regions that provide instructions for making proteins. That means that tweaking when and where the coding genes are activated early in life is particularly important. “You fundamentally need all these genes. It’s just that you want to tweak them up or tweak them down to get these nuanced changes in behavior,” Ostrander says.

The researchers examined where and when the gene variants associated with particular lineages were activated during development, and they found that many of them were turned on while the brain was forming. Most saliently, many of the gene variants associated with the sheepdog lineage were involved in axon guidance—that is, the process by which neurons connect to each other.

“To me, that was incredibly striking,” Ostrander says. “I certainly did not expect that. I thought they would be genes from all over the nervous system that did all kinds of different things. I didn’t expect one pathway to be so incredibly important in executing these behaviors.” For instance, modifications could have involved the number of cells in a certain part of the brain or the proportions of different cell types. “But really, it’s really how different parts of the brain talk to each other that’s the most important” for determining these behaviors, Dutrow explains.

The researchers found that a lot of the variation in brain and behaviors in the eight dogdoms has to do with motivation and learning. For instance, Dutrow says, scent hounds need to be self-directed learners who can follow their nose rather than be influenced by human direction. But gun dogs, who assist their handlers in finding and retrieving game, need to be closely in sync with their human handlers, she adds.

The gene variants associated with these lineages were not exclusively involved in brain and behavioral regulation, however. For example, in sight hounds, most of the gene variants were associated with the muscular system, the researchers determined.

Another important finding was that most of the genetic variants that the team identified were already present in wolves. “This goes back to the idea that the basic ingredients for all the variation we have in dogs today was largely present in a wolf ancestor,” Maclean says. But the changes from the wolf ancestor were different in the various dog lineages. “This is interesting to me because it really emphasizes the diversity of dogs and how evolution has progressed in different directions in these lineages, through both selection and [random] genetic drift,” Gnanadesikan says.

This study provides the foundation for more detailed research in each dog lineage and what makes that lineage unique. In addition, Ostrander hopes to take the behavior-related genetic variants that are different from one dog lineage to another and look at the variation in corresponding regions of the human genome. It’s likely that the genes that affect behavior in dogs also have an impact on behavior in humans, she says. Because dog breeds have stereotypical behaviors and not as much variation in genetic background as humans do, it is a “much better starting place for understanding what some of these regulatory regions really do,” Ostrander adds. Ultimately, these comparative studies could lead to new insights about human behaviors, including those associated with neuropsychiatric conditions, she suggests.