Team:Harvard/Human Practices
Human Practices
When we started this project, we asked ourselves how we could contribute to solving a problem that impacts millions of people worldwide. In order to do so, we had to create a solution that was effective and usable. The only way we could do this was to utilize a human-centered approach.
Question Design
The survey is separated into 6 sections: Demographic info, General understanding of the science, Favorability to synthetic biology and machine learning, Understanding and perception of COVID, Favorability to synthetic biology therapeutics, and Understanding and favorability to MOTbox.
Collecting demographic information allows us to better understand our data results and to ensure a representative sample population. We then survey the individual's general understanding of the different scientific concepts related to our project to better understand whether this understanding influences their favorability and perception about therapeutics. The Favorability to synthetic biology, machine learning, and COVID section helps us understand whether preconceived notions and opinions carry over to their thoughts about MOTbox. Finally, we allocated a section for MOTbox to present the idea and to evaluate respondent's understanding and favorability to the novel therapeutic.
With each section, we made sure to provide appropriate definitions for scientific terminology, so as to not assume any scientific background. Most questions required respondents to answer on a scale from strongly disagree to strongly agree, or very detrimental to very favorable. We found this strategy to be most effective towards surveying people's understanding and favorability towards different subjects.
Approval & Ethics
In order to ensure that our survey was ethical and that it maintained the ethical standards maintained by Harvard and iGEM, we put our survey and plan through the Harvard University Committee on the Use of Human Subjects. After communicating with Alicia McGovern, we submitted the proper paperwork as part of the Undergraduate Research Training Program (URTP). Our team member Frank D'Agostino had certification from CITI Program Social/Behavioral Research Training module, and thus was able to submit the paperwork and undergo the proper review process. The survey we sent was not considered in-person human research, and thus did not have to go through the entire IRB approval process as part of regulated research.
Additionally, all participants participated in our survey willingly. There were no incentives granted by the Harvard iGEM Team for filling out the survey, and participants were able to stop participating at any time they would have liked. They were also free to not answer any questions they did not want to. All questions were chosen carefully in order to take into consideration different backgrounds, identities, and groups.
Surveys
In creating any therapeutic device, we needed to answer a few critical questions. In particular, we needed to know how people perceive synthetic biology in order to determine whether or not they would ever use a DNA origami vaccine. Before creating any therapeutic, it is important to note whether or not patients would take such a vaccine in the first place.
To answer these questions, our team sent out a survey to 205 people through Survey Monkey Audience, asking them about their background knowledge of synthetic biology, and whether or not they would consider utilizing a treatment that involved it. From here, we worked on creating a design that would cater to our audience.
Our survey respondents, with a perfect gender balance, were representative across a variety of age levels and education backgrounds, though most respondents had at least a bachelor’s degree. Thus, respondents for our survey were for the most part representative of the communities we aim to serve.
What did we learn?
Approximately 60% of respondents are comfortable using a therapeutic derived in part from machine learning, but about 27% of respondents are uncomfortable with this. To take this information into account, we are planning a series of laboratory experiments once we can get into the lab to validate each of our components derived from machine learning.
Approximately 60% of respondents are comfortable using a therapeutic that uses DNA as a delivery mechanism, showing that our therapeutic is something that would be used by a wide variety of individuals.
If necessary, individuals place a premium on synthetic biology derived treatments, with 50% willing to devote a month of their salaries to getting a treatment for SARS CoV-2. This information shows that our treatment is viable for the communities we aim to serve.
After conducting this survey, we were able to determine that our therapeutic was something that individuals would be likely to use, so we moved forward with our design. Since some individuals voiced that they would not be entirely comfortable using a treatment derived in part from machine learning, we will be sure to validate our experimental design in the lab in the future.
Snapshot Into Our Survey Results
Concerns Raised
In our survey, we had one open-ended question that allowed users to submit any concerns they had. After carefully going through these responses, we decided to highlight some here and talk about how we thought about this in our design, and if it was not strategized as rigorously as we would have hoped, how we will work to quell these issues in future iterations and in Phase 2 next year.
Anonymous Response: "I don’t like the idea of DNA based delivery. Does this alter a person’s DNA ?? I trust machines for creating simulations but not for predicting how vaccines or treatments will work on humans. Humans are individuals where so many unseen reactions can occur that a machine might not'see'."
Interpretation and Actionable Item
The idea of using DNA to delivery things in the body can seem strange. A typical education teaches you that DNA simply stores the genetic information of the cell, and so it can be hard to visualize how an abstract concept such as DNA origami could play out. Additionally, it is completely valid to feel weary about computers designing vaccines or treatments, as there are many, many factors in play and in silico computations cannot generalize to the needs of each individual.
In order to alleviate this, we wish to send informational newsletters to people to teach them more about DNA origami and machine learning. Additionally, because of this comment, we worked especially hard to research the effects of DNA origami, and the literature strongly agrees that DNA origami structures in the body are harmless and are passed easily. Sentiments such as this one is why we chose DNA origami - we want a safe and reliable delivery mechanism that has little to no side effects.
Additionally, the use of machine learning is still a new concept to many people, and AI is a widely debated topic. To alleviate concerns, we wrote a small ethical statement about how ML should be used in the context of medical discovery/optimization/implementation, how it can be abused, and how we can regulate it so that it cannot be abused.
Ethical Statement on the Use of Machine Learning for Medical Use
Some of the most salient issues of implementing a machine learning framework within the drug/therapeutic development process are those that relate to the data sets used to train models to inform design, and the choices a human researcher makes in generation of models.
The accuracy of a model we generate can be verified by reproducing binding and neutralizing capability of antibody sequences outside the training set. Efficacy of the model in generating effective antibodies can be verified experimentally by in vitro experimentation, and eventually in animal models and clinical settings. However, there may be biases implicit within the dataset used to train the model that in vitro and animal model testing may be unable to detect. If the antibodies present in the dataset were generated primarily by a particular demographic, such as young people (where age is a significant factor in immune response to vaccination [Gustafson et. al., 2020]), or in a less clear manner (such as some genotypic commonality among the immune systems of those who generated the antibodies), the model could ultimately be biased to score antibodies generated by a particular group more highly, leading to clinical results that unnecessarily vary among the patient population in systematic ways.
There is also influence on the process of antibody design from the model types and hyperparameters a researcher decides to use in constructing their machine learning pipeline. This may vary by researcher, with their particular base of knowledge, background, and judgement criteria of the model.
One counter to the possibility of bias in a sequence training set is simply using a both large and sequence-diverse data set. This was the case to the best of our current ability through our use of the CoV-AbDab database, which continuously compiles SARS-Cov-2 antibody data from the entire population of existing manuscripts presenting such data.
Ultimately, though, we maintain that machine learning pipelines are merely a tool in the therapeutic design toolkit, and should not replace chemically and biologically informed therapeutic design, and extensive validation, redesign, and re-validation through the canonically experiment-driven route of therapy development. For this application of machine learning, any biases in training data sets or setting of hyperparameters should be addressed by a thorough experimental evaluation of a sufficiently large pool of antibody sequences predicted for efficacy by the pipeline.
In general, issues present where machine learning is applied at other phases of medical practice lie beyond the scope of our project. These include concerns on training data sets somewhat analogous to ours; for example, if a skin cancer diagnostic software is trained on a data set that includes far more images of lighter skin tones than darker skin tones, the model may fail to detect skin cancer more often for patients with darker skin tones [Adamson, 2018]. Demographic completeness of data sets is crucial, and can have life-and-death consequences when machine learning is applied to medicine. Also important are patient data privacy concerns when diagnostic software is trained on, for example, patients’ imaging results [Vayena et. al., 2018], or insufficient understanding when a practitioner must interpret a clinical treatment suggestion based on a software tool’s analysis of a patient’s biomarkers, and informed consent questions therein [Gerke et. al., 2020].
For the scope of our project, with respect to data privacy, our data on antibody sequences and qualitative binding affinities and neutralization capability derive from manuscripts which do not include identifying patient information. There is no intended insertion of a machine learning process into point-of-care decision-making; no machine learning informed decision would ever affect a patient before being running the lengthy and rigorous gauntlet of experimental evaluation.