Skip to main content
Loading Explore guides…
Clinical Trials · Acedra
Home
Explore
Medicine
Clinical Trials
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
304 segments
Copy
Hide video
0:01
And I think we'll begin. So I'm going to share my screen. I know people will be coming in shortly, but I at least want to get through our very quick PowerPoint, just talking about the different workshop series that we have. And obviously the first one is that we're talking about today is BERD. It's an introduction to clinical trials, and this is part one, and you can find this on our Core Competency workshop series where you'll actually see the advertisements.
0:31
Today we're going to have about 10 to 15 minutes with Dr. Wilding. And then we're going to go on to Clinical Trial Basics I and II with Dr. LaMonte, both from UB and excellent speakers as we've seen them in the past. I just want everyone to know that there's a PCORI series as well. The first two have actually finished. However, I have videos of them if you would like them. And then we have workshop three, four and five as well.
1:01
And that's also available on our CTSI UB workforce development website. The qualitative study design and data analysis workshop is also going on. That one's in person, but it is full because it is an in-person workshop. We will be giving it again either next year or the year after. So please kind of look out for that. And we have many core competency workshop series. In fact, this particular semester, it's just these three with PCORI, BERD clinical trials one, and
1:32
going on as well with qualitative study design and data analysis. And next semester we will have some as well, including scientific communication and responsible conduct of research and a few more.
1:45
I do want to tell you that there is a certificate of completion available, and if you do come to all four of these workshops, I can get that to you. If you do miss a workshop, please contact me if you'd like the certificate of completion. There is a short protocol. It's a written protocol you have to follow, but it's really not that long and I can make sure I get you the certificate of completion even after the series is completed.
2:10
I also want to tell you that CME credits are available, and it's going to be 6.5 hours of CME credits for anyone who wants them. And basically I just send the actual advertisement And I also send the attendance files to the CME department, and they will award those if you need them or want them. I also want to tell you that I'm going to be putting a clinical trials one workshop series survey one in the chat. Please, please, please try your personal best to complete that.
2:43
If you don't complete it, I will be sending it to everyone two other times. And the reason why is because these really help to improve our workshop series and we need it to report to our funding sources. I also want to say that I have a group that is in workforce development that I work very closely with, including Dr. Katia Noyes, Dr. David Jacobs, my colleague Liz Bengert, and then myself. And it's not just CTSI, however, that is putting on this series.
3:14
I'm actually going to turn it over to Dr. Wilding, who can put his PowerPoint slides up. And he's also going to talk about an exciting new department. I believe you're going to talk about that that is also sponsoring this series. And I'll hand it over to him. Catherine, can you see the right slide? Absolutely. You are all ready to go. Clinical trial minicourse series introduction. All right. Great. Hi, everyone. My name's Greg Wilding.
3:40
I am a prof in the Department of Biostatistics here at UB, and I've been here for a while. I came in 2002 when the department was actually formed. So I've been around for a bit, and I also direct two units that I'll talk about a little bit later in a few minutes. but what I want to do now is first, kind of talk about the series that you signed up for. This is what we call clinical trial mini course. Our goal with this series is really to turn those that attended into educated consumers of stat,
4:08
epi, and research design methodology as it applies specifically to clinical trial oriented research. Now our target audience is established researchers. So those that have done trials before and then also new researchers, those that have aspirations to do trials, perhaps in the near or the distant future. We, you know, we're thinking, of course, about our faculty here within not only UB, but the SUNY system as a whole,
4:34
but also our postdocs, our fellows, graduate students who are going to be interested in trials. And then a lot of the research staff that does the day to day work involving clinical trial research. And then when we design these, we also think about our industry partners as well. Now this is brought to you by the Workforce Development Core. And then two units which I'll get into after we kind of go through the schedule here.
4:57
Obviously today, October 22nd. So Dr. LaMonte will be talking about clinical trial basics. I kind of view that as the only prereq for the rest of the workshops assuming that you're not already very familiar with clinical trials in terms of concept. If you when you look at the schedule, if there's some that you want to skip, by all means don't feel you have to go to every one.
5:18
So to understand something that might take place later in November, it doesn't mean you have to attend every single workshop. We designed these to really be self-contained modules. Next after Dr. LaMonte on November 12th, Austin Miller from Roswell Park will talk about clinical trial outcomes and analysis concepts. I'll talk a little bit about randomization towards the middle of November. And Dr. LaMonte will come back and he'll talk about pilot trials.
5:45
We feel that's really important because when you're designing the pilot clinical trial, the thought process is a little bit different than a confirmatory trial. And so a lot more thought, I feel, has to be put into pilot trials and generally what we do currently. And then if you're interested in clinical trial data management, my data manager, Austin Glick, will be back towards the end of November to talk about that.
6:07
Now, this really comprises what we consider to be the intro series taking place during this fall. In the spring, on the right side, we have our tentative schedule. There's not dates there yet. We should come up with some dates, probably sometime in November December. But we'll be back. I'll be talking about advanced parallel group design. I'll be talking about crossover trials. A lot of people are really interested in step wedge nowadays.
6:30
And so we have a talk just on that by a faculty who's great and has lots of experience in that realm. Pragmatic trials. Adaptive designs, myself and colleague We'll be talking about that. You know, there's obviously a lot of complexity behind an adaptive trial, not only in terms of like the mathematics, even though we're going to keep it at a non mathematical level,
6:49
but also the implementation, it can be really tricky in the real world to implement an adaptive design and then time to event outcomes. There's a lot of complexities that come with that in the planning and the analysis of trials based on time to event outcomes. And Dr. Jihnhee Yu will cover that in the spring. Now, one thing to note is that trials are multidisciplinary efforts. Right. And also we're only going to give you a 10,000 foot view regarding these concepts and techniques.
7:17
We only have so many hours of exposure with you all. So if you're thinking about embarking on some clinical trial oriented research, I would suggest that you start gathering your team now and I can help you with some identifying some collaborators through a couple units that I direct. The first one is what we call the BERD core. This is within our Clinical and Translational Science Institute.
7:38
We consider this the hub for statistics and epi know how in the Buffalo area, the faculty, which I'll show you on the next slide, have very diverse experience and all have worked within the team science environment, which is really, really important when you're doing applied research. We operate according to two aims. The first one involves supporting innovative novel studies through a lot of the collaborative mechanisms that we create.
8:05
And the second one is partially what brings you this series. And this is really about training and workforce development.
8:13
Here's the faculty. Our BERD actually spans three departments across two institutes. Here at UB, we have the Department of Biostatistics and the Department of Epidemiology and Environmental Health.
8:26
And then we have a member who's a faculty within Roswell Park Comprehensive Cancer Center within their clinical trial development division. All the the personnel within the core, we have formal and comprehensive training degrees in BERD oriented fields. And then if you just look up our names in PubMed, you'll see all the diverse projects that we've been involved with. We can really handle any kind of database project that comes through the door.
8:52
Now, recently, what we did to kind of take UB to the next level is this past summer was we opened up a new office called the Office of Clinical Trial Development and Implementation. This is our centralized hub for clinical trial know how. This is funded by the vice president for health science office and with a physical location on South Campus. We have a mission, basically just to improve the planning and conduct of trials here at UB and across really the SUNY system.
9:20
And we do that through three goals, at least at this point we have three goals. First is really to build up the infrastructure. And that's currently what we're doing right now. We're spent a lot of effort building up infrastructure to support large clinical trials and centers that have a large clinical trial component to them. So, you know, we're creating SOP software libraries. A lot of the nitty gritty work is taking place right now,
9:43
because that is now something we could write into our grant applications as we go for funding for big clinical trials. Of course, we're going to support individual trials all the way from concept to reporting. And I'll talk a little bit more about that in a second. And then again, partially what brings this series to you today is we're always working on coming up with new skill building opportunities, not only for new trials, but established trials to help strengthen their research.
10:11
This office will better position not only UB, but SUNY as a whole as we embark on clinical trial oriented research. Here's the personnel. I've been working in trials for about 25 plus years across a number of areas, both within academia and industry. Austin Glick is our data manager. He's worked on very large NIH funded trials over the years. Joe Boccardo, he's our senior statistician. He's worked both within industry and academia for ten plus years.
10:42
On both clinical and non-clinical research projects. And then we have a couple students within the unit as well that help support the day to day work that needs to get done. Now, we're a pretty small unit at this point. We're gonna really grow. You know, these are people that have just been hired the last few months. We have some key university partners. Obviously the Department of Biostatistics would be, my home department, would be a key partner there.
11:06
There's a lot of faculty that have very high level clinical trial know how within that department. The Clinical and Translational Science Institute, that's a valuable resource, I think, for any UB investigator, due to its ability to help build teams and get access to resources, research information systems, they provide Redcap support on campus And they also have been involved in some of our projects in terms of database programing support.
11:31
And then of course, our clinical research office here at UB would be a natural partner for us. These, this is a representation of all of our experience. You know, we have lots of experience in evaluating not only drugs but medical devices, preventions, procedures, behavioral interventions, educational interventions, diagnostic tools and delivery systems. We've done trials from those early pilot trials all the way to those big confirmatory trials, which may be single site or multi-site.
12:01
We're well-versed in your typical design, like your parallel group, but we have a lot of know how in terms of things like step wedge designs, cluster randomized designs, individually randomized group treatment trials, but well versed in adaptive designs as well. And we work not only in explanatory trials, so these are kind of like your your trial 101's. Right. Like your basic trial in a very controlled environment, but also pragmatic trials where you're evaluating intervention in the real world.
12:30
On the right. I won't go through all those circles there, but you can see that we're very involved, really, in all aspects of the trial providing consistent support according to the developed SOP's. That's all I have to say. Now, if you are interested in working with us, either of the two units that I developed, feel free to contact me. There's my email at the bottom. You don't have to really write it down.
12:53
I'm easy to find. I'm in the Department of Biostatistics here at the University of Buffalo. You can contact me, and I can figure out which unit might be able to support you with your trial endeavors, or if not one of my two units, I can maybe find collaborators for you elsewhere within the university. Thank you very much for your time. Excellent. Thank you, Dr. Wilding. I appreciate that. I do want to at least introduce our speaker for today.
13:18
He is Dr. Michael LaMonte. He is director of graduate studies in the MS and PhD programs, a research professor, and he's in the Department of Epidemiology and Environmental Health in the School of Public Health and Health Professions at UB. And he is a wonderful speaker. We actually had some sessions with him last year, and I think you will enjoy this incredible series of workshops, Clinical Trial Basics one and two.
13:45
And I do want to tell you that if you would like a copy of the PowerPoint, please email Dr. LaMonte. I put his email in the chat and also please type your questions in the chat and a natural break will be occurring during the workshops. And I will address that with Dr. LaMonte at that time and turn it over to our esteemed speaker. Thank you. I guess I've paid you enough, then to say all those nice things about me. I appreciate it. Appreciate Greg's introductory comments about
14:15
the programs growing through the CTSI and the BERD, the new office around clinical trials and research. So I'm Mike and I'm an epidemiologist by training, and I'm going to spend the next period of time giving you, as Greg said, a mostly high level view of some of the basic parts of clinical trial thought process and the design, maybe just a little bit about implementation. But I just want to reiterate, as Greg said, we can provide some overview with the workshop series,
14:54
but the actual doing part requires a far more than we can engage you with in this form. And so if you're on the cusp of developing or thinking about, do I want to get into clinical trial type study designs We would certainly welcome having a consultation appointment through the BERD or the other offices Greg mentioned, where we can give you a little bit more directed advice to your question and your specific study.
15:28
But ultimately doing clinical trials is how you become good at doing clinical trials.
15:36
So these are the objectives that we'll try and get through this afternoon. My assumption is I point out here is that you have little or perhaps even no experience developing research studies, specifically clinical trials, as we'll discuss tonight. And then here are a few readings that I think are worthwhile to have available either on your bookshelf or electronically as a go-to for a little bit more embellishment on some of the concepts that I'll present this afternoon.
16:16
And I think this was sent out actually with the announcement to the workshop. So these citations would have been included there.
16:26
I just want to start and I often start talks that I give pertaining to methodology with a slide similar to this, just to remind us that science is a dynamic process, but it's a very systematic process when it's done well and it's a process for solving problems and sharing information, or pursuing knowledge, hopefully unbiased knowledge. But we're doing so in the face of uncertainty, in that we never can be 100% sure we've done it right,
17:01
or we have the correct answer, or we have the correct study population and so on and so forth. So uncertainty is always looming around us in our craft. In fact, Dr. Wilding, Dr. Miller, and the other statisticians, when they present their workshop pieces around specific statistical procedures, they will discuss testing hypotheses at a certain level of uncertainty. So not only do we recognize uncertainty is part of what we do,
17:33
but we can actually factor it into some of the testing that we do with respect to data collection and hypotheses. And scientists often will take their theories or hypotheses that are aimed at explaining what they observe in nature and then use a trial and error process. So we don't always get it right. And we have to go back to the drawing board, but we can learn from each step along the way.
18:00
So I think that's a good grounding in kind of what the idea of designing clinical trials or any sort of scientific research is aimed at. And one last comment I want to make is that at least the major paradigm we operate in currently in contemporary science is one in which we're really not setting forward to prove things. I kind of get a little bit anxious when somebody will say, well, that study proved this,
18:30
or the nightly news says Harvard investigators proved that. In fact, science is a process of refutation. We refute alternative explanations to the observations that we make in our science or our studies. And then on that basis, we move forward to try and understand well what else might be a possible explanation. And that uncertainty of not being able to refute every possible explanation for the phenomenon we observe is just the essence of what I'm trying to get at,
19:02
that we define as the scientific method. So we don't prove, we refute.
19:09
Now, study designs and clinical research kind of proceed along a continuum A large number of study designs fall into what we call the observational study suite. That can be something as simple as a case study from clinical practice through to a case control study or a prospective epidemiological cohort study. And when we leave the observational study design suite, we then, excuse me, embark into the experimental study designs where we find the clinical trial and the community trial.
19:45
That's what we're going to focus on this evening. But I want to make a point that no matter where you begin, as you proceed forward through the various study designs, and I'm listing common ones here, there's probably others I haven't listed by name, but I want you to think of something very important. Don't just think of this as a case study, a cohort study, a clinical trial.
20:09
What this progression of designs actually represents is a progression of evidence, strength and quality of evidence, for one thing, and that is to establish whether or not an independent variable is causally related to or causally affects a dependent variable or an outcome. So these study designs represent that strength of evidence moving from low to high, as I'm showing here. Now why is that important? Well, for example, we can take the same question.
20:43
I'm just throwing an example here. Is aspirin use associated with cancer? And we can answer that same question using progressively stronger study designs that as we move from one level of design to the next, and we collect information based on the specific design and proceed to the next, and so on Ultimately, we will have accrued a evidence base that can, based on the strength and the quality of the studies that were used to address this question
21:22
Ultimately, make a determination on whether or not aspirin is likely a causal agent for cancer or is not a causal agent for cancer. And that would be through the entire suite of designs, including what we'll talk about tonight, the randomized trial. The languages you see as each study question connects back to a design it changes somewhat. And the way those study questions are written give us a sense of what the nature of
21:53
the design is and how it differs from the previous design or the one subsequent. But it's important to realize that we build evidence over a series of study types and designs. We don't jump to just one or another and say, well, we've done it, we've answered the question, or we've proved X causes Y. It just doesn't work that way. And this progression of evidence is very important.
22:26
Many people consider the randomized trial design to be the gold standard for establishing cause and effect, as we utilize evidence from randomized controlled trials It provides us with the highest level of evidence in algorithms that many of you are probably familiar with, that allow us to take an evidence base and construe from it a evidence informed or an evidence based clinical practice guideline, public health policy or consensus statement, and by example
23:05
I'm sure several of you are familiar with the U.S. Preventive Services Task Force. This is a large body of experts who routinely will review evidence under a certain domain, and from that evidence, interpret it and then make a practice, a clinical practice or a public health practice guideline recommendation. If you put a bunch of smart people in a room and lay out evidence on the table and say, okay, go at it.
23:37
Determine what's important and what's not and come up with some conclusion
23:43
You could end up with chaos on your hands because if you leave it to smart individuals opinions
23:51
Certainly there's going to be areas where one opinion differs from another. So how do we do this then? How do we take a body of evidence
24:01
And evaluate it and refine it into a written and very clear
24:09
Evidence based practice guideline.
24:13
That is what I meant by the algorithms that are used for assigning level and class of evidence in what I think generally exists today Contemporarily, evidence based clinical practice and evidence based or evidence informed public health practice. You hear that terminology a lot. This happens to be an algorithm framework. The American College of Cardiology American Heart Association uses. And what I want to draw your attention to first is the very left side, the column
24:45
first column. The first thing that
24:50
These expert panels will do is they'll sort the evidence and they'll assign it first a level, level A, B, and C as I'm showing here. And there's other algorithms. This is for simplicity's sake. Level A is what I want to draw your attention to. The highest level. The highest quality of evidence is that which emanates from at least one well-controlled, well designed and implemented randomized controlled trial, or a meta analysis of several
25:22
Well-designed randomized trials or a randomized trial that's corroborated by high quality registry data.
25:33
I'm doing showing you this to point out that in these already established Guideline algorithms that are being used regularly, and we could point to the scientific literature and find documents for which this particular algorithm is cited is how the evidence was evaluated, and the guidelines within that particular document were written. Notice that the highest level is the randomized controlled trial.
26:04
The then what's done is based on the level of evidence A determination is made on what is called the class of the evidence, and that simply is the benefit, or the evidence for benefit far outweighing any evidence of risk And then as you move from left to right, you'll find eventually the lowest class is where the risk outweighs the benefit. And when you cross classify level of evidence and class of evidence, you then can see where guideline recommendations,
26:36
such as what I'm showing here is an example from the US PSTF on statin use for primary prevention of cardiovascular disease. That's where the guideline statements come from. An algorithm that looks something like this. Again, my point for tonight is the randomized controlled trial provides the highest level and quality of evidence in the current guidelines or algorithms that are used to produce guidelines.
27:05
Another important issue that I want to point out, which many of you may be familiar with, but if you haven't heard the term prior, the word equipoise. Equipoise is particularly relevant when we think about randomized controlled trials. When we do human experimentation, we don't jump to that design immediately. There has to be strong justification
27:31
To do experimentation or use an experimental Experimental design on humans. There should be the same consideration for animal research as well. But particularly for humans, the point of equipoise is there has to be
27:45
Uncertainty with respect to the weight of benefit versus risk in a given problem or question, such that the only way to move the needle forward and better understanding that balance is to think about a randomized controlled trial or experimental research design using humans.
28:11
Equipoise is founded on the idea of an evidence base accumulating over many of the early types of designs I showed on the prior slides, and ultimately amassing to the point where, within the paradigm, the experts conclude we cannot make a final determination on benefits versus risks unless we do a randomized controlled trial or some sort of experimental design using humans. So equipoise is a very important part of considering whether or not randomized
28:47
trials is an appropriate design for a given question at a given point in time, relevant to where the current scientific evidence is.
28:58
So this is an example of how we might word a research question that is connected to a randomized trial design. So very simply, what is the effect of an intervention on the incidence of outcome. Three important words there, effect. Remember the randomized trial is a design And we'll see why momentarily where we can think about the effect of The independent variable on the dependent variable, not simply an association or a correlation, but the effect of one thing on another.
29:43
Typically the effect of is an intervention of some sort. In almost all or the majority of situations, that intervention is presumed to be beneficial.
29:57
The effect of the intervention on often is the occurrence of a disease, a clinically manifest disease like heart attack or breast cancer, or dementia or mobility disability. So we're trying to see if the intervention affects the incidence of disease such that it's lower in the intervention group compared to a comparison or control group.
30:22
The second statement is very similar to the first. It just has a little bit more language in it, is the incidence of the outcome lower So it specifies a direction in whom, those who are randomized to the intervention group as compared to whom, compared to those who are randomized to a control group. And there's a lot of variation on how this can be even more refined, putting more detail in about the intervention, specifying the disease, specifying the control group.
30:54
But I want you to get a sense here, is that we're trying to establish cause and effect.
31:02
In terms of the hypotheses, as you may recall from previous workshops we've done, or in your own educational or professional area of work, it's actually the null hypothesis that's tested, and the null hypothesis is that there will be no difference in incidence between the intervention and control group.
31:22
The alternative hypothesis, the hypothesis we believe is true, but that we don't test statistically, is that the incidence of the outcome is going to be lower in the intervention group, or the incidence is going to be reduced as a consequence of the intervention compared to the control group. Okay. So very important to remember there are two hypotheses Just like with other study designs
31:47
But the statistical procedures that Dr. Wilding, Dr. Miller and others are going to show you over the next few workshop installations, those procedures are actually focused on only one of the hypotheses, and that's testing the null.
32:05
I want to just look at a couple of basic schematics of some of the common types of randomized trial designs. And you'll notice by name, a few of these are going to be part of the Spring Workshop series, where the presenters will spend an entire session breaking down this particular type of design and some of its nuances. Here, I'm just giving you an overview so you're familiar. The most simplest type of randomized trial design is a two group parallel group or two group parallel design.
32:39
You start with a base population. I'll tell you what that is momentarily. Eventually, from the base population, you end up with your study population. And that's the group of people who will ultimately be randomized. What happens when they get randomized? They end up in either the intervention group or the control group. And then both of those groups are followed up for occurrence of the outcome, with the hypothesis being that you're, the alternative hypothesis that the outcome rate is going
33:11
to be lower in the intervention group as compared to the control group. Parallel group design one that you may be familiar with that has gotten a lot of attention over the past several years is the SPRINT trial. This randomized controlled trials changed the landscape of clinical blood pressure evaluation and treatment in the United States. This was a simple parallel group,
33:37
two group randomized trial where a group of patients who met the eligibility were randomized into the intensive treatment group, which was the intervention versus usual care, and then they were followed up for various outcomes. A very simple trial design that had huge clinical impact in terms of practice and what we think is better control of blood pressure and reduction of clinical consequences owing to high blood pressure.
34:13
Another common design, and Dr. Wilding mentioned he's going to do a lecture specifically on this is called a crossover design. And some language will refer to this as a within person's design. And that's because the participants actually experience both of the groups, both the intervention that you're trying to test and the alternative group, which is often a standard of care or usual care. So once you have your study population and the crossover design,
34:48
those individuals will get randomized to be either in the new treatment or the standard of care, and they'll proceed in that group for a defined amount of time. Now, what then happens is they're going to cross over. They're going to change groups. Group one, who may start with the new treatment, crosses over to the standard treatment. And group two, who started with the standard treatment, crosses over to the new treatment.
35:18
Listen carefully. The crux of a crossover trial and you should bring this up in the spring If you go to the crossover trial design, if it's not discussed is the crux of this Being effective is there has to be a washout period, and the washout has to be adequate and preferably measurable so that you know the previous intervention is no longer physiologically acting on the system that you're trying to evaluate.
35:52
So you'd have to have some period of time and some measure that at the end of that washout period, any effect of the first intervention that was received is no longer operating, or it's at least reduced to a point where you're now comfortable when the two groups cross over, that the predominant impetus on the system is that second experience they have. One of the things we worry about, of course, as you might imagine, is the adequacy of the washout and whether or not there's any residual effects
36:28
of the prior treatment operating when they're encountering the second treatment. But this is a design that's used under certain circumstances. Give you an example. Quite a while ago, a double blinded, randomized crossover design was used as one of the first studies to test whether dietary sodium
36:55
And manipulation of dietary sodium could affect human blood pressure under free living circumstances. So the way this design worked is that each person would undertake a condition and each phase was one month. So the food that the people in this trial were going to eat or consume was provided to them, and their intake was monitored.
37:26
The sodium was highly controlled in the food provided, such that the lowest dose or the lowest amount of sodium 50 milli moles per day served as the comparison or the placebo, and then the other two conditions were incrementally higher amount of sodium. The way the cross-over design worked is the participants were randomized in terms of the order in which they would receive these groups.
37:53
Each phase lasted a month. There was a washout in between each phase of about two weeks, and they were able to use 24 hour urine measurements of sodium to determine that sodium levels had gone back down to a certain level after receiving the previous intervention in order for the person to cross over into the next. And this is just a snippet of the kind of results that came from this trial. These bars represent the average urinary sodium and blood pressure
38:30
For all the people that experienced the lowest sodium, the middle sodium, and the highest sodium and you can see for yourself there's a graded statistically significant increase in blood pressure with incremental, incrementally higher sodium intake. This was one of the first randomized trials in the US that showed dietary sodium can be manipulated and it can affect blood pressure in individuals that had hypertension.
38:57
Crossover design. Another really important design to be familiar with is called the factorial design. Factorial designs are complicated because they examine the intervention effects of more than one treatment simultaneously in a randomized population. One of the most famous factorial designs in the U.S., at least, was called the Positions Health Study. And this was done in the late 70s, early 80s. And this was really the first trial in the US to try and establish whether low
39:34
dose aspirin could be used for the primary prevention of coronary heart disease. This was done in a group of male physicians. But the other thing that this trial had was a secondary intervention, where they were also testing whether or not supplemental vitamin E could lower the risk of cancer. And so the way this proceeds now, in this factorial or two way randomized trial is it starts with 22,000 men who first got randomized to aspirin or a placebo condition,
40:09
the control condition for aspirin. After that initial randomization, those two groups then were further randomized to vitamin E supplementation or the control or placebo condition for vitamin E. And at the end of the day, you end up with this two by two table where you have the ability to look at both aspirin and vitamin E This group right here in relationship to the two study outcomes, you have aspirin only, you have vitamin E only
40:42
And then you have a true control group that got neither aspirin or vitamin E. This is a very efficient design to maximize
40:56
Resources needed to implement the trial, as well as maximize the information gleaned from the trial results with respect to two different interventions and two different clinical outcomes.
41:11
More recently a large scale factorial randomized trial was conducted. This was called the VITAL trial. And what was interesting about this, this trial studied two dietary supplements that the community has been convinced lowers their risk for cardiovascular disease and cancer. And VITAL tested this hypothesis that vitamin D
41:44
Or omega three fatty acids, which are basically fish pills, fish oil pills that give you high dose or medium to high dose omega fatty 3 acids, which are thought to be health enhancing where they tested both of those interventions on cardiovascular disease and cancer. So the first randomization was 20,000 individuals to vitamin D versus placebo. And then those two groups got randomized to the fish pills versus placebo. And again fish pill versus placebo.
42:17
So now you have people who got vitamin D and the fish pills vitamin D only the fish pills only and a true placebo group. And the follow up, the outcomes in this trial were cancer and CVD And as it turned out, neither supplement was associated with lower risk for either of the endpoints. But yet you can go into GNC tomorrow and you will see vitamin D with a bottle and a big heart on it that says heart healthy.
42:45
And we have clinical trial evidence now suggesting that that may not be the case, at least for the outcomes evaluated here.
42:54
A design I like to just mention, primarily because it was developed here at the University at Buffalo many, many years ago, is what's called the Zelen design. It happens to be being used in a large, pragmatic, randomized controlled trial. It's part of the National Women's Health Initiative currently here in the U.S. And the unique thing about the Zelen design is, is that it's what we call an opt out design. Let me explain this to you.
43:22
Marvin Zelen was a statistician in the statistics department here at UB when he developed and published this design in 1979. So you start with a group of people that are eligible based on already existing information or data. So this is a design that lends itself well for embedding it into a prospective cohort study where you've defined a group of people You've collected a whole bunch of information on them, you can characterize their health very well
43:54
And now you want to conduct a randomized trial within that cohort to test a hypothesis more in line with cause and effect. So you have your eligible group of people based on existing data. And then if they're eligible you can see here they go to this yes group. That group who are eligible to be in the trial of interest then gets randomized
44:19
To the control condition or the intervention. Now, you haven't said anything to anybody yet. You haven't invited them to be in the clinical trial yet. You haven't done any consenting. You basically pre randomize a group of eligible people into an intervention and control condition. And that point, those who got randomized to the intervention group would be asked, are they interested in being part of a study And the study would be explained to them. And basically it would be up to them to say no.
44:55
So they'd be opting out as opposed to opting or consenting in. So why would you do this? Well, there's a couple of reasons. The two big reasons is one thing that happens in certain types of clinical trials, dietary physical activity trials in particular, is that you typically get people who would be willing to enter the trial that have a propensity towards whatever that trial is testing. They like to exercise, they're interested in a certain diet pattern.
45:29
They're interested in taking a certain dietary supplement. Okay. So you kind of start off with a group of people who are slanted towards one direction, as opposed to a more general representation of people who you really want to test your intervention with them. So that's one reason you probably get a little bit better representation of people because you haven't asked them to come in.
45:56
You determine they meet your eligibility and now you're asking them, do you want to go out?
46:03
And what we found in the Women's Health Initiative The second big reason this design is something to think about when it's appropriate to be used is we had a very small, very small percentage of individuals who actually ended up opting out.
46:20
And so you tend to have a more preserved, excuse me, a more preserved working study population or functional study population that you'll actually conduct your intervention in because you tend to have fewer people that say no at this point, you're not losing anybody from the control group. They're just going to be followed passively for the outcome without necessarily even knowing that they're serving as a comparison group for another group of people that are under active intervention.
46:59
So the Zelen design or the opt out design.
47:05
Now moving to the next part of what I want to kind of talk about is clinical trials, as I'm sure many of you know, especially in human studies, tend to progress over defined phases.
47:21
The phase one design or the phase one trial is typically very small scale in terms of sample size, and it's really looking to evaluate safety more than anything else. Is there any obvious safety issues with a new intervention or device perhaps? Looking perhaps to try and better understand a dosing range and looking at side effects as part of the safety profile The phase two trial is bigger in terms of sample size and scale, and that's where we really start looking at efficacy.
47:59
Does the intervention work with respect to the outcome we're trying to change. But we're still also evaluating safety. Phase three is the largest of the scale in terms of testing the hypothesis that the intervention will reduce the incidence of disease. And so there we have to have statistical power to test the hypothesis. So many, many more people enrolled, larger scale, focusing on efficacy Does it work but still looking at safety considerations.
48:42
And then lastly the post-marketing study. Once an intervention or device is on the market, it's approved to be safe and it works Then we want to evaluate is it effective in the general population, Is a sufficiently high proportion of people using the intervention or taking it up appropriately And are we seeing expected results in the community setting?
49:11
One thing I want to be sure that you caught on to across all of these, whether it's the initial phase one or the post-marketing phase four: safety safety, safety is always part of what we're interested in when we're doing experimentation on humans.
49:32
Broad categories, then that the randomized trials will fall under are primary and secondary prevention Efficacy and effectiveness, and Dr. Wilding or Dr. Miller, I think, I can't remember which, will also spend some time talking to you about non-inferiority and superiority type trials. I'll leave them to discuss that with you. I just want to give you a working framework for these four general categories. So the primary prevention trial, its focus is to reduce the occurrence of first disease.
50:12
So in a group of individuals who are without the disease of interest, if you give them an intervention
50:20
Is the rate of developing the disease lower in the intervention group as compared to the control group?
50:30
Now, one of the things that you'll see when you talk with Dr. Wilding and Dr. Miller and look at some of the statistical considerations around trials a big an important balance of a trial is the sample size, the effect size and the length that the trial has to run. So the longer the trial has to run, the more expensive it's going to be, and the more opportunity for people to drop out or to change groups. We'll talk about that in a moment. And that would be deleterious to the trial.
51:05
The incident rate of the disease that you want to test by comparing Is the disease rate lower in the intervention versus the comparison group That's going to depend on how long the trial is and how big the group sizes are.
51:22
And the effect size is the difference in the incidence rate between the group that got the intervention and the control group. So they're all tied together. One of the things that trials designs often take advantage of is to ensure that there's an adequate amount of disease so that you can have a defined effect size, and that you don't have to have a trial go on and on and on forever To accrue cases is to enroll a group of individuals who have
51:57
predisposition for the disease, maybe because they have elevated risk factors for the disease. Now they don't have the clinical disease itself, but they have something that may put them at slightly higher risk for getting that disease. And so in the physician's health study, or, yeah, physicians health study, factorial disease, factorial design that I just talked about a few slides ago
52:23
Those men, actually, to be eligible for that trial, they had to be 40 to 65, which is the age we see a lot of heart attacks in men in the US, particularly 25, 30 years ago, and they had to have elevated cholesterol, which is a major risk factor for heart attack. So by having men who had a kind of high prior probability of ending up with a heart attack, they were able to test their intervention, probably with a smaller sample size
52:59
That would be the two groups randomized, aspirin versus control, and over a shorter period of time to accrue the cases because they were going to accrue cases at probably a higher rate than in the general population, because these individuals had predisposition for the disease because of their age, their sex, and they had a risk factor for it. Primary prevention. Secondary prevention is asking the question, does an intervention reduce the occurrence or
53:34
reoccurrence of a disease or improve prognosis in individuals who already have the disease? So the big difference between the primary prevention trial and the secondary prevention trial is the presence of disease. In the primary prevention, the disease you're interested in is absent from those who you enroll. In the secondary prevention trial, everybody that's in the trial has had an initial occurrence of the disease.
54:06
And so for secondary prevention, the physician health study might have asked instead, does daily low dose aspirin reduce the risk of having a second heart attack or dying from cardiac causes in the same group of men, except they already have had a heart attack. So they had a prior heart attack. They have existing disease. And you're seeing if this intervention reduces recurrence or improves prognosis. Secondary prevention. Efficacy trials.
54:43
The term efficacy, at least, the crux of it is you want you're trying to understand the extent to which an intervention
54:51
Produces the benefit, your expected effect under ideal circumstances.
54:57
So you want to know does it work? Is it safe? And efficacy trials are where we start to think about understanding mechanisms through which the intervention affects the occurrence of disease.
55:12
In contrast effectiveness trials You're taking an intervention that's already been determined to be efficacious. It works and to be safe, and you're now saying under a more real world or pragmatic setting in the community
55:33
Is the intervention obtained
55:36
By a reasonably high percentage of participants, and is the benefit of the intervention realized sufficiently high enough in the community setting. So, in other words, can we get people to do it and is it practical?
55:52
Effectiveness. You have to have efficacy before you can test effectiveness.
56:03
All right. Moving forward. This is a schematic that gives you a really simplified perspective of a very complicated process of enrolling a randomized trial population. So we start with the base population. This is basically who we want the findings of the trial
56:31
to reflect upon. The first thing we have to do once we define that is to determine who's interested in being in the trial we want to conduct, and immediately we'll lose some people from that base population because they won't be interested.
56:51
So from that base population, those who are interested, the next thing we have to determine are they willing to do
57:01
everything that would be required of them in the intervention And are they eligible? Do they meet the inclusion and exclusion criteria? So again, this pass at enrolling our trial population will yield the second biggest group of
57:23
loss if you will, or wouldn't be able to proceed, and this would be the group that's ineligible or not willing. They're not willing to do everything in the trial that's required of them, or they fail to meet the eligibility criteria.
57:39
At that point, we have a group who's eligible and willing
57:45
And it's at this point we would do the consent and something I want to be sure you understand is when you consent somebody into any study, but particularly into a randomized trial, you have to fully disclose to them
58:01
what will happen as a function of being in the trial What will be expected of them in terms of what they have to do while they're in the trial And most importantly, or at least right up there among the highest of importance You have to let them know that they may not get into the group that they want to get in. So part of the consenting process is you're going to tell them, what are the groups in the particular trial that they're being screened for enrollment.
58:35
And what are the expected outcomes of the group? What are the expected risks? And oh, by the way, there's a chance you'll be put into a group and it won't be the one you want to be in.
58:46
So at that point, that's another place where we may have people say, you know what? On second thought, if you can't guarantee me, then I'm going to go into the treatment group where I might get benefit Then I'm not interested in doing this. So we may lose some people at that point too.
59:04
But at this, somewhere around this point in time or this flow, if you will, through the enrollment cascade, we have an eligible and consented group of people. At that point, those individuals would undergo their baseline examinations, and then they would be randomized to the study groups. And the randomization basically gives each participant the same probability
59:32
of ending up in the treatment group or the control group.
59:38
Once into the groups, then the follow up on treatment intervention or on intervention follow up begins where the individuals are experiencing the conditions of the groups they were randomized to And as the follow ups moving forward, cases of disease are being recorded, adverse events or side effects are being recorded. Remember, this is where we start to understand safety and we could develop the safety profile of an intervention.
1:00:12
But also what's happening along the way is something we call losses. We have people randomized. They start the follow up interval and then at some point we don't know anything more about them with respect to whether they develop the disease or not. And this is something we really pay attention to and want to minimize as we're doing randomized trial work, because as these losses accrue and we have somebody who was randomized, but we don't know whether or not ultimately they had the disease or not
1:00:48
That starts to chip away at our ability to properly test the hypothesis from a statistical standpoint. We'll talk more about that a little bit more today and then quite a bit more in the forthcoming statistical workshops. And then lastly, the trial has a formal ending point, and that's called the closeout. That's at the time where the follow up would be over and we would unblind the two groups if this was a blinded design,
1:01:18
and the participants would be told which group they were in, and the investigators would know what group the participants were assigned to. And it's at that point that the data analysis and the trial reporting would take place. This is a very, very complex process. This schematic way oversimplifies the amount of effort it takes to move a group of individuals through the point of randomization and try and ensure that you keep some reflection of that base population of interest intact.
1:01:57
So now for the rest of the time I'm going to be with you, I'm going to go over a few aspects of the implementation of randomized trials, and I'm going to refer to two examples So you can see what I'm talking about in trials that have already been done. And the two that I'm going to use, both of them were primary prevention trials. Both of them were efficacy trials. And both of these clinical trials changed clinical practice in the United States immediately.
1:02:33
The first is the United States or the U.S. Diabetes Prevention Program. This was published in 2002 in New England Journal, and then in that same year, the primary results from the Women's Health Initiative randomized controlled menopausal hormone trials were published. And so I'm going to give you examples from these documents in terms of what the points I want to cover for the remaining time, what it actually looks like, based on what's published from a trial that's completed.
1:03:09
We can't there's no exercise we could possibly do in this workshop to give you some hands on with this. So the next best thing, I think, is pointing you somewhere where you can connect the dots for yourself.
1:03:22
So these are some of the things I'm going to try and get through, time permitting. We'll just go in order. We're going to start with selection of study sample. Now, I want to take a quick pause here. And we've gone for about an hour, a little less than an hour. So two things. First of all, are there any questions? Yes, I actually have three questions and I'm just going to read them. And then I think the person who asked them should come on because they can actually add more detail to them.
1:03:53
So the first and second question is from Marwa and she asks, but if they don't know, how would you collect data? So I don't know what that means. So I think she can talk about that. And she also is asking, are they getting standard treatment? So I think I will have her come on. And then I have another question from Vijay once we're done. Marwa, are you available to unmute? Yes. Great, thanks. Sorry. Hi Marwa. This is Mike. Yeah. Can you tell me what you mean by your question, please?
1:04:25
Sorry, I was just writing it out when you were actually giving the lecture. The first one is when the patients don't know this is the control group. The professor who developed this study from Buffalo University. Yeah. The Zelen design, right. Yes. So if the patients don't know they are the control group. So what if they don't share their medical history medical data? So later on, let's say year from now or two years from now, if they drop out
1:05:02
How would you have a control group? That's a great question. And I probably wasn't as clear as I should have been when I was talking through that schematic. So remember at the I'm going to go back to that Marwa. Remember here. Oops. Right here. The front end of the Zelen design requires that there's a group of people that you already have existing data on. So I suggested this is where we often will take an existing cohort study where you have collected information
1:05:38
on health characteristics and you're following them prospectively for clinical disease occurrence You take that cohort study and you nest within it, if you will, or embed within it the Zelen design for a nested or an embedded randomized trial. The way we get the data in the control group is they're being followed for outcomes as part of the parent study to begin with. So let's say annually there's a health update where they may report, oh yeah, I got cancer last year.
1:06:12
Or they do a follow up for mortality. You'll find out someone died. You already have the health information on them as part of the cohort structure, and you're getting the outcomes through some already ongoing mechanism that you're leveraging. So what you're basically doing is you're letting them go about their normal life and following them for their disease history. At the same time, you're intervening on a group from within that same cohort,
1:06:43
and you're going to compare the disease experience between the group you intervened on and the group who was just followed without intervention. Does that make sense? Yeah. Okay good. Yeah. Did you have another question? Yes. Another one is so I I thought like usually everyone gets the standard treatment and clinical trial is when the standard treatment isn't, you know, good enough. So when you explain that when you, when they get the consent when they're doing the consent
1:07:24
when the patients refuse to be in the clinical trial because what if they don't get the clinical trial intervention treatment right, but this is the this is the 50/50 chance. Why wouldn't they take it? Right. And that's a good question. And I think as investigators and the staff actually that are doing the recruitment and screening and answering these questions, doing the consenting we really, you know, we can speculate why somebody may choose
1:07:57
even though they have, as you say, 50/50 shot of being in one group or the other It isn't really in our prerogative to understand that, and we certainly wouldn't want to take that kind of an understanding and let it influence how we proceed at that point For group assignment, the group assignment has to proceed with the randomization plan and usually in a blinded manner such that only the person doing the randomization, it's usually the senior biostatistician or one of his or her designees.
1:08:35
Other than that, typically the participant and the other investigators and staff are not going to know which group. But the person may say, look, I've got X condition and I'm interested in this trial because if I get into the treatment group, I think from what I hear, your treatment might help me. But I only want to be in the trial if you can assure me that I'm going to get in the treatment group.
1:09:00
That's where you have some of this rethinking of people and saying, well, I don't want to be in it then. And at that point we have to let them go. It would be unethical for us to say, well, I'll guarantee that you'll get in the treatment group as long as you enroll.
1:09:15
Make sense? Yeah. Good. Thank you. Okay. Good question. A couple more questions. How about. Yeah. Let's see. Let me see how they how they go. If I can answer them quickly and then get restarted. Otherwise, we can wait till the end. Okay. Vijay's asking if possible at the end, so you can come back to this Can you brief about intent to treat analysis? Yes. In fact, Vijay, I'm going to talk about that briefly in a few slides.
1:09:44
And I know that that's going to be discussed on the statistical workshops either this semester and/or in the spring. But I am going to mention the intent to treat. Thank you very much. Well thank you, Vijay, it's good to see you. Great, and finally, Gillian would like you to go back to the last slide And could you summarize it, please. This one? Gillian, is this is like you were asking about? I just didn't know what he said about it. I didn't hear anything. I just saw it flash up and flash down.
1:10:20
But there's really nothing to summarize. Yeah. No. Gillian, all I was going to, all I did is say I'm hoping, time permitting, to say something about each of these These are the major components. Gotcha. There's a lot more behind this, but these are the major components that we think about when designing a trial or implementing a trial. So all I was doing was saying, I hope to cover these. And with that I think I will move forward
1:10:47
then. We can say. Yes, actually, we're all done with questions. So yes, please do so. Thank you. Okay so back to study selection or sample selection. So again we have this very first swath we have to define. And that's the base population. And as I mentioned that's the group who we want the results to generalize to. Now often, you know, in probably an unrealistic world we'd say, well we want to study everybody, everybody that's alive.
1:11:17
We want to test this intervention. And of course that's not practical. So what we have to do when we define that base population is we have to use restriction to focus it. Are we interested in a certain sex, so the Women's Health Initiative obviously was interested in women, postmenopausal women for example.
1:11:38
Are we interested in a certain age where older or younger may be excluded? And that narrows the focus. Prevalent disease. So in the diabetes prevention program, which is one of the examples I'll use here Everybody had to have pre-diabetes to get into that.
1:11:56
And they were wanting to understand in a high risk population defined by pre-diabetes, which is one step away from being clinically diabetic Does the intervention reduce the progression from prediabetes to clinical diabetes?
1:12:12
This restriction then, is further done or further honed by the inclusion and exclusion, and the inclusion and exclusion will often be partly to narrow the focus on whom you are intervening or testing the hypothesis, or kind of, if you will, refining the physiological system that you're trying to test the intervention within in terms of, of a human of certain phenotypic characteristics. And then exclusions are often, especially in experimental research, safety issues.
1:12:49
So if you're testing a drug intervention, any sort of a drug interaction that would be adverse or that would predispose somebody for a known side effect, that might be one of the exclusion criteria. Often unless the study is focused on women who are pregnant or planning to be pregnant, many trials will exclude individuals who are pregnant or safety concerns about both the mother and the fetus and so on.
1:13:21
One of the other things we do with this restriction is we can impose experimental control by design. We can get rid of extraneous factors that we don't want to influence either compliance with the intervention, safety of the intervention, as I just mentioned, or the intervention effect itself. So if we thought smoking would adversely affect the intervention we're trying to test, we may exclude current smokers
1:13:54
from the potentially eligible population. If we want to test menopausal hormone therapy in postmenopausal women, we are probably going to restrict men out of that base population of eligibility and so on. So once we have the base population, then we have to move through our flow as I showed you with that schematic to get down to the study population And this is who we're actually going to end up randomizing. So who does this represent?
1:14:28
Well, this represents people who would be eligible based on our inclusion and exclusions
1:14:35
And be willing, they'd be willing to or say they'd be willing to do what's required of them as part of the intervention. It may be as simple as taking pre-specified pills in a certain dose using a pill pack, a calendar pill pack. It may be doing a certain amount of exercise. It may be changing their diet. It may include not only the actual intervention activities, but then follow up activities. Maybe they have to return a questionnaire every six months or every year.
1:15:06
Maybe they have to have a clinic visit every year. Maybe as part of the clinic visit, they have to have an MRI done so they can't get claustrophobic when they go in the tube, and so on and so forth. So willing is very important that they understand what's going to be asked of them, and that we, as investigators, have a general assurance that with that understanding, they seem to be willing to undertake those activities. Once we have eligible and willing, then of course we consent them.
1:15:38
And that's a thorough, thorough explanation of the study design, the trial design, the objective, the question being asked, what the expected results are, and the implications of those results beyond the study population itself for the greater good, if you will.
1:15:58
And the fact that there's probably risks and benefit.
1:16:04
And then lastly that people get assigned a group randomly, so there's no guarantee that they'll be in the group they want. And this level of consideration, as you're going to hear from the statisticians, will come on for later workshops This is a very important level of enrollment because we have to ensure that the study population, the groups that got randomized, are large enough so that we have sufficient event rates for the disease.
1:16:35
That's what's going to drive our statistical power. And also that this group of people who we ultimately randomize have a high likelihood of compliance and completion of the intervention and the follow up interval. And that's something that we try to address with something called a run in phase. I'm not going to say much about that today. When I come back next time, I'm going to spend the entire workshop differentiating between pilot studies and run in phases.
1:17:11
So just kind of put that as a bookmark for now. And if you're interested in run in and pilots, come back for our November session.
1:17:20
So in the Women's Health Initiative, the primary question was
1:17:26
Does exogenous estrogen plus progestin, postmenopausal hormone therapy, reduce so it's directional, the risk of coronary heart disease in generally healthy postmenopausal women? Very explicit. Estrogen plus progestin We could have added the doses in here and the type to make it even more explicit. Reduce. It's a directional intervention, so it's beneficial, CHD is the outcome. And the group we're trying to understand the question in postmenopausal women.
1:18:04
So the reference population for WHI was very broad. It was the entire US population of postmenopausal women. That's the base population. They had a targeted enrollment of 27,500 women
1:18:21
With subgroup enrollments such that 20% of the randomized group would be from race and ethnic minorities. 30% would be women in early stages of menopause, less than 59 years of age, and 25% would be in late stages or further away from completing menopause 70 to 79. So how do you reach these people? Well, a huge amount of the trial budget was
1:18:55
allocated to trying to do campaigns using mass media, mailings, television ads, etcetera, etcetera to try and find age eligible women in communities across the U.S. there were 40 U.S. centers, and this took from 1993 to 1998 to complete. Now, I want to impress upon you for this trial, they screened over 300,000 women.
1:19:27
And identified a group of women who for this arm of the hormone trial
1:19:35
Finally was 16,608. 16% were from minority representation and they achieved the age targets.
1:19:44
So the exclusions that helped hone down the base population to those who ultimately got randomized, they excluded uncertain diseases because this was a primary prevention trial And because cardiovascular disease was the primary disease outcome, those who would be randomized could not have a history of cardiovascular disease. And they also had exclusion on cancer. As I tell you in a minute why. And there were also safety issues because they were using a drug as an intervention
1:20:20
And that drug could have certain hematological untoward effects or interact with other drugs. These had to be taken into consideration for safety purposes. And so that further helps hone down that base population into the group who ultimately got randomized. And the randomized sample was over 16,000 women As I said. For the Diabetes Prevention program the primary question was, is incidence of type two diabetes
1:20:54
I'll just refer to this as diabetes now, but its type two adult onset diabetes, is the incidence of this disease lower in high risk adults because they have prediabetes, who are treated with lifestyle or a drug, metformin. So the base population was all U.S. adults who had prediabetes. Again, this is a very broad base population they start with. Their targeted enrollment was 3000, half of which would be women, half from minorities, 20% older adults.
1:21:28
How did they find these people? Again, this was a national recruitment effort. They use culturally targeted recruitment, partly because diabetes does affect certain racial ethnic groups more so than others. So having representation of those groups in the trial was important. It took two years, and it was done over 27 U.S. centers.
1:21:53
They screened over 9500 to get their enrollment, which was 3234, a little bit higher than what their target was They achieved their quota for women, and they were just under their quota for racial ethnic minority representation in older age. And again, exclusions included the outcome of interest. You couldn't have diabetes in this trial because developing diabetes was the outcome of interest. And then certain other conditions that would be considered either unsafe or competing risks.
1:22:30
So that's a general picture how two very large and very successful primary prevention efficacy trials were done in the US in terms of how they recruited and enrolled.
1:22:45
So now that you have the people, how do you put them into their study groups? Well, first of all, the question is when. When do we allocate them into the intervention and control group?
1:22:57
Most often this is done, and I say most often because the Zelen design is an example contrary to this, but most often it's after you've determined who's eligible and willing and has consented
1:23:12
And has had the baseline assessment. Now some people will say, well Mike, if you going randomize Why does it matter when they have the baseline assessments. Those characteristics will distribute equally over groups. What else distributes though with the randomization is any nuances at the clinical centers that may have affected the baseline assessments.
1:23:34
And so by doing the baseline assessments first, if the Buffalo scale, for example, was reading five kilograms high for everybody that was weighed, that should distribute equally if the scale assessments were done prior to randomization. If it was done after randomization, it's conceivable that issues with the baseline assessments could accumulate
1:24:00
in one group versus the other and potentially form opportunity for bias. So that's why the baseline assessments typically are done prior to randomization. Now randomization as I already said is equal probability of group assignment.
1:24:18
This is the characteristic of the randomized trial that makes it so strong
1:24:25
To infer cause and effect or causal inference. Because each person had the same probability of group assignment, the extraneous factors, the baggage we carry with us that's unique to each of us, both those that get measured and the stuff that's unmeasured, like genotype, for example. All of that should be balanced between the two groups. So because that occurs if the randomization worked,
1:24:54
The process of randomization actually maximizes the comparison of the outcome between groups. When you look at the incidence of disease between the intervention group and the control group, if the randomization worked, the only explanation
1:25:12
that should be plausible is that the intervention effect
1:25:19
is the cause for the difference between disease experienced between the two groups.
1:25:24
Now, that's an ideal world. There are circumstances where even with perfect randomization, it worked, you've had really good balance at the point of randomization. Things can happen post randomization that can become challenging for that comparison of incidence between the two groups. And you're going to hear more about that when you hear about some of the statistical procedures.
1:25:50
I'm going to skip over the next slide for time sake, so I'm just going to pass by it.
1:25:56
So often in primary results papers from randomized trials, there'll be something that colloquially we call table one. So this is the table one from the Women's Health Initiative primary results paper. And what this does is it shows us all of the characteristics And this is a truncated picture of the table, all the characteristics of the participants
1:26:21
These were all measured prior to randomization. And then we're seeing the characteristics described as means or percentages after the two groups were randomized. And the question we want to ask is, is there balance between the groups? Are the mean age for example, is the mean age not different between those in the intervention and those in the control? Is the distribution of age proportionately not different?
1:26:54
Distribution of race and ethnicity proportionately not different, and so on and so forth. And in randomized trial reporting, typically you will see even in this descriptive table a hypothesis test and the resulting p value, because we actually do have pre-specified hypotheses about the randomization. Our hypothesis is that there should be no difference if the randomization worked on characteristics between the intervention and control group.
1:27:28
And so these p values, if we assume this was 0.05 alpha All of these are well above 0.05. And they give us assurance that the two groups were balanced in terms of these characteristics. The randomization worked. In the diabetes prevention paper you'd see the same basic type of setup
1:27:48
again truncated from the paper, just as an example, the placebo group and then the two intervention groups. They don't report p values in the table, but in the text it says that there were no significant differences in any of the characteristics across the three groups. So again, did the randomization work from what the investigators reported? Yes. It appears to have balanced characteristics across groups.
1:28:18
Okay. Blinding. You'll often hear, was it a blinded trial or was it an unblinded trial? So first of all, why do we blind? The whole purpose of blinding is to conceal which group a person is in.
1:28:33
So the obvious extension is you don't want the participant to know, quote unquote, if they're in the active intervention arm or the placebo control or comparison arm, okay. You don't want them to respond differently during follow up knowing which group they're in. Okay. That's the participant level. However, in
1:28:58
practicality, aside from a certain set of circumstances where the investigator particularly if it's if the trial has clinical implications that are acute in nature and there'd have to be clinicians aware of who's in which group That, notwithstanding, the blinding is often not only the participants, but also the investigators and the staff. And that's what we call a double blinded trial.
1:29:30
Essentially, the only person then that knows what group a participant is in or has been randomized to is the person who did the randomization. And as I said, that's usually the senior biostatistician or their designee. So then what does it mean to have a placebo control or a placebo blinded trial? Placebo is a form of blinding. It's a control condition that, for all intents and purposes, is totally indistinguishable
1:29:59
from the active intervention. So if it's a pill
1:30:05
That pill would be packaged. It would look like, it should taste like
1:30:12
It should be taken on the same schedule as the active intervention pill, except the placebo should exert no physiological effects that would have that would influence the outcome. In other words, it's inert. We used to call it a sugar pill, but that kind of came to be passé because giving people sugar probably wasn't at some point thought of as a good idea. Not that the pills were really sugar pills, but just to get away from that notion that it's a sugar pill, it's a placebo.
1:30:46
It's an inert condition that should not exert physiological effects. So not only is placebo important for blinding so that those in the control group
1:30:58
And those in the intervention group don't know which they're in on the basis of how the pill looks or how their activity in the group proceeds The other thing placebo is extremely important for is to establish adverse events, the safety profile.
1:31:17
These are a couple of examples. These were trials that were done years ago. The first upper part of the table I'm showing was one of the first antihypertensive trials in the US testing a beta blocker. And one of the concerns, from a safety standpoint for beta blockers in men was impotence. And so what they were able to determine, if you didn't see this placebo column and you saw 29%
1:31:48
frequency of reported impotence in the intervention group, you might say, oh my gosh, it's almost 1 in 3. That must be really bad. But then when you look at the reported impotence in a placebo control group and you see that that frequency is nominally the same and not statistically significant different We can conclude based on this placebo condition that at least in this trial, impotence was not likely an adverse safety issue.
1:32:19
In terms of the Physicians Health Study, which I mentioned a couple of times now that tested low dose aspirin for primary prevention of heart attack, One of the concerns was would it increase bleeding in the participants? And you can see when they sorted out the data and looked at the frequency of GI symptoms, much of which ended up being bleeding, there was a significantly higher
1:32:50
Amount of bleeding or GI symptoms in the intervention group as compared to the placebo control.
1:32:57
And it's actually this trial that began to move pharma towards having to put on aspirin bottles that one of the known side effects is an increase in GI symptoms and in GI bleeding. And that's because there was pretty clear evidence, given a placebo control, that this was not
1:33:22
a chance finding that this was systematically higher in one group compared to the other. So the placebo is very important not just for maintaining blinding, but also when you want to evaluate adverse events between the intervention and control.
1:33:41
Compliance and completion. This is huge. All the money, all the effort that gets us through that screening and enrollment algorithm, the process of randomization, the implementation of the intervention and all the costs that go around that both in terms of the investigation team and the participants The last thing we want to have happen is have really low compliance or completion, and be unable to definitively test the hypothesis.
1:34:13
So one of the things I think we have to be aware of, and maybe our inclination is to say, well, compliance and completion, that's all on the participants. That's on their shoulders. No, that's not true at all. Certainly the participants have a role.
1:34:28
Active participation and cooperation. We try and screen that at the front end As I mentioned, when we look at willingness, but there's certain characteristics we know will challenge participants. And two of them are trial duration and complexity. The longer a trial runs and the more complex the trial is with respect to what the participant has to do, the more likely we're going to run into issues of noncompliance and possibly non completion.
1:34:58
On the researcher side, compliance and completion is not a passive issue.
1:35:06
There's an investment allocation of resources and funding and personnel to do active monitoring and facilitate the participants remaining engaged and doing what's expected of them as part of the group they were randomized to. So how do we do this? Well, our selection process, interested They're interested. They're willing and they appear to be reliable. Well how do we know that they're going to be reliable? Well we can ask them. Of course. You're taking a person's word for it.
1:35:39
And that's where I'll make the connection of a dot between the reliability of the participant and the run in phase, or a feasibility study. And I'm going to talk more about these the next time I'm with you. But the main point here is that there are small little sub studies that we can do to identify people who give off, if you will, red flags that they could end up being a non-complier. And then we have a decision to make at that point. Do we dismiss them from the trial?
1:36:11
Do we retain them but keep a close eye on them? And I'll talk more about that the next time. Frequent contact is critical.
1:36:20
Once you have someone engaged in a trial, staying in touch with them, making their role, what they're doing as easy as possible. So packaging up their drugs. If it's a drug trial in calendar packs, where all they have to do is pop out the pill and it's on a calendar timeline. If there's a break in the treatment over a certain number of days, it's all factored in, making it easy on them. And of course, incentives. Many trials do provide incentives of some sort.
1:36:51
It could be results of testing that's being done as part of the trial that you're getting done for free, and can be shared with your own personal health care provider. It could be in the form of monetary or other sorts of gifts of appreciation.
1:37:06
But we also have to monitor. We as the investigators have to have a close eye on what's going on in the participants within the groups. Now, sometimes that's simply counting pills. So if the participants are coming in annually for examinations and you ask them to bring in their pill pack, you can count how many pills were dispensed, assuming that they haven't popped out pills, but they didn't really take them. Some bottles now have microchips in the cap so
1:37:36
every time the bottle is opened and closed, it's recorded in the microchip. Some bottles have a self weighing capacity. So you start with a certain weight with all the pills in them, and then it can be reweighed and so on. Periodic questionnaires. Just asking them, you know, if they're in an exercise trial How many sessions did you attend this week? Or if they were supposed to make a certain change in their diet
1:38:00
How many meals did you consume this week that had X number of this or that as part of the meal? That's a lot more subjective, of course, but it's something we do and we invest resource in. Some things we can track with biomarkers, whether it's a urine biomarker or a blood biomarker. There could be metabolites of drugs or metabolites of metabolic pathways that can be detected and give us a sense whether or not compliance is in place.
1:38:30
Noncompliance becomes a problem. What is noncompliance? That's when somebody leaves the group they were randomized in. So they either drop out of the group they were initially randomized to, and then either go into the alternative group or they just basically stop altogether and they're no longer engaged with the trial.
1:38:53
And Vijay if this is helpful, this notion of having what we sometimes refer to as noncompliance or crossover, where we're having people go into the opposite arm that they were originally randomized, or we're having people just stop altogether and they're no longer engaged in either arm This is where the intent to treat analysis or the intent to treat principle is used.
1:39:20
The intent to treat principle. And you'll hear more about this when Greg and Austin and Jeff talk about certain analytic techniques. But the idea here is the primary hypothesis is tested
1:39:33
on the assumption that all participants remained in the groups they were randomized to.
1:39:40
The hypothesis is tested, and then sometimes, perhaps frequently
1:39:47
as a secondary analysis, you might do a completion or a compliance based analysis, where you analyze the data, test the hypothesis where you sort or stratify the group into those who are at a certain level of compliance versus those who are below it. Sometimes 80% is used. So 80% compliant versus less than or equal to 80%, something like that. And you can make a determination then how likely do the primary results from the intent to treat model reflect the reality
1:40:20
of what actually happened in the conduct of the trial over the entire period of that?
1:40:28
In the Women's Health Initiative, there was a very interesting pattern of this. This graphic comes from the primary results paper. So they're disclosing all of their key features of the trial so the scientific community can digest this while interpreting the results. I want you to focus on the dark circle and the dark triangle. These two lines here. What you can see on the y axis is the percentage or the proportion that are dropping out of the intervention,
1:41:04
which is the dark circle, or the placebo, which is the dark triangle. And this is from time zero, which is randomization out to the first year. There was about 5% of the placebo that dropped out of the placebo arm, and about 10% of the intervention arm that dropped out of the intervention arm. And what you can see as time goes on is there is a somewhat increasing level of dropout from both groups,
1:41:33
and that at each time period, the drop out proportion is somewhat higher in the active intervention versus the placebo group.
1:41:43
Now, this is relevant to interpreting the findings, but I'm not going to focus on that part. What I want to make mention of is the nature of the trial itself often is something that contributes to patterns like this. So for example, postmenopausal women who may be experiencing vasomotor symptoms or colloquially, hot flashes at the time the WHI was done, which was in the early to mid 90s, early 2000s menopausal hormone therapy was often used to treat those symptoms, hot flashes.
1:42:26
And so you can imagine a woman who's in the placebo group who gets randomized, starts on the course of taking the placebo pill But doesn't get any relief from her hot flashes might make the decision Hm. I can't deal with this anymore. And I'm going to stop doing what you're asking me to do. And I'm going to go, perhaps to my own provider and ask them to prescribe for me The hormone therapy to alleviate my hot flashes.
1:43:00
Likewise one of the side effects of menopausal hormone therapy for a woman taking it can be cramping and distress in a gynecological area of the body and bleeding. So it's quite possible that somebody originally randomized to the active intervention of the menopausal hormone therapy experiences some of those symptoms and decides, I've had enough of this. I'm stopping and that person drops out. Whatever the reason is, you can see in this really well designed and implemented trial
1:43:37
As time went on, women were making decisions to stop the group they're originally randomized to.
1:43:48
In the Diabetes Prevention program, this is what they reported To give a sense of what compliance was like. So first of all, remember that one of the intervention arms of the DPP was a lifestyle intervention, dietary change, increased exercise, and stress management. And the investigators reasoned that this group should experience weight loss That was what they were trying to invoke as a way of reducing the rate of developing diabetes.
1:44:19
So what they did in their primary results paper is they showed change in weight across the three interventions, the placebo group, the drug group, and the lifestyle group. And what they showed and argue in their paper is we see the expected pattern. The lifestyle group lost weight almost immediately. And like with most weight loss programs, over time it starts to creep up.
1:44:44
That is, they start to regain weight. But at all time points, that weight change is most evident in the lifestyle compared to the other two groups. So they suggested this showed compliance with that intervention.
1:44:58
In terms of the pill, the metformin versus the placebo pill They showed this graph, which was adherence to the medication. And this was by pill counting at each six month exam. And unfortunately for them, you can see that first of all, in both groups, the compliance is less than 100%. It's less than 85%. It's about 80% initially in the placebo group But then it kind of squeaks down. But more importantly,
1:45:30
the adherence to the medication intervention was actually the lowest in the active intervention arm compared to the placebo. And so they talk about how this somewhat Concerning less than
1:45:49
less than, I wouldn't say ideal, but less than desired adherence may have affected the results of this particular arm. The comparison of the metformin drug against the placebo. Anyways, this is how they dealt with giving a sense of compliance in the DPP.
1:46:08
Outcomes. That's another very important part, of course, of the trial. What are we trying to test? We're testing an intervention to see if we can reduce the rate of a clinical event in some instances or improve some characteristic, like functional health.
1:46:29
Key things with thinking about ascertaining outcomes in clinical trials is the source of the information is critical.
1:46:38
Many times, clinical trials afford the opportunity to do actual clinical examinations on participants because they're coming back to the clinic or the study site periodically and some information on clinical endpoints can be obtained at that point of contact. Hospitalizations is another source of outcomes information. And for death, we now often rely on the National Death Index. Regardless of which mode of information collection,
1:47:08
the key to a very rigorous clinical trial outcome is that it has to be a standardized case definition
1:47:19
And that that case or that presumed case, once it's defined, it has to be adjudicated or verified using some standardized procedure.
1:47:29
And the data gatherers, the people that are getting the information about suspected event cases from either the clinical exam, the hospitals, or the NDI search. They need to be blinded. They shouldn't know if it's a control person or an intervention person.
1:47:48
Some of the concerns we have that can threaten the internal validity or introduce bias in a randomized trial design that pertains to outcomes is if the data gatherers are unblinded and they start thinking in the back of their mind, even subconsciously, if this is a control person, this must be a real heart attack, or this must be a real lung cancer case.
1:48:13
As opposed to, well, if this person's getting the intervention And in the back of my mind, I'm thinking this intervention is going to lower the risk of disease I might be thinking maybe this is an inaccurate potential case and less likely to proceed in identifying it or registering it accordingly. So blinding the data gatherers is very important. The other thing, if you remember on the schematic of the flow of a clinical trial to the point of randomization, and then once the follow up started,
1:48:46
I told you that we follow up primarily counting three things, counting incident cases of disease, counting adverse events, safety issues, and counting losses. Losses are people who get randomized and start the intervention and follow up
1:49:04
And then we lose touch with them. We don't know whether they actually became a disease case or not.
1:49:12
That is likely going to happen to some extent in most clinical trials. The issue is if there's a large amount of losses and sometimes there's a large amount of losses because of the nature of the intervention or the population. If they're really sick people you're doing an intervention on, or if it's a very complex intervention framework or it's a very long intervention Those can be things that lead to larger amounts of losses. But the key is, is the loss pattern correlated with randomization?
1:49:46
If there's more losses systematically in one group than another, then we have problems. And we have potentially a source of bias. Now that can be evaluated statistically. It can be treated at the analysis level statistically. But once it occurs you can't get rid of it. You have to understand it. You have to see what its impact is on the primary results. Which means, as I said a few slides ago, our active involvement as investigators monitoring the trial,
1:50:17
keeping track of the participants and doing a good job counting cases of the three things we're counting during follow up is critical because we can't determine how big a deal the losses are if we haven't counted all the losses. So this is an issue that ebbs and flows throughout the conduct of the active intervention and follow up period. And this is one of the issues we think heavily about with respect to the internal validity of the findings of a trial.
1:50:46
How much loss was there and was it clearly correlated with the randomized groups?
1:50:52
We have ten minutes remaining. Okay. So the primary outcome in WHI was a little bit interesting. The primary beneficial outcome in the Women's Health Initiative was to see if menopausal hormone therapy reduced the risk
1:51:10
of coronary heart disease compared to a placebo control.
1:51:16
The WHI also had what we what was called a primary adverse outcome. There was enough evidence in place that there was an assumption some proportion of women randomized to the active intervention would develop invasive breast cancer.
1:51:34
Remember. What does equipoise tell us? Equipoise tells us there's still uncertainty with respect to the balance of risks and benefits. So one of the benefits they were trying to understand in this trial was does menopausal hormone therapy benefit heart disease. Does it lower your risk for heart disease?
1:51:56
Against one of the known or really strongly suspected adverse effects of menopausal hormone therapy, which was increasing susceptibility to invasive breast cancer.
1:52:09
So the follow up was planned for eight and a half years. Now, this is a very important number for you to think about in this slide And when I get to the DPP. How did they come up with eight and a half years?
1:52:20
Well, that's a complicated part of the design in the trial. That eight and a half years was determined based on the intervention So the biological effect that menopausal hormone therapy would have on the two outcomes. How long would it take for the intervention to actually change the systems that lead to CHD or breast cancer? What the expected event rates were. So in the control group, for example, what would we expect the occurrence of CHD or breast cancer to be annually
1:52:54
In a group of women that weren't intervened on, and then collectively the intervention effect in the event rates
1:53:02
lead to the question of power. Statistical power to do the test of intervention of rates between the intervention and control group And that's driven by these two things intervention biology, the event rate, the sample size And then how long do you need to run the trial to accrue the event rate? It's complicated. And I hope it comes across that way. As Greg said earlier, you can't just go do a trial because you want to do one.
1:53:30
There's very, very important steps. And it's very experience based kind of research design. It's a very unique design in that regard and it takes a lot of different expertises to bear on each of these elements. Anyways. In WHI, the follow up was done by phone every six months, annually by a clinical examination And all of the cases that were suspected or identified as presumed cases were then verified and adjudicated.
1:54:05
The Diabetes Prevention Program. The outcome was one. The transition to clinical diabetes. The criteria for identifying it was, at the time, the American Diabetes Association criteria, you had to have both a positive fasting blood glucose test and a positive oral glucose tolerance test, which is the clinical gold standard in endocrinology for defining diabetes. They had a planned follow up for five years. They went through the same
1:54:35
considerations as I just outlined for WHI.
1:54:39
How long will it take for the interventions to affect glucose homeostasis? What's the expected incident rate of diabetes? What's the statistical power and sample size? And that helps inform how long they have to run the trial.
1:54:57
One of the things that's unique to the clinical trials as it pertains to the follow up interval, and actually the entirety of the trial itself, is unlike observational studies, For the most part, randomized trials are required to have what's called a data safety and monitoring board, or DSMB. This is a group of people that assure the first and foremost principle of human experimentation, which is first, do no harm.
1:55:25
So these people are external to the investigators.
1:55:30
They have really little stake in the trial from a scientific standpoint. They're not part of the team that will plan or develop or put out the results, but they're there to receive updates on how the trial's going, principally in the form of statistical data and then based on criteria that are developed by this group of people in collaboration with biostatisticians from the study team and the sponsor,
1:55:57
NIH, one of the institutes there, for example, they come up with a framework to monitor the effects and identify adverse effects and develop rules to stop the trial prematurely should that need to happen.
1:56:14
And that's a very complicated issue. I can't even begin to give you enough background on what that means to develop rules for stopping. Suffice it to say, this is part of what the board does. So they receive reports internally on a regular schedule, might be every six months It might be every year. What have you. This comes in the form of statistical data that typically only the primary senior biostatistician has knowledge of, because it has to be unblinded to look at those data for the DSMB.
1:56:50
And then they review that, and they make a determination on the safety of the trial and the integrity of the data. Should the trial stop, should it continue? In the women's health initiative I'm sorry. We have three minutes left. Okay. Let me I'll finish. Just. Yeah. So in the Women's Health Initiative, again, the primary beneficial outcome was coronary heart disease. The primary adverse outcome was invasive breast cancer.
1:57:17
But there were other outcomes that as part of the Women's Health Initiative hormone trial program, they were interested in. And so what they had to do was come up with a way of kind of balancing the idea of equipoise, the benefits against the risks, and summarize it as an overall effect size. And that's what they coined as the global index. And that was nothing more than a statistical indicator that balances the risks and the benefits
1:57:47
so that if the global index had a relative risk or a hazard ratio above one, that meant that on average, the risks were outweighing the benefits. If the hazard ratio or relative risk was below one, then the benefits on average outweighed the risks.
1:58:04
When the DSMB began looking at the data that was coming in and the interim results from the WHI Eventually they saw a picture that kind of looked like this. The overall indicator was that there was, on average, more risk or more harm than benefit. And as to such as it was described, I cut and paste this directly out of the primary results paper on the basis of the recommendation from the DSMB it was determined that early stopping of this,
1:58:42
estrogen plus progestin trial should be considered because the balance of risks outweighed the benefits. So this is a really unique situation of a randomized trial. I'm going to stop just a second. This group of individuals, the DSMB as they're monitoring the safety and the data integrity And as the trials proceeding and women are experiencing the intervention, the control group, cases are accruing of various end points, adverse events are occurring and so on
1:59:16
This group decided the trial after only five roughly five years of a planned eight and a half year follow up was starting to suggest that the risks were outweighing the benefits And so, as such, the job that group had was to make a recommendation based on those interim data. And in fact, the trial was stopped early. The stopping was a little bit more complicated than what I just described. But if you read this paper, there's more detail in it with respect to how this actually proceeded.
1:59:48
For the Diabetes Prevention Program I just want to point out that in the primary results paper, they also showed a table that kind of balances the risks against the benefits. They have a table of adverse events. You might not be surprised that the lifestyle group where exercise was part of it had a higher frequency of musculoskeletal symptoms. There were no differences in hospitalizations or deaths across the intervention. However shortly
2:00:23
Well, probably I think it was two years, roughly, prior to the end of the planned follow up, the DSMB of the DPP decided that the evidence was so strong that the two interventions were superior to the placebo in preventing the transition from prediabetes to diabetes, that they also recommended that the trial be stopped early not because there was harm being done, but because the benefits were there and it was not likely they were going to diminish,
2:00:55
and that it would be more important to get this information into the hands of the clinician and let them start using this in treating people with prediabetes to prevent the progression to full diabetes So the trial was stopped early. So two scenarios where randomized controlled trials, both primary prevention, both efficacy, were stopped early for different reasons. And that's the nature of trials. Now not all trials get stopped early.
2:01:22
A lot of trials see the entire course of their planned follow up and proceed from there.
2:01:28
I'm going to stop here. I was going to talk a little bit about some of the nuanced statistical data that come from a randomized trial, but because Greg and Austin and Jeff will be doing that in more detail, I think I'll stop here. Obviously, I'm at the time point but I won't go any further. Let's open it up for questions, please. Okay, well, I appreciate your thank yous. We, Greg, Catherine and all of us that are affiliated with the CTSI, the BERD,
2:02:04
and then the new office clinical trials research that Greg's leading, We thank you for joining us. It's really rewarding to do these workshops when there's somebody there listening. And if you don't have questions right now, or if you're running up on time and have to leave, please feel free to shoot an email to myself or Greg. I just want to go to this last slide. If you go to the Clinical Translational Science Institute website, you will find us, the BERD, right here.
2:02:37
The little icon is a graph with a normal distribution on it. And feel free to use the electronic way to get ahold of us Or you can email Greg directly as the director of the BERD and also a senior biostatistician in our School of Public Health. Thank you.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat