Skip to content

OT Futility Analysis

5 messages · Spencer Graves, Stephen A Roberts, Kevin E. Thorpe +1 more

Original
Kevin E. Thorpe
# 
I beg your pardon if this is too off topic.  I am posting here
since I hope to find an R solution to my problem.  Please indulge
me while I give a little background about what I'm trying to do.

I'm on a DSMB for a clinical trial.  The Steering Committee for the
trial has asked us to perform a futility analysis on their primary
outcome which is a time-to-event endpoint.  The trial was not designed
with group sequential methods, nor was any futility analysis spelled
out in the protocol.  Another thing which may be relevant is that
due to circumstances beyond the investigators' control, the trial
will stop recruitment prematurely unless there is some compelling
reason for them to find a way to continue the trial.  Lastly, the
trial has accrued not quite half of the planned sample size.

Admittedly, I don't have a vast amount of experience implementing
stopping rules.  In other protocols I have seen where futility
analyses have been planned but a group sequential design has not
otherwise been employed, conditional power has been used for the
futility rule.  So naturally, that was my first thought (although
I may well be wrong) in this case.  I have done RSiteSearch() with
the following terms (three different searches):

	futility analysis
	conditional power
	stochastic curtailment

Nothing that looked relevant to my problem jumped out at me.

I have read, somewhat recently, that there are problems with conditional
power, although I don't remember the details at the moment.  This
has prompted me to consider other approaches to the problem.

One simple thing that has occurred to me, although I don't know
what the implications are is to simply look at a confidence
interval around the hazard ratio for the treatment effect.  In
the event that the CI includes 1 and excludes any clinically
important difference, I would take that as an indication of
futility.

I would appreciate your comments on this and to learn of any more
formal methods, particularly of implementations in R.

Thank you for reading.

Kevin
4 days later
Spencer Graves
# 
What does this particular Steering Committee think a "futility 
analysis" is?  Do they have any particular reference(s)?  What do you 
find in your own literature review?

	  If it were my problem, I think I'd start with questions like that. 
Your comments suggested to me a confounding of technical and political 
problems.  The politics suggests the language you need to use in your 
response.  Beyond that, I've never heard before of a "futility 
analysis", but I think I could do one by just trying to be clear about 
the options the Steering Committee might consider plausible and then 
comparing them with appropriate simulations -- summarized as confidence 
intervals, as you suggest.

	  And I hope that someone else will enlighten us both if there are 
better options available.

	  Best Wishes,
	  spencer graves
p.s.  For any attorneys who may read these comments, the suggestions are 
obviously warranteed up to the amount you paid for it, which is nothing. 
  If you follow them and they turn out to be inappropriate, you will pay 
the price.  I encourage you to share the problems with me, so I can 
learn from the experience.  However, the limits of my liability are as 
already stated.
Kevin E. Thorpe wrote:

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Stephen A Roberts
    

    

      
      #
    

  


  

      
I would take the line that if they hadn't pre-specified any stopping rules, the only reason to stop is safety or new external data. I would be very suspicious of requests from the steering committee to stop for futility - they should be blinded so why are they thinking futility unless results have leaked? I would argue that they are obliged to finish the trial once they start.

This is an example of the need to sort out these things in advance - look up the stuff from the UK DAMOCLES project. The recent book edited by DeMets et al (Data Monitoring in Clinical Trials: A Case Studies Approach) is a good read on these sorts of issues and I think there is a more statistical book from the same group of authors.

 As far as software is concerned, futility analysis and conditional power are simply standard analyses with made up data and more-or-less justifiable assumptions.

Steve.

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Kevin E. Thorpe
    

    

      
      #
    

  


  

      
Thank you Spencer and Steve for your helpful comments.  If I may, I
would like to elaborate on some of the points you raise.

        
Stephen A Roberts wrote:

            

      
      
      
    

        
In general I agree with this.  In this case the request for a futility
analysis came from the sponsor (a drug company).  It is a classic case
of company B buys company A and wnats to stop R&D on company A's drugs.
Therefore the company was looking for a reason to stop.  Now that they
will stop producing the drug used in the trial, recruitment will end
before reaching its target.  Now the Steering Committee's point of
view is that if there is any reasonable hope, they would find some
other way to continue recruitment.  I am confident that results have
not leaked.  I am well aquainted with the data management and blinding
procedures in place for the trial.

            

      
      
      
    

        
Thanks for the reference.  My library has it, so will give it a look.

            

      
      
      
    

        
I am also interested if there are good alternatives to conditional
power for this type of scenario.

            

      
      
      
    

        
I did ask REPEATEDLY for guidelines from the steering committee, but
none came or are likely to come.  In fact, they wanted me to come up
with the recommendation, which I find entirely inappropriate, but here
I am.  So, I don't think I'm confounded between techincal and political.

Basically, they want to stop if there is a low chance of rejecting the
null hypothesis.  This is often referred to as conditional power or
stochastic curtailment.  I recently saw a paper by Scott Emerson
pointing out some problems (interpretation, relation to unconditional
power).

As far as references, I have used a book by Jennisen and Turnbull in
the past, but, as I recall, with the exception of stochastic
curtailment, it assumes the trial was designed with group sequential
methods.  I have also just found a 1988 Biometrics paper by Lan and
Wittes on the B-value which I will read.

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Marc Schwartz (via MN)
    

    

      
      #
    

  


  

      
On Wed, 2006-02-22 at 11:57 -0500, Kevin E. Thorpe wrote:

            

      
      
      
    

        
Kevin,

I am not sure if you received any offlist replies to your post. Given
the subject matter, I had considered that you might have.

You might find the following thread over in the MedStats group to be of
interest:

http://groups.google.com/group/MedStats/browse_frm/thread/144c97dc5cfc4f00?tvc=1

It discusses some of the issues of early stopping, in this particular
case due to "running out of funds". Some of the points raised below are
addressed in that thread.

MedStats, BTW, would be a good forum to consider for your query.

            

      
      
      
    

        
Has the decision to cease production already been made or is the sponsor
still open to being "sold" on the idea of keeping the study going,
pending the outcome of your further work?

If production of the study treatment has already ceased, the ability of
the SC to make a business case to the sponsor may be a forgone
conclusion if there is insufficient product available to continue.

            

      
      
      
    

        
should not provide you with any guidelines and the analysis should be
done independent of their input, since their input may be biased in
favor of the new drug. As I note below, the mere fact that the SC is
arguing in favor of continuing the study would suggest the possibility
of a priori bias. This is critical to consider, since there are no
pre-specified stopping rules.

In addition, these should not be done by you in isolation either and the
other clinical members of the DMC/DSMB should materially contribute to
the process. They should be just as clinically competent as any members
of the SC relative to putting forth reasonable assumptions upon which to
base any analyses.

            

      
      
      
    

        
J&T has a case study with survival analysis as I recall. I don't have it
at hand at the moment.

Another paper that you might find helpful is:

Modifying The Design of Ongoing Trials Without Unblinding
Gould and Shih
Statist. Med., 17, 89 ? 100 (1998)

I happened to locate a copy here during a Google search:

http://www2.umdnj.edu/~shihwj/papers/Gould%20and%20Shih%2098.pdf


There is also a Powerpoint presentation by Hung et al that you might
find of interest here:

http://www.fda.gov/cder/Offices/Biostatistics/Hung_etal_6/Hung_etal_6.PPT


BTW, though timing does not help you, there is a new book in process by
Proschan, Lan and Wittes due in June here:

http://www.springer.com/sgw/cda/frontpage/0,11855,4-10134-22-96964889-0,00.html


Two other documents that would be of value here, in a general guideline
sense, are the FDA's draft guidance on DMC's for sponsors (yes, I noted
you are "north of the border"):

http://www.fda.gov/cber/gdlns/clindatmon.htm

and of course ICH E9:

http://www.ich.org/cache/compo/475-272-1.html#E9

Both of which touch on the areas of interim analyses and early stopping.


It seems to me there there is a tension between the sponsor and the SC. 

In the former case, the sponsor is looking to make a business decision,
based upon what will clearly be less than perfect data, where
"scientific integrity" is or may be compromised. 

On the other hand, the SC is looking to argue for keeping the study
going and that presumably is based upon some a priori insight into the
likelihood of a favorable outcome for the new drug.

If there is to be a mid-course correction in the study (even though
there were no pre-specified stopping rules), there is a risk that the
mere decision to continue the study will somehow bias the future conduct
of it, given the present dynamics and "who knows what" about the
decision making process.

Some of the questions to be considered are what assumptions do you make
in the course of your assessment relative to future data and the need
for adjustments, if any, to the primary hypotheses of the study
(including alpha levels) given the knowledge that becomes available
during the un-planned interim analysis.

At the end of the day, you will need to consider the requirement to make
adjustments in the current study protocol, requiring re-submission and
re-approval by the requisite regulatory authority. It therefore would be
worthwhile to consider proactive communications with the regulatory
contacts for the study and discuss this situation with them to get their
"buy in" on any proposed approaches before taking any further steps.

<snip other content>

HTH,

Marc Schwartz