Research Synthesis: Health Technology Assessment

V1.0 researched and written by Elise Erickson, edited by Suerie Moon, last updated June 2018



The literature surrounding health technology assessment (HTA) is rich*. One challenge is that there is no set terminology or clear borders when referring to the issue of how to assess the value of a medicine, and its relationship to price. "HTA" is used in some of the literature, whereas others use "pharmacoeconomics" or "cost-effectiveness analysis," and others "value-based pricing." Adding to this challenge, it is a highly technical topic, which means that a significant amount of the literature may be less accessible to non-specialists.

Search terms 


Medicine/drug/pharmaceutical and health technology assessment, cost-effective analysis, pharmacoeconomics, value-based pricing.

Desk with Stethoscope

Contribute to improve and update this page!

We particularly welcome suggestions on gaps in the reviews and on interesting new research.

Synthesis of the literature


In general, the literature notes the increasing use of HTA in health decision-making as countries continue to spend more on health care, and usually promotes the use of HTA, while also recognizing and critiquing its limitations. Towse and Barnsley (2013) provide an approachable, general overview of the topic, and Henshall and Schuller (2013) provide an overview of the different ways to assess the value of a medicine (see in particular ‘Table 1.Summary of Core and Wider Elements of Value, and Approaches to Measurement and Valuation’). Sorensen, Drummond and Kanavos’ (2008) report "Ensuring Value for Money in Health Care: The Role of HTA in the EU" also provides a useful overview of the surrounding debate.

The WHO’s Guideline on Country Pharmaceutical Pricing Policies (2015), for instance, recommends using HTA as a tool for reimbursement decision-making, and for price setting and negotiation, but recognizes that HTA requires advanced technical capacity and notes the need for accompanying legislative/administrative frameworks, and transparency throughout the process. For a detailed list of studies carried out concerning HTA, see Annex J of the WHO Guidelines on Country Pharmaceutical Pricing Policies. Similarly, a report by Sorensen, Drummond, and Kanavos  (2008), which focuses on HTA in Europe, argues that HTA plays an important role in evidence-based decision making, but notes several weak points including the disparate roles of different HTA bodies, the need for more stakeholder involvement in the HTA process (i.e. greater involvement of patients and industry), and lack of transparency in the HTA process (2008).

Much of the discussion focuses on how best to calculate value (i.e. which factors should and should not be included), how to operationalize value-based pricing (VBP), and how to use it in health decision-making (World Health Organization (WHO) 2015Sussex, Towse, and Devlin 2013). Most studies look at HTA as an independent tool, rather than comparing it to other pricing tools. One study, however, compares HTA with reference pricing and found that HTA is superior (in that it is more nuanced), but it also noted the high cost of carrying out HTAs (Drummond et al. 2010).


Most of the studies focus on high income countries (Drummond et al. 2010). There are country-specific papers for the UK (NICE), Canada, Australia, Sweden, US, and Germany, among others; and some papers that compare processes across countries (Henry, Hill, and Harris 2005). There exist a handful of studies analyzing HTA agencies in particular, including in Europe, the UK, Australia, and many in Canada. While some studies introduce and describe the various agencies (e.g. Canada (Lefebvre, Lafeuille, and Tiggelaar 2017; Menon and Stafinski 2009; Paris and Belloni 2014), and Australia (Hailey 2009)), others examine more specific aspects of the agencies. In Canada, the Common Drug Review (CDR) was introduced in 2003 in place of the 18 separate review processes that previously existed (“CADTH Common Drug Review (CDR),” n.d.).


When examining the extent to which the listing decisions align with HTA body recommendations, Allen et al. (2016) found “moderate to substantial agreement” between the provincial listing decisions and the CDR. Comparing the quantity drug listings before and after the establishment of Canada’s CDR, Gamble et al. (2011), find that fewer drugs were listed after the agency’s creation, and that the time-to-listing fell in certain smaller Canadian provinces. A study commissioned by the Canadian Agency for Drugs and Technologies in Health examined how patient perspectives were assimilated into Common Drug Review (CDR) assessment reports and recommendations, finding that patient views were used to situate assessments and to interpret the evidence (Berglas et al. 2016). The study also stresses the importance of taking both the recovery process and health sustainability into account when assessing a drug. McCormick, Berescu, and Tadros (2018) examined recommendations for orphan drugs in Canada specifically, finding that positive recommendations have increased over time, usually conditional on a drop in price. In the UK context, a study by Dakin et al. (2015) suggests that NICE decision-making and thresholds for rejection had not changed significantly over time. When examining the extent to which NICE guidance is implemented, Sheldon et al. (2004) find that implementation varied depending on several factors including strong professional support, a clear evidence base, and no increase in cost. Looking at Sweden, a study of the Pharmaceutical Benefits Board (LFN)’s priority-setting procedure suggests that an accountable and reasonable procedure can bring about priority-setting that is “generally perceived as fair and legitimate by the major stakeholders and may increase social learning in terms of accepting the necessity of priority setting in health care” (Jansson 2007).


There are significantly fewer analyses of HTA in LMICs. A study of middle-income countries use of HTA found that their use of HTA is increasing, but developing at an uneven speed (Oortwijn, Mathijssen, and Banta 2010). Sivalal (2009) describe how Malaysia’s HTA agency has increased in terms of both size and resources since its creation in 1995, but note major challenges with regards to sustainability, including: a sufficient level of trained staff and appropriate awareness around the function and value of the agency. Previously, Sivalal (1998) describe an HTA training course in Malaysia shortly after its agency was created. The authors suggest this could be a useful resource for others designing HTA training courses in developing countries. Teerawattananon et al. (2009) describe the various factors that contributed to the development of HTA in Thailand, including studies on the poor distribution of health technologies, followed by an economic recession and the implementation of a universal health coverage plan.


Other studies compare HTA agencies across countries. For instance, a study comparing the value-added assessments (those that go beyond the HTA economic or clinical benefit assessment) between European countries found a number of similarities and differences with regards to the practices of value-assessment, and argued for greater transparency in the value-assessment criteria to enhance resource allocation and therefore societal welfare (Angelis, Lange, and Kanavos 2018). In fact, concerns over transparency in these processes was a reoccurring theme in the literature (Hailey 2009).  Clement et al. (2009) compare how effectiveness and cost-effectiveness are used in Canada’s CDR, the UK’s NICE, and Australia’s PBAC. They conclude that the listing decisions varied between the agencies—which is unsurprising considering that the agencies have different mandates and processes. In fact, the authors suggest that the differences in decisions are mostly driven by differences in processes and risk attitudes, rather than the interpretation of clinical or economic evidence. McMahon, Morgan, and Mitton (2006) highlighted various considerations for Canada’s CDR (just after CDR was created) based on lessons learned from the UK’s NICE.

Research gaps

  • More studies analyzing use or feasibility of HTA in low- or middle-income countries

  • Analyses of HTA as an input to medicines pricing and incentive for innovation​

Cited papers with abstract​

Henshall, Chris, and Tara Schuller. 2013. “HEALTH TECHNOLOGY ASSESSMENT, VALUE-BASED DECISION MAKING, AND INNOVATION.” International Journal of Technology Assessment in Health Care 29 (4): 353–59.

Abstract: Background: Identifying treatments that offer value and value for money is becoming increasingly important, with interest in how health technology assessment (HTA) and decision makers can take appropriate account of what is of value to patients and to society, and in the relationship between innovation and assessments of value.

Methods: This study summarizes points from an Health Technology Assessment International (HTAi) Policy Forum discussion, drawing on presentations, discussions among attendees, and background papers.

Results and Conclusions: Various perspectives on value were considered; most place patient health at the core of value. Wider elements of value comprise other benefits for: patients; caregivers; the health and social care systems; and society. Most decision-making systems seek to take account of similar elements of value, although they are assessed and combined in different ways. Judgment in decisions remains important and cannot be replaced by mathematical approaches. There was discussion of the value of innovation and of the effects of value assessments on innovation. Discussion also included moving toward “progressive health system decision making,” an ongoing process whereby evidence-based decisions on use would be made at various stages in the technology lifecycle. Five actions are identified: (i) development of a general framework for the definition and assessment of value; development by HTA/coverage bodies and regulators of (ii) disease-specific guidance and (iii) further joint scientific advice for industry on demonstrating value; (iv) development of a framework for progressive licensing, usage, and reimbursement; and (v) promoting work to better adapt HTA, coverage, and procurement approaches to medical devices.


Oortwijn, Wija, Judith Mathijssen, and David Banta. 2010. “The Role of Health Technology Assessment on Pharmaceutical Reimbursement in Selected Middle-Income Countries.” Health Policy 95 (2): 174–84.

Abstract: Objective: Middle-income countries are often referred to as developing or emerging economies and face multiple challenges of severe financial stresses in their health care sectors, and high disease burden. The objective of this study is to provide an overview of how health technology assessment (HTA) is used and organized in selected middle-income countries and its role in the process of pharmaceutical coverage.

Methods: We selected middle-income countries where HTA activities are evident: Argentina, Brazil, China, Colombia, Israel, Mexico, Philippines, Korea, Taiwan, Thailand, and Turkey. We collected and reviewed relevant information to describe the health care and reimbursement systems and how HTA relates to coverage decision-making of pharmaceuticals. This was supplemented by information from a structured survey among professionals working in public and private health insurance, industry, regulatory authorities, ministries of health, academic units or HTA.

Results: All countries require market authorization for pharmaceuticals to be sold and most countries have a national plan defining which pharmaceuticals can be reimbursed. However, the use of HTA in reimbursement decisions is still in its early stages with varying levels of HTA guidance implementation.

Conclusions: The study provides evidence of the development of HTA in coverage decision-making in middle-income countries. Increased health care spending and the resulting access to modern technology give a strong impetus to HTA. However, HTA is developing with uneven speed in middle-income countries and many countries are building on the organisational and methodological experience from established HTA agencies.


Sorenson, Corinna, Michael Drummond, and Panos Kanavos. 2008. Ensuring Value for Money in Health Care: The Role of Health Technology Assessment in the European Union. WHO Regional Office Europe.

Abstract: Not available


Sussex, Jon, Adrian Towse, and Nancy Devlin. 2013. “Operationalizing Value-Based Pricing of Medicines.” PharmacoEconomics 31 (1): 1–10.

Abstract: The UK Government is proposing a novel form of price regulation for branded medicines, which it has dubbed ‘value-based pricing’ (VBP). The specifics of how VBP will work are unclear. We provide an account of the possible means by which VBP of medicines might be operationalized, and a taxonomy to describe and categorize the various approaches. We begin with a brief discussion of the UK Government’s proposal for VBP and proceed to define a taxonomy of approaches to VBP. The taxonomy has five main dimensions: (1) what is identified as being of value, (2) how each element is measured, (3) how it is valued, (4) how the different elements of value are aggregated, and (5) how the result is then used to determine the price of a medicine. We take as our starting point that VBP will include a measure of health gain and that, as proposed by the UK Government, this will be built on the QALY. Our principal interest is in the way criteria other than QALYs are taken into account, including severity of illness, the extent of unmet need, and wider societal considerations such as impacts on carers. We set out to: (1) identify and describe the full range of alternative means by which ‘value’ might be measured and valued, (2) identify and describe the options available for aggregating the different components of value to establish a maximum price, and (3) note the challenges and relative advantages associated with these approaches. We review the means by which aspects of VBP are currently operationalized in a selection of countries and place these, and proposals for the UK, in the context of our taxonomy. Finally, we give an initial assessment of the challenges, pros and cons of each approach. We conclude that identifying where VBP should lie on each of the five dimensions entails value judgements: there are no simple ‘right or wrong’ solutions. If a wider definition of value than incremental QALYs gained is adopted, as is desirable, then a pragmatic way to aggregate the different elements of value, including both QALYs and benefits unrelated to QALYs, is to use a multi-criteria decision analysis (MCDA) approach. All approaches to VBP ultimately require the conversion of value, however assessed, into a monetary price. This requires assessment of the marginal values of all types of benefit, not just of QALYs. All stages of the VBP process are subject to uncertainty and margins of error. Consequently, the assessment of overall value can provide bounds to a price negotiation but cannot be expected to identify a precise value-based price.


Towse, Adrian, and Paul Barnsley. 2013. “Approaches to Identifying, Measuring, and Aggregating Elements of Value.” International Journal of Technology Assessment in Health Care 29 (4): 360–64.

Abstract: Background: Two general alternative approaches, cost-effectiveness analysis and the therapeutic added value approach, link the pricing and approval of drugs to value. Value as assessed by payers is a function of: benefit less cost, willingness to pay for benefit, and how they handle uncertainty.

Methods: This study uses international examples to explore the elements of value that can be included in the assessment of health technologies, approaches to scoring the elements of value and how they can be combined to make a decision.

Results: A range of value elements, measures, and approaches to aggregation are identified across different HTA systems. We show that seemingly arbitrary differences in measurement and aggregation can lead to significantly different outcomes, and argue that the choice of values, measures, and decision-making processes should be informed by the societal values that underpin a health system.

Conclusions: We identify three areas for further research to improve both health system and industry R&D decision making: (i) whether more consistency could be achieved across health systems on the elements of value that matter; (ii) the relative merits of discrete versus continuous measures of value; and (iii) how structured decision making (to aggregate the elements of value) could or should become.


World Health Organization (WHO). 2015. “WHO Guideline on Country Pharmaceutical Pricing Policies.”

Abstract: Not available


Drummond, M., B. Jonsson, F. Rutten, and T. Stargardt. 2010. “Reimbursement of Pharmaceuticals: Reference Pricing versus Health Technology Assessment.” In Eur J Health Econ.

Abstract: Reference pricing and health technology assessment are policies commonly applied in order to obtain more value for money from pharmaceuticals. This study focussed on decisions about the initial price and reimbursement status of innovative drugs and discussed the consequences for market access and cost. Four countries were studied: Germany, The Netherlands, Sweden and the United Kingdom. These countries have operated one, or both, of the two policies at certain points in time, sometimes in parallel. Drugs in four groups were considered: cholesterol-lowering agents, insulin analogues, biologic drugs for rheumatoid arthritis and "atypical" drugs for schizophrenia. Compared with HTA, reference pricing is a relatively blunt instrument for obtaining value for money from pharmaceuticals. Thus, its role in making reimbursement decisions should be limited to drugs which are therapeutically equivalent. HTA is a superior strategy for obtaining value for money because it addresses not only price but also the appropriate indications for the use of the drug and the relation between additional value and additional costs. However, given the relatively higher costs of conducting HTAs, the most efficient approach might be a combination of both policies.


Henry, David A., Suzanne R. Hill, and Anthony Harris. 2005. “Drug Prices and Value for Money: The Australian Pharmaceutical Benefits Scheme.” JAMA 294 (20): 2630.

Abstract: Not Available


Lefebvre, Patrick, Marie-Hélène Lafeuille, and Sean Tiggelaar. 2017. “Perspectives on the Common Drug Review Process at the Canadian Agency for Drugs and Technologies in Health.” In Decision Making in a World of Comparative Effectiveness Research, 167–79. Adis, Singapore.

Abstract: The Common Drug Review (CDR) is a federal review process that provides funding and adoption recommendations to Canadian provinces and territories on non-oncological drugs. This chapter will begin with providing an introduction to the Canadian Agency for Drugs and Technologies in Health (CADTH) and its role within the Canadian health-care system and will then describe and provide a commentary on the intricacies of the CDR process. The pathway of the CDR process is then outlined, from manufacturer submission, to the formation and evaluation of that submission by a review team, to the dissemination and publication of final recommendations from a pan-Canadian Drug Expert Committee. In addition to the CDR process pathway, details on key factors considered and desired in HTA submissions are outlined (large disease burden or an unmet need), as well as the recommended methodology manufacturers should consider when conducting clinical trials and cost-effectiveness models. This chapter then discusses CADTH’s performance, as reviewed by other organizations against fellow international HTA agencies. Based on the discussed strengths and limitations, the chapter concludes with providing future direction, encouraging CADTH’s continued focus on improved transparency and responsiveness while also urging them to conduct continued reviews (past the adoption milestone) that manage obsolescence and facilitate evidence translation.


Menon, Devidas, and Tania Stafinski. 2009. “Health Technology Assessment in Canada: 20 Years Strong?” Value in Health 12 (June): S14–19.

Abstract: Not available


Paris, Valerie, and Annalisa Belloni. 2014. “Value in Pharmaceutical Pricing - Country Profile: Canada.” OECD.

Abstract: Not available


Hailey, David. 2009. “The History of Health Technology Assessment in Australia.” International Journal of Technology Assessment in Health Care 25 (S1): 61–67.

Abstract: Objectives: To describe the development and application of health technology assessment (HTA) in Australia.

Methods: Review of relevant literature and other documents related to HTA in Australia. Results: Most HTA activity in Australia has been associated with provision of advice for the two national subsidy programs, Medicare, and the Pharmaceutical Benefits Scheme (PBS). National advisory bodies established by the federal government have had a prominent role. Assessments from the advisory bodies have had a major influence on decisions related to Medicare and the PBS, and in some other areas. Technologies without links to the national subsidy schemes, and those that are widely distributed, have been less well covered by HTA. To some extent these are addressed by evaluations supported by state governments, but details of approaches taken are not readily available.

Conclusions: HTA in Australia now has a long history and is well established as a source of advice to health decision makers. Challenges remain in extending the scope of assessments, developing more transparent approaches in some areas, and consistently applying appropriate standards.


“CADTH Common Drug Review (CDR).” n.d.

Abstract: Not available


Allen, Nicola, Stuart R. Walker, Lawrence Liberti, Chander Sehgal, and M. Sam Salek. 2016. “Evaluating Alignment between Canadian Common Drug Review Reimbursement Recommendations and Provincial Drug Plan Listing Decisions: An Exploratory Study.” CMAJ Open 4 (4): E674–78.

Abstract: Background: The CADTH Common Drug Review was established in 2002 to prepare national health technology assessment reports to guide listing decisions for 18 participating drug plans. The aim of this study was to compare the nonmandatory recommendations from the Common Drug Review in Canada with the listing decisions of provincial payers to determine alignment.

Methods: We identified the recommendations issued by the Common Drug Review from Jan. 1, 2009, to Jan. 1, 2015, and compared these with the listing decisions of 3 provincial public payers (Alberta, British Columbia and Ontario) that participate in the Common Drug Review and the recommendations from Quebec.

Results: We identified 174 medicine-indication pairs in CADTH Common Drug Review reports issued from Jan. 1, 2009, to Jan. 1, 2015; 110 of these met the inclusion criterion. Among the 110 medicine-indication pairs, listing decisions were available for 95 in Alberta, 102 in Quebec, 104 in Ontario and 106 in BC. There was moderate to substantial agreement between provincial listing decisions and Common Drug Review recommendations: 74.5% (κ = 0.47, 95% confidence interval [CI] 0.31-0.64) for Quebec, 78.8% (κ = 0.56, 95% CI 0.41-0.72) for Ontario, 78.9% (κ = 0.58, 95% CI 0.42-0.74) for Alberta and 81.1% (κ = 0.62, 95% CI 0.47-0.77) for BC.

Interpretation: Our study showed moderate to substantial agreement between Common Drug Review recommendations and provincial listing decisions. Future studies can build on this research by evaluating the concordance between Common Drug Review recommendations and listing decisions of all participating federal, provincial and territorial drug plans.


Gamble, John-Michael, Daniala L. Weir, Jeffrey A. Johnson, and Dean T. Eurich. 2011. “Analysis of Drug Coverage before and after the Implementation of Canada’s Common Drug Review.” CMAJ : Canadian Medical Association Journal 183 (17): E1259–66.

Abstract: Background: Canada’s Common Drug Review was implemented to provide publicly funded drug plans (provincial and federal) with a transparent, rigorous and consistent approach for assessing the clinical effectiveness and cost-effectiveness of new drugs. We compared uptake of drug coverage across jurisdictions before and after the implementation of the Common Drug Review.

Methods: Using the IMS Brogan formulary acceptance: monitoring and evaluation database, we identified new drug products in Canada five years before and five years after the first recommendation was made by the Common Drug Review. For each jurisdiction, we compared the proportion of drugs listed, the median time-to-listing and the agreement between the listing decisions of the drug plans and the recommendations of the Common Drug Review.

Results: We identified 198 new drugs approved for use in Canada between May 26, 1999, and May 27, 2009, of which 53 had a recommendation from the Common Drug Review. The proportion of drugs listed decreased after the introduction of the Common Drug Review for all participating drug plans (81.1% to 71.3% overall [p ≤ 0.01 for all plans, with the exceptions of Ontario and Quebec [p = 0.07]). The change in median time-to-listing between the periods before and after the Common Drug Review varied by jurisdiction, ranging from a decrease of 691 days to an increase of 250 days. The change in median time-to-listing was not statistically significant for most jurisdictions, with the exceptions of Saskatchewan (increased, Mann–Whitney U test p = 0.01) and New Brunswick, Prince Edward Island, and Newfoundland and Labrador (all decreased, Mann–Whitney U test p < 0.01).

Interpretation: There was a decline in the proportion of new drugs listed after the introduction of the Common Drug Review for both participating and nonparticipating jurisdictions. The introduction of the review was associated with a decreased time-to-listing for certain smaller provinces.

Conclusion: There was a decline in the proportion of new drugs listed after the introduction of the Common Drug Review, both for participating and nonparticipating jurisdictions. Our findings suggest that the Common Drug Review may have contributed to a streamlining of the process for reviewing drugs for certain jurisdictions. Specifically, patients in the provinces of New Brunswick, Prince Edward Island, and Newfoundland and Labrador may have benefited with earlier access to new drugs. Any substantial gains in savings or in the efficiency of publicly funded drugs plans to make listing decisions are important factors in maintaining the health and safety of Canadian patients. Future research evaluating the time-to-decision for both positive and negative listings would be an important outcome to measure from the perspective of the public.


Berglas, Sarah, Lauren Jutai, Gail MacKean, and Laura Weeks. 2016. “Patients’ Perspectives Can Be Integrated in Health Technology Assessments: An Exploratory Analysis of CADTH Common Drug Review.” Research Involvement and Engagement 2 (June): 21.

Abstract: Background: Since 2010, Canadian patient groups have contributed to the CADTH Common Drug Review (CDR). CADTH conducts health technology assessments of new drugs to support publicly funded drug plans’ reimbursement decisions. We explored whether, and how, patient insights were integrated into assessment reports and Recommendations by the CADTH Canadian Drug Expert Committee (CDEC).

Methods: We descriptively analyzed 30 consecutive assessments. One researcher identified a set of issues, insights, and desired treatment outcomes provided by patient groups for each included drug assessment. We tracked the presence of each identified patient insight in the relevant assessment protocol, in clinical trials as reported in the assessment, and in the CDEC Recommendations. Additionally, patient insights were categorized by topic and grouped into a three-tier framework to explore the observed juxtaposition between immediate treatment outcomes as seen in clinical trials and the insights from patients living with a chronic condition.

Results: In 30 drug assessments, 119 patient insights were identified. Of these insights, 89 were included in assessment protocols; 61 in reported clinical trial data; and 67 insights were reflected upon within the CDEC Recommendations. Patient insights within the first framework tier (health status achieved) were frequently included in all aspects of CDR assessments. Within the second tier (progress of recovery), although two-thirds of patient insights were included in protocols, only one-third was reflected in reported trial data or in CDEC Recommendations. Insights within the third tier, which address the long-term consequences of illness and treatment, were even less frequently addressed in all aspects of CDR assessments.

Conclusions: Patients’ perspectives need not be “considered” in isolation. Patient insights are used by CADTH reviewers to frame an assessment and used by CDEC to interpret the evidence. As health technology assessments should address the indirect and unintended consequences of a technology, as well as its direct and intended effects, drug assessments should consider the progress of recovery and sustainability of health, in addition to survival and immediate health achieved.


McCormick, John I., L. Diana Berescu, and Nabil Tadros. 2018. “Common Drug Review Recommendations for Orphan Drugs in Canada: Basis of Recommendations and Comparison with Similar Reviews in Quebec, Australia, Scotland and New Zealand.” Orphanet Journal of Rare Diseases 13 (January): 27.

Abstract: Background: Public payer reimbursement for non-oncology drugs in Canada, including orphan drugs, is based on recommendations by the Common Drug Review (CDR) (with the exception of Quebec). CDR has been criticized for negative recommendations for orphan drugs and contributing to delays in patient access to these drugs. However, it is unclear how CDR makes recommendations for orphan drugs and the role clinical and economic factors play in decision making. The objective of the present study was to analyze the basis for CDR orphan drug recommendations and to compare recommendations to those in other jurisdictions.

Methods: A list of orphan drugs reviewed by CDR (between 2004 and 2017) was compiled and final recommendations (list/do not list) assessed. The basis of each recommendation was categorized as clinical only, price only or combined clinical and price factors, based on the ranking of clinical and price parameters in recommendation summaries. The reimbursement status of the same drugs was determined in Quebec and other jurisdictions and level of agreement with CDR decisions assessed using a kappa analysis.

Results: Sixty eight orphan drug submissions were identified in the CDR database. Clinical, clinical and price and price parameters were the basis of 48.5%, 44.1% and 7.4% of the reviews, respectively, and corresponding positive recommendation rates were 45.5%, 86.7% and 40.0% (p = 0.0008); overall positive recommendation rate was 63.2%. Positive recommendation rate increased from 50.0% for drugs reviewed between 2004 and 2009 to 86.7% in 2016; however, 84.6% of the latter were conditional on a price reduction. Of the drugs reviewed by CDR, 80.9%, 88.2%, 80.9% and 58.8% were reviewed for the same indications by health technology assessment agencies in Quebec, Scotland, Australia and New Zealand, respectively, with positive listing rates ranging from 60.0% (Quebec) to 92.7% (Australia) with fair (kappa coefficient 0.3307) to poor (kappa coefficient 0.0611) agreement with CDR in listing decisions, respectively.

Conclusions: The positive CDR recommendation rate for orphan drugs was highest when clinical and price parameters supported the assessment. Over time there has been an increase in CDR positive recommendation rates for orphan drugs, although most are conditional on a price reduction. It is unclear if this change in CDR recommendations will impact equitable and timely access to orphan drugs across Canada.


Dakin, Helen, Nancy Devlin, Yan Feng, Nigel Rice, Phill O’Neill, and David Parkin. 2015. “The Influence of Cost-Effectiveness and Other Factors on Nice Decisions.” Health Economics 24 (10): 1256–71.

Abstract: The National Institute for Health and Care Excellence (NICE) emphasises that cost‐effectiveness is not the only consideration in health technology appraisal and is increasingly explicit about other factors considered relevant but not the weight attached to each.

The objective of this study is to investigate the influence of cost‐effectiveness and other factors on NICE decisions and whether NICE's decision‐making has changed over time.

We model NICE's decisions as binary choices for or against a health care technology in a specific patient group. Independent variables comprised of the following: clinical and economic evidence; characteristics of patients, disease or treatment; and contextual factors potentially affecting decision‐making. Data on all NICE decisions published by December 2011 were obtained from HTAinSite [].

Cost‐effectiveness alone correctly predicted 82% of decisions; few other variables were significant and alternative model specifications had similar performance. There was no evidence that the threshold has changed significantly over time. The model with highest prediction accuracy suggested that technologies costing £40 000 per quality‐adjusted life‐year (QALY) have a 50% chance of NICE rejection (75% at £52 000/QALY; 25% at £27 000/QALY).

Past NICE decisions appear to have been based on a higher threshold than £20 000–£30 000/QALY. However, this may reflect consideration of other factors that cannot be easily quantified.


Sheldon, Trevor A., Nicky Cullum, Diane Dawson, Annette Lankshear, Karin Lowson, Ian Watt, Peter West, Dianne Wright, and John Wright. 2004. “What’s the Evidence That NICE Guidance Has Been Implemented? Results from a National Evaluation Using Time Series Analysis, Audit of Patients’ Notes, and Interviews.” BMJ 329 (7473): 999.

Abstract: Objectives: To assess the extent and pattern of implementation of guidance issued by the National Institute for Clinical Excellence (NICE).

Design: Interrupted time series analysis, review of case notes, survey, and interviews.

Setting: Acute and primary care trusts in England and Wales.

Participants All primary care prescribing, hospital pharmacies; a random sample of 20 acute trusts, 17 mental health trusts, and 21 primary care trusts; and senior clinicians and managers from five acute trusts. Main outcome measures Rates of prescribing and use of procedures and medical devices relative to evidence based guidance.

Results: 6308 usable patient audit forms were returned. Implementation of NICE guidance varied by trust and by topic. Prescribing of some taxanes for cancer (P < 0.002) and orlistat for obesity (P < 0.001) significantly increased in line with guidance. Prescribing of drugs for Alzheimer's disease and prophylactic extraction of wisdom teeth showed trends consistent with, but not obviously a consequence of, the guidance. Prescribing practice often did not accord with the details of the guidance. No change was apparent in the use of hearing aids, hip prostheses, implantable cardioverter defibrillators, laparoscopic hernia repair, and laparoscopic colorectal cancer surgery after NICE guidance had been issued.

Conclusions: Implementation of NICE guidance has been variable. Guidance seems more likely to be adopted when there is strong professional support, a stable and convincing evidence base, and no increased or unfunded costs, in organisations that have established good systems for tracking guidance implementation and where the professionals involved are not isolated. Guidance needs to be clear and reflect the clinical context.


Jansson, Sandra. 2007. “Implementing Accountability for Reasonableness – the Case of Pharmaceutical Reimbursement in Sweden.” Health Economics, Policy and Law 2 (2): 153–71.

Abstract: This paper aims to describe the priority-setting procedure for new original pharmaceuticals practiced by the Swedish Pharmaceutical Benefits Board (LFN), to analyse the outcome of the procedure in terms of decisions and the relative importance of ethical principles, and to examine the reactions of stakeholders. All the ‘principally important’ decisions made by the LFN during its first 33 months of operation were analysed. The study is theoretically anchored in the theory of fair and legitimate priority-setting procedures by Daniels and Sabin, and is based on public documents, media articles, and semi-structured interviews. Only nine cases resulted in a rejection of a subsidy by the LFN and 15 in a limited or conditional subsidy. Total rejections rather than limitations gave rise to actions by stakeholders. Primarily, the principle of cost-effectiveness was used when limiting/conditioning or totally rejecting a subsidy. This study suggests that implementing a priority-setting process that fulfils the conditions of accountability for reasonableness can result in a priority-setting process which is generally perceived as fair and legitimate by the major stakeholders and may increase social learning in terms of accepting the necessity of priority setting in health care. The principle of cost-effectiveness increased in importance when the demand for openness and transparency increased.


Sivalal, Sadasivan, H. David Banta, Ellen F. M. ', T Hoen, and Jaudin Rusilawati. 1998. “A Training Course in Health Technology Assessment in Malaysia.” International Journal of Technology Assessment in Health Care 14 (4): 809–17.

Abstract: This article describes a 1-week training course in health technology assessment (HTA) presented in Malaysia by the Ministry of Health in 1996. Malaysia established an HTA unit in the Ministry of Health in 1995 and a National Health Technology Assessment Program in 1996. The purpose of the course was to develop HTA knowledge and skills in Malaysia, since these are largely lacking. The course consisted of didactic sessions and group work. Didactic sessions covered the principles of HTA. Group work was for the purpose of developing practical skills, and was based on reports from HTA agencies, published articles, and candidates for assessment suggested by course participants. Course participants were a mix of physicians, nurses, hospital administrators, and Ministry of Health officials. Experiences in this course may be helpful to others who wish to organize training courses in developing countries.


Sivalal, Sadasivan. 2009. “Health Technology Assessment in Malaysia.” International Journal of Technology Assessment in Health Care 25 (S1): 224–30.

Abstract: Objectives: Malaysia, as a rapidly developing country, has been facing tremendous pressures in its attempts to maximize scarce resources. Despite this problem, Malaysia has made great strides in developing its health services, and has successfully provided good access to the population to healthcare services, reduced the incidence of many communicable diseases, and improved life expectancies and other global indices of health care, some of which are comparable to that of developed countries.

Methods: The Health Technology Assessment (HTA) Unit was set up in Malaysia in August 1995 in the Ministry of Health Malaysia and has since grown tremendously in size and resources. To date, forty-three in-depth assessments have been carried out, and the recommendations of these assessments were subsequently implemented. In addition, approximately 140 rapid assessment reports were produced in response to requests from policy and decision makers. HTA has been able to provide input into formulation of national and Ministry of Health Malaysia policies, and provide a basis for clinical practice guidelines development, input into purchasing decisions, regulation of drugs, as well as advertisements related to health.

Results: A major challenge is sustainability of the program, to be able to have trained personnel competent to take on the demanding tasks of assessments and the sustained efforts that are required. In addition, there need to be constant efforts to create awareness of the utility of HTA so that its services are used and its full potential realized. The scope of services may also need to be expanded to include an early warning system.

Conclusions: Malaysia has successfully implemented a health technology program that has had major impact on policy formulation and decision making at various levels. Challenges may be faced in sustaining and developing the program further.


Teerawattananon, Yot, Sripen Tantivess, Jomkwan Yothasamut, Pritaporn Kingkaew, and Kakanang Chaisiri. 2009. “Historical Development of Health Technology Assessment in Thailand.” International Journal of Technology Assessment in Health Care 25 (S1): 241–52.

Abstract: Objectives: This study aims to review the development of health technology assessment (HTA), including the socioeconomic context, outputs, and policy utilization in the Thai setting.

Methods: This study was conducted through extensive document reviews including these published in both domestic and international literature.

Results: Evidence suggests that contextual elements of the health system, especially the country's economic status and health financing reforms, as well as their effects on government budgeting for medical and public health services, played an important role in the increasing needs and demands for HTA information among policy makers. In the midst of substantial economic growth during the years 1982 to 1996, several studies reported the rapid diffusion and poor distribution of health technologies, and inequitable access to high-cost technology in public and private hospitals. At the same time, economic analysis and its underpinning concept of efficiency were suggested by groups of scholars and health officials to guide national policy on the investment in health technology equipment. Related research and training programs were subsequently launched. However, none of these HTA units could be institutionalized into national bodies. From 1997 to 2005, an economic recession, followed by the introduction of a universal health coverage plan, triggered the demands for effective measures for cost containment and prioritization of health interventions. This made policy makers and researchers at the Ministry of Public Health (MOPH) pay increasing attention to economic appraisals, and several HTA programs were established in the Ministry. Despite the rising number of Thai health economic publications, a major problem at that period involved the poor quality of studies. Since 2006, economic recovery and demands from different interests to include expensive technologies in the public health benefit package have been crucial factors promoting the role of HTA in national policy decisions. Meanwhile, HTA capacity has been strengthened through the establishment of many health economic and HTA initiatives. An illustration of the work and contributions of the Health Intervention and Technology Assessment Program (HITAP) is provided. In this phase,

HTA policy integration has been enhanced through different mechanisms and organizations.

Conclusion: Over the past two decades a notable progression has been made in relation to the capacity building of HTA research and its policy utility in Thailand. Such development has been shaped by multiple factors. It is anticipated that experience gained among academics, health officials, and civil society organizations will be helpful not only in sustaining the momentum but also in improving formal HTA systems in the future.


Angelis, Aris, Ansgar Lange, and Panos Kanavos. 2018. “Using Health Technology Assessment to Assess the Value of New Medicines: Results of a Systematic Review and Expert Consultation across Eight European Countries.” The European Journal of Health Economics 19 (1): 123–52.

Abstract: Background: Although health technology assessment (HTA) systems base their decision making process either on economic evaluations or comparative clinical benefit assessment, a central aim of recent approaches to value measurement, including value based assessment and pricing, points towards the incorporation of supplementary evidence and criteria that capture additional dimensions of value.

Objective: To study the practices, processes and policies of value-assessment for new medicines across eight European countries and the role of HTA beyond economic evaluation and clinical benefit assessment. Methods A systematic (peer review and grey) literature review was conducted using an analytical framework examining: (1) ‘Responsibilities and structure of HTA agencies’; (2) ‘Evidence and evaluation criteria considered in HTAs’; (3) ‘Methods and techniques applied in HTAs’; and (4) ‘Outcomes and implementation of HTAs’. Study countries were France, Germany, England, Sweden, Italy, Netherlands, Poland and Spain. Evidence from the literature was validated and updated through two rounds of feedback involving primary data collection from national experts.

Results: All countries assess similar types of evidence; however, the specific criteria/endpoints used, their level of provision and requirement, and the way they are incorporated (e.g. explicitly vs. implicitly) varies across countries, with their relative importance remaining generally unknown. Incorporation of additional ‘social value judgements’ (beyond clinical benefit assessment) and economic evaluation could help explain heterogeneity in coverage recommendations and decision-making.

Conclusion: More comprehensive and systematic assessment procedures characterised by increased transparency, in terms of selection of evaluation criteria, their importance and intensity of use, could lead to more rational evidence-based decision-making, possibly improving efficiency in resource allocation, while also raising public confidence and fairness.


Clement, Fiona M., Anthony Harris, Jing Jing Li, Karen Yong, Karen M. Lee, and Braden J. Manns. 2009. “Using Effectiveness and Cost-Effectiveness to Make Drug Coverage Decisions: A Comparison of Britain, Australia, and Canada.” JAMA 302 (13): 1437–43.

Abstract: Context: National public insurance for drugs is often based on evidence of comparative effectiveness and cost-effectiveness. This study describes how that evidence has been used across 3 jurisdictions (Australia, Canada, and Britain) that have been at the forefront of evidence-based coverage internationally.

Objectives: To describe how clinical and cost-effectiveness evidence is used in coverage decisions both within and across jurisdictions and to identify common issues in the process of evidence-based coverage. Design, Setting, and Participants Descriptive analysis of retrospective data from the Common Drug Review (CDR) of Canada, National Institute for Health and Clinical Excellence (NICE) in Britain, and Pharmaceutical Benefits Advisory Committee (PBAC) of Australia. All publicly available information as of December 31, 2008, was gathered from each committee's Web site (data set begins in January 2004 [CDR], February 2001 [NICE], and July 2005 [PBAC]). Main Outcome Measure Listing recommendations for each drug by disease indication.

Results: NICE recommended 87.4% (174/199) of submissions for listing compared with a listing rate of 49.6% (60/121) and 54.3% (153/282) for the CDR and PBAC, respectively. Significant uncertainty around clinical effectiveness, typically resulting from inadequate study design or the use of inappropriate comparators and unvalidated surrogate end points, was identified as a key issue in coverage decisions. Recommendations varied considerably across countries, possibly because of differences in the medications reviewed; different agency processes, including the willingness to negotiate on price; and the approach to “me too” drugs. The data suggest that the 3 agencies make recommendations that are consistent with evidence on effectiveness and cost-effectiveness but that other factors are often important.

Conclusions: NICE, PBAC, and CDR face common issues with respect to the quality and strength of the experimental evidence in support of a clinically meaningful effect. However, comparative effectiveness and cost-effectiveness, along with other relevant factors, can be used by national agencies to support drug decision making. The results of the evaluation process in different countries are influenced by the context, agency processes, ability to engage in price negotiation, and perhaps differences in social values.


McMahon, Meghan, Steve Morgan, and Craig Mitton. 2006. “The Common Drug Review: A NICE Start for Canada?” Health Policy 77 (3): 339–51.

Abstract: Prescription drugs are one of the fastest growing cost components of modern health care systems. Efforts to control escalating costs while simultaneously maximizing population health outcomes have led many countries to implement restrictive criteria on the funding of certain drugs. While drugs are licensed for sale based on evidence of safety and efficacy versus a placebo, many funders now require evidence of clinical- and cost-effectiveness compared to existing drugs as part of their reimbursement criteria. In some countries, concerns about duplication of drug assessment and administrative effort across different jurisdictions have led to experimentation with various forms of centralized drug review processes. Centralized drug reviews strive to standardize, inform, and improve drug reimbursement decisions through critical assessments of comparative clinical- and cost-effectiveness. The ultimate objective is to inform formulary listing decisions that both maximize health outcomes and achieve good “value for money”. This paper describes the Common Drug Review (CDR), a uniquely Canadian version of a centralized drug review process, and compares it with the much-studied National Institute for Health and Clinical Excellence (NICE) in the United Kingdom. Through this analysis, which draws on prior critiques and experiences of NICE, we highlight several critical issues for pharmaceutical priority setting that must be considered in the operation and appraisal of centralized drug review processes. These include the selection of drugs for review, centralized versus decentralized decision-making, receptor capacity at local decision making levels, and public participation.


* For the purposes of this review, we have established three categories to describe the state of the literature: thin, considerable, and rich. 

 -   Thin: There are relatively few papers and/or there are not many recent papers and/or there are clear gaps

-   Considerable: There are several papers and/or there are a handful of recent papers and/or there are some clear gaps

-   Rich: There is a wealth of papers on the topic and/or papers continue to be published that address this issue area and/or there are less obvious gaps


Scope: While many of these issues can touch a variety of sectors, this review focuses on medicines. The term medicines is used to cover the category of health technologies, including drugs, biologics (including vaccines), and diagnostic devices.

Disclaimer: The research syntheses aim to provide a concise, comprehensive overview of the current state of research on a specific topic. They seek to cover the main studies in the academic and grey literature, but are not systematic reviews capturing all published studies on a topic. As with any research synthesis, they also reflect the judgments of the researchers. The length and detail vary by topic. Each synthesis will undergo open peer review, and be updated periodically based on feedback received on important missing studies and/or new research. Selected topics focus on national and international-level policies, while recognizing that other determinants of access operate at sub-national level. Work is ongoing on additional topics. We welcome suggestions on the current syntheses and/or on new topics to cover.

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.