Abstract
This paper is a slightly revised version of a paper prepared for the seminar on methods for impact evaluation of family planning programs held in Jaco, Costa Rica, May 14-16, 1997. The seminar was sponsored by the International Union for the Scientific Study of Population (IUSSP), the United States Agency for International Development (USAID), the Carolina Population Center of the University of North Carolina, and the Central American Population Program of the University of Costa Rica.
The goal of the seminar was to look at current methodological problems facing careful evaluation of the impact of programs, to examine some of the new methods that have been developed to address persistent issues, and to assess the methodological challenges posed by the expanded goals of many programs following the 1994 Cairo International Conference on Population and Development.
This paper was designed to serve as the background to discussions of current methodologies and issues by tracing the development and nature of methods for assessing impact that started soon after the first programs were initiated in the 1950s. The techniques discussed include standardization and trend analysis, the analyses of acceptor data, experimental designs, multivariate areal analysis, population-based surveys, and multilevel strategies.
The intent of the program sponsors and coordinators was to publish the collected papers but various contingencies intervened to make this infeasible. A description of the seminar and many of the papers are maintained on the University of Costa Rica website: http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm. As a background chapter, the original version contained references to many of the other chapters planned for the volume. As many of these papers appear on the website, relevant references are given to the authors and this website throughout the paper.
References
Ahlburg, D. and Diamond, I. (1996), ‘Evaluating the Impact of Family Planning Programmes’ in Ahlburg et al.(eds.), The Impact of Population Growth on Well- Being in Developing Countries. New York: Springer.
Bauman, K., Viadro, C., and Tsui, A. (1994), ‘Use Of True Experimental Designs For Family Planning Program Evaluation: Merits, Problems And Solutions.’ International Family Planning Perspective, 20(3):108-113.
Bauman, K. (1995), ‘How Effective Are Family Planning Programs? Evidence From True Experiments.’ Paper presented at the Annual Meeting of the Population Association of America, San Francisco, CA, April 5-7.
Bauman, K. and C. Suchindran. ‘Meta-analysis: an Overview with Application to Evaluations of Family Planning Programmes.’ (http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm)
Berelson, B. (1974), “World Population: Status Report, 1974. A Guide for the
Concerned Citizen.” Reports on Population/Family Planning, No, 15, January
Berelson, B., Anderson, R., Harkavy, O., Maier, J., Mauldin, P. and Segal, S. (1965), Family Planning and Population Programs: A Review of World Developments. Chicago: The University of Chicago Press.
Bertrand, J., Magnani, R. and Rutenberg, N. (1996), Evaluating Family Planning Programs, with Adaptations for Reproductive Health. Chapel Hill, NC: The EVALUATION Project.
Bertrand, J. and Lisanne F. Brown, (1997) ‘The Working Group on the Evaluation of Family Planning Operations Research Final Report’. Chapel Hill, NC: Evaluation Project, Carolina Population Center, University of North Carolina.
Bertrand, J. ‘Qualitative Methods for Evaluating Family Planning Programs.’
(http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm)
Bongaarts, J. (1986), ‘The Prevalence Method.’ In United Nations, The Methodology of Measuring the Impact of Family Planning Programmes on Fertility. Manual IX. Department of International Economics and Social Affairs, ST/SER.A/66/Add.1 New York: United Nations.
Bongaarts, J. (1994) ‘The Impact Of Population Policies: Comment.’ Population and Development Review 20(3): 616-20.
Bongaarts, J. ‘Role of Family Planning Programmes in Contemporary Fertility
Transitions.’ (http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm)
Brennan, E. ‘The United Nations Population Inquiries.
(http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm)
Buckner, B., Tsui, A., Hermalin, A. and McKaig, C., eds. (1995), A Guide to Methods of Family Planning Program Evaluation, 1965-1990. Chapel Hill, NC: The Evaluation Project
Bulatao, R. and Lee, R., eds. (1983), Determinants of Fertility in Developing Countries. (Two volumes.) New York: Academic Press.
Caldwell, J. and Caldwell, P. (1986), Limiting Population Growth and the Ford
Foundation Contribution. Dover, NH: Frances Pinter Ltd.
Casterline, J., ed. (1985a), The Collection and Analysis of Community Data. Voorburg, Netherlands: Statistical Research Institute.
Casterline, J. (1985b), ‘Community Effects on Fertility,’ in J. Casterline (ed.), The Collection and Analysis of Community Data. Voorburg, Netherlands: Statistical Research Institute, 65-75.
Casterline, J. (1987), ‘The Collection and Analysis of Community Data,’ in J. Cleland and C. Scott (eds.), The World Fertility Survey: An Assessment. Oxford: Oxford University Press.
Chamratrithirong, A., Hirschman, C. and Guest, P. (1992) ‘A Multilevel Analysis of the Determinants of Fertility in the Four Regions of Thailand.’ Asia Pacific Population Journal, 7(1): 51-64.
Chandrasekaran, C., and Hermalin, A. (1975), Measuring the Effect of Family Planning Programs on Fertility. Liege, Belgium: Ordina Editions.
Cleland, J. and Hobcraft, J. (1985), Reproductive Change in Developing Countries: Insights from the World Fertility Survey. Oxford: Oxford University Press.
Cleland, J., Phillips, J., Amin, S., and Kamal, G. (1994), Determinants of Reproductive Change in Bangladesh: Success in a Challenging Environment. Washington, DC: The World Bank.
Coale, A. and Watkins, S., eds. (1986), The Decline in Fertility in Europe. Princeton, NJ: Princeton University Press.
Cuca, R., and Pierce, C. (1977), Experiments in Family Planning: Lessons from the Developing World. Baltimore, MD: The Johns Hopkins University Press.
Curtis, S. and Diamond, I. (1995), ‘When Fertility Seems Too High for Contraceptive Prevalence: An Analysis of Northeast Brazil.’ International Family Planning Perspectives 21(2): 58-62.
Curtis, S. ‘Using Calendar Data to Study Contraceptive Use Dynamics.’
(http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm)
Cutright, P. (1983), ‘The Ingredients Of Recent Fertility Decline In Developing
Countries.’ International Family Planning Perspectives, 9(4): 101-109.
Davis, K. (1967), ‘Population Policy: Will Current Programmes Succeed?’ Science, 158: 730-739.
Demeny, P. (1979) ‘On The End Of The Population Explosion.’ Population and Development Review, 5(1): 141-62.
Donaldson, P. and Tsui, A. (1990), ‘The International Family Planning Movement.’ Population Bulletin, 45:3.
Easterlin, R. (1983), ‘Modernization and Fertility: A Critical Essay.’ In Bulatao, R. and Lee, R. (eds.) Determinants of Fertility in Developing Countries, Volume 2: Fertility Regulation and Institutional Influences. New York: Academic Press.
Easterlin, R. and Crimmins, E. (1985), The Fertility Revolution: A Supply-Demand Analysis. Chicago: The University of Chicago Press.
Entwisle, B., Hermalin, A., Kamnuansilpa, P., and Chamratrithirong, A. (1984), ‘A Multilevel Model of Family Planning Availability and Contraceptive Use in Rural Thailand.’ Demography 21, No, 4: 559-573.
Entwisle, B. and W.M. Mason (1985) ‘Multilevel Effects Of Socioeconomic
Development And Family Planning Programs On Children Ever (sic) Born.’
American Journal of Sociology 91(3):616-49.
EVALUATION Project (1996), ‘Monitoring Family Planning Programs.’ Wallchart. Chapel Hill, NC: The EVALUATION Project and the Population Reference Bureau.
Fisher, A., J. E. Laing, and John E. Stoeckel (1991), Handbook for Family Planning Operations Research Design. 2 nd edition. New York: The Population Council.
Fisher, A., R. Miller, and K. Miller ‘Situation Analysis: Assessing the Functioning and Quality of the Service Delivery Environment.’
(http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm)
Foreit, J., Gorosh, M., Gillespie, D., and Merrit, C.G. (1978), ‘Community-Based and Commercial Contraceptive Distribution: An Inventory and Appraisal.’ Population Reports: Family Planning Programs. Series J, No. 19, March 1978.
Foreit, J.R., J.E., Rosen, M. Ramos, E. Mostajo, and R. Monge. (1990), ‘The Impact of Service Delivery Frequency on Family Planning Program Output and Efficiency.’ Studies in Family Planning 21 (4): 209-215.
Freedman, R. and Takeshita, J. (1969), Family Planning in Taiwan: An Experiment in Social Change. Princeton, NJ: Princeton University Press.
Freedman, R., and Berelson, B. (1976), ‘The Record Of Family Planning Programs.’ Studies in Family Planning 7(1):1-40.
Freedman, R. and Freedman, D. (1992), ‘The Role Of Family Planning Programmes As A Fertility Determinant.’ In J.F. Phillips and J. A. Ross, eds., Family Planning Programmes and Fertility. Oxford: Oxford University Press.
Freedman, R. (1997), ‘Do Family Planning Programs Affect Fertility Preferences? A Literature Review’ Studies in Family Planning, 28/1: 1-13.
Gallen, M., and Rinehart, W., eds. (1986), ‘Operations Research: Lessons for Policy and Programs.’ Population Reports: Family Planning Programs, 14(2).
Gertler, P.J. and Molyneaux, J.W. (1994), ‘How Economic Development And Family Planning Programs Combined To Reduce Indonesian Fertility.’ Demography 31(1): 33-63
Gorosh, M. and Wolfers, D. (1979) ‘Standard Couple-years of Protection,’ in The Methodology of Measuring the Impact of Family Planning Programs on Fertility, pp. 48-62. New York: United Nations.
Guilkey, D. and I. Diamond, ‘Structural Models for Programs Evaluation.’
(http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm)
Harkavy, O. (1995), Curbing Population Growth: An Insider’s Perspectives on the Population Movement. New York: Plenum Press.
Hauser, P. (1967), ‘Family Planning and Population Programs: A Book Review Article.’ Demography 4 (1967): 397-414.
Hermalin, A.I. (1975), ‘Regression Analysis Of Areal Data.’ In C. Chandrasekaran and A.I. Hermalin (eds.), Measuring the Effect of Family Planning Programs on Fertility.
Liege, Belgium: Ordina Editions. Hermalin, A.I. (1979), ‘Multivariate Areal Analysis.’ In Manual IX: The Methodology of Measuring the Impact of Family Planning Programs on Fertility (Sales No. E.78.XIII.8). New York: United Nations Department of International Economic and Social Affairs.
Hermalin, A.I. (1982a), ‘Family Planning Programs: Effects On Fertility.’ In J.A. Ross (ed.), International Encyclopedia of Population. New York: Free Press.
Hermalin, A.I. (1982b), ‘Some Cautions In The Use And Interpretations Of Regression Analysis For The Evaluation Of Family Planning Programs.’ In Evaluation of the Impact of Family Planning Programs on Fertility: Sources of Variance (Sales No. E.84.XIII.9) New York: United Nations Department of International Economic and Social Affairs.
Hermalin, A.I. (1986), ‘The Multi-Level Aproach: Theory And Concepts.’ In Manual IX: The Methodology of Measuring the Impact of Family Planning Programs on Fertility, Addendum (Sales No. E.86.XIII.4) New York: United Nations Department of International Economic and Social Affairs.
Hermalin, A. and Entwisle, B. (1982), The Role of Surveys in the Analysis of Family Planning Programs. (Proceedings of a seminar held in Bogota, Colombia, October 1980.) Liege, Belgium: Ordina Editions.
Hermalin, A. and Khadr, Z. (1996), ‘The Impact of Family Planning Programs on Fertility: A Selective Assessment of the Evidence.’ Paper presented at the
conference, ‘Assessing the Past for the Future: Family Planning Policy, Programs, and Resources,’ June 1996, Bellagio, Italy.
Hernandez, D.J. (1984), Success or Failure? Family Planning Programs in the Third World. Westport, Conn.: Greenwood Press.
Himes, N. (1963), Medical History of Contraception. New York: Gamut Press, Inc.
Jain, A., (1995), ‘Implementing the ICPD’s Message,’ Studies in Family Planning, 26/5, 296-298.
Khuda, B. ‘Assessing the Policy Impact of Operations Research on the Bangladesh Health and Family Planning Programme.’ (http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm)
Kiser, C. (ed.) (1962), Research in Family Planning. Princeton, NJ: Princeton University Press.
Knodel, J., Chamratrithirong, A., and Debavalya, N. (1987), Thailand’s Reproductive Revolution: Rapid Fertility Decline in a Third-World Setting. Madison: University of Wisconsin Press.
Knowles, J., Akin, J. and Guilkey, D. (1994), The Impact Of Population Policies: Comment. Population and Development Review, 20(3):611-15.
Knowles, J.‘Cost-effectiveness and (http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm) Cost-benefit Analysis.’
Lapham, R.J. and Mauldin, W.P (1987) The Effects Of Family Planning On Fertility: Research Findings. In Lapham R.J. and G.B. Simmons (eds.), Organizing for Effective Family Planning Programs. Washington, D.C.: National Academy Press.
Lloyd, C. and Ross, J. (1989), Methods for Measuring the Fertility Impact of Family Planning Programs: The Experience of the Last Decade. Population Council Research Division Working Paper No.7.
Mason, W. (1986), ‘The Multi-Level Approach: Illustrative Example.’ In United Nations, The Methodology of Measuring the Impact of Family Planning Programmes on Fertility. Manual IX. Department of International Economics and Social Affairs, ST/SER.A/66/Add.1 New York: United Nations.
Mauldin, W.P. (1979), ‘Fertility Projection/Trend Analysis.’ In United Nations, The Methodology of Measuring the Impact of Family Planning Programmes on Fertility. Manual IX. Department of International Economics and Social Affairs, ST/SER.A/66/Add.1 New York: United Nations.
Mauldin, W.P. (1983), ‘Population programs and fertility regulation.' In R.A. Bulatao and R.D. Lee, eds., Determinants of Fertility in Developing Countries. New York: Academic Press.
Mauldin, W.P. and Berelson, B. (1978), ‘Conditions Of Fertility Decline In Developing Countries, 1965-75.’ Studies in Family Planning, 89-148.
Mauldin, W.P and Johnson-Acsadi, G. (1975), ‘Introduction,’ in C. Chandrasekaran and A.I. Hermalin (eds.), Measuring the Effect of Family Planning Programs on Fertility. Liege, Belgium: Ordina Editions, 1-15.
Mauldin, W.P. and Lapham, R. (1987), ‘The Measurement of Family Planning Inputs.’ In Lapham and Simmons (eds.) Organizing for Effective Family Planning Programs. Washington D.C.: National Academy Press.
Mauldin, W.P. and Ross, J. (1991), ‘Family Planning Programs: Efforts and Results, 1982-89.’ Studies in Family Planning 22(6):350-367.
McIntosh, C. A. and J. L. Finkle (1995), ‘The Cairo Conference on Population and Development: A New Paradigm?’ Population and Development Review, 21/2: 223-260.
Menken, J. (1975), ‘Simulation Studies.’ In C. Chandrasekaran and A.I. Hermalin (eds.), Measuring the Effect of Family Planning Programs on Fertility. Liege, Belgium: Ordina Editions, 97.
Mensch, B., Arends, M., Jain, A., and Garate, M. (1995), ‘Meeting Reproductive Goals: The Impact of the Quality of Family Planning Services on Unintended Pregnancy in Peru.’ Research Division, Working Paper No, 81. New York: The Population Council.
Montgomery, M. (1987), ‘A New Look at the Easterlin ‘Synthesis’ Framework.’
Demography, 24:4.
Montgomery, M. and Casterline, J. (1993), ‘The Diffusion of Fertility Control in Taiwan: Evidence from Pooled Cross-Section Time-Series Models.’ Population Studies, 47; 457-479.
Morris, L. and Anderson, J. (1982), ‘The Use of Contraceptive Prevalence Survey Data to Evaluate Family Planning Program Service Statistics.’ In A. Hermalin and B.
Entwisle (eds.), The Role of Surveys in the Analysis of Family Planning
Programs. (Proceedings of a seminar held in Bogota, Colombia, October 1980.) Liege, Belgium: Ordina Editions.
Ness, G.D. and Ando, H. (1984), The Land is Shrinking: Population Planning in Asia. Baltimore, MD: The Johns Hopkins University Press.
Nortman, D. (1970), ‘Population and Family Planning Programs: A Factbook.’ Reports on Population/Family Planning, July 1970.
Nortman, D. (1979), ‘Component Projection Approach I: A Computerized Model.’ In The Methodology of Measuring the Impact of Family Planning Programmes on Fertility, pp. 48-62. New York: United Nations, 1979.
Nortman, D. (1985), ‘Population and Family Planning Programs: A Compendium of Data through 1983.’ A Population Council Factbook, 1985.
Nortman, D. and Hofstatter, E. (1973), ‘Population and Family Planning Programs: A Factbook.’ Reports on Population/Family Planning, No. 2, September 1973.
Nortman, D. and Hofstatter, E. (1980), ‘Population and Family Planning Programs: A Compendium of Data through 1978.’ A Population Council Factbook, 1980.
Peterson, W. (1975), Population. (Third Edition) New York: Macmillan.
Phillips, J., Simmons, R., Koenig, M., and Chakraborty, J. (1988), ‘Determinants of Reproductive Change in a Traditional Society: Evidence from Matlab, Bangladesh.’ Studies in Family Planning, 19(6): 313-334.
Potter, R. (1979), ‘Analysis of Reproductive Process.’ In United Nations Manual IX: The Methodology of Measuring the Impact of Family Planning Programmes on Fertility, pp. 76-96. New York: United Nations.
Potter, R. (1981), ‘The Analysis of Cross-Method Variance in Assessing Family Planning Programme Effects on Fertility.’ In International Union for the Scientific Study of Population (IUSSP) International Population Conference, Manila. Volume 1. Liege, Belgium: IUSSP.
Pritchett, L. (1994), ‘Desired Fertility and the Impact of Population Policies.’ Population and Development Review, 20(1):1-56.
Rosero-Bixby, B. Entwistle, R. Rindfuss, S.J. Walsh ‘Geographic Information Systems (GIS) in Evaluation of Family Planning Programmes: Examples for Costa Rica and Thailand.’ (http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm)
Ross, J., J. Donayre, and R. McNamara, (1987), ‘Perspectives on Operation Research.’ International Family Planning Perspectives 13/4, 128-135.
Ross, J. and Forrest, J.D. (1978), ‘The Demographic Assessment of Family Planning Programs: A Bibliographic Essay.’ Population Index, 44(1).
Ross, J. and Lloyd, C. (1992), ‘Methods for Measuring the Fertility Impact of Family Planning Programs: The Experience of the Last Decade.’ In J. Phillips and J. Ross (eds.), Family Planning Programs and Fertility. Oxford: Clarendon Press.
Ross, J. and Mauldin, W.P. (1996), ‘Family Planning Programs: Efforts and Results, 1972-94.’ Studies in Family Planning, Vol. 27, No. 3: 137-147.
Ross, J., Mauldin, W.P., and Miller, V. (1993), Family Planning and Population: A Compendium of International Statistics. New York: The Population Council.
Rozenweig, M. and Wolpin, K. (1986), ‘Evaluating The Effects Of Optimally Distributed Public Programs: Child Health And Family Planning Interventions.’ American Economic Review, 76(3):470-82.
Schultz, P. (1992), ‘Assessing Family Planning Cost Effectiveness Applicability Of Individual Demand-Programme Supply Framework.’ In J. Phillips and J. Ross (eds.) Family Planning Programmes and Fertility. Oxford: Clarendon Press.
Schultz, P. (1997), ‘Sources of Fertility Decline in Modern Economic Growth: Is Aggregate Evidence on the Demographic Transition Credible?’ In M. Rosenzweig and O. Start (eds.) Handbook of Population and Family Economics. Amsterdam: Elsevier.
Seltzer, J.R. (2002) The Origins and Evolution of Family Planning Programs in Developing Countries. Santa Monica: RAND.
Sherris, J., ed. (1985), ‘The Impact of Family Planning Programs on Fertility.’
Population Reports: Family Planning Programs, XIII(1).
Simmons, R. and Simmons, G. (1987), ‘The Task Environment of Family Planning.’ In R. Lapham and G. Simmons (eds.), Organizing for Effective Family Planning Programs. Washington, DC: National Academy Press.
Steele, F. and M. Choe ‘Event History Analysis for Family Planning Programme Evaluation.’ (http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm)
Suchman, E. (1967), Evaluative Research: Principles and Practice in Public Service and Social Action Programs. New York: Russell Sage Foundation.
Tolnay, S.E. and Christenson, R.L. (1984), ‘The Effects Of Social Setting And Family Planning Programs On Recent Fertility Declines In Developing Countries: A Reassessment.’ Sociology and Social Research, 69(1): 72-89.
Tsui, A. (1985), ‘Community Effects on Contraceptive Use,’ in J. Casterline (ed.), The Collection and Analysis of Community Data. Voorburg, Netherlands: Statistical Research Institute, 77-99.
Tsui, A. (1996), Family Planning Programs in Asia: Approaching a Half-Century of Effort. Asia-Pacific Population Research Reports, No.8; April 1996.
Tsui, A.O. and Bogue, D.J. (1978), ‘Declining World Fertility: Trends, Causes,
Implications.’ Population Bulletin, 33(4): 1-55.
Tsui, A.O. and Hermalin, A.I. ‘Improving the Effectiveness of Family Planning
Programs: Measurement Needs and Issues.’ (http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm)
Tsui, A.O., Wasserheit, J. N. and Haaga, J. G., eds. Reproductive Health in Developing Countries, Washington, DC: National Academy Press.
United Nations (1973), The Determinants and Consequences of Population Trends: New Summary of Findings on Interaction of Demographic, Economic and Social Factors. Vol 1. New York: United Nations.
United Nations (1978), Methods of Measuring the Impact of Family Planning
Programmes on Fertility: Problems and Issues. (Sales No. E.78.XIII.2) New
York: United Nations.
United Nations (1979), Manual IX: The Methodology of Measuring the Impact of Family Planning Programmes on Fertility. (Sales No. E.78.XIII.8) ST/ESA/Ser. A/66. New York: United Nations.
United Nations (1982) Evaluation of the Impact of Family Planning Programs on Fertility: Sources of Variance (Sales No. E.81.XIII.9) New York: United
Nations.
United Nations (1985), Studies to Enhance the Evaluation of Family Planning Program. (Sales No. E.84.XIII.9) New York: United Nations.
United Nations (1986), Manual IX: The Methodology of Measuring the Impact of Family Planning Programs on Fertility, Addendum. (Sales No. E.86.XIII.4) New York: United Nations.
Vaessen, M. ‘Demographic and Health Surveys.’ (http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm)
Wawer, M.J., R. McNamara, T. McGinn and D. Lauro, (1991), ‘Family Planning Operations Research in Africa: Reviewing a Decade of Experience.’ Studies in Family Planning. 22/5: 279-293.
Westoff, C. ‘Unmet Need Revisited.’ (http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm) Wilkinson, M. ‘Using DHS Service Availability (http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm