Point Value for Project Components
Research: Topic (Problem identification)
Research: Literature review, causal theory/logic model and research questions
Research: Data collection and analysis
Project Research Components
These research components, each of which will be graded using the rubric below, include:
Topic (Problem Identification). The student will identify an issue area and program, policy or agency that will be the focus of the project. They will produce a short memo 1-2 pages that identifies and describes the program, policy or agency that will be the focus of the investigation. The professor will provide comment to help the students narrow the investigation if appropriate.
Literature review, causal theory/logic model and research questions. Students should conduct a literature review to inform the causal theory behind their research. This is similar to the logic model approach causal theories (i.e. theories about causes and effects; e.g. poverty increases crime or better teachers improve student outcomes) are important to both. Logic models combine causal theories into a single framework for a program, policy, or organization. They demonstrate how inputs and processes relate to outcomes and impacts; but most importantly, they outline the casual relationships between different elements of the program/organization and its impacts and outcomes on society. Ensuring that you have a clear theoretical model to test your statistical analysis against is critical to the research methods taught in this class. On the basis of this theory (or logic model), you will propose researchable questions. (Note that you will likely revise the researchable questions later in the project and for the capstone. Iterative development is probably the best approach here).
Data collection and analysis. Students will begin to analyze the data they have collected using SPSS and excel. These initial findings must be reported on using appropriate terminology, and following the correct procedures. This analysis will be supported with appropriate research and evidence from the academic literature, government data and reports, and reliable media reports. Analytical findings will be used to generate recommendations about policy, program or agency reform.
Each of these research components will be evaluated using the following rubric:
Rubric for Team Project Research Components (5% for each, except for Data Collection and Analysis, which is 10%)
Objective Unsatisfactory Partially Proficient Proficient Superior
Quality of underlying research
__/ 2 points
Does not meet minimum standard of quality. (0) Incorporates some research regarding program relevant to the concept, but lacks specificity or uses inappropriate sources. (0.5) Uses several sources of research from academic sources, reliable media outlets and from relevant sources. Does an adequate job of linking research in application. (1) Incorporates very high quality research; provides insight and originality in the integration of research to the application. (2)
Application of concept
__/ 2 points
Does not meet minimum standard of quality (0) Demonstrates partial understanding of concept(s) but lacks specificity or conceptual depth in application. (0.5) Solid understanding of concept(s) and valid application to the project at hand. (1) Sophisticated command of concept(s) and insightful application to the current project. (2)
Clarity of presentation.
__/ 1 points
Does not meet minimum standard of quality. (0) Frequent errors or ambiguities in writing or presentation impede understanding. (0) Presentation of concepts is clear and understandable. (0.5) Highly professional presentation with polished writing and few errors. (1)
All class work should utilize APA formatting.
General instructions for Literature Review, Causal Theory/Logic Model and Researchable Questions
There are a few different steps to this assignment:
1) Literature review, which informs the…
2) Causal theory / logic model, which informs the…
3) Researchable questions (which should be answerable with respect to the data you are collecting).
As you can see, you need multiple pieces to align here. This can be challenging, and you will have to work iteratively to ensure that your literature review connects to your causal theory and then to your researchable questions. I recommend sketching out these elements together and making sure they line up before writing up the individual sections.
Students should conduct a literature review to inform the causal theory behind their research. The causal theory behind your analysis is very similar to the logic model approach discussed in my version of PUB 500. In case you took PUB 500 with another professor, I have attached further discussion of logic models. These models demonstrate how inputs and processes relate to outcomes and impacts; but most importantly, they outline the casual relationships between different elements of the program/organization and its impacts and outcomes on society. To be clear, logic models are a structure for combining multiple causal theories into a single format. They are a standard format as they are used often in program evaluation, grant writing, and especially in the non-profit sector (though increasingly in the public sector). However, if this structure doesn’t fit your project, you can discuss the causal theories instead. 🙂
You can either submit a logic model for your policy/program, or agency, or a focused discussion of the causal relationships which sit behind your research questions. You don’t need both! But you do need at least one. 🙂 Either way, you need to discuss the reasoning and evidence behind these relationships (this is the most important bit!). Ensuring that you have a clear theoretical model to test your statistical analysis against is critical to the research methods taught in this class.
On the basis of this theory, you will propose researchable questions. Researchable questions are different from research questions. Research questions are what you want to find the answer to. Researchable questions are what you can find the answers to. Researchable questions are practical, and have the potential to be answered with available data.
Note that you will likely revise the researchable questions later in the project. Iterative development is probably the best approach here.
A quick note on how to write a literature review
Literature reviews for class projects should reference around 10 articles. This is just a rule of thumb for this class. 5 quality and relevant articles well-reviewed would be better than 15 slightly-related articles from non-academic sources and discussed in a haphazard fashion!
Literature reviews should focus on the academic literature, or reports written by politically-neutral think tanks or government agencies. Often studies include reference to news articles in their literature reviews. New articles are good for providing context to issues; however, they should not form the basis of a literature review.
One approach is to start with an “annotated bibliography”. Annotated bibliographies summarize each article in turn in terms of the main ideas, methods used, and a brief critical assessment of the article (what is good, what is missing, what questions remain, etc…).
However, an annotated bibliography is not sufficient to achieve the highest grades. Ideally students discuss the articles reviewed as a collective whole, rather than as stand-alone papers. This requires students to step back from the details and discuss the broader concepts and themes; what connects the articles and what is contrasting.
All of this takes some time! It is recommended to not leave this to the last minute, and to leave space between each of the steps. Please refer to some of the “best student papers” to get a sense of the kind of literature reviews others have done (this doesn’t mean they are ideal, just better examples!).
Data Collection and Analysis
Students will begin to analyze the data they have collected using SPSS and excel.
Please see the “Data” are of Blackboard. Some widely-used data options include: General Social Survey (1972-2014), Public Policy Institute of California Statewide Survey (1998-2016), or the Federal Employee Viewpoint Survey (2010). “A note on data collection” is provided to help guide you through that particular step. However, there are many issue-specific datasets available
These data should:
be able to answer your researchable questions,
Note that the questions and data should align, but it may take a few iterations to get this right.
apply to one of the statistical methods introduced in this class,
Please note that this most likely means multi-variate regression analysis, and should feature at least 3 independent variables and 1 dependent variables. Please see “a note on data collection” in the data area for advice on gathering data and the structure that data should take.
There are more complex extensions of multi-variate regression analysis discussed later in the class (such as logistic regression, diff-in-diff, instrumental variables, fixed effects), which you are welcome to undertake, but require additional assumptions and analytical steps.
be able to sufficiently control for confounding variables and/or answer critical questions.
Again, it will take a few iterations to get this right. Most multi-variate research projects go through dozens of iterations in their regression models before starting to think about publication! Just do your best and try to work towards improvements each time!
In addition to submitting the raw data in tabular form (ideally in an excel worksheet), you are expected to submit descriptive statistics as part of the write up (ideally in a well-formatted table).
These initial findings must be reported on using appropriate terminology, and following the correct statistical method procedures. Results should ideally be tabulated using a standard statistical analysis format – please see one example discussion here:
Note that this is just one way of presenting the results (there are many, as a quick google search will show you!). However, it is preferable to not simply present the SPSS results. SPSS results tables spit out lots of information, some of which is more necessary than the rest. By following the above guide, you can boil down your results and select the key indicators for your tables.
This analysis should be supported with appropriate research and evidence from the academic literature, government data and reports, and reliable media reports.
Analytical findings will be used to generate recommendations about policy, program or agency reform.
In sum, the required steps for this assignment are:
1. Gather data (see the Data section of Blackboard, and “a note on data collection” in particular)
2. Input the data into SPSS. Blackboard data is already set up for this. If you collect it yourself, data formatted in SPSS is best. If you piece the variables together yourself, make sure it fits the requirements of multi-variate analysis.
3. Generate a table of descriptive statistics for the data you will use. This can be automatically generated when doing multi-variate analysis. But please produce a table for your submission which summarizes your chosen variables.
4. Run a multi-variate regression analysis in SPSS. You have practiced this in Exercise 5. The steps for how to do it in SPSS are the same as for Exercise 5.
5. Put the SPSS results into Word tables, following the format outlined here:
6. Write up the results. Discuss the statistical significance of each coefficient, as well as the direction (positive or negative) and magnitude (the strength of the relationship between each variable and the dependent variable) of each coefficient. What is the overall model strength (R-squared). Now interpret the results. What are they telling us, both specifically and in terms of the research question more broadly.
Combined, this is a lot of steps! Multi-variate regression analysis is difficult and takes time to implement. My best advice is to take your time and work through each step carefully. 🙂
Project Final Reports
The student will next combine the various sections previously worked on into first a Draft and then, following feedback, a Final Report. This iterative approach aims to produce a high-quality report. Grading for the Final Report will take into account improvements made on the basis of feedback to the Draft.
Draft and Final Report Rubric (scores for draft and final reports are graded out of 100, are then converted to the overall grades of 10% for draft and final).
POLICY, PROGRAM, OR AGENCY OVERVIEW Unsatisfactory Partially proficient Proficient Superior
Problem definition and policy or program description
__/5 points Does not meet minimum standard of quality. (0) Identifies and describes program in general terms. (1)
Provides some detail about underlying issue/problem or goals of program; target population; stakeholders. References appropriate readings.
(3) Identifies and describes program, and sites evidence in describing the issue/problem or goals addressed, and key beneficiaries or stakeholders. Insightful use of class readings. (5)
Political, organizational, fiscal context
__/5 points Does not meet minimum standard of quality. (0) Contextual material is presented but is poorly integrated or gratuitous. (1) Some work integrating context in key areas of analysis. (3)
Context is integrated into problem definition, analytic criteria, choice of alternative set, etc. (5)
LITERATURE, THEORY, AND RESEARCH QUESTIONS Unsatisfactory Partially proficient Proficient Superior
__/5 points Does not meet minimum standard of quality (0) Literature review is incomplete, contains flaws or biases not acknowledged. (1) Literature review covers key texts, concepts, theory, and data. (3) A particularly insightful literature review with creative or innovative application to answer researchable questions. (5)
Causal theory/logic model
__/5 points Does not meet minimum standard of quality. (0) Causal theory/logic model is general; little development from earlier work. (1) Solid causal theory/logic model that shows refinement based on research. (3) Causal theory/logic model is refined and explicitly references research related to policy, program or agency. (5)
Does not meet minimum standard of quality. (0) Research questions are not clearly specified. (1) Questions are clearly defined, researchable, and relate to program. (3) Questions are also clearly linked to causal theory/logic model; articulate public service values. (5)
Discussion of research-ability and limitations
Does not meet minimum standard of quality. (0) Identification of at least one challenge. (1)
References key stakeholders, political context, ethics and/or other challenges to conduct analysis. (3)
Sophisticated discussion of researchable questions in analysis of political and/or organizational stakeholders and other challenges. (5)
DATA COLLECTION Unsatisfactory Partially
Proficient Proficient Superior
Valid and appropriate data and evidence
__/10 points Does not meet minimum standard of quality. (0) Data and evidence are not well-aligned to support research questions and analysis. (4) Data and evidence are appropriate to support research questions and analysis. (7) Refined and sophisticated data collection approach to support research questions. (10)
DATA ANALYSIS Unsatisfactory Partially
Proficient Proficient Superior
__/10 points Does not meet minimum standard of quality (0) Statistical model used does not answer appropriately research questions. (4)
Statistical modeling is capable, uses appropriate method and checks for most limitations (7) Sophisticated use of statistical modeling to produce insightful results while controlling for confounding variables and checking for statistical limitations. (10)
Communication of results
__/10 points Does not meet minimum standard of quality. (0) Results are not communicated in a clear an understandable manner. (4) Coherent discussion of results and appropriate use of tables and charts. (7) Refined and sophisticated presentation of results with insightful use of tables and charts. (10)
RECOMMEND-ATIONS Unsatisfactory Partially
Proficient Proficient Superior
Integration of analytical findings into recommendations
__/10 points Does not meet minimum standard of quality. (0) Mostly general discussion of how findings apply to recommendations. (4) Specific attention to linkages between findings and recommendations. (7) Specific and innovative integration of findings into recommendations. (10)
Reflection on strengths and weakness of analysis
__/10 points Does not meet minimum standard of quality. (0) Partial treatment of analysis strengths and weaknesses. (4) A solid critique of reform analysis. (7) An insightful critique of analysis that also shows how to address weaknesses. (10)
ARGUMENT-ATION Unsatisfactory Partially
Proficient Proficient Superior
Sequencing and structure
__/10 points Does not meet minimum standard of quality. (0) Basic organization detracts from argument; lacks acceptable executive summary. (4) A logically structured product with an acceptable executive summary. (7) Organization of analysis emphasizes thematic argument responsive to client issues. (10)
Grammar and style
__/5 points Does not meet minimum standard (0) Contains errors in grammar or syntax. (1) Grammar and style are clear and concise. (3) Grammar and style are highly professional and well-polished. (5)
__/5 points Does not meet minimum standard of quality. (0) Uses only section headings and bullets to break up argument and direct reader through argument. (1) Incorporates other visual elements such as charts, graphs, text charts, or other visual models to an adequate extent. (3) Particularly attractive and innovative use of text/graphic elements to include display of data; flow charts; maps, etc. Graphic elements help drive argument. Visual elements properly titled and discussed in text. (5)