In the past three years, there have been consistent claims that states were “cooking the books” with the monthly job estimates every time the news did not fit into someone’s particular narrative. The problem is that – while there are non-trivial flaws with the job estimates – the states simply cannot make up job numbers to mute bad news.
The job estimates and the unemployment rate are derived from two different surveys which use different units of analysis. Both surveys are conducted by the Census for the Bureau of Labor Statistics (BLS) during the week of the month that includes the 12th day. The unemployment rate is a product of the Current Population Survey (CPS) which measures the number of persons who are in the labor force and how many of them are employed and unemployed. There are 60,000 households surveyed nationally each month for the CPS. The job estimates are a product of the Current Establishment Survey (CES) which measures the number of jobs by industry. There are 140,000 establishments surveyed nationally for the CES each month.
The production of monthly job estimates is supervised and driven by the BLS. States cannot release job estimates that are not approved by the BLS. In the past, states were permitted to utilize analyst intervention to massage job estimates to better reflect the local knowledge officials had of their state and regional economies. This ability has been increasingly curtailed over the past several years – to the point where it functionally has not existed for more than a year.
The response rate in the CES survey is typically poor, and the production of the job estimates are made by a probability model created by the BLS. This model must take into account all of the myriad moving parts of the economy – such as breakdown of large firms versus small firms, the percentage of firms in different industries and sub-industries, the proper industry mix in a given state, etc. All of these considerations – which are not inherent in the CPS household survey – creates a product that has a greater degree of error in it than any analyst would like. Yet, it is the best we have. Firms are not required to participate in the survey. (Note that I am using “firm” and “establishment” to refer to the same thing in this blog – a specific business location that employs at least one person.)
In an nutshell, here’s how the production of the job estimates work from the states’ perspective:
1. The states receive from the BLS the CES responses from establishments in their states. Each state is given an estimate range that the state job estimate should be consistent with. The estimates of the states MUST total to the national estimate. This is another area that provides error since the national estimate is an estimate itself – not a total of estimates.
2. State LMI staff input the responses into the probability model created by the BLS.
3. State LMI staff check the responses to make sure that there aren’t obvious or potential mistakes or in the data.
4. State LMI staff incorporates any corrections from data (but ability to do this has been curtailed over the past few years).
5. The states transmit their estimates to BLS for approval.
6. BLS transmits approved estimates to the states. States MUST publish those estimates. BLS will publish them regardless of what the states do. This is a preliminary estimate – because responses continue to come in from establishments during this process. Those straggler responses (and there are typically a lot of them) are then incorporated into the revised estimates that are published the following month (but get a lot less attention – even though they are more accurate than the preliminary estimates).
For those still not convinced that states are not cooking the books, the role states play in the production of the job estimates is essentially over now. As of March 2011, the consolidation of authority for the job estimates at the national level (i.e., BLS) has been completed. Here’s the announcement on the BLS website:
Production of State estimates will transition from State Workforce Agencies to BLS
With the production of preliminary estimates for March, 2011, responsibility for the production of State and metropolitan area (MSA) estimates will transition from individual State Workforce Agencies to the BLS. State agencies will continue to provide the BLS with information on local events that may affect the estimates, such as strikes or large layoffs at businesses not covered by the survey, and to disseminate and analyze the CES estimates for local data users. This change is designed to improve the cost efficiency of the CES program and to reduce the potential for statistical bias in state and area estimates. A portion of the cost savings generated by this change is slated to be directed towards raising survey response rates in future years, which will decrease the level of statistical error in the CES estimates.
Refinement of procedures for producing State and MSA estimates
Estimates produced by the BLS at the statewide industry super-sector level will continue to utilize an improved outlier identification procedure that has been in effect since the production of January 2010 preliminary state CES estimates. Beginning with March 2011 preliminary estimates, this procedure also will be used in the estimation of detailed industry statewide estimates and MSA estimates. In addition at that time, the BLS will implement an improved imputation procedure for major survey non-respondents and a procedure to correct for differential response rates within an industry sector. The use of these procedures will allow BLS to rely less on individual analyst judgment and more on the use of standard statistical methodology. Statewide and MSA series with smaller sample sizes will continue to be estimated using a small domain model. Introduction of the new estimation procedures may result in more month to month variability in the estimates, particularly in smaller MSAs. For further information on the estimators see www.bls.gov/opub/hom/homch2.htm.
Published by this author at The Big Idea.