Quality Assurance

Given the large number of sites involved in EMNet studies, and the volume of data these sites collect, quality assurance is a high priority. In addition to tracking all ED visits during the enrollment period, all studies have relied on trained staff who collect data using structured interviews that are read directly to patients. Both the individual sites and the EMNet Coordinating Center (ECC) review forms several times before the forms undergo double data entry. The following is a list of quality assurance procedures:

  • Entry criteria. Sites are accepted into MARC-5 only if they are able to enroll at least 50% of consecutive patients (for 24/7 schedule x 2 weeks) and to contact at least 50% of enrolled patients. Every cycle, several sites fail to fulfill these criteria and their data are discarded. However, most of these sites are able, on a second chance, to fulfill these criteria.
  • Physician review. Physician investigators at the participating sites review all forms. After the forms arrive at the ECC, physicians employed by the ECC over-read the forms again to identify any potential inconsistencies.
  • Completeness of data. When the physician reviewers at the ECC note internal inconsistencies (e.g., discrepant triage and medication times), the reviewers flag these records and review them with the site investigator. When the site investigator cannot resolve the inconsistency, the item is recorded as missing. For example, most variables in the MARC asthma database pertaining to the in-ED interview have >95% completeness.
  • Study Registry (consecutive ED visits for condition studied during enrollment period). For all studies, the ECC requires that sites maintain a registry of all consecutive patients during the enrollment period that come to the ED with the condition being studied. This registry consists of several visit-related and demographic variables, including visit date and time, reason for visit, sex, and race. The ECC uses this data to confirm that sites have enrolled the required number of consecutive patients, as well as insuring that patients not enrolled do not differ demographically from enrolled patients. For greater accuracy, the ECC requires that sites use daily ED patient logs, rather than more readily available billing data. The ECC examined the accuracy of identifying consecutive ED asthma visits with hospital billing data, an approach commonly used for a variety of clinical research studies. We found 68% sensitivity and 9% false-positive rate at one site, and 88% and 24%, respectively, at another. (Note: Billing data were more accurate for asthma admissions.)
  • Telephone follow-up. The overall contact rate for MARC studies is approximately 80% at 2 to 4 weeks after the index ED visit. For example, MARC-4 required telephone follow-up during a 5-day window at 3 weeks after the index ED visit and 84% of subjects were successfully reached. At 3 months, during a 10-day window of allowed contact, the ECC staff successfully reached 66% of patients in MARC-4. In MARC-9, one-month follow-up was 81% and 6-month follow-up was 70%. (Please note that these EMNet studies were essentially voluntary, and they were NOT been conducted with long-term follow-up in mind.)

The ECC also has conducted “validation” studies for specific studies to insure high-quality data. Examples follow:

  • Reliability of selected variables. We re-contacted 50 patients in MARC-2 to ascertain the reliability of several variables (e.g., age at diagnosis, height, primary care provider status, and duration of menstrual cycle) and found >90% concordance between the original and telephone follow-up report.
  • Reason for coming to ED. Among 50 patients in MARC-2, two ECC reviewers independently assigned “free text” responses into pre-specified categories for a question asking the patient why he/she came to the ED (as opposed to another treatment setting). We found a kappa of >0.8 between the two reviewers.
  • Acute asthma relapse. In a similar exercise (n=100), we examined reviewer assignment of a more complicated definition of acute asthma relapse (patient report of asthma worsening and a physician intervention consistent with worsening) and found a kappa of >0.8 between the two reviewers.
  • Bronchiolitis relapse. To examine the reproducibility of a new relapse classification for MARC-25, we examined reviewer assignment of any urgent visit to an ED or clinic because of worsening bronchiolitis (i.e., parent report of bronchiolitis worsening and a physician intervention consistent with worsening). Among 100 randomly-sampled children who were discharged to home after an ED visit, we found a kappa of 0.98 between the two reviewers.