Educational Technology Priorities

In my role as CIO of Harvard Medical School, I oversee administrative, research and educational technologies for the school. Recently Information Technology and the Program in Medical Education agreed to enhance our IT governance by refining our charter for the Educational Technology oversight committee. I've included the complete charter below to give you an insight into the technology priorities of the Harvard Medical School teaching faculty.

Since 2001, Harvard Medical School has used Mycourses, a single portal for all educational content and applications. For a brief overview of Mycourses, you can click on Take a Tour. The website includes dozens of simulations, hundreds of videos, and thousands of PDFs, representing every handout for every course. We have a enterprise wide directory of all students and faculty, a self built student information system called Madris, and numerous applications which support faculty, staff and student needs.

Here's the Educational Technologies Committee charge identifying the priorities to make Mycourses even better.

Committee Charge:
* Oversee educational technology – existing and new - applications at HMS, including areas listed below.
* Identify annual academic and administrative priorities for educational technology, including new features (eg, enhanced user interfaces, curriculum database, on-line grading), improvements to existing applications (eg, surveys, test delivery)
* Develop an annual work plan and establish and monitor quarterly goals
* Meet bimonthly throughout the academic year

Areas of focus:
*Curriculum management
Improving and expanding the capabilities of MyCourses, making it more user friendly
Developing a search tool for curricular content across all four years of the curriculum, including information from MyCourses and the Course Catalogue

*Student information
Continued development of the MADRIS Student Information System

*Evaluation
Enhanced Evaluation of students by faculty
Enhanced Evaluation of faculty by students
Enhanced Course evaluation

*Virtual applications and simulation
Enhanced Virtual microscopy for pathology and histology
Enhanced Virtual patients
Enhanced Simulation

* Faculty information
Comprehensive teaching effort reporting
Tracking participation in faculty development
Tracking participation in evaluation

* Innovation
Support applications for external and HU funding for educational technology projects that enhance medical student education
Consider establishing HMS innovation funds to seed new initiatives that enhance the MD curriculum

Educational Technology Executive Committee Membership:
Director of Educational Technology and Software Development
Head Master, Academic Societies and Vice Chair, Curriculum Committee
Executive Assistant, PME Administration
Dean for Medical Education
Chief Information Officer
Associate Dean, Medical Education Planning and Administration
Registrar
Executive Director of Curriculum Programs
Senior Designer for Educational Technology
Educational Computing Software Support Specialist
Director, Primary Care Experience at BIDMC
Associate Master, Cannon Society
Master, Cannon Society

This governance committee will not only ensure our work is well aligned with the priorities of stakeholders, but it will also ensure enhanced communication about our progress between IT staff and faculty/staff/students.

I'm very interested in the priorities and experiences of other schools. If you'd like to share your priorities, please submit your experiences via our educational technology survey

I'll summarize the responses, keeping all identities confidential, in a future blog entry.

Cool Technology of the Week

I have an "-ology" problem.

Radiology, Cardiology, Pulmonology, Gastrology, Gynecology, and Endocrinology all have image management needs that require high bandwidth networks, short term high speed storage, and long term archival storage.

Radiology has a industrial strength GE Centricity PACS. The other "ologies" have heterogeneous applications from multiple vendors. As a CIO, I can no longer let 1000 wildflowers bloom in the world of image management. Why?

1. Each department would use its own image viewing software
2. Each department would need its own disaster recovery strategy
3. Each department would use its own records management/image retention rules
4. Each department would need its own capital budget for storage
5. Clinicians would not be able to have unified list of all imaging studies for patient or consolidate images across multiple institutions using different medical record numbers

How do I solve this problem? The answer is Long Term Archiving that is standards compatible and supports all the ologies. Teramedica's Evercore is one solution. IBM's Grid Medical Archive Solution is another. This concept is the cool technology of the week.

The idea behind these systems is simple. Each department can purchase the applications which interface to its imaging devices and support its workflow. The Departments own the "front end"

Each of these imaging systems supports a DICOM exchange to a long term archive. In the past, I've used content addressable storage with a proprietary API and DICOM broker for radiology, DVDs for echo, CDs for vascular, MODs for radiation oncology, etc.

All of this will be replaced with an enterprise image archiving approach which can

1. Provide one place for all images in the enterprise to be stored. IS can provide any storage hardware it wants - NAS, SATA disk, Data Domain archiving appliances etc.

2. Provide unified metadata for every image which can support a single application for consolidated image viewing of all studies from all the "ologies" at different institutions

3. Provide records management rules which enable deletion, information lifecycle management, and compression based on image type or department. For example, digital mammography needs to be kept 10 years, but we can move it from fast storage to slow storage when appropriate. CT images could be compressed after a year and deleted after 5 years.

Having unified storage, unified viewing, and unified management means that IS can now own the backend of image management and treat it as a utility, just as we do with other central file architectures.

The end result of this utility approach to long term image management is a win/win. Departments select the applications they need and the workflow they want. IS manages the security, integrity, and cost of storage centrally. The total cost of operating an enterprise image infrastructure is lower, the service levels are higher, and compliance with records retention policies are simplified.

I've been pursuing this concept for the past 5 years, but now the products are mature enough to make it a reality and I plan to do this as an FY09 project.

Playing the Japanese Flute

At the end of the day, when my email queue is empty, my Blackberry is silent, and my desk is cleared, I retreat to a quiet space to play the Japanese flute, called the Shakuhachi. This blog entry is a personal statement about playing the Shakuhachi and ensuring I keep my mental health while living the life of a CIO.

As I've said in previous blog entries, being a CIO is a lifestyle, not a job. It's a balancing act of keeping projects moving, customers happy, and budgets frugal. The average tenure of a CIO in most organizations is about 2-3 years. When I leave the office, I want to clear away the frustrations of the day and arrive home with the optimism and enthusiasm my family deserves. Having avocations outside of the office helps me maintain my mental clarity and positive outlook. Rock/ice climbing are great weekend activities to recharge the soul, and I'll blog about those later. Playing the Shakuhachi is a daily meditative experience that has been called "Blowing Zen".

The Shakuhachi is an end blown bamboo flute, which is a bit like a recorder without a mouth piece. It's created by hollowing out the root end of a single piece of bamboo. Sound is made by passing air over the blowing edge, enabling the player to produce 4 octaves using only the lips. Notes are fingered using 5 holes and sharps/flats are made by varying the position of the head or by partially covering the holes.

Shakuhachi sheet music is written entirely in Japanese Hiragana characters and not in western musical notation.

The instrument came into Japan from China at the end of the 7th century. From this period until the 12th century it was chiefly used as court music. From the 12th to 16th century it was a popular instrument among mendicant monks, who banded together in the 17th century to form the Fuke sect of Zen Buddhism and used the Shakuhachi as a spiritual tool. Their songs are called honkyoku and are meditation pieces rather than folk songs.

Learning to play the Shakuhachi requires years of study under a licensed Master. The Japanese say that it takes 3 years to learn to move your head appropriately and 7 years to make a note perfectly. One such Shakuhachi master, Phil Nyokai James, lives in Portland, Maine and offers lessons in the Boston area to 10 students every other week.

I play two flutes - a 1.8 (traditional 1.8 foot long) in the key of D made by Kobayashi Ichijo of Osaka and a 2.4 in the key of A (the picture above) made by Yamaguchi Shugetsu of Nara. In addition to playing the flute each night, I bring it to mountaintops (such as the summit of Mt. Fuji) and forests. The rich sound echos throughout natural settings and sounds truly spiritual.

A recent book about Mt. Monanock in New Hampshire describes the mysterious flute player of the mountain who is often heard but never seen. If you're hiking in New England and you hear a flute in the distance, it just might be the wandering CIO monk.

Standards for Secondary Uses of Data

Health Information Exchanges and Regional Health Information organizations typically focus on the exchange of patient data for clinical care. Use cases include providing clinical histories to Emergency Departments, pushing laboratory/radiology results from hospitals to physician offices, and supporting referral workflow between primary care clinicians/specialists.

However, secondary uses of data such as public health reporting, pharmacovigilance, biosurveillance and quality reporting can be equally important. The Social Security Administration disability evaluation process provides an great example of data exchange for secondary use that improves quality, saves money, and leads to increased patient satisfaction.

Today, the Social Security Administration pays over $500 million dollars per year to retrieve paper records and purchase consultative examinations when they are unable to obtain existing records in support of a disability application. Here's how it works:

1. A patient applies to the Social Security Administration for benefits related to a disability
2. The patient signs an authorization to release medical records at a local SSA office or submits the form via mail
3. The SSA forwards the authorization and a medical record request to hospitals via mail
4. Health Information Management staff at the hospital copies paper records or prints electronic records, then sends those records to the SSA. It's a manual, costly process. Records are generally sent via mail, but some providers use fax or the SSA website to upload non-standard file formats, like Word, which the SSA converts to images for use in their current system. These images are just pictures, not data, and therefore are not searchable.
5. Staff at the SSA manually review the paper records to verify diagnoses, medications, lab results, and other observations which document disability
6. The application is manually reviewed, then an administrative decision is made. The entire process takes about 6 months

With interoperable data standards, the new process could be:

1. A patient applies to the Social Security Administration for benefits related to a disability
2. The application is entered into SSA's disability claims system
3. The patient authorization is digitized by a local SSA office and stored centrally
4. The case is transferred electronically to the State Disability Determination Service - who will determine whether the claimant is medically disabled according to SSA's rules.
5. At the same time as the electronic case is transferred, SSA's system automatically sends the digitized authorization to a hospital along with an electronic query to verify patient records are present - without any human intervention
6. The hospital verifies the authorization and sends an electronic clinical summary securely with the SSA
7. SSA receives the clinical summary, formats it in a document and automatically saves in the electronic disability folder - again, without any human intervention. At the same time, a rules engine reviews the data. Depending upon the data received, the system alerts the case adjudicator to take appropriate steps.

For example, the hospital includes an ICD 9 diagnosis code of 153.2, which is Malignant neoplasm of the descending colon, in the clinical summary. The summary also includes a secondary ICD 9 diagnosis code of 197.7, which is a secondary malignant neoplasm of the liver.
The rules engine automatically creates an alert to advise the adjudicator to consider the listing for Colon Cancer with distant metastasis. The adjudicator sees the alert when he/she opens the case for the first time.

8. The adjudicator finds the associated operative and pathology reports in the clinical summary and makes a decision on the case immediately. A process that used to take weeks will be accomplished in a matter of days.

Perhaps some day, with sophisticated enough clinical summaries and rules engines, a system could be developed to automatically make a decision on some disability claims based upon electronic health records.

Although the specific use case for exchange of data between hospitals and the SSA has not been in scope for HITSP, the interoperability specifications developed for biosurveillance, another secondary use of data, work very well. Specifically

1. The PIX/PDQ transaction can be use to transfer patient demographic information and verify patient records are present
2. The Continuity of Care Document provides a clinical summary of problems, medications, allergies, and laboratories
3. The XDR standard provides secure transport of the CCD from hospital to the SSA

Of course, patient privacy must always be protected with any data exchange. Technical security standards enforce privacy policy and the social security administration workflow is predicated on patient authorization, acceptance of the signed authorization by the hospital and transmission of records to SSA only after patient identity has been verified.

The fact that HITSP interoperability specifications have been recognized by Secretary Leavitt means that standards for labs, medications, clinical summaries, transport, and security are available to meet the interoperability requirements of clinicians, patients, hospitals, labs, pharmacies, and government agencies. 2008 is the tipping point for interoperability now that standards are available, government and hospital stakeholders are aligned, and the business case for data exchange is clear.

BIDMC is currently working on a pilot with SSA to implement the HITSP standards and the workflow described above. A successful pilot could lead to wide adoption of data sharing in support of the disability process and integration of these workflows into the Nationwide Health Information Network. Best of all, the enhanced service to patients will likely result in lower overall costs, making implementation fundable from the savings of eliminating paper record transfer.

We'll be live later this year and I'll share all our experiences.

Electronic Health Records for Non-Owned Doctors - Funding

Over the past two months, I've written about Electronic Health Record project management, architecture, planning, technology, and staffing, but the hardest part of the entire project is "What's it going to cost and who's going to pay". Stark safe harbors allow hospitals to support up to 85% of startup and implementation costs, excluding physician office hardware.

I asked the leaders of every major EHR rollout project in Eastern Massachusetts to comment on their funding model. Here's their feedback by institution:

BIDMC
BIDMC and Beth Israel Deaconess Physicians Organization have selected eClinicalWorks as the EMR, have outsourced desktop/centralized hosting to Concordant, and have a combination of insourcing/outsourcing to the Massachusetts eHealth Collaborative for practice consultation. BIDMC financial modeling of all these costs is consistent with the industry standard experience of $40,000-$60,000 per clinician including office hardware. Subsidies will come from BIDMC and local hospitals. With maximum Stark allowable subsidies, our computation of the minimum legal cost per clinician is $15,000, so clinicians will be asked to pay at least that much.

Caritas
Caritas has selected eClinicalWorks as the EMR. Caritas would like to subsidize 85% of the allowable costs. However, Caritas leaves the amount of the subsidy up to local hospital CEOs, since they will ultimately pay the bill and decide which community practices are the most strategic for rollout.

Children's Hospital Boston
Children's and Pediatric Physician's Organization at Children's have selected eClinicalWorks as the EMR and the hospital is hosting the application in their data center, maintaining it via existing IT staff. Six new FTEs in the primary care network will provide practice consulting and implementation services. Children's has outsourced desktop support to The Ergonomic Group. Children's experience has paralleled the BIDMC financial model and has roughly the same cost structure, subsidizing costs to the maximum that Stark allows.

Mt. Auburn Hospital
Mt. Auburn and Mt. Auburn Cambridge IPA have selected eClinicalWorks as the EMR and the physician's organization is hosting the servers in a co-location facility. The hospital has subsidized allowable costs and physicians are paying for all office hardware. The proportion of subsidy for costs of hosting, licenses, training and implementation is still a work in progress. Mt. Auburn notes that part of the cost equation should be consideration of loss of productivity during initial implementation. This is a good point and no other organization has computed this as part of the cost equation.

New England Baptist Hospital
New England Baptist hospital and its physicians have selected eClinicalWorks as the EMR and the hospital is hosting the application in a co-location facility. The hospital has hired 4 FTEs, managed by the hospital CIO, to assess office needs, install hardware and implement software. An MOU has been developed detailing the relationships between the parties, services to be provided, and funding allocation to ensure compliance with Stark. The hospital currently subsidizes 80% of the Stark allowable costs. The physicians are funding their portion of the costs via a combination of payer withholds, PHO funding, and direct physician payment.

Partners Healthcare System
Partners Healthcare offers their home built Longitudinal Medical Record or GE Centricity as the EMR. LMR is hosted centrally and GE Centricity is hosted in clinician offices. As a system, Partners uses withholds in its pay for performance contracts to fund EMRs. EMR adoption is a criterion for remaining in the PCHI network. No subsidy is provided from the central corporation. However, on the local level, Physician Hospital Organizations are subsidizing EMRs. Thus, Partners docs who are in community receive their subsidy and their incentive through pay for performance contracts, which could be a Partners physicians-hospital organization like Newton Wellesley, North Shore or could be a non-Partners physicians-hospital organization like Emerson, Hallmark.

Winchester Hospital
Winchester Hospital has committed to assist its physicians with the EHR implementation process and has budgeted $5 million, including $1.5M for a 4 FTE implementation support team, portal infrastructure development and associated connectivity. The remainder of the money is available for affiliated physicians to purchase and host the EMR of their choice. Winchester will subsidize up to 85% of the cost of EMR licenses and implementation. The support team primarily offers project management and is staffed out of a joint venture funded by the hospital and the IPA. Winchester hosts the hardware for the 15 practices that are under the corporate umbrella (Winchester Physician Associates). The remainder of the practices are hosting applications in their offices with support from number of hardware/networking vendors. The IPA has its own incentive program unrelated to the hospital donation.

Thus, the overall consensus in the community is to subsidize the maximum or nearly the maximum of Stark allowable costs and to provide centralized project management with either insourced or outsourced implementation/practice consulting services. Most organizations are providing central hosting in a hospital or co-location facility.

In two weeks, I'll provide an overview of the funding and staffing model for ongoing support of these practices once they are live.

The Impact of e-Prescribing

In 2007, we went live with integrated e-Prescribing within our enterprise electronic health record via the MA-Share rxGateway, our statewide health information exchange collaboration of payers and providers. We had to redefine workflows, cleanup old prescription data and refine the our existing applications to adapt to the new features of e-Prescribing (eligibility checking, formulary enforcement, medication history display and prescription routing). The following is a summary of the impact of e-prescribing on our General Internal Medicine practice in the first year:

Time and Resource Impact:
1. Prior to full implementation of e-prescribing, Medical Assistant call-in of prescriptions averaged 350 prescriptions per day. We've reduced this to 80/day and we'll further reduce this to 30/day by next month when all residents go live with e-Prescribing.

2. Each call-in averages 4 minutes per prescription and this equals 23 hours or 3 FTE worth of work per day, approximately $96,000.00 of salary. This has been reduced to 0.66 FTE of Medical Assistant work per day or $21,000.00 salary.

3. The Medical Assistant staff are now available to more consistently perform the core work required to support the patients, providers, and practice. In the past, the lack of control over the daily volume of prescriptions resulted in unpredictable exam room support.

4. We experienced significant improvement in efficiency and patient satisfaction in the time for prescriptions to reach the pharmacy. With e-prescribing, prescriptions travel quickly to pharmacies versus up to 2 days for the rx to be called to the pharmacy.

5. We have also seen a decrease in medication errors, in terms of wrong patient, wrong medication, wrong dose since e-prescribing has decreased the potential for "communication errors"

6. We are able to track prescriptions more efficiently. With the paper call-in system, rxs were being called in by many people. Now we can look in our EMR and quickly determine where a prescription is in the process (i.e. in queue, transmitted successfully, transmission failed, etc)

By redirecting our Medical Assistance staff from prescription refills to other tasks we have:

* Piloted a standardized patient visit check out process that is going well
* Provided consistent documentation of vital signs for all patients.
* Helped with fee ticket entry improving the timeliness of charge capture
* Improved examination room turn around time, resulting in decreased waiting room stays

All of these workflow and efficiency improvements have reduced nurse stress level substantially. In addition, nurses are very happy about the decreased opportunity for error and the rework this caused with errors in prescription call ins.

e-Prescribing has been a win/win/win for providers, payers and patients. Massachusetts is now the number #1 e-Prescriber in the country and the MA-Share rxGateway enables payers and providers to collaboratively implement e-Prescribing quickly. It's now live at BIDMC and Partners. It will soon be live at Children's and we've offered it to all other hospitals in the state.

Cool Technology of the Week

A few days ago, I received an email message that got my attention

"Hi John,

There is an Eastern European website/blog offering a set of sites that have been 'hacked' and are being sold to anyone who wants to 'take-over' the site for $7 to $10. One of your sites is on the list. You may want to scan this site for possible SQL injection vulnerabilities/attacks.

PS Other bloggers attest to the validity of the offer and the 'reliability' of this hacker. "

A SQL Injection attack occurs when a database-backed web application does not filter inappropriate user input and executes that input against the database. For example, suppose a phone directory application asks for last name as part of employee lookup. A hacker might type "Select social_security number from employees" instead of entering a last name and the query against the database, if correctly structured, could return confidential information.

I did not want to pass along my credit card number to a hacking site, so I asked our security consulting firm, Third Brigade to check it out.

The attacks noted on the Eastern European website started early in March and by Wednesday March 12, 2008, 10,000 Web pages were compromised. In the first step, dynamic web pages are infected with a link to a malicious JavaScript file. A file downloads and executes code that tries to install password-stealing software on the desktops of people who visit the sites. Users who visit one of these infected websites may unknowingly execute malicious code. This code attempts to exploit known vulnerabilities for which patches are available but may not have been applied to the victim's system. Once the code executes on the desktop, specific malware is downloaded which sends passwords and other sensitive information to the attacker.

Here's a diagram of how it's done:

Step 1: The attacker attacks the Web Site using SQL injection. This involves information gathering queries followed by the specific attack that injects the link. The link is injected into string fields of the database so that dynamic web page content generated from the database will contain the link.

Step 2: Once the attack is successful, the attacker injects malicious code to the Web Site.

Step 3, 4 and 5: Users visit the website and are redirected to a malicious site which attacks the desktop using known vulnerabilities in operating system, browser and ActiveX plugins.

Step 6: Once successful, the attacker connects to various other sites to download malware (keylogger, password stealer etc)

Step 7: The attacker has sensitive information and has complete control of the desktop

Here's a diagram of how to protect against it:

* At BIDMC, we chose to implement Third Brigade's Host Based Intrusion Protection software, the Cool Technology of the Week. Third Brigade’s SQL Injection smart filter provides generic protection against SQL Injection attacks.

* In addition, Third Brigade has released a specific exploit filter which identifies if a Web Site has been compromised and is serving malicious content to unsuspecting users.

*Third Brigade provides protection against these Web Site attacks that are highly sophisticated and in some cases encoded using evasive techniques like URI encoding, double encoding, mixed case and non minimal UTF-8 encoding.

* Install filters for known vulnerabilities in Browsers, Operating Systems and ActiveX Plugins

* Install filters which prevent the user from accessing sites serving malicious pages. In this case, we released a specific protection which detects if the user visits a site that has malicious javascript in it.

* Install filters which block domains which download the malware on the target machine.

* Install filters detecting existence of known malware on the machine.

We're deploying these Third Brigade agents on our web clusters and SQL clusters so that even vulnerable applications and operating systems will be protected against many types of attacks. As I've said before, security is a journey requiring constant vigilance. The Third Brigade Host based Intrusion Protection systems are important components that we'll incorporate into all our future application releases.

Buying Great Tea

This is an "I Tea" rather than "IT" blog entry. Just as I've written about Sake, here's a guide to buying great tea.

6 years ago when I became vegan, I also eliminated coffee and most caffeine from my lifestyle. Other than the 2 weeks of headaches, insomnia, GI upset, and irritability, quitting coffee was no problem at all.

I replaced coffee with green tea. To get a sense of the caffeine in beverages, here's a helpful chart:

Brewed coffee (8 oz) 60-120 mg
Double espresso (2oz) 45-100 mg
Red Bull (8.2 oz) 80 mg
Jolt 71 mg
Instant coffee (8 oz) 70 mg
Pepsi One 55 mg
Mountain Dew 55 mg
Tea - black (8 oz) 45 mg
Pepsi (12 oz can) 38 mg
Coca Cola (12 oz can) 34 mg
Tea - oolong (8oz) 30 mg
Tea - green (8 oz) 20 mg
Tea - white (8 oz) 15 mg
Decaf coffee (8 oz) 1-5 mg

Thus, I went from 4 cups of strong brewed coffee (480mg caffeine) to 2 cups of green tea (40mg caffeine) per day. If I stop drinking green tea, I have no withdrawal symptoms at all.

The major categories of "true" tea are Black tea, Oolong tea, Green tea and White tea, which are made from the leaves of Camellia sinensis. Herbal teas are made from numerous ingredients which may or may not include caffeine.

White tea, such as Silver Needle, is made from fresh leaves with minimal oxidation and very little processing.
Green tea, such as Japanese Sencha, has minimal oxidation during processing.
Oolong , such as Chinese Guan Yin Buddha, ranges from 10% to 70% oxidation.
Black tea, such as Russian Caravan, is the most oxidized and typically has the most caffeine.

Green tea has a subtle flavor and comes in various grades.

Gyokuro- The highest grade Japanese green tea which is cultivated in the shade. It has a very, complex, grassy flavor.

Matcha - A ground green tea, grown in a similar manner to Gyokuro. It is used in the formal Japanese tea ceremony and as the flavoring in green tea ice cream. It is not typically consumed as everyday tea.

Sencha - The first and second seasonal growth (flush) of green tea, which is made from leaves that are exposed directly to sunlight. Sencha is served as a special occasion household tea in Japan.

Bancha - The third or fourth flush of green tea, harvested between summer and autumn. Bancha is the everyday tea of Japan.

Genmai cha - Bancha or sometimes Sencha mixed with roasted brown rice. Match is typically added to make the color a more intense green.

Hoji cha - A strong charcoal roasted green tea.

Kukicha - "Stick tea" made from tea stalks by harvesting buds, leaves and twigs.

I drink all of these but Gyokuro is my weekend tea, Sencha is my morning tea, and Kukicha is my every day tea.

Here's how to brew the perfect cup of tea:

Start with cold, clean tasting water. I use reverse osmosis filtered water since our Wellesley water is filled with carbonates, magnesium, and iron.

Pre-heat the tea pot by rinsing it with hot water. Add 1 teaspoon of tea leaves per 8 ounces of water. White and Green tea require lower brewing temperatures, much cooler than boiling water. Steep time is important. The longer a tea steeps, the more bitter it gets. To make stronger tea, add more tea leaves, not hotter water or a longer steep time. Here's a rough guide

White Tea 180 degrees F 4 minutes
Green Tea 160 degrees F 2 minutes
Oolong Tea 190 degrees F 4 minutes
Black Tea Rolling Boil 4 minutes

My favorite green tea is Gyokuro Asahi, a particularly flavorful green tea. I brew it at 160 degrees for 2 minutes in tea pots from Iwachu, a Japanese ironworks known for its great teaware.

Now you know how to make a great cup of tea!

Social Networking Software for Performance Improvement

I've written several articles about the IT tools needed for performance measurement and the use of social networking tools for collaboration

Recently, Beth Israel Deaconess melded these two concepts to create a community wide collaboration for performance improvement.

The idea is simple. If problems can be solved by investigating the root cause in real time instead of just developing workarounds long after the incident, performance can be improved significantly. Our requirements for an IT solution were

1. A threaded discussion forum with issue tracking features that anyone in the organization could add/edit/view
2. Workflow which enabled escalation, documentation of responses, and a history of all entries related to the thread
3. Easy access to incident reporting and adverse event tracking systems for documentation of patient care related issues

Dell successfully did something like this with their IdeaStorms site which enables users to propose performance improvement ideas and then promote/demote them via voting. Dell's site was created by SalesForce.com using their Software as a Service platform. We considered using a Salesforce.com platform but we needed tight integration with our existing applications.

We had to implement this infrastructure on a very tight timeframe at very low cost. We wanted a toolkit that integrated with Active Directory for authentication, was compatible with all browsers/operating systems, and leveraged the talents of our existing developers.

We chose to use Windows SharePoint Services 3.0 which is integrated in Windows Server 2003 and does not require additional licensing fees. We found the platform to be a good foundation for the project, especially because our developers could leverage its out-of-the-box collaboration and communication features. Over the 2 weeks of development time, we found that requirements changed frequently as we delivered functionality that prompted users to brainstorm about additional possibilities. Rather than a traditional development project that used specifications set in stone, this project was a Rapid Application Development exercise of creating a prototype, enhancing it, testing it, gathering feedback, enhancing it, testing it, etc.

The end result was a portal called the BIDMC Spirit Portal with easy access to all our issue tracking applications, including problem logging, resolution workflow, a blog for successful cases and enterprise wide collaboration. A typical workflow is illustrated by our "Case of the Week"

Problem:
At 8:30 a.m. a staff nurse on Farr 9 needed to page the medical house staff with a question about a patient admitted overnight from the Emergency Department (patient arrived on floor between 3:30 and 4:00 a.m.). The nurse paged the resident listed as covering, but that beeper was forwarded to another resident who stated he was not covering. That resident instructed the nurse to call another resident who also stated she was not covering. The nurse paged the attending physician of record who gave the nurse two additional options to page. At this point, the Nurse Manager on Farr 9 became involved, entered the issue in the web-based BIDMC Spirit portal and paged the Chief Medical Resident for help in determining coverage.


Root Cause:
There were a higher number of medical admissions than usual overnight. The patient was assigned to a different medical firm (team of residents, interns, medical students and attending physicians) than the usual firm that usually covers Farr 9 patients. The provider order entry order set did not indicate the correct firm coverage.

Solution After Investigation:
The immediate issue was fixed and the correct team assignment was notified, but it took 30-45 minutes.

Action Plan:
A Hospitalist and the Chief Medical Resident, worked on solutions with the Nurse Manager to prevent the issue from occurring again.

• Medical firms were reassigned to support increased medical volume on Farr 9.

• Farr 9 RN staff were educated about medical staff coverage and how to find medical call schedules on the intranet. We also ensured all medical teams had entries in our web-based paging system.

• Medical house staff updated our order entry order sets to accurately reflect team coverage.

• As a back up, the medical house staff agreed to either evaluate the patient if a critical issue is occurring or locate correct coverage as opposed to the nurse another intern/resident to page.

• We automated the paging system so that it generates an automated alert to the medical admitting resident once a bed is assigned for patients admitted from the Emergency Department

All of this was coordinated via BIDMC Spirit Portal discussion forums, blogs, and issue tracking applications.

As you can see, Social networking meets Performance Improvement meets Rapid Application Development. This infrastructure, which has now been used to support 300 real time problem solving events, again demonstrates the power of Web 2.0 for the healthcare enterprise!

Electronic Health Records for Non-owned Doctors - Creating the Model Office

There are many reasons for implementing electronic health records - enhancing quality, clinical integration, reducing redundant testing, and building workflow efficiency with technologies such as e-Prescribing.

Today, we have 300 closely affiliated non-owned doctors in 150 practices. Although these doctors are not employed by our system, they are part of the Beth Israel Deaconess Physician's Organization (BIDPO) which is a coordinated provider network of nearly 1500 physicians. Among other quality and safety measures, BIDPO focuses its Pay for Performance efforts on advanced diabetes care, appropriate radiology test ordering, and use of e-Prescribing.

Gathering metrics about physician practices requires consistent clinical and process data about the care delivered. This means that all our practices should maintain ICD9 codified problem lists, accurate medication histories, standard-based result reporting, and structured data about lifestyle choices such as smoking behavior. Unless our clinicians have a consistent way to record this data, quality and pay for performance reporting will be impossible since clinicians would implement their own unique ways of recording problems, medications, allergies and notes.

Hence, we're designing the "model office" configuration for eClinicalWorks 8.0, the version of the EMR software we'll be implementing. This means that before our first goal live, we're developing data dictionaries that will be used in all practices. We're loading the decision support rules which will provide identical alerts and reminders to every clinician. We're integrating all the lessons learned from the Massachusetts eHealth Collaborative rollouts to design the idealized configuration of each EMR screen which will ensure the best quality data capture and enhance the likelihood that we can do performance measurement as a clinically integrated community of clinicians.

Part of our model office also includes idealized workflow made possible by interoperability. Each office will have

1. New England Health EDI Network (NEHEN) which provides electronic links to our regional payers for HIPAA transactions including benefits/eligibility, referral/authorization, claim submission, and claim status
2. MA-Share RxGateway for e-Prescribing features such as formulary enforcement, community drug history, and routing to retail/mail order pharmacies.
3. Results interfaces for labs and radiology from local hospitals
4. Ordering of BIDMC-based testing including SafeMed radiology decision support
5. Quality and performance reporting via build in eClinicalWorks 8.0 data query system. Since our model office includes a standard configuration of all data dictionaries and input screens, we'll be able to use a federated approach to performance measurement. Here's how it works.
a. BIDPO has been given the authority to collect performance data by all BIDPO clinicians using eClinicalWorks
b. BIDPO devises a query such as "How many patients in each practice with diabetes have a hemoglobin a1c greater than 7"
c. At night, while the office is closed, the query runs on each clinician's database
d. The aggregate counts (not individual patient identified data) are returned to the medical director for use in pay for performance measurement and medical management

Creating the model office ensures that patients will receive the same quality care wherever they go, that doctors will be empowered with decision support tools, and that we'll be able to measure performance at all our practices as if they were a single integrated entity. We believe the up front work to design the model office will make our use of eClinicalWorks more effective, make training easier, and serve as a foundation for all our interoperability efforts. As we complete the model office designs, I'll post the details.

Records Management policies

Developing comprehensive, enterprise-wide policies by consensus in any organization is challenging. Over the past 3 years, BIDMC has developed a policy on records management that I think is valuable to share. This policy ensures compliance with applicable laws and regulations, accreditation standards, and additional requirements governing the management of records that are created or received for patient care, research activities, hospital operations, legal, or other purposes.

Given recent high-profile corporate scandals involving the inappropriate management of business records such as Arthur Andersen (Enron), Morgan Stanley (Sunbeam), and even the recent inappropriate disposal of hospital records in Florida, corporations are realizing the need to establish and enforce vigorous records management policies. This is especially important within highly-regulated industries like healthcare, which are subject to a wide variety of competing (and sometimes conflicting) Federal, State, local, and other standards.

At BIDMC, the Office of Business Conduct, Office of General Counsel, and Health Information Management collaborated to create a records management policy for BIDMC.

They also began the careful and time-consuming process of coordinating with specific departments to research and identify the appropriate standards governing the records created and used by each department. To begin, they prioritized records related to patient care and human resources, because records in these two areas are very highly regulated and often contain particularly sensitive information. Work with additional departments is continuing.

Important aspects of the new policy include –

*Coverage of records related to patient care including those maintained apart from the hospital’s official medical record, e.g. radiology films, scans, EEG/EKG tracings and so forth.

*Coverage of records related to non-patient care activities that serve research, legal, business, and other operational functions.

*Destruction procedures for hospital medical records as well as other medical, financial, and business records that contain personal information of our patients and employees.

*Formal procedures for the management of records relevant to an investigation, audit, or other legal proceeding.

*The policy applies equally to electronic and paper records and provides guidance for the appropriate use of digital scanning and other imaging technologies to replace original documents.

*It clarifies the retention requirements for e-mail and other communications containing important messages of significant business, legal, or medical value.

*Ensures compliance with new e-discovery laws such as the December 2006 Federal “Rules of Civil Procedure” that clarified the rights and obligations of parties related to the discovery of electronic records.

*It promotes efficient and cost-effective records management techniques. The proposed policy will prevent the unnecessary accumulation of records that are no longer needed or otherwise useful. At BIDMC, we spend millions per year storing paper and electronic records.

*Requires appropriate handling of the personal information entrusted to BIDMC by our patients and employees and thereby decrease their risk of identity theft.

The entire text of the policy and its supporting appendix A and appendix B took us years to develop. Hopefully, our work on this policy will be helpful to you.

Cool Technology of the Week

Numerous authors have suggested that six degrees of separation connect every human on the planet.

In healthcare IT, I'm convinced it's one degree, since we all seem to know each other.

Understanding the six degrees of separation of healthcare in Eastern Massachusetts can be challenging with our numerous providers, private payers, public payers, and academic affiliations.

The Cool Technology of the week is OrgScope, an organizational
mapping tool invented by the folks at NetAge. It's built on StarTree,
an elegant hyperbolic viewer originally invented at Xerox PARC by
Ramana Rao. NetAge has used OrgScope to map the Boston Healthcare
environment.

What's a hyperbolic viewer? Per one leader in the development of hyperbolic geometry, David P. Dobkin of Princeton, hyperbolic space provides a new perspective on viewing objects - "As you fly toward things, you get more and more detail." It's a bit like the New Yorker cover shown above - as you change focus, the scale around you changes proportionally, increasing the detail of your immediate environment and making more distant objects disappear.

To test it out, click on the Boston Healthcare Network initial model - Tree version , then on Harvard University, then on Harvard Medical School. Drag the Harvard Medical School icon to the center of your screen. Try moving the Harvard Medical School icon counter clockwise to expand the network and clockwise to shrink it. You'll find Partners and Caregroup, then see how Beth Israel Deaconess, where I work, relates to the entire network.

In addition to organizational network modeling, OrgScope includes analytics to help understand the layers of organizations.

Potential applications of this technology include visualizing the details of people within organizations, trust relationships within a health information exchange, and even the nationwide health information network.

I found this hyperbolic viewer much easier than an org chart for navigating a large number of complex relationships and look forward to the potential uses of this technology for visualizing our increasing connectedness in healthcare.

Risky Business

This is one of those episodic blog entries that delves into some aspect of my personal life. I've often asked if I'm a risk taker, given my alpine mountaineering passion, rock/ice climbing activities, and my travel around the world

Here's the way I think about my life. Let's say there is an activity that results in 100% chance of death (mortality). Let's call that 1 Mort.

Is it possible to skydive without parachute and survive? Highly unlikely, so such an activity is probably .99999 Mort.

How about flying on domestic airlines? Over the past 20 years, the fatalities per plane flight in the US have been 1 in 7 million. This means that if I took a non-stop flight every day, I would average 19,000 years before succumbing to a fatal crash. Even frequent flyers face minimal cumulative risk over their careers. What's my risk? Last year I took 166 non-stop flights which is .0000237 Morts - 23.7 microMorts.

I'm 45 years old, so let's look at the CDC mortality rate data.

All causes of death 3.52 milliMorts
1 Malignant neoplasms .867 milliMorts
2 Diseases of heart .696 milliMorts
3 Accidents (unintentional injuries) .426 milliMorts
4 Intentional self-harm (suicide) .169 milliMorts
5 Chronic liver disease and cirrhosis .151 milliMorts
6 Cerebrovascular diseases .119 milliMorts
7 Human immunodeficiency virus (HIV) disease .115 milliMorts
8 Diabetes mellitus .10 milliMorts
9 Chronic lower respiratory diseases (J40-J47) .056 milliMorts
10 Assault (homicide) .053 milliMorts

The National Safety Council (NSC) publishes an annual summery of fatalities and accident statistics called Injury Facts.

All accidents have a risk of .56 milliMorts , which is very close to the CDC accident data above for 45 year olds of .426 milliMorts

Motor vehicle accidents are .15 milliMorts per year

My life insurance policy prohibits me from from operating a private plane, skydiving and scuba diving. Per my review of the literature

Operating a single engine private aircraft has a risk of .35 milliMorts
Skydiving has a risk of .30 milliMorts
Scuba diving has a risk of .02 milliMorts

Thus, perceived high risk behaviors are just twice as risky as driving in Boston and are half the risk of having a 45 year old heart.

The medical literature suggests that alpine mountaineering causes 1.87 deaths per 1000 climbing days. Last year, I did 5 days of alpine climbing. That's 9.35 milliMorts compared to the baseline all cause rate for 45 year olds of 3.52 milliMorts

The bottom line is that my highest risk behavior is just 2.65 times more risky than breathing as a 45 year old.

Thus, for the moment, I'll continue to climb, fly commercial aircraft, and do a few alpine ascents every year. At 9.35 milliMorts per year, I've got 107 years to go before I reach a Mort!

Update on the 2008 Keep Me Awake Projects

Now that Spring is around the corner, it's time for an update on the "keep me awake at night projects" for 2008. Here's the first quarter progress on these 10 challenging issues:

1. Electronic Health Records for non-owned doctors - We've spent the first few months of 2008 planning the project, refining the budget, building partnerships, and establishing project management. I believe this extra up front time (and investment) will enhance our likelihood of success and avoid major time/budget overruns in the future. Each week I post a blog entry about some aspect of this project and the entire 11 article "pre-golive" series will be done by April. We're on track for a pilot this Summer, but there are still many unknowns because few hospitals in the US have offered Software as a Service Electronic Health Records to their non-owned clinicians. This project will still keep me up at night until our pilot sites are completed and we've secured all the funding needed for full production rollout.

2. Storage as a utility - After a 2 year journey, we've completed our storage as a utility designs, creating 3 tiers of storage using EMC SAN, SATA NAS, Data Domain de-duplication storage and Acopia file virtualization. I am now confident that we can store, archive, backup and retrieve the institution's data in two geographically distant data centers, providing the rapid recovery times mandated by our disaster recovery plan. I can remove this issue from my keep awake list.

3. e-Prescribing - BIDMC recently completed its latest release of e-Prescribing functionality within our home-built enterprise ambulatory electronic health record. Now a clinician can view payer formularies, retrieve community drug history with allergy/drug interaction checking, do medication reconciliation , route prescriptions to retail/mail order pharmacies and check patient insurance eligibility while writing for a medication. This leverages the state-wide electronic prescribing gateway built by MA-Share which connects payers, pharmacies, RxHub, Surescripts, and providers. In early March, Massachusetts was recognized as the number one e-Prescriber in the country . Although I will work very hard to continue the rollout and adoption of ePrescribing in Massachusetts, I can remove this issue from the keep awake list.

4. Data sharing for clinical care among a community of caregivers - We've recently gone live with the notion of pushing standards-based clinical summaries among caregivers throughout the state . This early stage is a pilot and although it has been technologically successful, the real success will be increasing the number of providers and hospitals using it. This keep awake project has gone from a technology issue to an education/communication issue, building a community of users sending summaries for patient care coordination. It's still on my list.

5. Security - Security will always be on my keep awake list. Over the first quarter of 2008, we've implemented a pilot of host-based intrusion detection from Third Brigade. The idea is that we have software running on each server which prevents common attacks such as buffer overflow, cross-server scripting and SQL Injection. This adds a layer of protection that defends us against attacks even if the operating system or application is vulnerable. Our plan is to rollout this technology throughout the data center and eventually across all our desktops. As fast as we innovate, hackers and spammers innovate, so we'll vigilantly keep adding more security projects every year.

6. RFID and Bar coding - We recently completed our enterprise rollout of Cisco Location Based Services and Pango Networks active RFID tags for tracking mobile assets such as wheelchairs, IV pumps, and EKG machines. Today we have about 350 tagged assets and recently acquired 900 tags to deploy in the upcoming weeks. Our FY08 operating budget includes an additional 500 tags to be purchased by the end of September. For bar coding, we recently completed our pay for performance goal of a bar coded wrist band on every ED/inpatient/ambulatory surgery patient and by June we will have bar coded our unit dose medications, enabling us to develop an electronic medication administration record (eMAR) system by next year. Once that eMAR system is live, I will remote this one from my keep awake list.

7. Providing decision support - I recently provided an update on our Performance Measurement activities and Decision Support tools . The keep me awake portion of this is our upcoming demonstration of secure, deidentified, data sharing for decision support and clinical research across all the Harvard hospitals as part of the Clinical Translational Science Awards (CTSA) program of NIH. This effort requires us to obtain IRB approval from several Harvard hospitals, build standards-based middleware layer, and create an easy to use graphical tool for data browsing. No patient identified data will be involved, but the technology, policy, and organizational challenges needed to go live by July 1 will keep me awake.

8. Compliance requirements for revenue cycle workflows - Every day new compliance requirements add more projects to the IT department. Recently I was asked if all our applications are Payment Card Industry (PCI) compliant. The good news is that we do not store credit card data on hospital systems, we only have secure credit card transactions running over our networks to third party firms. Without persistent credit card data, our exposure to fraud is lessened. Over the first quarter, we've added more electronic data interchange transactions via our NEHEN gateway, enhanced our coding software/interfaces, and provided all the necessary data to meet our state/federal reporting obligations. Thus, for the moment, I feel good about our compliance and it is not keeping me awake.

9. Internal and external websites - We've been hard at work implementing a new web content management system and I am confident that in 2008 we will launch our new external and internal websites. There is a great deal of work to do to transition from static HTML pages and our home built content management system, so this one stays on the awake list until Fall.

10. Disaster recovery - We've made substantial progress with our disaster recovery efforts and we now have our most mission critical systems backed up by a redundant data center. Just as security is journey, so is disaster recovery. The entire plan, our progress, and our phasing for the future is outlined in this presentation. Although disaster recovery will always be on the awake list, I can rest a bit more knowing that our major clinical systems are not only redundant within a data center, they are redundant across two data centers.

So the report card on the first quarter is that a few of the keep me awake projects are off the list, and a few more will be off the list by Summer and Fall. As long as we are making forward progress on all these items, I'm happy, since the trajectory is more important than our position on any given day.

Electronic Health Records for non-owned doctors - staffing the project

This is my sixth post in the series about Electronic Health Records for non-owned doctors. This week, I'll discuss the complexities of staffing the project, since we've had to weigh insourcing verses outsourcing, costs, and service levels to arrive at a balanced staffing plan. We've divided the staffing into 10 categories and below I describe the strategy for each.

In creating this staffing model, we had several considerations

*The project scope is not fixed. We're starting with 300 clinicians and may implement over 1000. Thus, we need a staffing model which is scalable on demand. We may need to flex the size of our teams up or down depending on implementation schedules.
*In Massachusetts, at this time, it is challenging to hire and maintain healthcare informatics staff because of intense competition among hospitals implementing CPOE/EHRs and local software companies expanding their healthcare IT workforce
*Outsourcing can be a way to rapidly expand capacity but it requires diligent management and tends to be more expensive than hiring internal staff.

1. Project Management and Financial Management
We have a fulltime Project Director and have leveraged our existing IS fiscal staff to manage the budgets. We will partner with the Massachusetts eHealth Collaborative to operate a project management office under the direction of our Project Director but jointly and flexibly staffed with BID and MAeHC personnel. This will allow us to:
a. Complement our existing knowledge of hospital and employed-practice deployments with outside expertise regarding non-owned ambulatory practices
b. Ramp up staff strength quickly as implementation intensity grows
c. Ramp down smoothly as deployment transitions to long-term support. This model will also allow us to rapidly and seamlessly plug in the temporary project management and staffing gaps that will inevitably arise during the course of the project.

2. Technical Design and Engineering
We elected to insource design and engineering for our servers, storage, and network design. We collaborated with our vendors - HP, EMC, and Cisco, as well as our infrastructure implementation partner, Concordant, on these designs. We assigned .25 FTE of the manager of our Ambulatory Applications group to this task.

3. Central Site Construction
We elected to outsource construction of the central hosting facility to Concordant for a fixed price. They acquired the co-location space, installed all equipment, and took responsibility for establishing power/cooling/network connectivity to the site. They will also manage the central site during the pilot phase. We are paying for a deliverable rather than paying per diem rates for time or purchasing FTEs.

4. Office Hardware Deployment
We elected to outsource office hardware deployment to Concordant by purchasing a team of people which scales in direct relation to the number of offices we are implementing. Purchasing a deployment team rather than working on per diem rates means that we paying for FTEs assigned in direct relation to the deliverables.

5. Practice Consultation
We elected to insource and outsource practice consultation. An MAeHC senior consultant will directly manage the practice consultant team, which will comprise both BID and MAeHC consultants. These practice consultants will be assigned to individual practices and will provide end-to-end project management of practice-level implementations, to ensure that all activities associated with the implementation are synchronized. They will also work with individual practices on optimizing workflow and translating that optimized workflow into appropriate software configuration, hardware layout, and training approach. As with the project management function, this is a flexible insource/outsource model that allows us to scale up and down rapidly, tap into existing expertise and apply it to the project right away, and maintain an adaptive but robust capability to meet program changes as they arise.

6. Training
We elected to insource and outsource training. eClinicalWorks will provide the training for all our initial pilot sites then train our trainers. Going forward a combination of eCW and insourced trainers will serve our sites, with supervision/quality control of all training to be provided by eClinicalWorks.

7. Central Site Operations
We elected to outsource central site operations to Concordant via a "lights out data center" model coupled with systems and application administration. They provide monitoring tools and problem escalation 24x7 rather than hiring a specific number of staff to manage our installation. This enables us to leverage the fact that they are providing support to other customers and projects, keeping our costs low and avoiding the need for us to hire fractional FTEs to provide data center support. The challenge with hiring internal staff to do this is that we expect data center needs to be higher during our initial implementation because of the build/change activities, then markedly reduce during our steady state operations. Outsourcing this function to Concordant, which spreads FTEs over many customers, enables us to flex our needs easily.

8. Help Desk, Tier 1 and Tier 2 Support
We elected to insource and outsource telephone support. Concordant will be the initial single point of contact for all phone calls, doing Tier 1 support such as password resets and then triaging Tier 2 Support . They will handle infrastructure Tier 2 issues and escalate others (such as EHR best practice questions) to our insourced staff. Our staff will escalate eClinicalWorks specific issues to eCW as needed. By focusing on resolving as many questions as possible remotely and dividing support between Concordant, our staff, and eCW, clearly defining the tasks of each group, we minimize our costs.

9. Field Support
We elected to outsource field support to Concordant, using a shared staff model. This optimizes our costs, coverage and flexibility since here again Concordant spreads FTEs over multiple customers.

10. Security auditing
Security has been built into our project from the very beginning as part of our infrastructure design, application configuration, and staffing model. We elected to outsource security auditing to an expert ethical hacking and security firm, Third Brigade for a fixed price. By hiring an expert group to do this, we provide another layer of vigilance and control, ensuring we have an outside party validating our configurations, providing host based intrusion protection, and monitoring our systems.

Thus, by dividing the staffing of the project among the members of our "dream team" - BIDPO/BIDMC, Concordant, Massachusetts eHealth Collaborative, eClinicalWorks and Third Brigade - we have achieved an affordable, scalable, balanced insource/outsourcing staffing model.

More to come as we test this model in production this Summer!

Conflicts of Interest 2008

In all of my jobs and activities I try to be very transparent about all I do, both good and bad. In the interest of full disclosure, I thought it would be useful to describe all my affiliations, my disclosures, and my attempts to avoid conflicts of interest.

Having just finished my tax returns, I can tell you that my income is just a W2 from BIDMC, which invoices Harvard for the time I spend there. I did this to avoid having two paychecks and FICA deducted twice.

Any income I receive from speaking engagements or consulting I donate to BIDMC to minimize conflict of interest.

In 1993, I created a family trust and all my savings/investments are managed in that trust by an independent third party. The trust does not invest in technology related stocked or bonds.

Harvard has very strict rules about working with for-profit companies. No Harvard logos can be used in association with a commercial product. No organizational endorsements are permitted i.e. "Harvard thinks this is the best product on the market!". Case studies are fine and objective comments about functionality are permissible i.e. "I tested 5 products and found product x met my specific functional criteria."

Anytime I do case studies, they are reviewed by Public Affairs at Harvard Medical School for appropriateness. Anytime I have a question about serving as an adviser or board member, I inform Corporate Compliance at CareGroup and seek permission. Here are the roles I serve in for-profit companies and the financial arrangements:

Google Health Advisory Council - For the past year, I have served as an unpaid advisor on the Google Health Advisory Council. Since several individuals working on the Council are from non-profits and share my feelings about objectivity, Google circulates a form among the group so we can specify our preferences about being paid for our time. I have not accepted any personal compensation for the meetings, phone calls, policy advice and standards work.

Healthline Medical Advisory Board - Healthline has a built a search engine that uses controlled vocabularies/ontologies in an attempt to deliver more specific search results to users i.e. it knows that brains contain neurons and thus a search on neurons also looks at diseases of the brain. I serve on their advisory board to offer suggestions about how to incorporate medical knowledge in the search process. I have not accepted any compensation. I was asked to serve as an adviser by one of my former professors, who leads the Medical Advisory Board.

Whole Health Advisory Board - Whole Health provides workplace based medical services and wellness management. They use electronic health records throughout their clinics to optimize quality and safety. I was asked to serve as an adviser by a colleague at Harvard Business School. I have not accepted any compensation.

SafeMed Board of Directors - I rarely serve on the Boards of companies, but I made an exception with SafeMed because I so strongly believe in their mission - to make decision support available to doctors, patients and payers using a web-services architecture. They are the decision support behind Google Health, informing patients about medication interactions. I have not received any cash compensation for my Board service and will receive stock options, as is typical for Board members. These options will be declared with Harvard and CareGroup and I will recuse myself from any decisions regarding purchasing of SafeMed products. At present, CareGroup uses SafeMed as part of our Blue Cross funded pay for performance initiatives and CareGroup did not purchase products from SafeMed as part of that effort.

ePocrates Board of Directors - ePocrates is handheld prescribing software that runs on Treos, Blackberries and the iPhone. Many attending physicians, residents, and medical students at Harvard download the free version of ePocrates. Since it is the most popular mobile software used by clinicians at BIDMC and Harvard, I agreed to serve on the Board of Directors. I have not received any cash compensation for my Board service and did receive stock options, as is typical for Board members. These options were declared on the Harvard and CareGroup conflict of interest statements. CareGroup and Harvard have not purchased any products from ePocrates and I will recuse myself from any purchase decisions if any product selection is done in the future.

Blackberry Advocate - In 2007, I flew to a New York City SOHO studio and spent a day speaking about the way mobile devices impact my life. It was unscripted and captured my honest feelings about life as a highly connected human. What did I do to keep my objectivity? If you go to the Blackberry website, you'll see that it calls me "John Halamka, CIO and Emergency Physician", specifically removing my institutional affiliations. I worked very hard with Harvard and CareGroup counsel to ensure all materials did not include organizational endorsements, logos, or sales literature. The Blackberry media campaign in magazines, videos, and elevators was very professionally done. I did my best to express my personal belief in the utility of wireless devices while minimizing conflicts. When I walked on the set, I was given union scale wages for the day ($100 plus residuals) as would be done for any TV appearance by an extra and I donated that to BIDMC as well as disclosed it to Harvard and BIDMC corporate compliance.

These are all my for-profit company activities. I hope my attempt to isolate any conflicts of interest sound reasonable to you. The only companies from this list that I have mentioned in my blog are Google and SafeMed. For my Google entry on February 28, 2008, I specifically mentioned my Advisory Council service. Regarding my SafeMed entry of November 12, 2007, I began my Board service on February 1, 2008, so this entry is my declaration.

If there is anything I can do to adhere to other best practices, ensuring my objectivity, let me know!

Cool Technology of the Week

This week, the Massachusetts Health Information Exchange, MA-SHARE, went live with secure sharing of clinical summary records from provider to provider at Beth Israel Deaconess Medical Center, Lahey Clinic, Northeast Health Systems and Boston Children's using the recently recognized national standards harmonized by HITSP.

Here's how it works.

When a patient registers for care, their primary care giver is captured by the registration clerk. We store the National Provider Indentifier of the clinician.

When a patient is discharged from the hospital or the emergency department, a comprehensive clinical summary is automatically prepared in Continuity of Care Document format including medications, diagnoses, procedures, and all followup issues.

This electronic document is sent via SOAP/HTTPS to a hospital-hosted gateway. That gateway uses the National Provider Identifier to look up the primary care giver's institutional affiliation in our statewide provider index. The summary is then routed to the gateway of the receiving institution via the internet via SOAP/HTTPS. Once it arrives, it is routed to the clinician's Electronic Health Record, Fax machine, or secure Email box.

The end result is that we are ensuring appropriate followup, medication reconciliation, and communication among physicians using a standards-based electronic document and the internet. Further details are available in this overview.

In Masssachusetts, we exchange 100 million HIPAA transactions per year via our New England Health EDI Network (NEHEN) gateway. Massachusetts was recently recognized as the #1 e-Prescriber in the country and part of our accelerated adoption has been our use of an e-Prescribing Gateway. Now the secure summary exchange gateway is live. All three of these gateways are built on the same application platform, which is running at hospital systems and payers throughout the state. There are no transaction fees, and no charges to the patient. The software is developed and maintained via contributions from provider and payer organizations which derive value from its use. In the case of secure document exchange, we can eliminate mailing discharge summaries and ED notes, saving hospitals like BIDMC $100,000 per year.

National standards, the internet, and a business model for health information exchange in production. That's cool!

Buying Great Sake

On occasion, I'll write blog entries about my avocations - rock and ice climbing, playing the Japanese flute, drinking green tea, winemaking, mushroom hunting and finding the perfect Sake.

Today's entry is about buying Sake, which can be very confusing.

Sake is made from rice, water, Koji (Aspergillus Oryzae, a starch dissolving mold), and yeast. It is not a distilled beverage and is generally between 15% and 17% alcohol. Sake takes approximately one month to brew, followed by a six month aging period and is meant to be consumed soon after purchase. Try to buy a sake bottled within the last year. There is no such thing as a vintage Sake.

Interpreting a Sake label can be challenging. The most important elements reflect the addition of alcohol and the processing of the rice. You'll find one of two designations about alcohol

Honjozo - Sake to which a small amount of distilled alcohol is added
Junmai- No added alcohol

Premium sake is brewed with special Sake rice in which the starch component (the shinpaku or "white heart") is concentrated at the center of the grain, with proteins, fats, and amino acids located toward the outside. With increased milling, sake makers can remove the fats, proteins, and amino acids that lead to unwanted flavors and aromas in the brewing process. Ginjo-shu (premium Sake) has at least 40% or more milled away. Daiginjo (super premium Sake) has at least 50% or more milled away. Sake bottles list Seimaibuai, the amount of the rice grain left after milling.

Another important element is the level of dryness. The Sake Meter Value (Nihonshu-do) is the specific gravity of a Sake. It indicates how much of the sugars created from the starches in the rice were converted to alcohol, and how much remained to contribute to sweetness. By historical convention, the higher the number, the drier the Sake. What is the range? In theory, it is open-ended. In practice, + 10 or so is quite dry, -4 or so is quite sweet, and +3 or so is neutral.

Acidity affects how the flavor spreads, and also the sensation of sweet and dry. The range is quite narrow, with 0.7 being low and 2.0 being quite high. 1.2 or so is average.

Good Sake is made from special Sake rice. There are dozens of types of Sake rice, which is different from eating rice, but only a handful that are truly worth remembering. The most important of these is Yamada Nishiki.

Yeast mainly affects fragrance, and then flavor. There are dozens of yeast strains, each with its own subtly different characteristics, mostly affecting fragrance, but also flavor. Learning to discern the characteristics of the various yeast strains is part of the fun of learning about Sake.

Other elements you may find on the label are

Amakuchi -Sweet in flavor
Jizake - Sake from smaller sake breweries
Karakuchi - Dry in flavor
Nigori - Unfiltered sake which has a white milky color
Yamahai - Sake brewed in a way that allows wild yeasts to grow. This is rarely done since the aroma is "gamey".

My favorite Sake is Wandering Poet, a Junmai Ginjo made by Rihaku Shuzo, Shimane Prefecture. Brewed slowly at low temperatures using traditional brewing techniques and Yamada Nishiki rice with 45% milled away, it has a well-rounded flavor, extraordinary fragrance, and a clean finish.

Let's analyze the label:
*Junmai Ginjo means no alcohol added and at least 40% of the rice grain has been milled away. I tend to prefer Junmai Ginjo instead of Junmai Daigingo, because the Ginjo has more intense flavor and character. I tend to eat a lot of raw/unrefined foods and I prefer my sake a little less refined.
*Sake Meter Value/Nihonshudo +3 means that this is neither overly dry nor sweet
*Alcohol 15.2% means that it is relatively low in alcohol for a sake
*Acidity 1.6 means that it is a crisp sake, relatively high in acid, that goes well with food
*Seimaibuai 55% means that 45% of the rice grain has been milled away
*Yamada Nishiki means that a premium rice has been used
*Yeast # 9 means that a specific strain of yeast was used. Many ginjo yeasts are #9-based strains which creates a great fragrance and a consistent fermentation.

Sakes are available at many liquor stores throughout the US, but often they are from inexpensive American producers. The best Sakes are available online and from a few regional specialty stores that import fine Sake. Interestingly Trader Joes often has fine Junmai Ginjo type Sakes at a great price.

Sake should be served chilled, not warmed. Warming is only done with inferior Sakes to hide the impurities.

So drop by your nearby Trader Joes, pickup a bottle of a Junmai Ginjo Sake and serve it chilled. You'll have a great Sake experience.

The CareGroup Network Outage

On November 13, 2002 at 1:45pm, Beth Israel Deaconess Medical Center went from the hospital of 2002 to the hospital of 1972. Network traffic stopped. No clinician could order medications or labs electronically. No decision support was available. Luckily, no patients were harmed. Here's the story of what happened and our lessons learned.

In the years after the 1996 merger of the Beth Israel and Deaconess Hospitals, operating losses caused several years of capital starvation (the network budget in 2002 was $50,000 for the entire $2 billion dollar enterprise). We were not available to invest in infrastructure, so our network components were beyond their useful life.

However, it was not this infrastructure underinvestment that was the root cause of the problem, it was my lack of enterprise network infrastructure knowledge. I did not know what I did not know. Here are the details:

1. Our Network topology was perfectly architected for 1996. Back in the days when the internet was a friendly place where all internal and external collaborators could be trusted, a switched core (layer 2) that transmitted all packets from place to place was a reasonable design. After 1996, the likelihood of denial of service attacks, trojans, or other malware meant that networks should be routed and highly segmented, isolating any bad actors to a constrained local area. At the time of our outage, a data flood in one part of the network propagated to every other part of the network - a bit like living downstream from a dam that collapsed. A well meaning researcher created a napster-like application that began exchanging hundreds of gigabytes of data via multicast to multiple collaborators. The entire network was so saturated that we could not diagnose the root cause of the data flood. I did not know that a switched core was a point of failure.

2. Our Network team was manged by a very smart engineer who did not share all his knowledge with the network team. Much of our network configuration was poorly documented. With the knowledge of our network isolated to one person, we had a single point of human failure. I did not know that this engineer was unfamiliar with the best practices for routed/redundant network cores, routed distribution layers and switched access layers isolated into vlans with quality of service configurations to prevent monopolization of bandwidth by any one user or application. We brought in a Cisco partner, Callisma, to document the network, but the network failure occurred before they were finished.

3. I did not know about spanning tree algorithms, hot standby routing protocols (HSRP), and Open Shortest Path First (OSPF). During the outage, I approved configuration changes that actually made the situation worse by causing spanning tree propagations, flooding the network with even more traffic.

4. I did not establish a relationship with the vendor (Cisco) that enabled them to warn me about our vulnerabilities. A relationship with a vendor can take many forms, ranging from a sales driven vendor/client adversarial relationship to a collaborative partnership. In 2002, Cisco was just another vendor we purchased from. Today they are a collaborative partner with a seat at the table when we plan new infrastructure.

5. I did not know that we needed "out of band" tools to gain insight into the problems with the network. In effect, we required the network to be functional to diagnose problems with the network.

6. We did not have a robust, tested downtime plan for a total network collapse. When the outage occurred, we rapidly designed new processes to transport lab results, orders, and other data via runners from place to place.

7. We did not have a robust communication plan for responding to a total network collapse. Email, web-based paging, portals, and anything that used the data network for communication was down. Voice mail broadcasts using our PBX and regular phones (not IP phones) turned out to save the day.

8. When we diagnosed the problem, we explored many root causes and made many changes in the hope that we'd find the magic bullet to cure the problem. In the end, we ended up fixing many basic structural problems in the network, which took 2 days and eventually solved the problem. A more expedient solution would have been to reverse all changes we made in our attempts to fix the network once we had stopped the internal attack. When a crisis occurs, making changes on top of changes can make diagnosis and remediation even more difficult.

9. We did not have an enterprise wide change control process to ensure that every network configuration, server addition, and software enhancement was documented and assessed for impact on security, stability, and performance. Today we have a weekly change control board that include includes all IS departments, Cisco engineering services, and IS leadership to assess and communicate all configuration changes.

10. I was risk averse and did not want to replace the leadership of the network team for fear that terminating our single point of human failure would result in an outage. The price of keeping the leadership in place was a worse outage. I should have acted sooner to bolster leadership of the team.

Despite the pain and stress of the outage, there was a "lemons to lemonade" ending. Without this incident, the medical center would never have realized the importance of investing in basic IT infrastructure. If not for the "perfect storm", we may have limped along with a marginal network for years.

Today, we receive annual capital funding to support a regular refresh of our technology base, and we are asked to introduce change at a pace that is manageable. People in the medical center still remember the outage and are more accepting of a tightly managed infrastructure such as locked down workstations, security precautions, disaster recovery initiatives, and maintenance downtime windows.

During and immediately following the event, I presented this overview of the outage to senior management. Shortly after the outage, I worked with CIO Magazine to document the events and lessons learned.

I hope that my experience will help other organizations prevent network and other IT outages in the future.

Electronic Health Records for non-owned doctors -scalable infrastructure

This is my fifth entry about our Electronic Health Record project for non-owned doctors. As I've described, the scope of the project is to implement a highly reliable, secure, feature rich, well supported, but affordable electronic health record for private practices. Today's entry is about building the scalable centralized Software as a Service hosting infrastructure to meet these goals.

A key design requirement for the project is scalability. Our projected customer base is 300 clinicians and we have a fixed start up budget. However, we must design the infrastructure in a way that can cost effectively support the smallest amount of adopters as well as scale to thousands if our project is wildly successful. We debated two possibilities (metaphorically speaking)

a. Build a hotel, not knowing if anyone will ever check-in
b. Build a housing development, where the limits of expansion are only defined by available land

We decided on choice "b", starting with a robust foundation and adding new equipment and storage as we add clinicians. We standardized our central site equipment on products from HP EMC and Cisco, with guidance from our infrastructure partner, Concordant, and our equipment supplier CDW, ensuring it was easy to plug in additional hardware on demand. We invested a significant amount of time designing the central hosting facility, doing it right the first time. Over the years, I've seen CIOs rush through the design phase, only having to rebuild the infrastructure later when application performance did not scale. We partnered with our vendors to build something special, that if successful, could be a model for other medical centers and communities.

Considerations in designing our hosting infrastructure included:
* Supporting a user base that is remote, unmanaged, and diverse. We need to be able to identify any performance issues via end to end monitoring of all components
* Meet important security and privacy restrictions, as well as address liability issues (who is responsible for what)
* Understand infrastructure costs for a) start up b) additional capacity that occurs in bursts or steps, and c) variable requirements as practices go live.
* Provide connectivity to external parties (labs, claims, etc...) through interfaces which create additional security and performance complexities
* Address the limitations and performance of "last mile" connectivity through publicly available internet access (DSL, cable, etc.)

The infrastructure choices we made are:

Virtualized servers - VMware was the natural choice because of the scalability design goals. VM and V-Motion technologies also play an important part in redundancy, failure recovery and disaster recovery

Physical services - We debated rack mounted verses blade servers and elected to use powerful small footprint HP rack servers connected to fast multi-tiered storage. We computed the economics of blade servers verses rack mounted servers and the use of VMWare made small powerful rack servers the most cost effective solution.

Storage - We purchased a Clariion CX3-20 series SAN. We will go live with 11.1TB of total storage (2.1TB of fast, Tier 1 storage for database transactions and 9TB of secondary, Tier 2 storage for files). A single CX3-20 will allow us to expand in a modular fashion to accommodate up to 1200 practices. We'll also be leveraging a disk to disk backup strategy, using tape only for disaster recovery.

Network Infrastructure – We implemented a high speed network backbone with multiple paths for redundancy using
* Cisco Integrated Services Routers (ISR) 2811s for internet connectivity
* Adaptive Security Appliances (ASA) 5520 for Firewalling, Intrusion Protection and IPSec VPN Client termination
* Catalyst 4948 Switches for Server connectivity and layer 3 routing
* MDS 9000 Series Multilayer SAN Switches for SAN connectivity

Security – We incorporated physical, technical and administrative controls to protect confidentiality, integrity and availability.

SSL Accelerators - We are using Array Networks TMX-2000, the hardware recommended by eClinicalWorks to optimize web server performance.

Redundancy & Disaster Recovery - One of the real challenges to this project is the price sensitivity of our private clinicians. We needed to build a world class system at a price that all clinicians could afford. Redundancy and disaster recovery is like life insurance - it's a great investment only if you need it. We had to balance our infrastructure investment with total cost of ownership, given the fixed hospital contribution and physician frugalness. In the end we used the equation

Risk=likelihood of bad events * impact of bad events.

We believe that it is much more likely that a component will fail than an entire data center be destroyed, so we elected to build a highly redundant infrastructure in a single data center for now, expanding to a secondary data center once we have sufficient clinicians signed up to fund the new infrastructure. Networking gear, servers, power and cabling are duplicated within a commercial co-location facility. Storage is disk to disk redundant. Tapes are moved offsite nightly. Once the hardware is up, we'll work with Concordant, Cisco, EMC, HP, Array Networks, and the Co-Location facility to test physical hardware and operating system/database software redundancy. Then we'll install eClinicalWorks and run the redundancy tests again. We'll also engage Third Brigade at that time for intrusion/security testing.

We've written a comprehensive disaster recovery plan and if we lost the co-location facility due to disaster, we would recover the tapes from offsite storage and build a replica of the hosting environment (VMWare plays a key role here) and restore the data. The recovery point and recovery time objectives for this plan will be clearly communicated to all who sign up for service. The customer base for our Software as a Service solution is mainly small practices which operate Monday through Friday 8am-6pm. Our disaster recovery plan includes a solid practice/workflow specific contingency/downtime plan. We will also perform a mock downtime as part of each implementation.

By creating a highly redundant single data center with rack mounted servers, two tiered storage, virtualization and offsite tape backup, we believe we've balanced scalability, affordability, and maintainability. We go live this Summer and I'll let you know if we were right!