The Challenges of a Software Legacy

Ever year as I prioritize new application development, I remind my governance committees that 80% of my staff resources are devoted to keeping existing systems stable, secure, and error free. These staff maintain infrastructure and add incremental improvements to support compliance with new rules, new standards, and new workflow requirements.

As I reflect on Microsoft's current challenges with Vista, I sympathize with their dilemma. On the one hand, the user community expects each upgrade to offer bold new features and innovation. On the other, users expect all their Windows 98, NT, 2000, and XP software to work flawlessly.

The amount of engineering required to ensure this backward compatibility is enormous and in large part explains the difficulty the Microsoft Operating System Development group has in releasing something that is boldly new.

Industry analysts point to the speed of innovation of Google or the fact that Canonical's Ubuntu is rapidly converging on the Windows feature set. Both Google and Canonical have the advantage of little legacy compatibility support.

I've experienced this same burden of a software legacy several times.

At Harvard Medical School, we introduced Mycourses.med.harvard.edu as the educational portal in 2001. Each year through Mycourses, we deliver 60,000 streaming videos, thousands of documents, and hundreds of simulations. Per my recent post about Educational Technology priorities, there is a desire to enhance usability and add many new features. The challenge is that we need to maintain existing features while innovating. This is like changing the wings on a 747 while it's flying. A perfect example is our Surveybuilder and Testbuilder. These web-based applications in Mycourses evolved over years based on hundreds of user feature requests. At this point, adding new features will likely break old features. Our best approach is to rewrite them from scratch, based on a streamlined set of user requirements. Thus, we'll evolve the existing Mycourses into a new portal framework, then rewrite the applications over time. This will be evolution rather than revolution. Some people will comment that our pace of innovation is slower in 2008 than it was in 2001. This is the reality of an existing legacy of highly functional software.

At BIDMC, we launched our intranet in 1998. At that point in web history an intranet was just a list of links and not a highly interactive Web 2.0 infrastructure supporting blogs, wikis, forums, and new media. In 2007 we introduced a new intranet based on many modern features such as single sign on, user customization, support for RSS feeds, and Sharepoint features. Today, of our 5000 users, 4000 use the old portal and 1000 use the new portal. Our user survey indicated that the average user just does not want to change. Learning a new portal is more effort than is justified by the new features. In 2008, we're relaunching the portal again, making the look and feel more similar to the old portal but supporting many new collaboration and content management features. In 1998, launching the intranet was simple because there was nothing to compare it with. In 2008, it's very hard because we have to support the legacy of highly functional old portal features while moving forward.

In both cases, we're leveraging our governance processes to build top down and bottom up support for change. We hope that by creating an urgency to change, a vision for the future and a guiding coalition, we'll be able to overcome the burden of our software legacies.

Thus, Vista may have its warts, but I understand the struggle Microsoft faces.

Electronic Health Records for Non-owned doctors - Support

This is my tenth entry about providing electronic health records for non-owned doctors. The previous entries have described the efforts to go from vision to live implementation. The subject of this post is support after go live and ongoing operational funding. As with my post about implementation funding, I've asked all the implementers of EHR projects in Massachusetts to comment on their plans.

BIDMC
At BIDMC, we'll provide a central help desk (Concordant), outsourced desktop/network support (Concordant), and ongoing application support (internal staff, Mass eHealth Collaborative staff and eClinicalWorks). Clinicians will pay a fixed monthly rate for this service. We'll centrally contract for all these services, so the cost will be as low as possible. BIDMC may pay for the ongoing operation of the centrally hosted eClinicalWorks system (i.e. rent in the co-location data center, server support staff) and this is still under discussion.

Caritas
Cartias is evaluating their strategy for ongoing support. They are considering the possibly of reassigning members of the implementation team to support as implementation is completed. The have not yet identified a specific funding model for support, but are considering an approach similar to BIDMC.

Childrens
Children's will provide a similar model to BIDMC. The help desk function and first tier application support will be outsourced to a third party vendor (The Ergonomic Group). They will escalate to eClinicalWorks as necessary. Ergonomic will also manage and support network operations at each of the practice sites. Children's will support the central hosting site hardware and infrastructure. Children's will also support all network operations inside the core data center. Clinicians will pay a fixed monthly rate for this service.

Mt. Auburn Hospital/MACIPA
Mt. Auburn/MACIPA will provide a central help desk and ongoing hardware/application support. They are currently retraining clinicians to help them increase the utilization of the product, given that during the initial training there is only so much a physician can absorb. They also intend to also hold classes at the IPA periodically. Post live financial support is still being discussed.

New England Baptist Hospital
NEBH will provide an outsourced help desk, ongoing hosting, and application support. Clinicians will pay for non-Meditech interfaces, software maintenance, and connectivity/support to billing companies.

Partners
Partners will follow the same model as BIDMC, with clinicians funding ongoing support services.

Winchester
Community physicians will fund ongoing software and hardware support. The team in Highland Management (joint venture between the hospital and IPA) will provide guidance in the development of templates and the use of the system for reporting to meet P4P goals and clinical integration. Winchester IT will also be involved in the development of interfaces and the transfer of patient data for care delivery.

This post marks the conclusion of my first series about electronic health records for non-owned physicians.

Today, the BIDMC Finance Committee approved our pilots, so we'll be moving forward with all the plans I've outlined. This is a major milestone for our project that enables all our contracts, service level agreements and spending to progress.

My next series about this topic will start in July as our pilots go live. I'm sure there will be many more lessons learned to share including comments on budgets, practice workflow transformation and loss of productivity. I hope these first 10 posts about planning the project have been useful to you!

Tamperproof prescriptions

On May 27, 2007, Congress enacted regulations requiring the use of tamper-resistant prescription pads. This primarily affects patients covered by state Medicaid programs.

Following several meetings internally, with state agencies and with professional organizations, BIDMC elected to use tamper proof paper stock in printers which produce electronic prescriptions (not e-prescribing, which is exempt from the regulation). The new stock will be used to write all prescriptions, regardless of payer/insurer and contains the features proposed by MassHealth as "standards" for Tamper-Resistant Prescription Pads:

It has a greenish hue.

It is perforated (twice) so that up to three prescriptions can be printed on one sheet.

When photocopied, the word "VOID" will appear in multiple locations.

The backside has Rx icons printed in thermochromic ink. This means the icon will disappear when rubbed with your finger and then reappear when you stop.

We're implementing this in two phases. From April 1, 2008 to October 1, 2008, we can still use plain paper, but we've modified prescription printing as permitted by the new regulation :

“Quantity Border and Fill (for computer generated prescriptions on paper only), i.e. Quantities are surrounded by special characters such as an asterisk to prevent alteration; e.g. QTY **50** and Value may also be expressed as text, e.g. (Fifty), (optional).”

“Refill Border and Fill (for computer generated prescriptions on paper only), i.e. Refill quantities are surrounded by special characters such as an asterisk to prevent alteration; e.g. QTY **5** and Value may also be expressed as text, e.g. (FIVE), (optional).”

Here is a link to an example of our new printed prescriptions. This is live now.

We're also providing a Notice to Dispensing Pharmacies which we're attaching to prescriptions.

By October 1, 2008, we'll replace all plain paper with the tamperproof paper as required by the regulation.

This has been an effort requiring coordination among IT, providers, administration, pharmacies, and government. It's been quite complex and we're hopeful that our planning, communication, and phased implementation effort will be successful.

Cool Technology of the Week

One of the challenges of being a CIO is the "application is slow, can you fix it" phone call. Generally, the network is blamed first, but there are many layers that all need to be examined - desktop, network, server, storage, database, active directory, internet service provider etc. For example, a complaint about email slowness can be caused by a multitude of factors.

We recently worked with our electronic health record infrastructure partner, Concordant, to do an end to end application performance analysis.

The tools they employed were:

WhatsUp Gold for Network and Server Monitoring
Windows Performance Monitor for Server and Client Monitoring
OPNET Ace for End to End Network traffic analysis
Computer Associates eHealth for Network Monitoring

The general approach they used covered three domains. They began by identifying and defining the problems from a user perspective. This helped to identify issues related to system performance versus non-technical issues that amplified the technical issues and affected user perception of performance, e.g. training, improper usage of the application. They used multiple subject matter experts to focus on the different domains to ensure they had the in-depth knowledge to evaluate each of them.

The three investigation domains and key focus areas within each domain were:

Client
End User Observation & Interviews
Client device performance analysis
Device configuration & Log review
Device specification analysis per application vendor recommendations

Network
WAN link utilization
Device performance analysis
Device configuration & Log review
Packet Loss & Latency analysis
Traffic Analysis

Infrastructure
Server and Storage performance analysis
Device configuration & Log review
Service and Process performance analysis
Device specification analysis per application vendor recommendations

The findings from the assessment did not identify a "magic bullet" issue that caused performance issues, but instead identified multiple smaller issues that combined to impact system performance.

In my experience of troubleshooting complex IT systems, I've found that the comprehensive approach outlined above works very well.

If I had to choose one simple approach to determine the cause of application performance issues, I would:

1. Check to see that the desktop, the server, and the database all have their network cards set to Auto, since performance problems are often network card duplex mismatches

2. Install OPNET agents on the client and server. More often than not, OPNET rapidly identifies root causes of application performance issues.

Based on my positive experience with OPNET, including in this particular project, I'm naming OPNET as the cool technology of the week. Now I can respond to the "application is slow" question with an OPNET answer.

Climbing New Hampshire

Why do humans climb mountains?

Because they're there? To regain a sense of challenge and opportunity for heroism that we've lost in the modern world? To get away from cell phones, email and the non-stop flow of information that is part of our internet culture?

On March 15, 2008, I finished ascending all the New Hampshire peaks over 4000 feet high during winter. Winter hiking/climbing poses some unique challenges such as how to stay warm at the top of Mt. Washington when it's -20F and the wind is blowing 50 mph. How to avoid avalanches. How to stay hydrated when even boiling water freezes over the course of a hike. How to ascend 12 foot snow drifts for 15 miles at a time. And the most dangerous - how to drive from Boston to New Hampshire on ice covered freeways without getting hit by a skidding bus.

Like everything I do, there is method in this madness. I think of alpine climbing as a kind of puzzle - an outdoor version of Sudoku - which cleanses my mind from the concerns of the work week. What gear is needed to stay warm but not too warm, since sweat freezes solid and can rapidly cause hypothermia? What route is safest? What techniques are best to ascend steep ice, deep snow, and tree covered terrain? What pace is best to manage time and energy, ensuring a successful trip? Reaching the summit is optional, but returning to the car is mandatory.

In New Hampshire there are 48 mountains above 4000 feet. Records have been kept for the past 50 years and I'm the 360th person to have completed the winter ascents of all the New Hampshire peaks.

Here's a typical schedule. Pack the night before with just the right amount of gear to be safe. I typically carry about 9 pounds of food, clothing, water and rescue equipment that I've refined over the years. Here's my gear list. I get up at 5am, eat a bowl of oatmeal, and fill a liter bottle with boiling water for drinking on the ascent. I pick up my climbing partner, who is also a physician, and drive to the White Mountains, typically a 2-3 hour commute depending on the mountain destination. All of gear has to be carefully laid out so that when we arrive at the trailhead, clothing layers can be added without losing body warmth and we can rapidly get started with the hike/climb. I generally like to start off a bit chilled so that the initial run up the trail gives me a body temperature that's just right. Along the trail, hats, gloves, and body insulating layers are added or removed as needed to stay just the perfect temperature.

Our typical journeys are 10-20 miles with 4000 feet of vertical gain. Depending on the depth of snow drifts, the ice, and the bushwacking through tree canopies, it can be quite taxing. I typically carry light snowshoes for deep snow and crampons for traversing ice. I wear a boot within a boot (Scarpa Alpha double plastic boots) to keep my feet warm.

Near the summit, the temperatures and the wind are so severe that goggles and facemasks are needed to keep your eyeballs from freezing. We typically summit, stay just a few minutes and then descend. Remember that the summit is only halfway back to the car.

Each year, several people die while winter hiking in the White Mountains. Most are reckless or significantly under prepared. My climbing partner and I are very safe and will not take risks. If we feel avalanche danger is too high or weather conditions are too severe, we will turn around. The good news is that ascending 48 peaks prepares you for a variety of conditions and events. We recently ascended Wildcat A through chest deep snow, hiking straight up the mountain because the trails were completely invisible in this year's heavy snow fall. The last mile took us 4 hours of hard work, with 3 steps backward for every 4 steps forward. At the end of the hike, we were exhausted by the effort. Only later did we find out that we were the first to ascend the mountain the past 30 days due to extreme conditions. I'm glad we did not know that the ascent was impossible before we started!

After Hours Pay for IT Professionals

I was recently asked how we fund "on call" pay and subsidize remote access for our IT staff.

At BIDMC, our policy is that we have standardized on call pay for those who carry a beeper and support our critical systems.

We reimburse some IT employees for half of their monthly home internet service cost (approximately $30/month) if deemed necessary to do their job, assuming that only 50% of a home internet connection will be used for business. Also, we reimburse cell phones and Blackberries by adding the amount of an appropriate monthly plan (decided by their Director/Manager) to employee paychecks. Employees pay the bills themselves, eliminating the administrative burden of reimbursements.

I asked my IT colleagues at other hospitals in Boston for permission to publish their policies.

At Boston Medical Center, they provide on call pay but employees must pay for their own internet access. They previously funded internet access, but dropped internet reimbursement as home connections became more ubiquitous. They are currently paying for a "team" on call cell phone but may ask employees to use their own phones in the future.

At Partners Healthcare, they currently pay for on call support and offer a stipend to staff to cover their home internet access. They are investigating the best practices at other healthcare IT organizations.

At Children's Hospital of Boston, they have two hourly rates for on call support. The higher rate (referred to as Tier I) is paid to on call staff who are paged more frequently. The second rate (Tier II) is paid to anyone who participates in the on call rotation but is paged infrequently. Children's pays for home internet access of on-call staff who frequently log in remotely to perform systems management.

As IT staffing becomes increasingly virtual, it's clear that our policies on paying for beeper call, remote access, mobility technologies, and home office equipment will evolve.

Electronic Health Records for Non-Owned Doctors - Implementation Order

This is the ninth entry in my series about providing electronic health records for non-owned clinicians. We'll call this one "triaging the practices". Since we have 300 non-owned clinicians who need electronic health records, where do we begin? If new clinicians join the Beth Israel Deaconess Physician's Organization during our rollout, how do their practices fit into the rollout?

We need specific triage rules to decide on the order of implementation.

In a for profit business, some metric like referral volume might be used, but in the non-profit healthcare world, such an approach would be a violation of Stark anti-kickback rules.

In our case, we want to ensure the highest quality care, coordinated through the use of interoperable electronic records. We want to ensure decision support is enabled for those who needed it most. We want to invest our effort into those practices which require the most clinical integration with the hospital to ensure high performance medicine. Based on a quality/safety approach, a rational implementation order would be:

1. Primary Care Physicians are the first priority - PCPs see a high volume of patients and are the "air traffic controllers" for care, ensuring coordination among all the clinicians a patient sees. An EHR enables an accurate problem list, an up to date medication list and alerts/reminders for wellness care. We want every patient's PCP to have the benefits for an EHR. Yes, we know that the first few months of using an EHR will impact a PCPs productivity, but our experience with other EHR implementations is that with appropriate training and a "model office" configuration, productivity rapidly returns to baseline.

2. Specialists who serve as a kind of primary care giver, managing diseases such as congestive heart failure, cancer and diabetes are also a priority to be early EHR users. Tracking diabetes care requires data coordination among endocrinologists, Ophthalmologists, and Vascular surgeons. Ob/Gyns are primary care givers. Chronic diseases such as COPD and CHF require coordination among pulmonologists, cardiologists and PCPs. Specialties that require significant care coordination with primary care givers or deliver primary care themselves include Cardiology, Ophthalmology, OB/GYN, Dermatology, Orthopaedics, Urology, Gastroenterology, Surgery, Pulmonary, Neurology, Endocrinology, Vascular, and Rheumatology.

3. As we are rolling out EHRs to these PCPs and specialists, it's likely that new clinicians will become affiliated with BIDMC. As we plan our rollout calendar, we will need to stay flexible so that new PCPs get priority and the specialists who most benefit from care coordination are placed ahead in the queue.

This approach to triage ensures that patients and providers get the maximal benefit from our efforts as we rollout 6 practices per month starting this Summer. We may need to refine our rules even further as we learn more from our rollouts:

* PCPs with a closer geographical location to BIDMC/a local hospital go first, since they have the most data interoperability needs. * Clinicians near retirement may choose not to be early adopters and may want to stay on paper.
* Some practices may more easily adapt to new technology than others and should go sooner

Over the next few months, the hospital and the physician organization will finalize the triage rules based on quality, safety and data sharing benefits, so it is very clear that we are Stark compliant and can easily explain to every non-owned physician when their EHR will be implemented based on objective criteria. I'll let you know how it goes!