Now that Spring is around the corner, it's time for an update on the "keep me awake at night projects" for 2008. Here's the first quarter progress on these 10 challenging issues:
1. Electronic Health Records for non-owned doctors - We've spent the first few months of 2008 planning the project, refining the budget, building partnerships, and establishing project management. I believe this extra up front time (and investment) will enhance our likelihood of success and avoid major time/budget overruns in the future. Each week I post a blog entry about some aspect of this project and the entire 11 article "pre-golive" series will be done by April. We're on track for a pilot this Summer, but there are still many unknowns because few hospitals in the US have offered Software as a Service Electronic Health Records to their non-owned clinicians. This project will still keep me up at night until our pilot sites are completed and we've secured all the funding needed for full production rollout.
2. Storage as a utility - After a 2 year journey, we've completed our storage as a utility designs, creating 3 tiers of storage using EMC SAN, SATA NAS, Data Domain de-duplication storage and Acopia file virtualization. I am now confident that we can store, archive, backup and retrieve the institution's data in two geographically distant data centers, providing the rapid recovery times mandated by our disaster recovery plan. I can remove this issue from my keep awake list.
3. e-Prescribing - BIDMC recently completed its latest release of e-Prescribing functionality within our home-built enterprise ambulatory electronic health record. Now a clinician can view payer formularies, retrieve community drug history with allergy/drug interaction checking, do medication reconciliation , route prescriptions to retail/mail order pharmacies and check patient insurance eligibility while writing for a medication. This leverages the state-wide electronic prescribing gateway built by MA-Share which connects payers, pharmacies, RxHub, Surescripts, and providers. In early March, Massachusetts was recognized as the number one e-Prescriber in the country . Although I will work very hard to continue the rollout and adoption of ePrescribing in Massachusetts, I can remove this issue from the keep awake list.
4. Data sharing for clinical care among a community of caregivers - We've recently gone live with the notion of pushing standards-based clinical summaries among caregivers throughout the state . This early stage is a pilot and although it has been technologically successful, the real success will be increasing the number of providers and hospitals using it. This keep awake project has gone from a technology issue to an education/communication issue, building a community of users sending summaries for patient care coordination. It's still on my list.
5. Security - Security will always be on my keep awake list. Over the first quarter of 2008, we've implemented a pilot of host-based intrusion detection from Third Brigade. The idea is that we have software running on each server which prevents common attacks such as buffer overflow, cross-server scripting and SQL Injection. This adds a layer of protection that defends us against attacks even if the operating system or application is vulnerable. Our plan is to rollout this technology throughout the data center and eventually across all our desktops. As fast as we innovate, hackers and spammers innovate, so we'll vigilantly keep adding more security projects every year.
6. RFID and Bar coding - We recently completed our enterprise rollout of Cisco Location Based Services and Pango Networks active RFID tags for tracking mobile assets such as wheelchairs, IV pumps, and EKG machines. Today we have about 350 tagged assets and recently acquired 900 tags to deploy in the upcoming weeks. Our FY08 operating budget includes an additional 500 tags to be purchased by the end of September. For bar coding, we recently completed our pay for performance goal of a bar coded wrist band on every ED/inpatient/ambulatory surgery patient and by June we will have bar coded our unit dose medications, enabling us to develop an electronic medication administration record (eMAR) system by next year. Once that eMAR system is live, I will remote this one from my keep awake list.
7. Providing decision support - I recently provided an update on our Performance Measurement activities and Decision Support tools . The keep me awake portion of this is our upcoming demonstration of secure, deidentified, data sharing for decision support and clinical research across all the Harvard hospitals as part of the Clinical Translational Science Awards (CTSA) program of NIH. This effort requires us to obtain IRB approval from several Harvard hospitals, build standards-based middleware layer, and create an easy to use graphical tool for data browsing. No patient identified data will be involved, but the technology, policy, and organizational challenges needed to go live by July 1 will keep me awake.
8. Compliance requirements for revenue cycle workflows - Every day new compliance requirements add more projects to the IT department. Recently I was asked if all our applications are Payment Card Industry (PCI) compliant. The good news is that we do not store credit card data on hospital systems, we only have secure credit card transactions running over our networks to third party firms. Without persistent credit card data, our exposure to fraud is lessened. Over the first quarter, we've added more electronic data interchange transactions via our NEHEN gateway, enhanced our coding software/interfaces, and provided all the necessary data to meet our state/federal reporting obligations. Thus, for the moment, I feel good about our compliance and it is not keeping me awake.
9. Internal and external websites - We've been hard at work implementing a new web content management system and I am confident that in 2008 we will launch our new external and internal websites. There is a great deal of work to do to transition from static HTML pages and our home built content management system, so this one stays on the awake list until Fall.
10. Disaster recovery - We've made substantial progress with our disaster recovery efforts and we now have our most mission critical systems backed up by a redundant data center. Just as security is journey, so is disaster recovery. The entire plan, our progress, and our phasing for the future is outlined in this presentation. Although disaster recovery will always be on the awake list, I can rest a bit more knowing that our major clinical systems are not only redundant within a data center, they are redundant across two data centers.
So the report card on the first quarter is that a few of the keep me awake projects are off the list, and a few more will be off the list by Summer and Fall. As long as we are making forward progress on all these items, I'm happy, since the trajectory is more important than our position on any given day.
Selasa, 11 Maret 2008
Senin, 10 Maret 2008
Electronic Health Records for non-owned doctors - staffing the project
This is my sixth post in the series about Electronic Health Records for non-owned doctors. This week, I'll discuss the complexities of staffing the project, since we've had to weigh insourcing verses outsourcing, costs, and service levels to arrive at a balanced staffing plan. We've divided the staffing into 10 categories and below I describe the strategy for each.
In creating this staffing model, we had several considerations
*The project scope is not fixed. We're starting with 300 clinicians and may implement over 1000. Thus, we need a staffing model which is scalable on demand. We may need to flex the size of our teams up or down depending on implementation schedules.
*In Massachusetts, at this time, it is challenging to hire and maintain healthcare informatics staff because of intense competition among hospitals implementing CPOE/EHRs and local software companies expanding their healthcare IT workforce
*Outsourcing can be a way to rapidly expand capacity but it requires diligent management and tends to be more expensive than hiring internal staff.
1. Project Management and Financial Management
We have a fulltime Project Director and have leveraged our existing IS fiscal staff to manage the budgets. We will partner with the Massachusetts eHealth Collaborative to operate a project management office under the direction of our Project Director but jointly and flexibly staffed with BID and MAeHC personnel. This will allow us to:
a. Complement our existing knowledge of hospital and employed-practice deployments with outside expertise regarding non-owned ambulatory practices
b. Ramp up staff strength quickly as implementation intensity grows
c. Ramp down smoothly as deployment transitions to long-term support. This model will also allow us to rapidly and seamlessly plug in the temporary project management and staffing gaps that will inevitably arise during the course of the project.
2. Technical Design and Engineering
We elected to insource design and engineering for our servers, storage, and network design. We collaborated with our vendors - HP, EMC, and Cisco, as well as our infrastructure implementation partner, Concordant, on these designs. We assigned .25 FTE of the manager of our Ambulatory Applications group to this task.
3. Central Site Construction
We elected to outsource construction of the central hosting facility to Concordant for a fixed price. They acquired the co-location space, installed all equipment, and took responsibility for establishing power/cooling/network connectivity to the site. They will also manage the central site during the pilot phase. We are paying for a deliverable rather than paying per diem rates for time or purchasing FTEs.
4. Office Hardware Deployment
We elected to outsource office hardware deployment to Concordant by purchasing a team of people which scales in direct relation to the number of offices we are implementing. Purchasing a deployment team rather than working on per diem rates means that we paying for FTEs assigned in direct relation to the deliverables.
5. Practice Consultation
We elected to insource and outsource practice consultation. An MAeHC senior consultant will directly manage the practice consultant team, which will comprise both BID and MAeHC consultants. These practice consultants will be assigned to individual practices and will provide end-to-end project management of practice-level implementations, to ensure that all activities associated with the implementation are synchronized. They will also work with individual practices on optimizing workflow and translating that optimized workflow into appropriate software configuration, hardware layout, and training approach. As with the project management function, this is a flexible insource/outsource model that allows us to scale up and down rapidly, tap into existing expertise and apply it to the project right away, and maintain an adaptive but robust capability to meet program changes as they arise.
6. Training
We elected to insource and outsource training. eClinicalWorks will provide the training for all our initial pilot sites then train our trainers. Going forward a combination of eCW and insourced trainers will serve our sites, with supervision/quality control of all training to be provided by eClinicalWorks.
7. Central Site Operations
We elected to outsource central site operations to Concordant via a "lights out data center" model coupled with systems and application administration. They provide monitoring tools and problem escalation 24x7 rather than hiring a specific number of staff to manage our installation. This enables us to leverage the fact that they are providing support to other customers and projects, keeping our costs low and avoiding the need for us to hire fractional FTEs to provide data center support. The challenge with hiring internal staff to do this is that we expect data center needs to be higher during our initial implementation because of the build/change activities, then markedly reduce during our steady state operations. Outsourcing this function to Concordant, which spreads FTEs over many customers, enables us to flex our needs easily.
8. Help Desk, Tier 1 and Tier 2 Support
We elected to insource and outsource telephone support. Concordant will be the initial single point of contact for all phone calls, doing Tier 1 support such as password resets and then triaging Tier 2 Support . They will handle infrastructure Tier 2 issues and escalate others (such as EHR best practice questions) to our insourced staff. Our staff will escalate eClinicalWorks specific issues to eCW as needed. By focusing on resolving as many questions as possible remotely and dividing support between Concordant, our staff, and eCW, clearly defining the tasks of each group, we minimize our costs.
9. Field Support
We elected to outsource field support to Concordant, using a shared staff model. This optimizes our costs, coverage and flexibility since here again Concordant spreads FTEs over multiple customers.
10. Security auditing
Security has been built into our project from the very beginning as part of our infrastructure design, application configuration, and staffing model. We elected to outsource security auditing to an expert ethical hacking and security firm, Third Brigade for a fixed price. By hiring an expert group to do this, we provide another layer of vigilance and control, ensuring we have an outside party validating our configurations, providing host based intrusion protection, and monitoring our systems.
Thus, by dividing the staffing of the project among the members of our "dream team" - BIDPO/BIDMC, Concordant, Massachusetts eHealth Collaborative, eClinicalWorks and Third Brigade - we have achieved an affordable, scalable, balanced insource/outsourcing staffing model.
More to come as we test this model in production this Summer!
In creating this staffing model, we had several considerations
*The project scope is not fixed. We're starting with 300 clinicians and may implement over 1000. Thus, we need a staffing model which is scalable on demand. We may need to flex the size of our teams up or down depending on implementation schedules.
*In Massachusetts, at this time, it is challenging to hire and maintain healthcare informatics staff because of intense competition among hospitals implementing CPOE/EHRs and local software companies expanding their healthcare IT workforce
*Outsourcing can be a way to rapidly expand capacity but it requires diligent management and tends to be more expensive than hiring internal staff.
1. Project Management and Financial Management
We have a fulltime Project Director and have leveraged our existing IS fiscal staff to manage the budgets. We will partner with the Massachusetts eHealth Collaborative to operate a project management office under the direction of our Project Director but jointly and flexibly staffed with BID and MAeHC personnel. This will allow us to:
a. Complement our existing knowledge of hospital and employed-practice deployments with outside expertise regarding non-owned ambulatory practices
b. Ramp up staff strength quickly as implementation intensity grows
c. Ramp down smoothly as deployment transitions to long-term support. This model will also allow us to rapidly and seamlessly plug in the temporary project management and staffing gaps that will inevitably arise during the course of the project.
2. Technical Design and Engineering
We elected to insource design and engineering for our servers, storage, and network design. We collaborated with our vendors - HP, EMC, and Cisco, as well as our infrastructure implementation partner, Concordant, on these designs. We assigned .25 FTE of the manager of our Ambulatory Applications group to this task.
3. Central Site Construction
We elected to outsource construction of the central hosting facility to Concordant for a fixed price. They acquired the co-location space, installed all equipment, and took responsibility for establishing power/cooling/network connectivity to the site. They will also manage the central site during the pilot phase. We are paying for a deliverable rather than paying per diem rates for time or purchasing FTEs.
4. Office Hardware Deployment
We elected to outsource office hardware deployment to Concordant by purchasing a team of people which scales in direct relation to the number of offices we are implementing. Purchasing a deployment team rather than working on per diem rates means that we paying for FTEs assigned in direct relation to the deliverables.
5. Practice Consultation
We elected to insource and outsource practice consultation. An MAeHC senior consultant will directly manage the practice consultant team, which will comprise both BID and MAeHC consultants. These practice consultants will be assigned to individual practices and will provide end-to-end project management of practice-level implementations, to ensure that all activities associated with the implementation are synchronized. They will also work with individual practices on optimizing workflow and translating that optimized workflow into appropriate software configuration, hardware layout, and training approach. As with the project management function, this is a flexible insource/outsource model that allows us to scale up and down rapidly, tap into existing expertise and apply it to the project right away, and maintain an adaptive but robust capability to meet program changes as they arise.
6. Training
We elected to insource and outsource training. eClinicalWorks will provide the training for all our initial pilot sites then train our trainers. Going forward a combination of eCW and insourced trainers will serve our sites, with supervision/quality control of all training to be provided by eClinicalWorks.
7. Central Site Operations
We elected to outsource central site operations to Concordant via a "lights out data center" model coupled with systems and application administration. They provide monitoring tools and problem escalation 24x7 rather than hiring a specific number of staff to manage our installation. This enables us to leverage the fact that they are providing support to other customers and projects, keeping our costs low and avoiding the need for us to hire fractional FTEs to provide data center support. The challenge with hiring internal staff to do this is that we expect data center needs to be higher during our initial implementation because of the build/change activities, then markedly reduce during our steady state operations. Outsourcing this function to Concordant, which spreads FTEs over many customers, enables us to flex our needs easily.
8. Help Desk, Tier 1 and Tier 2 Support
We elected to insource and outsource telephone support. Concordant will be the initial single point of contact for all phone calls, doing Tier 1 support such as password resets and then triaging Tier 2 Support . They will handle infrastructure Tier 2 issues and escalate others (such as EHR best practice questions) to our insourced staff. Our staff will escalate eClinicalWorks specific issues to eCW as needed. By focusing on resolving as many questions as possible remotely and dividing support between Concordant, our staff, and eCW, clearly defining the tasks of each group, we minimize our costs.
9. Field Support
We elected to outsource field support to Concordant, using a shared staff model. This optimizes our costs, coverage and flexibility since here again Concordant spreads FTEs over multiple customers.
10. Security auditing
Security has been built into our project from the very beginning as part of our infrastructure design, application configuration, and staffing model. We elected to outsource security auditing to an expert ethical hacking and security firm, Third Brigade for a fixed price. By hiring an expert group to do this, we provide another layer of vigilance and control, ensuring we have an outside party validating our configurations, providing host based intrusion protection, and monitoring our systems.
Thus, by dividing the staffing of the project among the members of our "dream team" - BIDPO/BIDMC, Concordant, Massachusetts eHealth Collaborative, eClinicalWorks and Third Brigade - we have achieved an affordable, scalable, balanced insource/outsourcing staffing model.
More to come as we test this model in production this Summer!
Sabtu, 08 Maret 2008
Conflicts of Interest 2008
In all of my jobs and activities I try to be very transparent about all I do, both good and bad. In the interest of full disclosure, I thought it would be useful to describe all my affiliations, my disclosures, and my attempts to avoid conflicts of interest.
Having just finished my tax returns, I can tell you that my income is just a W2 from BIDMC, which invoices Harvard for the time I spend there. I did this to avoid having two paychecks and FICA deducted twice.
Any income I receive from speaking engagements or consulting I donate to BIDMC to minimize conflict of interest.
In 1993, I created a family trust and all my savings/investments are managed in that trust by an independent third party. The trust does not invest in technology related stocked or bonds.
Harvard has very strict rules about working with for-profit companies. No Harvard logos can be used in association with a commercial product. No organizational endorsements are permitted i.e. "Harvard thinks this is the best product on the market!". Case studies are fine and objective comments about functionality are permissible i.e. "I tested 5 products and found product x met my specific functional criteria."
Anytime I do case studies, they are reviewed by Public Affairs at Harvard Medical School for appropriateness. Anytime I have a question about serving as an adviser or board member, I inform Corporate Compliance at CareGroup and seek permission. Here are the roles I serve in for-profit companies and the financial arrangements:
Google Health Advisory Council - For the past year, I have served as an unpaid advisor on the Google Health Advisory Council. Since several individuals working on the Council are from non-profits and share my feelings about objectivity, Google circulates a form among the group so we can specify our preferences about being paid for our time. I have not accepted any personal compensation for the meetings, phone calls, policy advice and standards work.
Healthline Medical Advisory Board - Healthline has a built a search engine that uses controlled vocabularies/ontologies in an attempt to deliver more specific search results to users i.e. it knows that brains contain neurons and thus a search on neurons also looks at diseases of the brain. I serve on their advisory board to offer suggestions about how to incorporate medical knowledge in the search process. I have not accepted any compensation. I was asked to serve as an adviser by one of my former professors, who leads the Medical Advisory Board.
Whole Health Advisory Board - Whole Health provides workplace based medical services and wellness management. They use electronic health records throughout their clinics to optimize quality and safety. I was asked to serve as an adviser by a colleague at Harvard Business School. I have not accepted any compensation.
SafeMed Board of Directors - I rarely serve on the Boards of companies, but I made an exception with SafeMed because I so strongly believe in their mission - to make decision support available to doctors, patients and payers using a web-services architecture. They are the decision support behind Google Health, informing patients about medication interactions. I have not received any cash compensation for my Board service and will receive stock options, as is typical for Board members. These options will be declared with Harvard and CareGroup and I will recuse myself from any decisions regarding purchasing of SafeMed products. At present, CareGroup uses SafeMed as part of our Blue Cross funded pay for performance initiatives and CareGroup did not purchase products from SafeMed as part of that effort.
ePocrates Board of Directors - ePocrates is handheld prescribing software that runs on Treos, Blackberries and the iPhone. Many attending physicians, residents, and medical students at Harvard download the free version of ePocrates. Since it is the most popular mobile software used by clinicians at BIDMC and Harvard, I agreed to serve on the Board of Directors. I have not received any cash compensation for my Board service and did receive stock options, as is typical for Board members. These options were declared on the Harvard and CareGroup conflict of interest statements. CareGroup and Harvard have not purchased any products from ePocrates and I will recuse myself from any purchase decisions if any product selection is done in the future.
Blackberry Advocate - In 2007, I flew to a New York City SOHO studio and spent a day speaking about the way mobile devices impact my life. It was unscripted and captured my honest feelings about life as a highly connected human. What did I do to keep my objectivity? If you go to the Blackberry website, you'll see that it calls me "John Halamka, CIO and Emergency Physician", specifically removing my institutional affiliations. I worked very hard with Harvard and CareGroup counsel to ensure all materials did not include organizational endorsements, logos, or sales literature. The Blackberry media campaign in magazines, videos, and elevators was very professionally done. I did my best to express my personal belief in the utility of wireless devices while minimizing conflicts. When I walked on the set, I was given union scale wages for the day ($100 plus residuals) as would be done for any TV appearance by an extra and I donated that to BIDMC as well as disclosed it to Harvard and BIDMC corporate compliance.
These are all my for-profit company activities. I hope my attempt to isolate any conflicts of interest sound reasonable to you. The only companies from this list that I have mentioned in my blog are Google and SafeMed. For my Google entry on February 28, 2008, I specifically mentioned my Advisory Council service. Regarding my SafeMed entry of November 12, 2007, I began my Board service on February 1, 2008, so this entry is my declaration.
If there is anything I can do to adhere to other best practices, ensuring my objectivity, let me know!
Having just finished my tax returns, I can tell you that my income is just a W2 from BIDMC, which invoices Harvard for the time I spend there. I did this to avoid having two paychecks and FICA deducted twice.
Any income I receive from speaking engagements or consulting I donate to BIDMC to minimize conflict of interest.
In 1993, I created a family trust and all my savings/investments are managed in that trust by an independent third party. The trust does not invest in technology related stocked or bonds.
Harvard has very strict rules about working with for-profit companies. No Harvard logos can be used in association with a commercial product. No organizational endorsements are permitted i.e. "Harvard thinks this is the best product on the market!". Case studies are fine and objective comments about functionality are permissible i.e. "I tested 5 products and found product x met my specific functional criteria."
Anytime I do case studies, they are reviewed by Public Affairs at Harvard Medical School for appropriateness. Anytime I have a question about serving as an adviser or board member, I inform Corporate Compliance at CareGroup and seek permission. Here are the roles I serve in for-profit companies and the financial arrangements:
Google Health Advisory Council - For the past year, I have served as an unpaid advisor on the Google Health Advisory Council. Since several individuals working on the Council are from non-profits and share my feelings about objectivity, Google circulates a form among the group so we can specify our preferences about being paid for our time. I have not accepted any personal compensation for the meetings, phone calls, policy advice and standards work.
Healthline Medical Advisory Board - Healthline has a built a search engine that uses controlled vocabularies/ontologies in an attempt to deliver more specific search results to users i.e. it knows that brains contain neurons and thus a search on neurons also looks at diseases of the brain. I serve on their advisory board to offer suggestions about how to incorporate medical knowledge in the search process. I have not accepted any compensation. I was asked to serve as an adviser by one of my former professors, who leads the Medical Advisory Board.
Whole Health Advisory Board - Whole Health provides workplace based medical services and wellness management. They use electronic health records throughout their clinics to optimize quality and safety. I was asked to serve as an adviser by a colleague at Harvard Business School. I have not accepted any compensation.
SafeMed Board of Directors - I rarely serve on the Boards of companies, but I made an exception with SafeMed because I so strongly believe in their mission - to make decision support available to doctors, patients and payers using a web-services architecture. They are the decision support behind Google Health, informing patients about medication interactions. I have not received any cash compensation for my Board service and will receive stock options, as is typical for Board members. These options will be declared with Harvard and CareGroup and I will recuse myself from any decisions regarding purchasing of SafeMed products. At present, CareGroup uses SafeMed as part of our Blue Cross funded pay for performance initiatives and CareGroup did not purchase products from SafeMed as part of that effort.
ePocrates Board of Directors - ePocrates is handheld prescribing software that runs on Treos, Blackberries and the iPhone. Many attending physicians, residents, and medical students at Harvard download the free version of ePocrates. Since it is the most popular mobile software used by clinicians at BIDMC and Harvard, I agreed to serve on the Board of Directors. I have not received any cash compensation for my Board service and did receive stock options, as is typical for Board members. These options were declared on the Harvard and CareGroup conflict of interest statements. CareGroup and Harvard have not purchased any products from ePocrates and I will recuse myself from any purchase decisions if any product selection is done in the future.
Blackberry Advocate - In 2007, I flew to a New York City SOHO studio and spent a day speaking about the way mobile devices impact my life. It was unscripted and captured my honest feelings about life as a highly connected human. What did I do to keep my objectivity? If you go to the Blackberry website, you'll see that it calls me "John Halamka, CIO and Emergency Physician", specifically removing my institutional affiliations. I worked very hard with Harvard and CareGroup counsel to ensure all materials did not include organizational endorsements, logos, or sales literature. The Blackberry media campaign in magazines, videos, and elevators was very professionally done. I did my best to express my personal belief in the utility of wireless devices while minimizing conflicts. When I walked on the set, I was given union scale wages for the day ($100 plus residuals) as would be done for any TV appearance by an extra and I donated that to BIDMC as well as disclosed it to Harvard and BIDMC corporate compliance.
These are all my for-profit company activities. I hope my attempt to isolate any conflicts of interest sound reasonable to you. The only companies from this list that I have mentioned in my blog are Google and SafeMed. For my Google entry on February 28, 2008, I specifically mentioned my Advisory Council service. Regarding my SafeMed entry of November 12, 2007, I began my Board service on February 1, 2008, so this entry is my declaration.
If there is anything I can do to adhere to other best practices, ensuring my objectivity, let me know!
Kamis, 06 Maret 2008
Cool Technology of the Week
This week, the Massachusetts Health Information Exchange, MA-SHARE, went live with secure sharing of clinical summary records from provider to provider at Beth Israel Deaconess Medical Center, Lahey Clinic, Northeast Health Systems and Boston Children's using the recently recognized national standards harmonized by HITSP.Here's how it works.
When a patient registers for care, their primary care giver is captured by the registration clerk. We store the National Provider Indentifier of the clinician.
When a patient is discharged from the hospital or the emergency department, a comprehensive clinical summary is automatically prepared in Continuity of Care Document format including medications, diagnoses, procedures, and all followup issues.
This electronic document is sent via SOAP/HTTPS to a hospital-hosted gateway. That gateway uses the National Provider Identifier to look up the primary care giver's institutional affiliation in our statewide provider index. The summary is then routed to the gateway of the receiving institution via the internet via SOAP/HTTPS. Once it arrives, it is routed to the clinician's Electronic Health Record, Fax machine, or secure Email box.
The end result is that we are ensuring appropriate followup, medication reconciliation, and communication among physicians using a standards-based electronic document and the internet. Further details are available in this overview.
In Masssachusetts, we exchange 100 million HIPAA transactions per year via our New England Health EDI Network (NEHEN) gateway. Massachusetts was recently recognized as the #1 e-Prescriber in the country and part of our accelerated adoption has been our use of an e-Prescribing Gateway. Now the secure summary exchange gateway is live. All three of these gateways are built on the same application platform, which is running at hospital systems and payers throughout the state. There are no transaction fees, and no charges to the patient. The software is developed and maintained via contributions from provider and payer organizations which derive value from its use. In the case of secure document exchange, we can eliminate mailing discharge summaries and ED notes, saving hospitals like BIDMC $100,000 per year.
National standards, the internet, and a business model for health information exchange in production. That's cool!
Selasa, 04 Maret 2008
Buying Great Sake
On occasion, I'll write blog entries about my avocations - rock and ice climbing, playing the Japanese flute, drinking green tea, winemaking, mushroom hunting and finding the perfect Sake.
Today's entry is about buying Sake, which can be very confusing.
Sake is made from rice, water, Koji (Aspergillus Oryzae, a starch dissolving mold), and yeast. It is not a distilled beverage and is generally between 15% and 17% alcohol. Sake takes approximately one month to brew, followed by a six month aging period and is meant to be consumed soon after purchase. Try to buy a sake bottled within the last year. There is no such thing as a vintage Sake.
Interpreting a Sake label can be challenging. The most important elements reflect the addition of alcohol and the processing of the rice. You'll find one of two designations about alcohol
Honjozo - Sake to which a small amount of distilled alcohol is added
Junmai- No added alcohol
Premium sake is brewed with special Sake rice in which the starch component (the shinpaku or "white heart") is concentrated at the center of the grain, with proteins, fats, and amino acids located toward the outside. With increased milling, sake makers can remove the fats, proteins, and amino acids that lead to unwanted flavors and aromas in the brewing process. Ginjo-shu (premium Sake) has at least 40% or more milled away. Daiginjo (super premium Sake) has at least 50% or more milled away. Sake bottles list Seimaibuai, the amount of the rice grain left after milling.
Another important element is the level of dryness. The Sake Meter Value (Nihonshu-do) is the specific gravity of a Sake. It indicates how much of the sugars created from the starches in the rice were converted to alcohol, and how much remained to contribute to sweetness. By historical convention, the higher the number, the drier the Sake. What is the range? In theory, it is open-ended. In practice, + 10 or so is quite dry, -4 or so is quite sweet, and +3 or so is neutral.
Acidity affects how the flavor spreads, and also the sensation of sweet and dry. The range is quite narrow, with 0.7 being low and 2.0 being quite high. 1.2 or so is average.
Good Sake is made from special Sake rice. There are dozens of types of Sake rice, which is different from eating rice, but only a handful that are truly worth remembering. The most important of these is Yamada Nishiki.
Yeast mainly affects fragrance, and then flavor. There are dozens of yeast strains, each with its own subtly different characteristics, mostly affecting fragrance, but also flavor. Learning to discern the characteristics of the various yeast strains is part of the fun of learning about Sake.
Other elements you may find on the label are
Amakuchi -Sweet in flavor
Jizake - Sake from smaller sake breweries
Karakuchi - Dry in flavor
Nigori - Unfiltered sake which has a white milky color
Yamahai - Sake brewed in a way that allows wild yeasts to grow. This is rarely done since the aroma is "gamey".
My favorite Sake is Wandering Poet, a Junmai Ginjo made by Rihaku Shuzo, Shimane Prefecture. Brewed slowly at low temperatures using traditional brewing techniques and Yamada Nishiki rice with 45% milled away, it has a well-rounded flavor, extraordinary fragrance, and a clean finish.
Let's analyze the label:
*Junmai Ginjo means no alcohol added and at least 40% of the rice grain has been milled away. I tend to prefer Junmai Ginjo instead of Junmai Daigingo, because the Ginjo has more intense flavor and character. I tend to eat a lot of raw/unrefined foods and I prefer my sake a little less refined.
*Sake Meter Value/Nihonshudo +3 means that this is neither overly dry nor sweet
*Alcohol 15.2% means that it is relatively low in alcohol for a sake
*Acidity 1.6 means that it is a crisp sake, relatively high in acid, that goes well with food
*Seimaibuai 55% means that 45% of the rice grain has been milled away
*Yamada Nishiki means that a premium rice has been used
*Yeast # 9 means that a specific strain of yeast was used. Many ginjo yeasts are #9-based strains which creates a great fragrance and a consistent fermentation.
Sakes are available at many liquor stores throughout the US, but often they are from inexpensive American producers. The best Sakes are available online and from a few regional specialty stores that import fine Sake. Interestingly Trader Joes often has fine Junmai Ginjo type Sakes at a great price.
Sake should be served chilled, not warmed. Warming is only done with inferior Sakes to hide the impurities.
So drop by your nearby Trader Joes, pickup a bottle of a Junmai Ginjo Sake and serve it chilled. You'll have a great Sake experience.
Today's entry is about buying Sake, which can be very confusing.
Sake is made from rice, water, Koji (Aspergillus Oryzae, a starch dissolving mold), and yeast. It is not a distilled beverage and is generally between 15% and 17% alcohol. Sake takes approximately one month to brew, followed by a six month aging period and is meant to be consumed soon after purchase. Try to buy a sake bottled within the last year. There is no such thing as a vintage Sake.
Interpreting a Sake label can be challenging. The most important elements reflect the addition of alcohol and the processing of the rice. You'll find one of two designations about alcohol
Honjozo - Sake to which a small amount of distilled alcohol is added
Junmai- No added alcohol
Premium sake is brewed with special Sake rice in which the starch component (the shinpaku or "white heart") is concentrated at the center of the grain, with proteins, fats, and amino acids located toward the outside. With increased milling, sake makers can remove the fats, proteins, and amino acids that lead to unwanted flavors and aromas in the brewing process. Ginjo-shu (premium Sake) has at least 40% or more milled away. Daiginjo (super premium Sake) has at least 50% or more milled away. Sake bottles list Seimaibuai, the amount of the rice grain left after milling.
Another important element is the level of dryness. The Sake Meter Value (Nihonshu-do) is the specific gravity of a Sake. It indicates how much of the sugars created from the starches in the rice were converted to alcohol, and how much remained to contribute to sweetness. By historical convention, the higher the number, the drier the Sake. What is the range? In theory, it is open-ended. In practice, + 10 or so is quite dry, -4 or so is quite sweet, and +3 or so is neutral.
Acidity affects how the flavor spreads, and also the sensation of sweet and dry. The range is quite narrow, with 0.7 being low and 2.0 being quite high. 1.2 or so is average.
Good Sake is made from special Sake rice. There are dozens of types of Sake rice, which is different from eating rice, but only a handful that are truly worth remembering. The most important of these is Yamada Nishiki.
Yeast mainly affects fragrance, and then flavor. There are dozens of yeast strains, each with its own subtly different characteristics, mostly affecting fragrance, but also flavor. Learning to discern the characteristics of the various yeast strains is part of the fun of learning about Sake.
Other elements you may find on the label are
Amakuchi -Sweet in flavor
Jizake - Sake from smaller sake breweries
Karakuchi - Dry in flavor
Nigori - Unfiltered sake which has a white milky color
Yamahai - Sake brewed in a way that allows wild yeasts to grow. This is rarely done since the aroma is "gamey".
My favorite Sake is Wandering Poet, a Junmai Ginjo made by Rihaku Shuzo, Shimane Prefecture. Brewed slowly at low temperatures using traditional brewing techniques and Yamada Nishiki rice with 45% milled away, it has a well-rounded flavor, extraordinary fragrance, and a clean finish.
Let's analyze the label:
*Junmai Ginjo means no alcohol added and at least 40% of the rice grain has been milled away. I tend to prefer Junmai Ginjo instead of Junmai Daigingo, because the Ginjo has more intense flavor and character. I tend to eat a lot of raw/unrefined foods and I prefer my sake a little less refined.
*Sake Meter Value/Nihonshudo +3 means that this is neither overly dry nor sweet
*Alcohol 15.2% means that it is relatively low in alcohol for a sake
*Acidity 1.6 means that it is a crisp sake, relatively high in acid, that goes well with food
*Seimaibuai 55% means that 45% of the rice grain has been milled away
*Yamada Nishiki means that a premium rice has been used
*Yeast # 9 means that a specific strain of yeast was used. Many ginjo yeasts are #9-based strains which creates a great fragrance and a consistent fermentation.
Sakes are available at many liquor stores throughout the US, but often they are from inexpensive American producers. The best Sakes are available online and from a few regional specialty stores that import fine Sake. Interestingly Trader Joes often has fine Junmai Ginjo type Sakes at a great price.
Sake should be served chilled, not warmed. Warming is only done with inferior Sakes to hide the impurities.
So drop by your nearby Trader Joes, pickup a bottle of a Junmai Ginjo Sake and serve it chilled. You'll have a great Sake experience.
The CareGroup Network Outage
On November 13, 2002 at 1:45pm, Beth Israel Deaconess Medical Center went from the hospital of 2002 to the hospital of 1972. Network traffic stopped. No clinician could order medications or labs electronically. No decision support was available. Luckily, no patients were harmed. Here's the story of what happened and our lessons learned.
In the years after the 1996 merger of the Beth Israel and Deaconess Hospitals, operating losses caused several years of capital starvation (the network budget in 2002 was $50,000 for the entire $2 billion dollar enterprise). We were not available to invest in infrastructure, so our network components were beyond their useful life.
However, it was not this infrastructure underinvestment that was the root cause of the problem, it was my lack of enterprise network infrastructure knowledge. I did not know what I did not know. Here are the details:
1. Our Network topology was perfectly architected for 1996. Back in the days when the internet was a friendly place where all internal and external collaborators could be trusted, a switched core (layer 2) that transmitted all packets from place to place was a reasonable design. After 1996, the likelihood of denial of service attacks, trojans, or other malware meant that networks should be routed and highly segmented, isolating any bad actors to a constrained local area. At the time of our outage, a data flood in one part of the network propagated to every other part of the network - a bit like living downstream from a dam that collapsed. A well meaning researcher created a napster-like application that began exchanging hundreds of gigabytes of data via multicast to multiple collaborators. The entire network was so saturated that we could not diagnose the root cause of the data flood. I did not know that a switched core was a point of failure.
2. Our Network team was manged by a very smart engineer who did not share all his knowledge with the network team. Much of our network configuration was poorly documented. With the knowledge of our network isolated to one person, we had a single point of human failure. I did not know that this engineer was unfamiliar with the best practices for routed/redundant network cores, routed distribution layers and switched access layers isolated into vlans with quality of service configurations to prevent monopolization of bandwidth by any one user or application. We brought in a Cisco partner, Callisma, to document the network, but the network failure occurred before they were finished.
3. I did not know about spanning tree algorithms, hot standby routing protocols (HSRP), and Open Shortest Path First (OSPF). During the outage, I approved configuration changes that actually made the situation worse by causing spanning tree propagations, flooding the network with even more traffic.
4. I did not establish a relationship with the vendor (Cisco) that enabled them to warn me about our vulnerabilities. A relationship with a vendor can take many forms, ranging from a sales driven vendor/client adversarial relationship to a collaborative partnership. In 2002, Cisco was just another vendor we purchased from. Today they are a collaborative partner with a seat at the table when we plan new infrastructure.
5. I did not know that we needed "out of band" tools to gain insight into the problems with the network. In effect, we required the network to be functional to diagnose problems with the network.
6. We did not have a robust, tested downtime plan for a total network collapse. When the outage occurred, we rapidly designed new processes to transport lab results, orders, and other data via runners from place to place.
7. We did not have a robust communication plan for responding to a total network collapse. Email, web-based paging, portals, and anything that used the data network for communication was down. Voice mail broadcasts using our PBX and regular phones (not IP phones) turned out to save the day.
8. When we diagnosed the problem, we explored many root causes and made many changes in the hope that we'd find the magic bullet to cure the problem. In the end, we ended up fixing many basic structural problems in the network, which took 2 days and eventually solved the problem. A more expedient solution would have been to reverse all changes we made in our attempts to fix the network once we had stopped the internal attack. When a crisis occurs, making changes on top of changes can make diagnosis and remediation even more difficult.
9. We did not have an enterprise wide change control process to ensure that every network configuration, server addition, and software enhancement was documented and assessed for impact on security, stability, and performance. Today we have a weekly change control board that include includes all IS departments, Cisco engineering services, and IS leadership to assess and communicate all configuration changes.
10. I was risk averse and did not want to replace the leadership of the network team for fear that terminating our single point of human failure would result in an outage. The price of keeping the leadership in place was a worse outage. I should have acted sooner to bolster leadership of the team.
Despite the pain and stress of the outage, there was a "lemons to lemonade" ending. Without this incident, the medical center would never have realized the importance of investing in basic IT infrastructure. If not for the "perfect storm", we may have limped along with a marginal network for years.
Today, we receive annual capital funding to support a regular refresh of our technology base, and we are asked to introduce change at a pace that is manageable. People in the medical center still remember the outage and are more accepting of a tightly managed infrastructure such as locked down workstations, security precautions, disaster recovery initiatives, and maintenance downtime windows.
During and immediately following the event, I presented this overview of the outage to senior management. Shortly after the outage, I worked with CIO Magazine to document the events and lessons learned.
I hope that my experience will help other organizations prevent network and other IT outages in the future.
In the years after the 1996 merger of the Beth Israel and Deaconess Hospitals, operating losses caused several years of capital starvation (the network budget in 2002 was $50,000 for the entire $2 billion dollar enterprise). We were not available to invest in infrastructure, so our network components were beyond their useful life.
However, it was not this infrastructure underinvestment that was the root cause of the problem, it was my lack of enterprise network infrastructure knowledge. I did not know what I did not know. Here are the details:
1. Our Network topology was perfectly architected for 1996. Back in the days when the internet was a friendly place where all internal and external collaborators could be trusted, a switched core (layer 2) that transmitted all packets from place to place was a reasonable design. After 1996, the likelihood of denial of service attacks, trojans, or other malware meant that networks should be routed and highly segmented, isolating any bad actors to a constrained local area. At the time of our outage, a data flood in one part of the network propagated to every other part of the network - a bit like living downstream from a dam that collapsed. A well meaning researcher created a napster-like application that began exchanging hundreds of gigabytes of data via multicast to multiple collaborators. The entire network was so saturated that we could not diagnose the root cause of the data flood. I did not know that a switched core was a point of failure.
2. Our Network team was manged by a very smart engineer who did not share all his knowledge with the network team. Much of our network configuration was poorly documented. With the knowledge of our network isolated to one person, we had a single point of human failure. I did not know that this engineer was unfamiliar with the best practices for routed/redundant network cores, routed distribution layers and switched access layers isolated into vlans with quality of service configurations to prevent monopolization of bandwidth by any one user or application. We brought in a Cisco partner, Callisma, to document the network, but the network failure occurred before they were finished.
3. I did not know about spanning tree algorithms, hot standby routing protocols (HSRP), and Open Shortest Path First (OSPF). During the outage, I approved configuration changes that actually made the situation worse by causing spanning tree propagations, flooding the network with even more traffic.
4. I did not establish a relationship with the vendor (Cisco) that enabled them to warn me about our vulnerabilities. A relationship with a vendor can take many forms, ranging from a sales driven vendor/client adversarial relationship to a collaborative partnership. In 2002, Cisco was just another vendor we purchased from. Today they are a collaborative partner with a seat at the table when we plan new infrastructure.
5. I did not know that we needed "out of band" tools to gain insight into the problems with the network. In effect, we required the network to be functional to diagnose problems with the network.
6. We did not have a robust, tested downtime plan for a total network collapse. When the outage occurred, we rapidly designed new processes to transport lab results, orders, and other data via runners from place to place.
7. We did not have a robust communication plan for responding to a total network collapse. Email, web-based paging, portals, and anything that used the data network for communication was down. Voice mail broadcasts using our PBX and regular phones (not IP phones) turned out to save the day.
8. When we diagnosed the problem, we explored many root causes and made many changes in the hope that we'd find the magic bullet to cure the problem. In the end, we ended up fixing many basic structural problems in the network, which took 2 days and eventually solved the problem. A more expedient solution would have been to reverse all changes we made in our attempts to fix the network once we had stopped the internal attack. When a crisis occurs, making changes on top of changes can make diagnosis and remediation even more difficult.
9. We did not have an enterprise wide change control process to ensure that every network configuration, server addition, and software enhancement was documented and assessed for impact on security, stability, and performance. Today we have a weekly change control board that include includes all IS departments, Cisco engineering services, and IS leadership to assess and communicate all configuration changes.
10. I was risk averse and did not want to replace the leadership of the network team for fear that terminating our single point of human failure would result in an outage. The price of keeping the leadership in place was a worse outage. I should have acted sooner to bolster leadership of the team.
Despite the pain and stress of the outage, there was a "lemons to lemonade" ending. Without this incident, the medical center would never have realized the importance of investing in basic IT infrastructure. If not for the "perfect storm", we may have limped along with a marginal network for years.
Today, we receive annual capital funding to support a regular refresh of our technology base, and we are asked to introduce change at a pace that is manageable. People in the medical center still remember the outage and are more accepting of a tightly managed infrastructure such as locked down workstations, security precautions, disaster recovery initiatives, and maintenance downtime windows.
During and immediately following the event, I presented this overview of the outage to senior management. Shortly after the outage, I worked with CIO Magazine to document the events and lessons learned.
I hope that my experience will help other organizations prevent network and other IT outages in the future.
Electronic Health Records for non-owned doctors -scalable infrastructure
This is my fifth entry about our Electronic Health Record project for non-owned doctors. As I've described, the scope of the project is to implement a highly reliable, secure, feature rich, well supported, but affordable electronic health record for private practices. Today's entry is about building the scalable centralized Software as a Service hosting infrastructure to meet these goals.
A key design requirement for the project is scalability. Our projected customer base is 300 clinicians and we have a fixed start up budget. However, we must design the infrastructure in a way that can cost effectively support the smallest amount of adopters as well as scale to thousands if our project is wildly successful. We debated two possibilities (metaphorically speaking)
a. Build a hotel, not knowing if anyone will ever check-in
b. Build a housing development, where the limits of expansion are only defined by available land
We decided on choice "b", starting with a robust foundation and adding new equipment and storage as we add clinicians. We standardized our central site equipment on products from HP EMC and Cisco, with guidance from our infrastructure partner, Concordant, and our equipment supplier CDW, ensuring it was easy to plug in additional hardware on demand. We invested a significant amount of time designing the central hosting facility, doing it right the first time. Over the years, I've seen CIOs rush through the design phase, only having to rebuild the infrastructure later when application performance did not scale. We partnered with our vendors to build something special, that if successful, could be a model for other medical centers and communities.
Considerations in designing our hosting infrastructure included:
* Supporting a user base that is remote, unmanaged, and diverse. We need to be able to identify any performance issues via end to end monitoring of all components
* Meet important security and privacy restrictions, as well as address liability issues (who is responsible for what)
* Understand infrastructure costs for a) start up b) additional capacity that occurs in bursts or steps, and c) variable requirements as practices go live.
* Provide connectivity to external parties (labs, claims, etc...) through interfaces which create additional security and performance complexities
* Address the limitations and performance of "last mile" connectivity through publicly available internet access (DSL, cable, etc.)
The infrastructure choices we made are:
Virtualized servers - VMware was the natural choice because of the scalability design goals. VM and V-Motion technologies also play an important part in redundancy, failure recovery and disaster recovery
Physical services - We debated rack mounted verses blade servers and elected to use powerful small footprint HP rack servers connected to fast multi-tiered storage. We computed the economics of blade servers verses rack mounted servers and the use of VMWare made small powerful rack servers the most cost effective solution.
Storage - We purchased a Clariion CX3-20 series SAN. We will go live with 11.1TB of total storage (2.1TB of fast, Tier 1 storage for database transactions and 9TB of secondary, Tier 2 storage for files). A single CX3-20 will allow us to expand in a modular fashion to accommodate up to 1200 practices. We'll also be leveraging a disk to disk backup strategy, using tape only for disaster recovery.
Network Infrastructure – We implemented a high speed network backbone with multiple paths for redundancy using
* Cisco Integrated Services Routers (ISR) 2811s for internet connectivity
* Adaptive Security Appliances (ASA) 5520 for Firewalling, Intrusion Protection and IPSec VPN Client termination
* Catalyst 4948 Switches for Server connectivity and layer 3 routing
* MDS 9000 Series Multilayer SAN Switches for SAN connectivity
Security – We incorporated physical, technical and administrative controls to protect confidentiality, integrity and availability.
SSL Accelerators - We are using Array Networks TMX-2000, the hardware recommended by eClinicalWorks to optimize web server performance.
Redundancy & Disaster Recovery - One of the real challenges to this project is the price sensitivity of our private clinicians. We needed to build a world class system at a price that all clinicians could afford. Redundancy and disaster recovery is like life insurance - it's a great investment only if you need it. We had to balance our infrastructure investment with total cost of ownership, given the fixed hospital contribution and physician frugalness. In the end we used the equation
Risk=likelihood of bad events * impact of bad events.
We believe that it is much more likely that a component will fail than an entire data center be destroyed, so we elected to build a highly redundant infrastructure in a single data center for now, expanding to a secondary data center once we have sufficient clinicians signed up to fund the new infrastructure. Networking gear, servers, power and cabling are duplicated within a commercial co-location facility. Storage is disk to disk redundant. Tapes are moved offsite nightly. Once the hardware is up, we'll work with Concordant, Cisco, EMC, HP, Array Networks, and the Co-Location facility to test physical hardware and operating system/database software redundancy. Then we'll install eClinicalWorks and run the redundancy tests again. We'll also engage Third Brigade at that time for intrusion/security testing.
We've written a comprehensive disaster recovery plan and if we lost the co-location facility due to disaster, we would recover the tapes from offsite storage and build a replica of the hosting environment (VMWare plays a key role here) and restore the data. The recovery point and recovery time objectives for this plan will be clearly communicated to all who sign up for service. The customer base for our Software as a Service solution is mainly small practices which operate Monday through Friday 8am-6pm. Our disaster recovery plan includes a solid practice/workflow specific contingency/downtime plan. We will also perform a mock downtime as part of each implementation.
By creating a highly redundant single data center with rack mounted servers, two tiered storage, virtualization and offsite tape backup, we believe we've balanced scalability, affordability, and maintainability. We go live this Summer and I'll let you know if we were right!
A key design requirement for the project is scalability. Our projected customer base is 300 clinicians and we have a fixed start up budget. However, we must design the infrastructure in a way that can cost effectively support the smallest amount of adopters as well as scale to thousands if our project is wildly successful. We debated two possibilities (metaphorically speaking)
a. Build a hotel, not knowing if anyone will ever check-in
b. Build a housing development, where the limits of expansion are only defined by available land
We decided on choice "b", starting with a robust foundation and adding new equipment and storage as we add clinicians. We standardized our central site equipment on products from HP EMC and Cisco, with guidance from our infrastructure partner, Concordant, and our equipment supplier CDW, ensuring it was easy to plug in additional hardware on demand. We invested a significant amount of time designing the central hosting facility, doing it right the first time. Over the years, I've seen CIOs rush through the design phase, only having to rebuild the infrastructure later when application performance did not scale. We partnered with our vendors to build something special, that if successful, could be a model for other medical centers and communities.
Considerations in designing our hosting infrastructure included:
* Supporting a user base that is remote, unmanaged, and diverse. We need to be able to identify any performance issues via end to end monitoring of all components
* Meet important security and privacy restrictions, as well as address liability issues (who is responsible for what)
* Understand infrastructure costs for a) start up b) additional capacity that occurs in bursts or steps, and c) variable requirements as practices go live.
* Provide connectivity to external parties (labs, claims, etc...) through interfaces which create additional security and performance complexities
* Address the limitations and performance of "last mile" connectivity through publicly available internet access (DSL, cable, etc.)
The infrastructure choices we made are:
Virtualized servers - VMware was the natural choice because of the scalability design goals. VM and V-Motion technologies also play an important part in redundancy, failure recovery and disaster recovery
Physical services - We debated rack mounted verses blade servers and elected to use powerful small footprint HP rack servers connected to fast multi-tiered storage. We computed the economics of blade servers verses rack mounted servers and the use of VMWare made small powerful rack servers the most cost effective solution.
Storage - We purchased a Clariion CX3-20 series SAN. We will go live with 11.1TB of total storage (2.1TB of fast, Tier 1 storage for database transactions and 9TB of secondary, Tier 2 storage for files). A single CX3-20 will allow us to expand in a modular fashion to accommodate up to 1200 practices. We'll also be leveraging a disk to disk backup strategy, using tape only for disaster recovery.
Network Infrastructure – We implemented a high speed network backbone with multiple paths for redundancy using
* Cisco Integrated Services Routers (ISR) 2811s for internet connectivity
* Adaptive Security Appliances (ASA) 5520 for Firewalling, Intrusion Protection and IPSec VPN Client termination
* Catalyst 4948 Switches for Server connectivity and layer 3 routing
* MDS 9000 Series Multilayer SAN Switches for SAN connectivity
Security – We incorporated physical, technical and administrative controls to protect confidentiality, integrity and availability.
SSL Accelerators - We are using Array Networks TMX-2000, the hardware recommended by eClinicalWorks to optimize web server performance.
Redundancy & Disaster Recovery - One of the real challenges to this project is the price sensitivity of our private clinicians. We needed to build a world class system at a price that all clinicians could afford. Redundancy and disaster recovery is like life insurance - it's a great investment only if you need it. We had to balance our infrastructure investment with total cost of ownership, given the fixed hospital contribution and physician frugalness. In the end we used the equation
Risk=likelihood of bad events * impact of bad events.
We believe that it is much more likely that a component will fail than an entire data center be destroyed, so we elected to build a highly redundant infrastructure in a single data center for now, expanding to a secondary data center once we have sufficient clinicians signed up to fund the new infrastructure. Networking gear, servers, power and cabling are duplicated within a commercial co-location facility. Storage is disk to disk redundant. Tapes are moved offsite nightly. Once the hardware is up, we'll work with Concordant, Cisco, EMC, HP, Array Networks, and the Co-Location facility to test physical hardware and operating system/database software redundancy. Then we'll install eClinicalWorks and run the redundancy tests again. We'll also engage Third Brigade at that time for intrusion/security testing.
We've written a comprehensive disaster recovery plan and if we lost the co-location facility due to disaster, we would recover the tapes from offsite storage and build a replica of the hosting environment (VMWare plays a key role here) and restore the data. The recovery point and recovery time objectives for this plan will be clearly communicated to all who sign up for service. The customer base for our Software as a Service solution is mainly small practices which operate Monday through Friday 8am-6pm. Our disaster recovery plan includes a solid practice/workflow specific contingency/downtime plan. We will also perform a mock downtime as part of each implementation.
By creating a highly redundant single data center with rack mounted servers, two tiered storage, virtualization and offsite tape backup, we believe we've balanced scalability, affordability, and maintainability. We go live this Summer and I'll let you know if we were right!
Langganan:
Postingan (Atom)