Email is Lousy

Email sucks...

There I said it, and for as much as I rely on this post internet wonder drug, Email is surely one of the worst inventions of mankind for certain types information exchanges. You know the kind of exchanges where people used to get together face to face and iron out differences in a discussion. Sure email is fantastic for those times when one party needs to inform another party of some static topic. The fields in the file are as follows, the phone number of that person is, the directions to the party are, all are easily transferred to their recipients via email with little or no problems. Other discussions that have deeper meaning or have a bidirectional element woven into their fiber reveal a startling shortcoming in the typical email exchange. Most importantly email lacks that all important human element where one party can gauge the mood of the other side and adjust the conversation accordingly. In some ways email great strength for one task is also its downfall when applied to another task. Email is fast but its not immediate, it also has that ‘say it now and listen to it later’ element. This is all well and good for unidirectional information exchanges but falls apart in any form of multidirectional information exchange. Add this shortcoming to the lack of email to convey mood, and you have a recipe for trouble. I write this not to convey some great understanding onto the internet masses. I am sure this is not news to a great many of you folks out there in internet land, but more as a reminder to my own thickness on the subject and a warning to myself to avoid using the medium for things that it is ill suited for, maybe then I will avoid issues like last weeks debacle.

Mapping Data in HIPAA HELL

The following is not meant to be a review of any real depth but as the blogs title suggests, its more of a summation of my findings on a recent problem...


Its been a while now since I have relied on tools to help with something as simple in concept as taking a file and reading its contents into some form of database. I used to always take the approach that tools were for wusses for these kinds of things. Boy was I WRONG! There I said it, I was wrong, wasn't the first time, and won’t be the last either. My particular leverage point on this tipping was reading some HIPAA AnsiX12 834 eligibility files. Now HIPAA is a wonderful standard. (You know what they say about standards). Add to this mess the fact that implementation of this 'standard' often leads to the decidedly non-standard. In this case we were tasked with reading in some membership files cranked out of a state system for public sector healthcare.

I looked into a number of tools; each had to address the following needs.

Be able to produce code that would perform the function without any need to pay royalties to anybody. (Most of our code is actually in the ownership of the client thus the tools needed to create code that could also be in the client’s ownership)

Be able properly read the odd and downright nasty formatting quirks of a typical HIPAA ANSI X12 file.

Not cost an upper and lower appendage (something under 5 grand would be nice)

What I found, was a product called MAPFORCE from a company called ALTOVA. Mapforce is a powerful mapping and conversion tool designed to provide a nice GUI onto the mapping process. Using it is basically as simple as selecting some form of input and some form of output and at its simplest level connecting elements of the input to elements of the output.

Part of the tools power lies in its almost complete collection of EDI and X12 standards awareness. Other sources of power come from its ability to connect to just about any Database available via OLEDB/ODBC. Mapforce also handles traditional Flat ASCII text files with ease, and the last great thing in the software tool are the built-in functional blocks that the user can employ to translate/transform/conditionally act on the data as its passing from input to output.


The software allows you to take these bits of functional blocks and collapse them into user defined functional blocks that perform the task of many hidden behind a single function. Initially these user defined functions only allowed a single output, but during my testing and learning Altova released a new version (SP3) that allowed user defined functions to be multi output as well as multi input, a very welcome addition to the tools arsenal of weapons that can be brought to a problem.

Along the way, I encountered some problems. Most notable was the fact that transitioning from SP2 to SP3 broke the software in regards to reading one of my files. This is where I exercised the companies support option which is solely through the email. Now this has not ever been one of my favorite ways to get support on something, but I can say that its execution with the company ALTOVA was about as good as it gets, and I did get my problem resolved. Most of the real issues I encountered with support stemmed from their need for data, and not being able to freely share the problem files because of the personal health care information contained within the offending data. This would have been a problem with anyone using any form of support. In the end the company was able to help me resolve the issue and along the way fixed some things that were also not a critical problem but minor annoyances. Overall excellent and I can say the best email support I have ever experienced.

In the end, Mapforce proved to be more than I had expected. They have a complete working version available online that times out in 30 days otherwise no strings attached. The code generation can handle XSLT, XQuery, C#, Java or C++. The templates handle every kind of nutty file you might be able to throw at it. The programmability is very powerful. The support is more than adequate, and it does not cost an arm and a leg. Just what the doctor ordered. Now does this mean I will abandon my desire to write everything myself?

Hmmmm Maybe.....

Naaaaaaa.

Sysadmins I feel your pain

From an earlier post made in December 2005 elsewhere...

This past Saturday’s festivities proved to be a challenge ultimately ending in failure. You see, we attempted to migrate our crusty old Linux Mail/Web/Nat/Firewall/File server to a shiny new Windows Server 2003 machine doing all the same stuff.

We have used Linux for this task in the past, but since we are largely a Microsoft shop and our clients are also largely Microsoft in their own infrastructures we decided that perhaps it was time to eat our own dog food as it were. My one reservation was using Exchange for messaging (read EMAIL).

After setting up server 2003 on a new dell server class machine in-house and configuring that system for exchange, (Making it a domain as a requirement because exchange appears to need Active Directory, so we are a tiny company with Active directory setup oh the joys of that are indescribable -cough. In order to be a domain you need to configure the machine to also be a DNS server, never mind that your have your DNS potentially hosted elsewhere windows wants to be everything, so you allow it to at least think its the authoritative DNS which it really isn't)

At this point exchange is configurable and we can seemingly get attached to it via Outlook or should I call it LOOKOUT!. So on the morning of the migration we start copying email from our old SENDMAIL server via IMAP to the pristine exchange server via LOOKOUT. I should say moving mail because LOOKOUTS default is to copy then remove the message from the source, by the time we discover this little ditty its to late for some high profile mail accounts like the presidents. (Before you go blasting me YES I have a backup but that’s not the point).

On putting the server in place we set out to configure the firewall and NAT features. Numerous documents tell you various conflicting things but it appears that we need the full ISA version of the Microsoft firewall product, in order to be able to get exchange access remotely without requiring the remote site to connect to a VPN. In fact we also need the ISA for the full VPN class anyway so we go about setting up ISA product. Here is where we begin to get into trouble. Seems that ISA product pretty much takes over the machines IP/PORT traffic. The web server no longer responds and the DHCP server on this machine is no longer handing out leases. We futz about for some time and manage to get NAT happening again via the Wizards in the ISA product. We solve the DHCP issue by getting an internal database server to also hand out leases. The web server is however another matter The ISA product is clobbering port 80 traffic and there is no document that we can find that points out how to allow 80 to pass through. What we do find is three different accounts on how to move the IIS server to some nonstandard port like 8080 and then bridge 80 to it via a custom rule and a TCP listener. Ok we get we happening but when you attempt to connect to the web exchange client the 8080 redirect cause traffic to round trip back to your web browser and back to the server at 8080 where ISA blocks its because 8080 is not configured and you cant find how to configure a port to just pass through. I am sure the problem would not even exist if you did not attempt to have the one box running web server and firewall; Microsoft’s answer to all stability issues appears to be, add another server. I already said we were a small firm and 4 servers to support Mail, Nat/Firewall, Web and File/Print are out of the question. Don’t even get me started on the mass of contradictory documentation available from Microsoft in a number of their sanctioned places. I can say that nothing we read from the firm was conclusive and correct all day. The most valuable resources for us were third party accounts from what I am sure are forcefully balding others of our ilk who after navigating the same waters we were in to some form of safety, felt compelled to recount their own tales of woe in the hope that some other poor souls might be saved.

For all of you out there who helped with your writings, I thank you. I hope the hair plugs work for your newfound baldness. We on the other hand were forced to turn back. I realize that were are not true Sysadmins. I also realize that we were definitely in over our heads with this setup, but the poor documentation and general attitude of you should NOT be doing all this in one BOX does seem to preclude the companies products in very small operations like ours. Don't get me wrong I Love much of Microsoft's offerings. Visual Studio .Net is perfect for us and its result are perfect for our clients. Their infrastructure stuff however leaves me under-whelmed. Now, where are those Mail backups...