Subject: Re: [erlang-questions] List Question

>From what I've been able to glean about the HL7 message
format, there are two aspects:
* the basic syntax is the multi-level delimited thingy but
* there is also *semantics*, predefined message types,
an assortment of data types, rules for mapping data
types to trees and so on.

>From a quick look at the Elixir HL7 parser, they have
taken steps to handle some (but perhaps not all) of the
*semantics* of HL7 and don't just give a tree of strings,
but more structured data.

Parsing the multi-level delimited syntax is trivial.
Dealing with the semantics is not.
I think that in figuring out for yourself how to work
with HL7 messages in Erlang, the starting point would
- what message types do you want to handle?
- what kinds of data occur in them?
- how do you want to represent those kinds of
data in Erlang?
- do you actually want to represent *every* field
at all? Some might not be relevant to you.
- would you be streaming messages through a
system (like some sort of pub/sub queueing
middleware), summarising messages, storing
them, or what?
- what does a type declaration for a message type
look like in HL7? Is there some way to automatically
derive parsing code from that?

What I'm getting at with the last point is that there
is ASN.1 support for Erlang. Give it an ASN.1
definition, and you get Erlang code out the other end.
I am particularly thinking of the PADS project:

PADS: Processing Arbitrary Data Streams

Kathleen Fisher and Bob Gruber, AT&T Labs

Slides in ppt

Transactional data streams, such as sequences of stock-market buy/sell orders,
credit-card purchase records, web server entries, and electronic fund transfer
orders, can be mined very profitably. As an example, researchers at AT&T have
built customer profiles from streams of call-detail records to significant
financial effect.

Often such streams are high-volume: AT&T's call-detail stream contains roughly
300 million calls per day requiring approximately 7GBs of storage space.
Typically, such stream data arrives ``as is'' in ad hoc formats with poor
documentation. In addition, the data frequently contains errors. The
appropriate response to such errors is application-specific. Some applications
can simply discard unexpected or erroneous values and continue processing. For
other applications, however, errors in the data can be the most interesting
part of the data.

Understanding a new data source and producing a suitable parser are crucial
first steps in any use of such data. Unfortunately, writing parsers for this
kind of data is a difficult task, both tedious and error-prone. It is
complicated by lack of documentation, convoluted encodings designed to save
space, the need to handle errors robustly, and the need to produce efficient
code to cope with the scale of the stream. Often, the hard-won understanding of
the data ends up embedded in parsing code, making long-term maintenance
difficult for the original writer and sharing the knowledge with others nearly

The goal of the PADS project is to provide languages and tools for simplifying
data processing. We have a preliminary design of a declarative data-description
language, PADSL, expressive enough to describe the data feeds we see at AT&T in
practice, including ASCII, binary, EBCDIC, Cobol, and mixed data formats. From
PADSL we generate a tunable C library with functions for parsing, manipulating,
and summarizing the data. In joint work with Mary Fernandez and Ricardo Medel,
we are working to integrate PADS and XQuery to support declarative querying of
data sources with PADS descriptions.

The PADS project moved from AT&T to

I say this in all seriousness: if I had a need to process
GB of HL7 data, I would start by seeing if PADS was adequate
to describe it, and if so I'd write an HL7->Erlang data
translator in C or ML (as PADS has C and ML versions).
If not, I'd see what ideas I could steal from PADS.

Using a declarative data language to describe the message types
I was interested in would be an up-front cost, but it would
hugely simplify later maintenance.

erlang-questions mailing list
[email protected]

Programming list archiving by: Enterprise Git Hosting