Prefiniti: Architecture and Updates

The old Prefiniti codebase (WebWare.CL and Prefiniti 1.0/1.5/1.6) was bleeding-edge at the time of its original implementation (circa 2007-2009), as it used a technique called AJAX (Asynchronous Javascript And XML), which allowed all navigation operations within the site to load only the parts of the page that were to be changed.

Essentially, Prefiniti implemented what today would be called a “container/fragment” approach, where a single container page’s DOM contains “div” elements with a specific ID attribute into which “fragment” pages would be loaded. In the case of Prefiniti, the container pages were called webwareBase.cfm, appBase.cfm, Prefiniti-Steel-1024×768.cfm, or prefiniti_framework_base.cfm (depending on which Prefiniti version we’re discussing). What all of these container pages have in common is a pair of HTML div elements called sbTarget and tcTarget, which stand for “sidebar target” and “time collection target”, respectively. sbTarget is normally a left-hand navigation sidebar containing an accordion control, while tcTarget is the main element to which application content is loaded and rendered. It is so named because the time collection component of Prefiniti was the first to use AJAX techniques.

There is a utility function written in JavaScript, called AjaxLoadPageToDiv(), which would take as arguments the ID attribute of a DOM element, and a URL which would be loaded into and rendered within that DOM element. If the DOM element was tcTargetAjaxLoadPageToDiv() would look within the loaded document for XML tags wwafcomponent, wwafsidebar, wwwafdefinesmap, wwwafpackage, and wwaficon. These tags (where wwaf stands for WebWare Application Framework) would determine the component name, contextual sidebar, package name, and icon of the content being loaded, and trigger a recursive load of the appropriate sidebar fragment into sbTarget.

The difficulty with this approach arose from the legacy of the application: the direct predecessor of WebWare.CL/Prefiniti was a simple order form for customers to order land surveys from a local surveying firm, Center Line Services. This original application did not use AJAX at all, and employed some legacy techniques in its use of server-side rendering, which I’ll explain here:

Prefiniti is implemented in a programming language and application server known as ColdFusion. Upon receiving an HTTP request for a ColdFusion template, which is denoted by a .cfm file extension, ColdFusion looks in the current directory for a file called Application.cfm, which it will run and render prior to the requested template. Application.cfm’s job is to set up session variables, application timeouts, cookies, etc. for things like user authentication and maintaining application state. If Application.cfm is not found in the same directory as the requested template, ColdFusion will traverse all parent directories up to the site’s document root until it finds one. Once Application.cfm is run and rendered, ColdFusion will run and render the template that was requested, and then look for OnRequestEnd.cfm (using the same directory traversal rules as used by Application.cfm), and run and render it.

This is not a bad technique, except that the original application on which WebWare.CL/Prefiniti was based used Application.cfm to render DOCTYPE, html, head, and body elements, along with a site header, navigation menubar, and a toolbar, and OnRequestEnd.cfm would close these tags, while any requested template would fill in the rest of the page body as appropriate.

The problem with this manifested when AjaxLoadPageToDiv() would request a fragment to be loaded into tcTarget and sbTarget, the fragment also being a ColdFusion template. Application.cfm would be processed in the normal way, and the header, navbar, and toolbar–which was only supposed to exist at the top of the page, above the sbTarget and tcTarget div elements–would be repeated within both sbTarget and tcTarget.

At this point in the application’s development, Application.cfm had grown tremendously complex, and I, as a relatively green CF developer, couldn’t figure out how to move the visual content out of it and into the container template (webwareBase.cfm et. al.) in order to fix the problem correctly. My solution at the time was to place fragments into subdirectories (tc, workFlow, socialnet, businessnet, etc.) of the document root, each subdirectory having an empty Application.cfm file within it, to prevent rendering of the parent Application.cfm within sbTarget and tcTarget. This worked, except that page fragments no longer had access to any session state, including the ID of the currently logged-in user.

My solution to this problem was to generate JavaScript on the server-side that would create front-end JS variables for each needed session variable, and have that JS code run when the application’s login form was submitted, and have AjaxLoadPageToDiv() pass all of those variables to fragment pages as part of the HTTP query string. This meant that all form submissions required custom JavaScript to build a GET request that would collect form fields’ values and submit them to the back-end, which is a horrible abuse of GET (the HTTP standards require that such submissions be POSTed instead, placing the form fields within the body of the request, rather than in the URL). It also meant that session timeouts were handled poorly, security problems were many, and adding new features to the application was complex and difficult, requiring a great deal of JavaScript code that bloated the initial load of the application to unreal proportions.

In the current re-factor of Prefiniti, these problems have nearly all been mitigated. Visual rendering has all been moved out of Application.cfm and into prefiniti_framework_base.cfm, the empty Application.cfm templates in the application subdirectories (tc, workFlow, socialnet, etc.), have all been removed, and page fragment templates now have full access to session state. The process to strip out dependencies on GET requests and huge query strings is in progress, and most of the JavaScript bloat will thus be easy to remove, future-proofing the application and making it secure, and much easier to maintain and extend. This also has the benefit that the server-side modules for core framework functionality and database I/O can be loaded once for the entire application and made available to page fragments with no additional effort.

UI updates are also on the way, by way of Bootstrap 4, making Prefiniti a modern, responsive, and mobile-ready platform for web applications.

Here’s to the future!

Advertisements

Why UTF-8 is a train wreck (or: UNIX Doesn’t Represent Everyone)

This post won’t go into the gory details of Unicode or the UTF-8 encoding. That ground has been covered better elsewhere than I could ever hope to here. What we’re looking at today is almost as much political as technical, although technical decisions play a huge part in the tragedy. What I am positing today is that UTF-8–for all its lofty compatibility goals–fails miserably in the realm of actual, meaningful compatibility.

The supposed brilliance of UTF-8 is that its code points numbered 0-127 are entirely compatible with 7-bit ASCII, so that a data stream containing purely ASCII data will never need more than one byte per encoded character. This is all well and good, but the problem is that aside from UNIX and its derivatives, the vast majority of ASCII-capable hardware and software made heavy use of the high-order bit, specifying characters for code points 128-255. However, the UTF-8 encoding either chokes on or specifies control characteristics using the high-order bit, to include encoding whether or not the character specified will require a second byte.  This makes 7-bit ASCII (as well as encodings touting 7-bit ASCII compatibility) little more than a mental exercise for most systems: like it or not, the standard for end-user systems was set by x86 PCs and MS-DOS, not UNIX, and MS-DOS and its derivatives make heavy use of the high-order bit. UNIX maintained 7-bit purity in most implementations, as mandated by its own portability goals, and UTF-8’s ultimate specifications were coded up on a New Jersey diner placemat by Ken Thompson, the inventor of UNIX, and Rob Pike, one of its earliest and most prolific contributors. UTF-8 effectively solved the problem for most UNIX systems, which were pure 7-bit systems from the beginning. But why should UTF-8’s massive shortcomings have been foisted upon everyone else, as if UNIX–like many of its proponents–was some playground bully, shoving its supposed superiority down everyone else’s throats?

It should not. The UNIX philosophy, like functional programming, microkernels, role-based access control, and RISC, has its merits, but it is not the only kid on the block, and solutions like UTF-8 that just happen to work well in UNIX shouldn’t be forced upon environments where they only break things. Better to make a clean break to a sane, fixed-width encoding like UTF-32, perhaps providing runtimes for both ASCII (including its 8-bit extensions) and the new encoding to allow software to be ported to use it piecemeal. At least with something like UTF-32, data from other encodings can be programmatically converted to it, whereas with UTF-8 with its two-bit 8th-bit meddling, there’s no way of knowing whether you’re dealing with invalid code points, kludgey shift characters, or some ASCII extension that was used for a meaningful purpose.

ArcaOS 5.0: UNIAUD Update

I have discovered that part of my UNIAUD  audio driver problem can be solved. Using the command line UNIMIX.EXE, I can manually set the speaker output level. Turns out that there was actually sound being generated, but only to the headphone jack.

There’s still another problem, however: desktop sounds repeat and are very noisy and filled with static.

I will be publishing a few screenshots of ArcaOS in the coming days.

VistA Innovation?

VistA cannot evolve if its MUMPS code is viewed as the unfortunately obsolete back-end for Node.js applications.

If we buy into the current prevailing wisdom that we should essentially leave VistA’s MUMPS code in maintenance mode, enshrining its current structure and shortcomings, we are implicitly asking for it to be slowly phased out, and replaced with something else.

Adding blobs of red and mouse support to ScreenMan forms is not VistA innovation.

Building hundreds of new RPC broker calls for consumption by Javascript code is not VistA innovation.

Building tools to paper over the cracks in KIDS and DIFROM is not VistA innovation.

Writing web frameworks that expose MUMPS globals and VistA RPCs is not VistA innovation.

Even if you use every DevOps tool and agile methodology that is trending on Reddit while you’re doing these things, it’s not VistA innovation.

We can wax eloquent at great length saying that lab and scheduling are the keys to the kingdom, but the very best lab and scheduling packages really aren’t VistA innovation.

We are at this point essentially putting lipstick on a pig. The pig may be a tremendously powerful and intelligent wild boar that can do thousands of things normal pigs can’t do, but wrestling with it will still leave a bruise.

That’s not to say that DevOps tools, web frameworks, packaging ideas, or any of these projects and ideas aren’t innovative. They are, and everyone who does that work deserves praise and appreciation for it. But these are accessories. Nice, useful, pretty, and even essential accessories. But are they VistA? No. VistA is 30,000+ MUMPS routines, written in a style that was in vogue during the Reagan administration.

VistA’s entire MUMPS codebase needs to be refactored. Not replaced, but refactored in a way that reflects all the great and useful techniques that computer science has taught us since the underground railroad went mainstream. And yes, I mean APIs. I mean separation of concerns. I mean (perhaps controversially) that the SAC needs to quit forbidding mnemonically useful identifiers, and instead start forbidding us to leak data through local variables. Well-defined interfaces, that cannot speak to any component of the software that is more than one layer of abstraction away from it. Interfaces forming a strong contract between the software and the developers who develop against it.

MUMPS is more than up for the task. We have scoped variables with NEW. We have call by value and call by reference. We can pass complex objects to and return complex objects from well-defined methods in the form of MUMPS extrinsic functions and the glorious dot operator. Every modern implementation of the MUMPS language supports at least 31-character identifiers and large routines, so that routine names like ZZTQPRL3 are now not only unnecessary, but indefensible.

VistA cannot survive if we have the hubris to maintain that its design is sacrosanct, and superior by definition to new technologies. Along with this, we can no longer pretend that medical software is any different from other complex software, nor can we lie to ourselves and say that MUMPS–or hierarchical database technology in general–is inherently superior to other database technologies in our domain, and finally, we cannot continue insisting that advances made in programming methodology and software architecture don’t apply to us.

It’s been asserted–but not once effectively proven or even rationalized–that these computer science concepts (layers of abstraction, interface contracts, APIs, and separation of concerns) somehow don’t apply to medical software, or to VistA. I’ve personally heard arguments ranging from “APIs are an attack vector” to “VistA is a living organism, unlike any other software.”

Poppycock. Absolute rubbish. So completely wrong as to be comical.

First, VistA is utterly loaded with APIs. Every time someone calls into Kernel or FileMan, that’s an API. Every time someone writes a new RPC, that’s an API. And every one of them is as much an “attack vector” as it is in modern software. The only real difference is that ours aren’t well-architected, ours don’t separate concerns, ours are poorly documented, ours require way too many arguments, and ours have horrible names that nobody can remember.

Second, software is software is software. The things that make an operating system unmaintainable make an EHR unmaintainable. The things that make a word processor maintainable make an EHR maintainable. Even the argument that hierarchical databases are somehow inherently better-suited to medical data than relational databases (or network databases, or any other database) is specious and silly. Perhaps this was arguably true in the 1970s, but it is not true today. Every data structure that you can represent in FileMan, you can represent in Oracle or MySQL or DB2, with minimal fuss. Look at Practice Fusion. Look at Amazing Charts. The hip, new EHRs are all based on modern databases. It can be done.

It’s been argued that MUMPS’ lack of schema makes it easier to change the database to match the evolution of medical data without re-writing the software. Again, rubbish. Once FileMan is in the picture, we are right back to employing a schema that requires UI modifications once we change it. FileMan enforces its own schema on data organization. True though it is that external modules like ScreenMan make it relatively easy to propagate schema changes into the user interface, this same sort of ease exists in relational databases with technologies like ORM, LINQ, and others. And today, there are methodologies that make it even easier to propagate schema changes all the way up to the UI. If software developers employs proper separation of concerns and strong interface contracts, changes to the schema are transparent to the UI.

VistA admits of no such discipline.

In VistA, user interface, business logic, schema definition, and data storage are tangled together like Christmas lights in the box in Grandma’s attic. You can’t even programatically define a new FileMan file; it’s all done through interactive green-screen UIs, and distributed in KIDS builds, the installation of which are notoriously error-prone.

MUMPS has the facilities to make all of these nightmares disappear, and where it shines is in throughput, robustness, and scalability. It has great facilities for indirection, data hiding, abstraction, and all the other tools we need to make VistA even more awesome than it is. Just takes some time and dedication. It’s also fast. Extremely fast. Like, bullet train fast.

And VistA is awesome. The developers in its community are awesome and have tons of enthusiasm. But today, its core infrastructure needs some serious attention. MUMPS and VistA are kind of like a gorgeous Tudor mansion: scores of beautiful, ornate, and useful rooms, but all the pipes leak, the wallpaper is peeling, and the light in that one downstairs bathroom is always flickering for some reason. And we’ve lost the blueprints.

The VA wants to bulldoze the house and put up a shopping mall, a Best Buy, and a McDonald’s. In the meantime, they’ll throw some glue behind the wallpaper and set up buckets underneath the leaky pipes.

But the house is free for public consumption and improvement! So instead of doing what they’ve been doing, let’s fix the plumbing, put in some new wallpaper, and fix the electrical system. And while we’re at it, we can add central heating and a gourmet kitchen.

That is VistA innovation.

Contradictions

What I want as a computer user vs. what I want as a computer programmer are often polar and contradictory opposites.

As a programmer, I want computer languages and operating systems that give me complete, direct, and possibly dangerous control over the hardware. The power to not only create blisteringly fast and efficient code, but also to bring down the whole machine on a whim. If I can break the hardware in the process, all the better.

This is the essence of my earliest exposure to programming: writing BASIC programs in MS-DOS, with assembly language for the performance-critical sections. Translating the assembly language into the hexadecimal digits representing machine code, loading these codes into a BASIC string, and calling the address of the string directly. Getting into the muck of segmented memory, PEEKing and POKEing memory directly, engaging in dirty tricks to push the hardware to its limits, writing self-modifying code to squeeze every bit of power out of every CPU cycle. The antithesis of the UNIX philosophy.

As a user, I want all of the above to disappear, and programmers to be forced into high-level, safe, and nominally interpreted languages, on protected-mode operating systems that erect impregnable walls between software and hardware. As a user, I essentially want my computer to be as much of an appliance as my toaster.

If I get what I want as a programmer, the users’ lives become frustrating, and the audience and ultimate reach of computing is reduced.

If I get what I want as a user, most programmers’ lives become tedium and drudgery, as they’re reduced from engineers to technicians.

However, if I get what I want as a programmer, perhaps computers become once again the exclusive domain of geeks who will never again equate a CD-ROM tray to a cupholder or ignore the distinction between the web and the Internet, or refer to crackers as hackers. Doesn’t sound like a bad outcome, from my perspective.

It’s probably better for humanity that I don’t get my way. Just don’t take away my old DOS machine and my old assembler, lest I become even more of a curmudgeonly old fart than I already am.

Hmm.

Memories and Dynamic Textboxes

Around 2000, I had the dubious honor of enhancing–under contract–a VB6 application that interfaced to MODBUS radio telemetry units. Instead of using a ListBox or some other appropriate control, it employed a control array of TextBox controls to visualize raw sensor voltages from remote units.  The kicker to all this was that the code was all attached directly to a timer control that would go out and poll the units (this subroutine was about 20 pages long, mixing up MODBUS parsing, UI updates, radio communication, and virtually every part of the application munged together). When this code needed to do something with the received data, it read it back from the dynamic text boxes it had created during its poll cycle.

Dynamic textboxes are bad, mmmkay?

Dynamic textboxes are bad, mmmkay?

I refactored the code into a much better system and UI without dynamic textboxes. The new UI showed a tree view of all remote units and allowed reporting on each one, as well as fancy charts and graphs. Each sensor on each remote unit could have a custom math formula to turn its raw voltages into human-readable data. My version also logged data to a SQL Server database for archival and later analysis.

I was supposed to be hired by the company that originally made the software in order to turn it into a full-fledged SCADA (systems control and data acquisition) suite, but various situations with them and the organization to which I was contracted precluded that job move.

I have long since moved into the Linux and medical systems world, most recently doing Node.js development and EHR support. But this, my first programming job, has always stuck with me as a real “baptism by fire” moment, with which many fun memories are associated. I still have a fond place in my heart for VB6–with all its warts–but the process of creating from memory the little UI mock-up for the image I used on this post (which was done in VB6 in a Windows NT 4.0 virtual machine on my Linux box) makes me realize how far we’ve come and why we should never hope to go back.

What I want in a platform…

Just a small wishlist/rant:

Hardware

No more software buttons. Why do hardware engineers feel the need to make my volume controls, mute button, and power button require intervention from the operating system? Do it in hardware. I really don’t want a volume change to have to wait on the kernel to schedule the handling of its interrupt.

Whether this is a laptop, desktop, or server, make the chassis out of steel that doesn’t flex. I’m not so weak that I need a one-ounce laptop to be satisfied with life.

Casework should be toolless, and components modular easily field-upgradable–even on a laptop. Come up with several form factors for a laptop chassis, and allow the guts to be upgraded over time. Standardize, standardize, standardize. The largest one or two form factors for a laptop should not make weight a consideration.

Plenty of USB ports is a definite must, but I also want internal 10/100/1000 Ethernet and one RS-232 serial port. Also, give me a modular bay that can accommodate a 5.25″ device (optical drive, or even a floppy).

Instruction Set and CPU Architecture

A nice, orthogonal CISC instruction set with lots of flexible datatypes and addressing modes. Lots of general-purpose registers. Give me an instruction to copy bytes, words, etc. directly from RAM to RAM. Don’t make me sidestep through a register. If you can’t do this in pure hardware, microcode it. I don’t particularly care if we’re big-endian or little-endian, but I do like the POWER ISA’s ability to switch endianness. I want a flat virtual memory address space. If we have hardware multitasking (which we should) and the ability to support both little- and big-endian byte ordering, allow me to have different endianness per task segment. Allow CPU microcode to be software-upgradable.

I want several rings of privilege, and although I’m aware that ring transitions require lots of work, please optimize them as much as humanly possible. Don’t give me CALL instructions that are so inefficient as to be less usable than combinations of many more primitive instructions.

Give me lots of I/O channels (with DMA, obviously). I/O processing should be offloaded properly to a specialized ASIC or ASICs. Each I/O bus (PCIe, VME, SCSI, etc.) should be connected to an I/O processor ASIC, and this processor should have a uniform API  in the firmware (see below) abstracting away the differences between the various attached buses.

Give me massive L1 cache, and good support for multiple CPUs. Multiple complete CPU support should be given more priority than multiple cores/threads per CPU.

Firmware

I want a firmware with a fully-realized CLI. Give me a dedicated Ethernet interface for firmware, and allow me to natively redirect console output with VNC over this Ethernet connection, or via ssh if the console is not in graphical mode. The CLI should give me the ability to enumerate all hardware devices, configure things like SCSI IDs, memorable device aliases, Ethernet MAC addresses, etc. I should be able to boot from any partition on any connected storage with a simple command. I should also be able to dump registers and do kernel debugging through this CLI.

The firmware should have an API that is fully supported in all CPU modes without hacks like virtual 8086 mode. This API should allow me to enumerate and access all devices and memory easily. It should be available to a first-stage bootloader.

More to come…