Need A CF App for Tracking Errors From Multiple Applications? Check Out BugLogHQ

Last month, during a brief lull in my work projects, I decided to tackle an issue I'd been putting off for awhile:  finding a better way of managing errors caught and reported by my ColdFusion applications.  All of my apps are designed to email me a full report of any errors that reach the global error handler, and while that means I'm immediately notified if there's a problem I need to address, the error message folder in my email account ends up serving as a de-facto error archive.  And now that we're (slowly) moving towards building applications as a team, it makes sense to have the errors stored in a centralized location.

After doing a bit of research, I ended up at the home page for the open-source bug logger BugLogHQ, created by Oscar Arevalo.  Based on the project changelog list, it sounded like it might have the features I wanted, but there wasn't a lot of explanatory documentation on the site and the screenshot images were thumbnails that couldn't be enlarged.  And it didn't support Oracle, which is our primary database engine at work (and a perusal of the Google Groups forum for BugLogHQ confirmed that).

Despite those factors, I decided it was worth trying to add Oracle support to the codebase so I could run it and try it out.  I'm glad I did:  BugLogHQ is a well-designed, robust application for managing errors in a way that will still let me keep track of what's happening with my apps.  Now that I've seen how it operates and what it can do, I'm kind of surprised that I had to do research to come across it:  either it completely slipped under my radar or it needs more publicity/exposure in the community.

Several things I like about BugLogHQ:

• I like the fact that it's easy to get up and running.  Download the files, put them in a "buglog" directory under the webroot (or use a mapping), run the SQL install script for your database of choice (mySQL, PostgreSQL, MS SQL 2000/2005, MS Access, and now Oracle) update the config file with your datasource name and database engine, and simply point your web browser to that "buglog" directory.  After logging in (which forces you to change the default password for the built-in admin user account), you can verify everything's working by going into the settings area of the app and running the tests, which all mimic the methods your other applications (referred to in BugLog parlance as "client" apps) can use to transmit errors to BugLogHQ.  Simple and straightforward.

• Just like most email clients have rules for filtering and acting on incoming email messages, BugLogHQ provides some rules for managing the errors that come in.  You can set up multiple "mailAlert" rules that will forward on particular errors (filtered by error type/severity, the source app's hostname or application name, or keywords in the configured error message text) to an email address of your choosing, or you can create a mailRelay rule that relays every error received to an email address.  And the emails sent out by BugLogHQ aren't just notifications about the errors:  they provide the full error details so you don't have to log into BugLogHQ to get more information.  There are also rules for discarding certain errors or for monitoring "heartbeat" transmissions from an app, so if the app is offline for a certain length of time BugLogHQ can report the problem.

• The methodology BugLog uses to report errors is very well-designed.  If your client app is also a CFML app, you simply instantiate a single CFC (with settings for interacting with the BugLogHQ web service you want to utilize) in either the application scope or a bean factory, then call it with its notification function whenever you want to send information to the BugLogHQ app.  If the client app is reporting an error, you can send BugLogHQ a line of message text, the error struct (the raw "cfcatch" struct), the type/severity of the error, and a struct of any additional information you want as part of the report (like the affected user's ID or name). Here's how I called it from the main.error() controller function in the FW/1 application I used during my testing (where I've instantiated the CFC as a service object):

  public void function error(rc) {
      var extraInfo= {username= session.user.getUsername()};
      variables.bugLogService.notifyService("Error report from TestApp1",request.exception,extraInfo,"INFO");
  }
  

  All of those parameters are optional, so you don't need an error struct if you're reporting something that's a status or informational message rather than an error.  If there's a problem with sending the report to BugLogHQ, it'll fallback to sending an email containing all that information to whatever address you specify so the error doesn't get lost.

• I also appreciate how BugLogHQ "eats its own dogfood".  If an error occurs in BugLogHQ itself, it'll record the error just like it would for any client app, and if the error occurs during the recording process it'll report the error via email.

• BugLogHQ utilizes a scheduled task (with a default interval of 2 minutes) referred to in the app as the "BugLog Listener Service" to denote and process incoming error messages.  When the task fires, the error messages that have been received since the last run of the task are processed and then written to the error database.  If there's a problem with writing to the database (which I understandably encountered as I was tweaking the code to support Oracle), it'll maintain the errors in the queue until the next iteration (and you can view the contents of the queue within the BugLogHQ dashboard).  Sometimes you need to restart that service in order for certain changes to take effect. If you somehow forget to reactivate the service, it'll be restarted when it receives a new error report from a client app.

• If one or more of your clients apps are NOT CFML-based web applications, there are library files provided that let you submit errors from PHP or Python apps, or straight from Javascript.  I haven't tried out those files yet, but we do have a few PHP apps in our shop now that could make use of the PHP option.

• There are configuration options for tying BugLogHQ to a JIRA instance, providing error summaries via RSS or a digest email, and setting an API key to ensure that only your apps can submit data to BugLogHQ.

Bottom line:  if you don't yet have an error-logging application or other mechanism for storing and managing errors from your application(s), you should take a look at BugLogHQ.  Oscar and his contributors have done a great job with this application.

dirtyFields jQuery Plugin Updated with New Options/Functions

I've made some updates to my dirtyFields jQuery plugin.  Here's the rundown:

• Added two new public functions:
  • getDirtyFieldNames() returns an array of all currently dirty fields.
  • updateFormState() checks the clean/dirty state of all form fields in the specified container
• Made two changes to how the CSS class denoting a dirty/changed form is applied:
  • Added a new configuration option ("self") to apply the class to the actual form element.
  • Split the single option for applying a style to a changed text input and select drop-down into two separate options for granular control (if you used the textboxSelectContext option with a previous version of the plugin, you will need to update your code).
• Added three new configuration options to control plugin behavior:
  • The denoteDirtyFields option controls whether or not the dirty CSS class is applied to the form elements that are dirty/changed.
  • The exclusionClass option specifies the name of the CSS class that, when applied to a form field, will exclude that field from being processed by the plugin.
  • The ignoreCaseClass option specifies the name of the CSS class that, when applied to a form field, will instruct the plugin to do a case-insensitive evaluation of the current and original states of the field.

All of these changes were implemented in response to code suggestions made a team of developers (listed in the GitHub readme.txt file) who modified the plugin to meet their specific needs in one of their intranet sites.

Quick ColdFusion Tip: Use ClearParams() When Doing SQL Inserts in a Loop

Today I had the need to write code that would loop over a recordset, parse the data in each iteration, and then insert the transformed data into another table.  Nothing I haven't done countless times, but this time I was using the query functions in cfscript to do my inserts.  I knew the resulting code was going to be a bit slow (as inserting thousands of rows takes time), but when I ran it it was a LOT slower than I expected.

After adding some code to time the various parts of my routine, I discovered that the time it took for each insert transaction was steadily growing with each iteration of the loop, to the point where it was taking 500+ milliseconds per insert.  But why?

Then I saw the problem.  I had forgotten to invoke the query object's clearParams() function at either the beginning or end of my loop.  Apparently ColdFusion will let you create a query parameter with the same name attribute using addParam() - as was happening in my loop - and not throw an error (which is what I would have expected to happen), but it leads to a performance issue with the SQL execution.

In the few times where I've reused a query object with different parameters, I've been careful to use clearParams(), but I simply overlooked it this time.  Lesson learned.

 

Adding a Time Selector to the jQuery UI Datepicker Widget

One of the functional requirements for the voter registration application I blogged about recently was that the application should not allow further registrations between the registration deadline (October 16 at 9pm) and a date after the election specified by the state Board of Elections.  For the initial run of the application, I simply hard-coded the deadline and restart date into the application logic, knowing full well I couldn't leave it that way unless I wanted to personally change the code year after year....which I don't.

So this week I set out to write a tool within the administrative interface of the application that would allow a non-programmer to update the deadline and restart date every year.  The jQuery UI Datepicker widget is my tool of choice when it comes to having users enter or edit a date, but I've used a few different approaches to having users enter a time of day.  This time around, I decided to see I if could find something comparable to the Datepicker widget for setting the time.

What I found was a rather sweet plugin called the Timepicker Addon that adds a set of time controls to the jQuery UI Datepicker.  If you customize your jQuery UI download to include both the Slider and Datepicker widgets, you can present the time controls as sliders, like so (without the Slider widget, you get select boxes):

The plugin comes with a number of configuration settings so you can do things like adjust the time increments, change how the time is displayed, and allow the user to denote the time zone associated with the time value.  Once I had the plugin configured the way I wanted, I simply had to write some code to validate the date and time string submitted from the form field, and I was done.  Very cool.

Simple Triangles with CSS

Recently, I had to create JavaScript-based replacements for 2 Java applets in an application I inherited.  One of the applets was a collapsible hierarchy tree of organizational units that used the conventional triangle icons (pointing right for closed, pointing down for open) to indicate state.

In past applications, I'd simply created actual triangle icons of the appropriate color for this sort of thing, but this time I decided I wanted to do it in pure CSS. 

After a quick search on the web (The Shapes of CSS is a nice resource on this topic) and a bit of trial-and-error, I came up with some CSS triangles that offset nicely from their tree items (and coded a blank triangle for items without children).  They work in all the common modern browsers and in IE as far back as IE 7:

.arrowNone {
  border-left: 5px solid transparent;
  border-right: 5px solid transparent;
  border-top: 10px solid transparent;
  float:left;
  height: 0; 
  margin-right:5px;
  width: 0; 	
}

.arrowOpen {
  border-left: 5px solid transparent;
  border-right: 5px solid transparent;
  border-top: 10px solid #666;
  float:left;
  height: 0; 
  margin-right:5px;
  margin-top:6px;
  width: 0; 
}

.arrowClosed {
  border-bottom: 5px solid transparent;
  border-left: 10px solid #666;
  border-top: 5px solid transparent;
  float:left;
  height: 0; 
  margin-right:5px;
  margin-top:4px;
  width: 0; 
}

Granted, they're not the smoothest of triangles, but they worked well enough for this purpose.

Getting Out the Vote, ColdFusion-Style

Normally I don't blog about the actual ColdFusion web applications I create at the University of Maryland College Park (UMCP).  Most of them are extranet applications inaccessible to the public and computerize business processes that would take too long to explain.  But my most recent application doesn't fit the normal mold and it's received some attention from the local media because of what it does.

The application is an online application form for registering to vote in the State of Maryland in November designed specifically for our students at the university.  It was commissioned by our undergraduate Student Government Association (SGA) as part of a larger, multi-faceted effort to get out the vote this year, and supported by the university administration and the State Board of Elections (who provided the guidelines for how the data would be collected and sent to them).

The application runs on Adobe ColdFusion 9 and uses FW/1 as its application framework.  The model is comprised of record and service objects, all written in script syntax, with the service objects instantiated via ColdSpring.  Most of the client and server-side validation is handled by ValidateThis, and the application is styled with Bootstrap as well as some custom CSS and a bit of jQuery.

After reading the introduction page, students log into the application using our university's single-sign on solution.  As soon as they're logged in, the system pulls information about the student from our LDAP system.  That information is used to pre-populate certain form fields (name, date of birth, etc.) and it also affects the behavior of the application.  For example, if the LDAP information indicates that the student has a home ("permanent") address in Maryland that is different from their current on-campus / near-campus address, it will show both addresses to the student and let them choose which one they want to use as their residential/voting address. 

But if the student only has one Maryland address on file, they're only presented with one set of address form fields.  And in both cases, the addresses pulled from LDAP are formatted to conform to the way the state recognizes voter street addresses, making it more likely that the submitted address exactly matches an address on file with the state (that part took awhile: the state parses street addresses in a rather unique manner).

In short:  if the LDAP information is accurate and up-to-date, most students can complete the online registration without ever typing a key on the keyboard.

Perhaps the coolest part of the application is the last step prior to the review page.  In that step, the student has to acknowledge that they understand the legal requirements to vote.  They acknowledge this by clicking on a checkbox next to an image of their electronic signature.  When students enroll at UMCP and get their student photo ID, they write their signature on an electronic pad, and the signature is stored in binary format in a database table.  It was the fact that we had these signatures on file that made this project viable, because the new Maryland law that permits online voter registration made the submission of a signature a mandatory requirement.  And thanks to ColdFusion's image manipulation functions, rendering that binary data as an image takes but one line of code.

Admittedly, we did run into one issue with the signatures:  some of the older ones were stored in a proprietary format that couldn't be read outside of the ID card system.  But we found an option in that software that let us export large batches of those signatures as JPEG files, and from there I used ColdFusion's image functions to reduce each image to a reasonable size and write them to a new database table in binary format.

Once the student reaches the review page and submits their voter registration application, the form data is written to a table record and their electronic signature is output as a file to a directory outside the web root.  And every night a scheduled task runs that writes out all the new registration records to a data file in the format specified by the state and uploads that file and the related signatures files to a server run by the state.

And, like most of the applications I build, there is also an administrative interface that certain individuals can log into to view some anonymous registration statistics and to update the list of politicial parties currently recognized by the state (as they change from year to year).

The system went live October 1st, and as of right now over 1650 students have used it to submit voter registration applications.  The SGA is promoting the application at every opportunity and they're hoping to get to 2,000 registrations before the October 16th deadline.  While that might not seem like a huge number, I believe it would be the highest number of on-campus registrations since the SGA started doing voter registration drives several years ago.

Once we hit the October 16th deadline, we'll shut down the application until after the election has past, then reopen it.  The goal is to let the system accept registrations year-round, and I already have a few small improvements planned based on user and stakeholder feedback.  I also need to add a tool or two to the administrative interface so the stakeholders can run the system without my help. 

But otherwise, I'm pretty happy with how it turned out, and so are the SGA and the other stakeholders.

Tip on Combining Column Filtering and Column Hiding with jQuery DataTables Plugin

I'm a big fan of the jQuery plugin DataTables.  I use it in a number of projects to enhance HTML tables, making them sortable and searchable.

DataTables provides a number of advanced options and functions that let you customize the table's functionality.  One of these functions is fnFilter(), which you can use to filter the table contents based on the presence of a value in a particular column.  The most common use case for this function is to add text inputs to the header or footer of each column, and bind the use of the function to the keyup event, as in the official DataTables example:

$("tfoot input").keyup( function () {
    /* Filter on the column (the index) of this element */
    oTable.fnFilter( this.value, $("tfoot input").index(this) );
} );

 

...The "oTable" is a reference to the DataTables-enhanced table, the first parameter is the value of the input box in the footer, and the second parameter is the index value (position) of the column in the table.

DataTables also provides the fnSetColumnVis() function for showing/hiding columns, which is helpful if you have a table with a lot of columns and want to let the user get rid of the columns they don't need at that moment.  It takes two parameters, the index position of the column you want to hide and true or false to set the visibility state of the column:

oTable.fnSetColumnVis( 1, false );

 

Both are very useful functions, but you have to be careful when you enable both column filtering and column hiding on a single table. Hiding a column takes it out of the DOM, and therefore it changes the index position of every column to the right of the hidden one. While that makes sense, the problem is that DataTables keeps internal track of the original DOM positions and will execute the filter against the column with that original index value.

So say you have a table with columns A, B, and C going left to right.  DOM index positions start with 0, so column A's position is 0, column B's is 1, etc..  If you hide column A, then column B now has index position 0 and column C has position 1.  When you try and filter column C, fnFilter gets the current position of column C (1) and runs the filter against the column that originally had index position 1...which is column B.  So the table gets filtered based on column B even though the user entered their filter term in the footer of column C, which is obviously not what you want.

You can work around this issue by recording the initial index position of each column in an attribute in the input tag, and use that value as the second parameter for fnFilter().  So if you used an attribute like "colPos":

...
    <tfoot>
        <tr>
            <th><input type="text" value=" colPos="0"/></th>
            <th><input type="text" value=" colPos="1"/></th>
            ...
         </tr>
    </tfoot>
...

 

...then you'd rewrite your keyup bind function like so:

$("tfoot input").keyup( function () {
    /* Filter on the column (the index) of this element */
    oTable.fnFilter( this.value, $(this).attr("colPos") );
} );

 

That will keep the column filters tied to their original columns.