Monitoring Log4J messages on a browser via server push

Introduction

Most of the Java projects I’ve worked on have used Log4J as the logging framework. We normally change the severity level to ERROR on lo4j.xml before deploying our web application to a production system. But when something goes wrong, you may want to enable debugging for a short time and see the messages, and then turn the level back. There are a couple of ways you can change the severity level on the fly. You can either use a jsp file as explained here or a JMX solution as explained here or here

But we also wanted to take peek at the log data in real time so when we see an error being generated we can change the level back to ERROR and look at server’s log files for debugging. Something like doing a tail –f logfile.log from the server. In some places, developers are not given SSH access to server, and needs to contact the DBA for any log files. Even if the developer has access to it, SSH access may be limited to internal traffic, and if you want to fix an issue from home, you’re out of luck. A browser based solution would be ideal in these cases. So here we are discussing such a solution, which should be very easy to implement in your own project.

Implementation

• Place the liveLogger.jar on your WEB-INF/lib folder
• Add the following to your web.xml (before the closing web-app tag)

<context-param>
<param-name>LIVE_LOGGER_MAX_CLIENTS</param-name>
<param-value>10</param-value>
</context-param>
<servlet>
<servlet-name>LiveLogger</servlet-name>
<servlet-class>uk.co.jaimon.LiveLogger</servlet-class>
</servlet>
<servlet>
<servlet-name>LiveLogFeeder</servlet-name>
<servlet-class>uk.co.jaimon.LiveLogFeeder</servlet-class>
</servlet>
<servlet>
<servlet-name>LogSelector</servlet-name>
<servlet-class>uk.co.jaimon.LogSelector</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>LiveLogger</servlet-name>
<url-pattern>/logger/*</url-pattern>
</servlet-mapping>
<servlet-mapping>
<servlet-name>LiveLogFeeder</servlet-name>
<url-pattern>/liveLogFeeder</url-pattern>
</servlet-mapping>
<servlet-mapping>
<servlet-name>LogSelector</servlet-name>
<url-pattern>/livelog</url-pattern>
</servlet-mapping>

Monitoring the logs

Once you ran your application, point the browser to, http://<<your server>>/<<your context>>/livelog to get started. I’m assuming that you’ve a valid log4j.properties or log4j.xml file in your classpath. This utility itself uses Log4J for logging with the logger name LiveLogger. Here is a sample screen shot of what you should be seeing on the first page. This is where you can select a logger to change its severity or start streaming log data.

Once you select the logger, you’ll see log data being displayed on the left hand pane as shown below. You can also use this page to change the severity level of your selected logger. Once done, change the level back and stop the monitoring by clicking on Stop button.

How does it work?

Once we select the logger, livelog.html file opens an invisible iframe connecting to LiveLogFeeder servlet to enable a comet style server push. Different styles of comet programming are well documented, so I’m not going to repeat it here. I’m using the streaming technique here. The feeder servlet adds a custom appender to our selected logger, which will keep collecting log messages on to a queue. The servlet then pushes these messages in batch to a client queue with some pause (500 milliseconds by default), so your browser doesn’t freeze up. A JavaScript function on the client side process these queues in batch and puts each message on top of existing once, so the last one gets displayed on top.

Issues with streaming a lot of data to a browser

If there is an influx of messages, then it would severely affect the performance of your application. So I’ve limited the number of messages we can receive on the queue to a 1000.

Your browser will also struggle if we keep inserting a large number of items to the list. So there is a 1000 items restriction here as well. JavaScript function which places the message will also remove any old messages.

Since we are adding an appender, it is important we remove it as soon the client stops or the connection gets lost. We’re achieving this in two ways. On the client side, we’ve added a window.onbeforeunload JavaScript event to stop the monitoring when the user closes the tab or window, or going in to another website. On the server side, most JEE servers will throw an IOException when we try to write to a closed connection. So we can remove the appender and get out of our servlet once we get an exception. OC4J behaves slightly differently though, where it takes another minute or so for it to generate an IOException, when the connection gets lost. I’ve also added a bit of code to write some characters, if we haven’t written to the stream in the last minute to make sure we generate an IOException on closed connections.

Making changes to the configuration parameters

At the moment, the only thing that is configurable via web.xml is the maximum number of clients allowed. There are a few other parameters, which I haven’t bothered to get it from web.xml. These are listed here, and I believe all of them are self explanatory.

• maxNoOfMessagesToWriteOnce (LiveLogFeeder.java)
• intervalBetweenWrites (LiveLogFeeder.java)
• maxMessageQLength (LMAppender.java)
• MAX_LOG_ITEMS_TO_DISPLAY (livelog.js)

My Netbeans project files can be downloaded from here, if you want to make any changes and create the jar file. There is also an Ant target “createjar” which generate the jar file. You’ll need to change the location of the jar file though, as it is hard coded at the moment. If you are using any other IDE, create a new project and copy/paste contents of src and web folders to relevent locations.

Making improvements

There is scope for improvement in a number of places.

On the server side, we could use a separate thread to write messages to the client to reduce some of the overheads. But as you already know, creating threads from Servlet is a no-go area. Two JSRs (236 and 237) were created as possible solutions, unfortunately none of them made it to JEE6. While JSR 236 remains inactive, last year JSR237 was withdrawn. There is a commonj project, which can be utilized as a possible solution.

On the client side, inserting and removing elements via JavaScript takes the most time. I’m sure there are better ways to implement this than using insertBefore and removeChild methods in a loop.

I would appreciate, if you could let me know of any improvements you make to this project.

Files to download

Compiled jar file : liveLogger.jar

Source code (As a Netbeans project) : LiveLoggerV01.zip

Advertisements

Managing Vimeo video player via Javascript using Moogaloop API

Vimeo Moogaloop JavaScript Integration

Vimeo is a great web site for sharing videos. It doesn’t have any 10 minute video length restriction as in YouTube. It also has some developer tools to assist in integrating vimeo with your own site. My interest was in the Moogaloop API, and using JavaScript to load a list of files and control the player.

The Moogaloop JavaScript example you can download from vimeo wasn’t working for us (As of Dec 2009).  As it turns out, we weren’t the only people having this problem as this forum post shows. In that forum, one user posted a working example at this blog. His example was working for the first time, but not able to reload another movie. The issue was that the DOM element for the container was getting replaced the first time a movie loads, so the next call to embedSWF will fail due to non-existence of a named element. My minor fix was to check for element existence before calling embedSWF and if not found, create it with a document.createElement call.

In this post we are going to extend on that, and showing a sample where we get a list of movies for a particular user and display it in a list with thumbnail images. Clicking on each link will load the movie in the player area, and will start playing (if auto play is enabled).

To  demonstrate Moogaloop’s event system, I’m displaying a “Please wait” message at the top when somebody clicks on Play, and when we are past the first second of playing, we are  hiding this message. Please see Moogaloop’s API documentation for more details. Although I’ve added two event listeners to show the capabilities, onProgress event should only be used with some caution, as it fires a lot of events every second. If you are using it, make sure the code doesn’t run for more than a few milliseconds.

I’ve also included some wrapper methods to call Moogaloop’s API like play, stop, pause and seek at the end of the JavaScript file. One thing to remember is seek will only work with already buffered data. For example, if the movie is just loaded, and there is only 5 seconds worth data in the buffer, then seekTo(10) won’t work at all.

Using it with other JavaScript libraries

If you are including this script with any other JavaScript code, make sure the global variables I’ve used here don’t create a variable name conflict. Normally I tend to use private JavaScript variables as much as possible. But this post is not intended to be used as a library, but as a starting point to get Vimeo integrated, I haven’t isolated the code as I’ve done in my other blogs here and here.

See it in action

For a live demo, please click here.

Download code

You can download all the required  files as a zip from here.

Cheers,

Jaimon

Making Scrollable Tables with fixed headers

********** Please click here for an update script with additional functionalities like multiple header freezing, column freezing and multiple tables ***********

Although it is a common requirement, it’s not always straight forward to make scrollable tables with fixed headers. Here, we are going to try a few JavaScript solutions that could be suitable for most cases. I need to emphasize on that, because you might need to make a few tweaks to make it work in your environment, especially tables generated by web  frameworks.

Implementation

To implement these JavaScript solutions, your tables should have an ID and width. Width can be a fixed value or a percentage. Since the width can be specified in percentage, it should also work when window get resized. It has been designed to introduce this functionality with minimal changes to your existing pages. So for the JavaScript functions to work, you only need to add the following lines to your HTML pages,

<script type="text/javascript" src="fxheader1.js"></script>
<script type="text/javascript">
fxheaderInit('dataTable',300);
</script>

Make sure you add the second JavaScript call after the table element. Replace dataTable with the ID of your table, and 300 with the height you require. Instead of specifying the height as a fixed value, there are ways to stretch the table to fill the page based on other components on the page. Last solution make use of such a method.

Browser compatibility

It’s been tested on the following browsers all running on Windows XP.

• IE6
• IE7
• IE8
• Firefox 2.0
• Firefox 3.5
• Google Chrome

The big table

Here is an example of a table we are going to introduce the scrolling functionality. Well, it is not really big, just big enough to test our functionality 🙂

http://jaimonblog.appspot.com/datascroller/bigTable.html

Solution 1: Simple CSS only scrolling without fixing headers

With the help of a bit of CSS, we can implement a simple scroller. Enclose the table element within a div container with an overflow value of auto and a fixed height. Something like

<div style=’overflow:auto;height:300px;’>
<table …. />
</div>

You can see the demo at http://jaimonblog.appspot.com/datascroller/simpleScroller.html

Solution 2: Scrolling with fixed headers using table cloning

You can see a working demo at http://jaimonblog.appspot.com/datascroller/fixedHeaderFullClone.html

Download fxheader1.js and include it in your project as explained on the implementation section

How does it work?

• First we put a scrollable div container around the table element as in our first solution.
  • create a div element
  • clone the table element using cloneNode(true)
  • added the cloned table to this div
  • replace the table element with this div
• Create a container on top of the table to place the headers.
• Attach scrollHeader function to the table container div’s onscroll event, so that we can align the heading when the table is scrolled horizontally.
• We then clone the table and place it in the header container. Container height is set, so that only the header row is visible.
• We then set a negative margin top on the original table to hide the actual header.
• Attach fxheader function to window onresize event, so it works when window get resized.

Please note that, there are more than one way to do add a div element around our table element using JavaScript. You could speed up the process by adding these divs manually on the page as in Solution 1, because cloneNode function can be expensive. Make sure to use the correct IDs, so that rest of the code can find these. Please see the function addScrollerDivs in fxheader1.js for more details.

Solution 3: Scrolling with fixed headers using fast table cloning

You can see a working demo at  http://jaimonblog.appspot.com/datascroller/fixedHeaderFastClone.html

Download fxheader2.js and include it in your project as explained on the implementation section to get started.

Solution 2 will give you very accurate column widths at the expense of doing a full table cloning. If you have a table with lots of rows, this can take a while. If your rows have elements with unique IDs, you will have to change it on the cloned node to avoid  ID collision. Although it will work, it will lead to a sluggish user experience on pages with really large tables.

In solution 3, we are doing a fast clone without any child elements. Then we add the first row to it and set individual column width separately based on the offsetWidth property of each cell. You’ll have to offset cell margin/padding when setting the width. In our example, I deduct 3 from offsetWidth and that works fine for our table in all browsers I’ve tested.

Everything else is similar to solution 2.

Solution 4: Adding scrolling to table generated by ADF Faces 10g with fixed header (With and without pagination support) and automatic height stretching

Although in theory it is similar to solution 3, I had to make a few changes to make it work for a table generated by ADF Faces. The problem is ADF Faces creates three HTML tables for each af:table element. First and third tables are created for pagination support. It doesn’t create an ID for the actual data table itself, so we need to access it as a childNode of the container element. If pagination is there, we need to place this above our fixed header. Since we have the scrolling functionality, I thought the second pagination controls below the data table is not necessary, and I’m removing it from the page.

When cloning table data here, we have to update ID field of each element to avoid ID collision. We also have to set the correct selected index value for pagination selector combo box. Another functionality I’ve implemented here is the ability to auto stretch the table content to fit the page. You should pass in the ID of your last container component to make it work. Please see adjustHeight function in fxheader.js for more details.

As with the second and third solutions, you could add the following line to your JSPX page to get this working.

<afh:script source="fxheader.js" />
<afh:script text="fxheaderInit('testTable','footerContent');" partialTriggers="testTable" />

 

Where testTable is the ID of your af:table element and footerContent is the ID of your last container component for stretching. Partial Triggers are required here if used with pagination.

You can download a fully functioning JDeveloper project from http://jaimonblog.appspot.com/datascroller/ADF10gScroller.zip

Please note that it is created with  JDeveloper version 10.1.3.3. If you are using 11g, then fixed header functionality is already part of it. (Although there are a few issues with it at the time of writing this. One such issue is discussed here http://forums.oracle.com/forums/message.jspa?messageID=4032492#4032492)

As I’m not able to provide a demo link, you can see some screen shots of this solution here,