What are good tests for any well-behaved OCAP application?
One place to start is by devising tests based on specifications such as the CableLabs document "OCAP Well-Behaved Applications", issued to the public on July 22, 2008. A copy is available from the CableLabs website at the URL
In this OEDN book the concepts in the CableLabs document are extracted and transformed into tests. In the first step we put on our system engineering hats and read the document for testable ideas. Each concept is numbered and is placed on its own page.
Next each concept is treated like a requirement - not in the legal sense but more in the sense of an guideline for the application to be well-behaved. The concept is expressed as an "assertion" which is the basic testable unit often used by professional software testers. This wording style makes it easier to evaluate if the guideline has been met or not.
Some examples of assertions are:
and when applied to an OCAP application:
Finally test procedures describe a recipe for demonstrating if the assertion is true for a particular application. Written procedures list a sequence of steps and expected results. At the end of the steps it should be clear if the application meets the guideline. Wherever it is pragmatic automated test procedures are preferred to manual ones.
Our objective is to accumulate an archive of tests for well-behaved OCAP applications. Naturally, application designers will want to virtually walk through the tests to ensure they have not missed or violated any of the assertions. And, application testers can add to their arsenal of other tests.
We invite you, the OEDN community, to contribute to these tests. Help us grow the suites.
Title: No intentional blocks in XLet entry points
Number: 10
Specification reference: (Section 6.1.1 page 6)
"There are five entry points defined by the Xlet interface.
Assertions:
Test Procedures:
Task - Expected result
Title: No argument constructor does nothing
Number: 20
Specification Reference ( Section 6.1.1.1 page 6)
" (the no-argument constructor for the base class of the application) constructor is not expected to do anything, and need not be implemented by the application. Instead, all application initialization is expected to be handled by the initXlet() method."
Assertion
There is no executable code in the no argument constructor for the base class of the application.
Test Procedures
Task - Expected result
Alternative procedure
Task - Expected result
Title: Extended I/O uses background threads
Number: 30
Specification Reference: (page 7, section 6.1.1.2)
"It is recommended to use background threads for any extended I/O operations at this point, returning from initXlet() as quickly as possible."
Assertions
If the application performs any I/O operation in initXlet() then background threads are used.
Test Procedures:
Task - Expected result
If the application performs any I/O operation in initXlet() then background threads are used.
Alternative procedure
Task - Expected result
If the application performs any I/O operation in initXlet() then background threads are used.
Title: initXlet() proscribed functionality
Number: 40
Specification reference: (Section 6.1.2 page 7)
“While it is possible for applications to receive AWT focus events or user input events while in the paused state, an application SHALL NOT request focus, install AWT or DVBEvent listeners, or make itself visible within initXlet().”
Assertions:
1. The application does not request focus
2. The application does not install AWT
3. The application does not install DVBEvent listeners
4. The application does not make itself visible within initXlet()
Test Procedures:
1. Manually inspect the source code
2. Use Java reflection API or other tools to inspect the application .class files
Task – Expected result
1. The prohibited invocations are not found.
Title: Applications should perform certain operations within startXlet()
Number: 50
Specification reference: (Section 6.1.3 page 7)
“Applications SHOULD perform the following operations within startXlet():
• Allocate all major blocks of memory;
• Start any background threads for loading data from the file system;
• Install all event listeners;
• Make the initial GUI components visible;
• Request AWT focus, if using AWT events;
• Otherwise prepare the application for execution.
… The MHP specification indicates that it will be called immediately after the successful return of initXlet() and also when returning from a paused state."
Assertions:
1. Within startXlet() the application allocates all major blocks of memory
2. Within startXlet() the application starts any background threds for loading data from the file system.
3. Within startXlet() the application install all event listeners
4. Within startXlet() the application makes the initial GUI component visible
5. Within startXlet() the application requests AWT focus, if using AWT events
6. The application calls startXlet() immediately after the successful return of initXlet
7. The application calls startXlet() immediately after the successful return from the paused state
Test Procedures:
1. Manually inspect the source code.
2. Use Java reflection API or other tools to inspect the application .class files
Task – Expected result
1. The invocations are found within startXlet()
Title: startXlet() re-initializes only those released while paused
Number: 60
Specification reference: (Section 6.1.2 page 7)
“Applications SHOULD take care to only re-initialize objects or data that were released while in the paused state.”
Assertions:
1. The application only re-initialized objects that were released while in the paused state.
2. The application only re-initialized data that were released while in the paused state.
Test Procedures:
1.
2. Manually inspect the source code.
3. Use Java reflection API or other tools to inspect the application .class files
Task – Expected result
1. The invocations are found within startXlet()
Title: pauseXlet() releases resources
Number: 70
Specification reference: (Section 6.1.1.4 page 8)
“Applications SHOULD release all reserved DAVIC resources during pauseXlet().
Applications SHALL be prepared to lose reserved resources while in the paused state that were not released in pauseXlet().
Applications SHALL NOT be visible while paused, SHALL hide their UI, and SHALL NOT request AWT focus during pauseXlet()."
Assertions:
1. Within pauseXlet() the application realease all reserved DAVIC resources
2. Within pauseXlet() the application is prepared to lose reserved resources that were not released in pauseXlet()
3. While in pauseXlet() the application is not visible
4. While in pauseXlet() the application hides their UI
5. While in pauseXlet() the application does not request AWT focus.
Test Procedure 1:
1. Manually inspect the source code.
2. Use Java reflection API or other tools to inspect the application .class files
Expected result
1. The invocations are found within pauseXlet() to release resources
2. The application does not request AWT focus within pauseXlet()
Test Procedure 2:
1. Execute the application
2. Obtain Focus
3. Force the application to enter the paused state
4. Attempt to trigger the application via controls on the application GUI
Expected result
1. The application is not visible
2. The application does not have focus
Title: destroyXlet() follows MHP directions
Number: 80
Specification reference: (Section 6.1.1.5 page 8)
“Applications SHALL follow the directions for this state transition as described in [MHP] clause 11.7.1.2:
• Cause any threads that they have created to exit voluntarily;
• Stop, deallocate and close any JMF players that they have created;
• Stop and destroy any JavaTV service selection ServiceContext objects that they created;
• Release any other scarce resources that they created, e.g., NetworkInterfaceControllers if they do any
tuning;
• Flush any images using the Image.flush() method;
• Xlets shall not cause any unnecessary delay in their Xlet.destroyXlet method;
• De-register any event listeners.
In addition to these tasks, an application SHALL terminate any active timers. Also, an application SHALL NOT sleep during destroyXlet()."
Assertions:
1. Within destroyXlet() the application causes any threads that they have created to exit voluntarily
2. Within destroyXlet() the application s tops, deallocates and closse any JMF players that they have created
3. Within destroyXlet() the application stops and destroys any JavaTV service selection ServiceContext objects that they created
4. Within destroyXlet() the application releases any other scarce resources that they created
5. Within destroyXlet() the application flushes any images using the Image.flush() method
6. Within destroyXlet() the application does not cause any unnecessary delay
7. Within destroyXlet() the application de-registers any event listeners
8. Within destroyXlet() the application terminates any active timers
9. Within destroyXlet() the application does not sleep
Test Procedures:
1. Manually inspect the source code.
2. Use Java reflection API or other tools to inspect the application .class files
Expected result
1. The appropriate invocations are found within destroyXlet()
2. The appropriate invocations are not found within destroyXlet()
Title: Self termination follows MHP directions
Number: 90
Specification reference: (Section 6.1.1.5 page 8)
“An application can terminate itself by calling XletContext.notifyDestroyed(). In this case, the implementation willNOT call the destroyXlet() entry point, so the application SHOULD perform all of these functions before calling notifyDestroyed().”
Assertions:
1. Prior to calling notifyDestroyed() the application causes any threads that they have created to exit voluntarily
2. Prior to calling notifyDestroyed() the application s tops, deallocates and closse any JMF players that they have created
3. Prior to calling notifyDestroyed() the application stops and destroys any JavaTV service selection ServiceContext objects that they created
4. Prior to calling notifyDestroyed() the application releases any other scarce resources that they created
5. Prior to calling notifyDestroyed() the application flushes any images using the Image.flush() method
6. Prior to calling notifyDestroyed() the application does not cause any unnecessary delay
7. Prior to calling notifyDestroyed() the application de-registers any event listeners
8. Prior to calling notifyDestroyed() the application terminates any active timers
9. Prior to calling notifyDestroyed() the application does not sleep.
Test Procedures:
1. Manually inspect the source code.
2. Use Java reflection API or other tools to inspect the application .class files
Expected result
1. The appropriate invocations are found prior to calling notifyDestroyed()
Title: Components change highlighting as focus obtained and lost
Number: 100
Specification reference: (Section 6.2.3 Focus Switching Behavior page 10)
“Applications that display a UI without the presence of a mouse or other pointing device generally highlight the focused Component by changing the color of the Component. … Applications that display a UI SHALL implement either the FocusListener or the WindowListener interface to determine when it loses focus or is no longer the top HScene on the stack and SHALL remove the highlight from any display elements when either of these conditions occurs.”
Assertions:
1. Focused components are highlighted
2. The application implements either the FocusListener or the WindowListener interface
3. When the application loses focus the focused components are no longer highlighted
4. When the application is no longer the top HScene on the stack the focused components are no longer highlighted
Test Procedure 1:
1. Manually inspect the source code.
2. Use Java reflection API or other tools to inspect the application .class files
Expected result
1. One of the interfaces is found
Test Procedure 2:
1. Execute the application
2. Obtain focus
3. Lose focus
Expected result
1. Each component when it receives focus is highlighted
2. Each component when it loses focus is no longer highlighted
3. When the application loses focus the focused components are no longer highlighted
4. When the application is no longer the top HScene on the stack the focused components are no longer highlighted
Title: Keys not defined in OCAP are not expected
Number: 110
Specification reference: (Section 6.3.3 OCAP Key Processing page 12)
“Such keys may or may not be passed to the OCAP environment; therefore, OCAP applications SHALL NOT expect to receive keys that are not defined in [OCAP] section 25.”
Assertions:
1. Only keys defined in section 25 of the OCAP specification are used by the application.
Test Procedures:
1. Manually inspect the source code.
2. Use Java reflection API or other tools to inspect the application .class files
Expected result
1. For each key used by the application it is matched to a key on the list of OCAP defined keys.
Title: Application functions properly with only the Focus Keys
Number: 120
Specification reference: (Sections 6.3.4 and 6.3.4.1 page 13)
“An application should be designed to function properly with only the Focus Keys. An application SHALL reserve other keys only while in focus and SHALL release all such keys when focus is lost or a release request is received."
…
Applications SHALL only reserve the Focus Keys events while they are in focus. When an application detects that it no longer has focus, it SHALL release any reservations on these events.”
Assertions:
1. When the application has focus is the only time it attempts to reserve the focus keys.
2. When an application detects that it no longer has focus it releases any reservations on these events.
Test Procedures:
1. Manually inspect the source code.
2. Use Java reflection API or other tools to inspect the application .class files
Expected result
1. The only paths in the code which lead to key reservation are found when the application is in the focused state.
Title: Focus gained or lost is detected
Number: 130
Specification reference: (Section 6.4.4 Stack Events page 17)
“Applications can obtain a ResourceServer from an OcapScene, through which they can register a ResourceStatusEventListener. This listener will provide notification when the HScene is "in front" or has lost its front most position in the z-ordered stack of HScenes.
Alternatively, the rules that tie AWT focus to the top HScene on the stack allow an application to more deterministically use the FOCUS_GAINED and FOCUS_LOST events to determine both when the application loses focus and when its HScene is no longer on the top of the stack."
Assertions:
1. The application obtains a ResourceServer or uses FOCUS_GAINED and FOCUS_LOST events
Test Procedures:
1. Manually inspect the source code.
2. Use Java reflection API or other tools to inspect the application .class files
Expected result
1. The invocations are found within the application
Title: Active rendering rules are followed
Number: 140
Specification reference: (Section 6.5 page 17)
“While this technique is essential for animations, several rules apply to those application doing active rendering, to prevent those actions from disrupting AWT's control of the other applications being rendered:
• An application SHALL NOT perform any active rendering unless that application has the front-most HScene on the stack of visible HScenes.
• Even when doing active rendering, an application SHALL paint its entire display (subject to clipping) when paint() is invoked.
• When doing active rendering, the application SHALL be responsible for all display updates, including clearing or erasing of previous contents, if necessary.
• Active rendering is not expected to play well with inter-scene blending. An application SHOULD disable inter-scene blending via HScene.setBackgroundMode() when doing active rendering."
Assertions:
1. The application does not perform any active rendering unless the application has the front-most HScene
2. When doing active rendering and paint() is called the application paints its entire display
3. When doing active rendering the application is responsible for all display updates
4. The application does not disable inter-scene blending via HScene.setBackgroundMode()
Test Procedure:
1. Manually inspect the source code.
2. Use Java reflection API or other tools to inspect the application .class files
Expected result:
1. The code follows the rules for active rendering.
Title: Pixel burn-in is avoided
Number: 150
Specification reference: (Section 6.6 Screen Saving page 17)
“In order to avoid pixel burn-in, applications that present a UI SHALL NOT present a static graphic for more than 10minutes at a time."
Assertions:
1. The application does not present a static graphic for more than ten minutes at a time.
Test Procedures:
1. Run the application
2. Transition the application to each of its graphic states
3. Wait at least ten minutes
Expected result
1. The graphic is not static