TIPS EXPERIENCE FROM THE FIELD ON DESIGNING A PIXELSENSE APPLICATION
Since the release to market of the first Microsoft Surface device (Aka Surface V1) and the today Samsung SUR40 with Microsoft PixelSense technology, I have been working on building different types of application for such devices. By looking a bit behind (even if I am not used to do that in general) I have explore and validate different form of approach and rules on such technology , in order to bring to ending customers what they were really expecting to get.
Those rules and guide lines that I have experiment during all those past years, this is something that I will describe here. It has perfectly work in all my projects and could be apply or adjust in order to cover why not yours.
Rules 1 : Understanding customer objectives
One of the most important rule in my mind is to really understand main objectives of your customers, on having a dedicated PixelSense application. At first it might sounds an evidence but when you look deeper it is the key point entry. This understanding will indicate the type of application that you are going to build and set up. Depending on customers objectives the application interaction and design goal will be totally different and you will be able to setup a compromise between the “Woww” effect and application performance.
Rules 2 : Always remember basic questions before anything else
- What is the purpose of integrating the device in a selected location?
- Does the location is suitable to receive the device? ( lighting condition, traffic,..)
- Who will be your application users?
- What type of application will be more suitable based on Rule 1
- How the environment will be ? ( colors, style, branding,..)
Rules 3 : Learn from those who have tried before
Do not jump too quickly into programming your design, but instead raise your head and takes some time to explore what is already existing. To help developers, the Surface team has written an important set of documents to puts you in the right track. The meeting point is the Microsoft PixelSense Design and Development. The Surface SDK 2.0 is the set of tools and sample that will help you building an application but keep in mind that any good programmer can use and understand an SDK but designing a great application is another story and this is the point where you will spend most of your time in a project.
Rules 4 : Forget what you have previously learned on designing traditional application
For some, it is difficult to jump to a new design approach and they always refer to what they have done before and wish to get the same result. You cannot imagine how many times I had to explain to customers or design agency (most of them coming from the web) that the design approach of an application in PixelSense is totally different. This is something where you need to explain again and again until the point you manage to convince them and established a full trust relationship, relying on the experience you provide.
“It takes time to build a great experience, so start with simple thing that you can validate easily”
Your successful application will start from the Microsoft® Surface® 2 Design and Interaction Guide that you absolutely need to cover and get a better idea of what all of this means.
Rules 5 : Do not hesitate to go back to school with your favorite tools
When you have validated the objectives of your customer application define in Rule1, your first favorite tool on your application design will be a pair of scissors and sheets of papers.
During my first application design, I remember the time where I was starting right away my application interface by opening visual studio and start creating windows, controls… When building application for PixelSense this approach will definitely make you lose a lot of time and effort.
- Build a preliminary brain storming with your team on the main aspect and key point of the application
- List on a white board all crazy ideas you might have on your project ( you cannot imagine how a crazy idea can became an excellent one when it is added to others)
- Draw main user interface sketch flow without going in details
Now it’s time to go back to school. Simply remember the time when you was at the kinder garden.
- Take your scissors and paper and start to build what you think your interface can be
- Use your paper as it was the application itself
- Review the size of element if you think they are not correct
- Validate your preliminary design based on what you visually see
Measure each validated visual by keeping in mind that 1cm ~= 22 pixels
“Do not imagine that the time spend around the device with your scissors is useless. It will save you a lot of tedious effort on sizing your visual element based on content flow”
Rule 6 : The Woww factor against application performance
Who have never start to think of his application by already imagine the type of different animation that could be done in order to bring to the application the famous Woww effect ?
Let me stop a while on this simple question: Does “Wow effect” really means animation?
Absolutely not, and this is one of an important point that comes to me when designing an application.
“What my customer is really expecting when he was telling me “Make it wow”
In all of my first PixelSense project, I was permanently thinking of that 3 words and with the time, they disappears from my though for the simple reason that this famous wow factor will come naturally by bringing to live customers mains objectives.
The success of an application will be identify when your customer will start to use your application and will naturally pronounced one word: “Woww”.
Points that make me talk about this are essentially based on followings:
- Too much animation will kill your application
- Reserved animation for content transition and make it fast enough for rich experience. For example an opacity rendering should not go over 0.5 s.
- Do not overload your application for unnecessary things that brings no added value for the general concept
- Always gives priority to application performance
Rule 7 : Avoid using “Forever” storyboard on TagVisualizer
This is one of the rule I recently try to explain to the final customer on one of my last project. As every potential customers, they took major of their ideas from different videos they could catch on the web and explore different application type that can exist, until one of them could be apply to their own use.
So far nothing special, instead a normal behavior when you are new to this world of interaction. The main part of the application was to interact fully with different objects and bring respective content. Nothing special either in that sense except one single request that makes me start to doubt a bit: Each object visual was requested to be animated during the whole presentation of content.
We start to build the requested animation which was based on quite many different layer of successive animated object in different time line. To reach the customer demand we had to set up no more than 8 different storyboard to reach the goal. So far so good we get the animation in place and it behaves as customer expected to get it:
- The object his place on the table
- Different circle object appears around the object and rotate asynchronously
- So particles moves around illustrating user activities during the whole time object was on the table
That was working quite nicely until the time we place more similar object on the table. At that time we could easily notify the weak response of the application during content manipulation. My first doubt start to come true and I could verify this by exploring the CPU usage which reach over 85%.
After some different testing, by simply modifying the storyboard animation to run only for few seconds, the CPU usage drop to 45%
Of course there is not only the fact of stopping a storyboard with make such measure possible but that was the goal of this single test only. Having effect on storyboard consume a lot of resource as well and can be avoided whenever possible.
Rule 8: Always bring alternative to object interaction
Interacting with object is the must have feature for a PixelSense application offering better user experience in a natural way. Objects gets build according to the impact and image to be deliver to the audience and they are in close relation to direct information content attached.
I have been face to a personal bad experience in an earlier application during an event where we bring some nice object specially built to interact with the application I was showing. At the beginning it was cool until the time you let visitors playing around the table and suddenly discover that some objects are not anymore present.
“A missing object should not avoid accessing to it’s initial content”
I do not know what I get in mind when I was preparing the application, but I get a strange feeling that I had to be able to access object content without the object. And this is what I have done and saved my demo at that time.
Rule 9 : Keep track of your main application information
PixelSense application goes to different state during the run and provide different useful information. Keeping track of those information will help you during the whole application level to get those states available at any time.
Here is below one sample class model we used to store such information during the whole application life time:
The main interesting thing here is that this class is a part of a MVVM architecture and its properties can be accessible at any time.
Rule 10 : Do not hesitate to disable unwanted touch input if needed
After releasing an application and deploy it to the customer location, I always spend time to clearly look at users and the way they interact with the application in order to bring fine tuning during validation process on site. During that phase, I write down all that works as well as all that does not work.
I was of course expected different type of people around my application but also people wearing different type of clothes.
“Why am I talking about clothes here as we are talking about designing an application? “
Well, have you ever present the result of your application to your manager and ask him to try it? I am sure yes and at that time you have also notice that his long tie touching the screen disturbed his interaction with requested content. This is what could happen in real as well, as you are not expected to wear only tee-shirt to use your application correctly.
At this time you need to adjust you application in the environment it is used. For such you simply need from your main application windows to disable all touch input that your application do not need.
The following sample is filtering only finger input and store Finger position and orientation in an application state as seen in Rule 9:
Private void SurfaceWindow_PreviewTouchDown (object sender, TouchEventArgs e)
If (e.Device.GetIsFingerRecognized ())
this.contactPosition = e.Device.GetPosition (this);
this.contactDegreeOrientation = e.Device.GetOrientation (this) +90;
AppState.Instance.FingerOrientation = this.contactDegreeOrientation;
AppState.Instance.FingerPosition = this.contactPosition;
With this simple code, the big tie will not be seen any more as a touch input.
Rule 11 : Use a single SurfaceWindow for better portability and flexibility
In PixelSense application design and WPF in general we should forget the multi window approach and going more to a single Window showing different type of content as illustrated below:
This architecture will offer flexibility and easier portability. With such approach we have a single main input entry for the whole application and content type gets displayed based on different criteria and directly activated through data binding. The power of the MVVM pattern here takes all it’s importance.
Let’s keep In mind as well that the Surface SDK 2.0 is compatible with Windows 7 platform. By having one single input window for the whole application the compatibility change of the application for Windows 7 compatibility will be resume only by providing a Surface Window for PixelSense app and a standard WPF Windows for Window7.
The only change to switch between those environments will be done from the App.xaml.cs file by overriding the App_Startup as shown below:
Void App_Startup (object sender, StartupEventArgs e)
if (InteractiveSurface.PrimarySurfaceDevice.IsTagRecognitionSupported == true)
//define Surface environment
StartupUri = new Uri (@”/PropertySense;component/SurfaceMainWindow.xaml”, UriKind.Relative);
//define Windows 7 environment
StartupUri = new Uri (@”/PropertySense;component/MainWindow.xaml”, UriKind.Relative);
Rule 12 : Drag & Drop operation are far from being natural if not shown
During whole application design where drag and drop was in place, I have notice how complicated it is to make such operation intuitive and evident for users when they are in front of your application for the first time.
By this I really mean that you let users manipulate your application and simply see them, without showing or saying anything. We make this simple test where we provide a simple view of photos equivalent as the one below:
First they see an horizontal slider so they understand easily that sliding from left to right can give access to more content. The next thing they most all do is touching a picture and wait. All users were doing exactly the same thing until I decide to show them that they can drag.
From that simple test and great feedback results, you need to start to reconsider that interaction. It does not mean that the drag/ drop interaction is bad in this condition but more how to have it more intuitive to users. I start to explore different approach as:
- Add to the interaction the Tap gesture on all element in the list
- Display a Notification message to the user at a certain time
- Display text information inside the visual element being manipulated
- Simulate the drag operation when no activities on the list
From those possible solution, I came from the fact that simulating the interaction with a small storyboard sequence was the most effective way to bring users to repeat that gesture
This scenario keeps me remember all time that people never do what you are expecting them to do in some situation.
Rule 13 : Users are the main source of inspiration
When you have completed the deployment of your application, after a certain time of use, always collect feedback of users manipulating your application. If you have the possibility to go at the location, do not hesitate and simply open your eyes. You will surely see something that you will have never think of or never seen.
“By looking at others using your app, you are already thinking of your next application design”
Rule 14 : Users are the main source of inspiration
When you build a Surface application for Microsoft PixelSense device and have in mind the multi user approach, cleaning teh screen is not necessary the first thing you are thinking of. This comes once again when you look at users the way they use the app. In all application I have made I could notice that users allways open content, but most of the time never close it after they have seen it. This is the reason why after some time of user your main application screen is full of content, the the next user coming to use your app will be confuse on where to start.
Base on that fact we need to try to follow those simple aspect of your content :
- Always provide a close button to be able to close manually your visual content
- Provide an inactivity time out on your content in order that after a certain time of no use, content will be closed automatically
Hoping that I could help you a bit in your reflexion based on my succesfull deployement so far where those rules gets extracted. Of course they engaged only my own personal opinion based on proven scenarios.