Archive

Archive for the ‘PixelSense’ Category

PowerPoint Presentation : Overview of Microsoft Surface Application Design

May 13, 2013 1 comment

There are a lot of sources of information on the net where you can get different topics based on Surface application design. For one case where I need to present to a customer an overview on designing Surface Application, I though it would be much easier to get basic information, group in a presentation that I coud reuse at any time.

This is what I have done so far and customer personnal attending such presentation really appreciate . Clear enough, concentrate with the essential.

Note that this is not a support for developping an application but more a starting point on what you should know, mixed with my own experience from the field.

As it has been really usefull work for me, I decide to share it with you.

You can view the presentation here

Good luck !

Advertisements

Starting your PixelSense application efficiently : Loading Controls

May 12, 2013 Leave a comment

When you start to build application for the SUR40 and Microsoft PixelSense, all used to work really perfectly well on the simulator and your development PC. When you start to install and run your application directly on the device in Surface mode, you start to face a challenging but important issue:

Get your application started before the Surface Shell timeout

All of us as found the famous startup time-out handle by the Surface shell. It could happen that your application appears to never start in that mode or it start but not all the time.

Never block the main UI thread

The entry point of your Surface application is the main surface window, and this is also the main UI thread for your application. If you perform some long operation within the loading process of that main Window, you are blocking the main UI thread and you will definitely get the famous time-out message

How does it really works

A Microsoft Surface application must not block the UI thread for more than 5 seconds. If it does so, the system closes and tries to restart the unresponsive applications twice. If it must close the application a third time because of unresponsiveness, Surface returns to Launcher and does not try any further restarts.

Loading your application while using potentially long loading of controls

When you start to build an application, you usually start to fill up your SurfaceMainWindow.xaml grid with the different control that you think your need to load.
The following code snapshot shows that scenario:


Potentially Locked Main Surface windows loading process

<s:SurfaceWindow x:Class=”SurfaceAppStartup1.SurfaceWindow1″

xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation&#8221;

xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml&#8221;

xmlns:s=”http://schemas.microsoft.com/surface/2008&#8243;

Xmlns:controls=”clr-namespace:SurfaceAppStartup1.Controls”

Title=”SurfaceAppStartup1″

>

<Grid x:Name=”Content” Height=”Auto”>

<s:SurfaceScrollViewer x:Name=”_mainHScrollViewer” ScrollChanged=”_mainHScrollViewer_ScrollChanged”

Width=”{Binding Element Name=Main, Path=Width}”

PanningMode=”None”

HorizontalScrollBarVisibility=”Hidden”

VerticalScrollBarVisibility=”Hidden” DataContext=”{Binding}”>

<StackPanel x:Name=”AppPanel” Orientation=”Horizontal” Background=”Transparent” >

<!– Load background Window –>

<controls:BackgroundImage DataContext=”{Binding}”/>

<!– Load WordMapControl  –>

<controls:WorldMapControl/>

</StackPanel>

</s:SurfaceScrollViewer>

</Grid>

</s:SurfaceWindow>

From the sample above, you can see that we are instantiated directly in Xaml the loading of the WorldMapControl UserControl as <controls:WorldMapControl/>. In this case if the loading time that that control is longer than plan your main UI thread will be block and you will get the Surface Shell Time-Out.

Solution : Use of Content control and Binding

From the sample above, we simply replace the <controls:WorldMapControl/> by the code below :

<ContentControl DataContext=”{Binding Source={StaticResource Locator}, Path=WorldMapViewModelStatic}”

Content=”{Binding MapControl, UpdateSourceTrigger=PropertyChanged}” />

You can guess from code above that we are using binding and an MVVM pattern to reach our goal. By doing so, the main UI thread will not be block during the loading of the control. Instead it will prepare a Content control that will have its Content property bind to a MVVM property named MapControl. When the MapControl property change, the control will be injected in the ContentControl.

What to do then while my user will wait for control to load?

This is a really interesting question and there are different approach. First avoid using a simple loading screen without adding value for the user. Remember that as soon as a Surface application has started, the application is interactive and user expect to do something with it:

  1. Provide simple content information in relation with your application that users can already explore
  2. Indicate visually that remaining content are currently loading
  3. When the remaining controls have completed and the full application is ready,bring attention to users with visual component or effect that something more can be done

Important :

When all your application initialization content is done, your application must call the Microsoft.Surface.ApplicationServices.SignalApplicationLoadComplete method to dismiss the opening activity when the application is ready to appear.

You can download sample project which illustrate this from the msdn code sample.

Hoping this small post will be helpfull for you.

TIPS EXPERIENCE FROM THE FIELD ON DESIGNING A PIXELSENSE APPLICATION

May 3, 2013 4 comments

Since the release to market of the first Microsoft Surface device (Aka Surface V1) and the today Samsung SUR40 with Microsoft PixelSense technology, I have been working on building different types of application for such devices. By looking a bit behind (even if I am not used to do that in general) I have explore and validate different form of approach and rules on such technology , in order to bring to ending customers what they were really expecting to get.

Those rules and guide lines that I have experiment during all those past years, this is something that I will describe here. It has perfectly work in all my projects and could be apply or adjust in order to cover why not yours.


Rules 1 : Understanding customer objectives

One of the most important rule in my mind is to really understand main objectives of your customers, on having a dedicated PixelSense application. At first it might sounds an evidence but when you look deeper it is the key point entry. This understanding will indicate the type of application that you are going to build and set up. Depending on customers objectives the application interaction and design goal will be totally different and you will be able to setup a compromise between the “Woww” effect and application performance.


Rules 2 : Always remember basic questions before anything else

  • What is the purpose of integrating the device in a selected location?
  • Does the location is suitable to receive the device? ( lighting condition, traffic,..)
  • Who will be your application users?
  • What type of application will be more suitable based on Rule 1
  • How the environment will be ? ( colors, style, branding,..)


Rules 3 : Learn from those who have tried before

Do not jump too quickly into programming your design, but instead raise your head and takes some time to explore what is already existing. To help developers, the Surface team has written an important set of documents to puts you in the right track. The meeting point is the Microsoft PixelSense Design and Development. The Surface SDK 2.0 is the set of tools and sample that will help you building an application but keep in mind that any good programmer can use and understand an SDK but designing a great application is another story and this is the point where you will spend most of your time in a project.


Rules 4 : Forget what you have previously learned on designing traditional application

For some, it is difficult to jump to a new design approach and they always refer to what they have done before and wish to get the same result. You cannot imagine how many times I had to explain to customers or design agency (most of them coming from the web) that the design approach of an application in PixelSense is totally different. This is something where you need to explain again and again until the point you manage to convince them and established a full trust relationship, relying on the experience you provide.


“It takes time to build a great experience, so start with simple thing that you can validate easily”

Your successful application will start from the Microsoft® Surface® 2 Design and Interaction Guide that you absolutely need to cover and get a better idea of what all of this means.


Rules 5 : Do not hesitate to go back to school with your favorite tools

When you have validated the objectives of your customer application define in Rule1, your first favorite tool on your application design will be a pair of scissors and sheets of papers.

Cisors

During my first application design, I remember the time where I was starting right away my application interface by opening visual studio and start creating windows, controls… When building application for PixelSense this approach will definitely make you lose a lot of time and effort.

  1. Build a preliminary brain storming with your team on the main aspect and key point of the application
  2. List on a white board all crazy ideas you might have on your project ( you cannot imagine how a crazy idea can became an excellent one when it is added to others)
  3. Draw main user interface sketch flow without going in details

Now it’s time to go back to school. Simply remember the time when you was at the kinder garden.

  1. Take your scissors and paper and start to build what you think your interface can be
  2. Use your paper as it was the application itself
  3. Review the size of element if you think they are not correct
  4. Validate your preliminary design based on what you visually see

Measure each validated visual by keeping in mind that 1cm ~= 22 pixels

cut


“Do not imagine that the time spend around the device with your scissors is useless. It will save you a lot of tedious effort on sizing your visual element based on content flow”


Rule 6 : The Woww factor against application performance

Who have never start to think of his application by already imagine the type of different animation that could be done in order to bring to the application the famous Woww effect ?

Let me stop a while on this simple question:  Does “Wow effect” really means animation?

Absolutely not, and this is one of an important point that comes to me when designing an application.


“What my customer is really expecting when he was telling me “Make it wow”

In all of my first PixelSense project, I was permanently thinking of that 3 words and with the time, they disappears from my though for the simple reason that this famous wow factor will come naturally by bringing to live customers mains objectives.

The success of an application will be identify when your customer will start to use your application and will naturally pronounced one word: “Woww”.

Points that make me talk about this are essentially based on followings:

  • Too much animation will kill your application
  • Reserved animation for content transition and make it fast enough for rich experience. For example an opacity rendering should not go over 0.5 s.
  • Do not overload your application for unnecessary things that brings no added value for the general concept
  • Always gives priority to application performance


Rule 7 : Avoid using “Forever” storyboard on TagVisualizer

This is one of the rule I recently try to explain to the final customer on one of my last project. As every potential customers, they took major of their ideas from different videos they could catch on the web and explore different application type that can exist, until one of them could be apply to their own use.

So far nothing special, instead a normal behavior when you are new to this world of interaction. The main part of the application was to interact fully with different objects and bring respective content. Nothing special either in that sense except one single request that makes me start to doubt a bit: Each object visual was requested to be animated during the whole presentation of content.

We start to build the requested animation which was based on quite many different layer of successive animated object in different time line. To reach the customer demand we had to set up no more than 8 different storyboard to reach the goal. So far so good we get the animation in place and it behaves as customer expected to get it:

  • The object his place on the table
  • Different circle object appears around the object and rotate asynchronously
  • So particles moves around illustrating user activities during the whole time object was on the table

That was working quite nicely until the time we place more similar object on the table. At that time we could easily notify the weak response of the application during content manipulation. My first doubt start to come true and I could verify this by exploring the CPU usage which reach over 85%.

After some different testing, by simply modifying the storyboard animation to run only for few seconds, the CPU usage drop to 45%

Of course there is not only the fact of stopping a storyboard with make such measure possible but that was the goal of this single test only. Having effect on storyboard consume a lot of resource as well and can be avoided whenever possible.


Rule 8: Always bring alternative to object interaction

Interacting with object is the must have feature for a PixelSense application offering better user experience in a natural way. Objects gets build according to the impact and image to be deliver to the audience and  they are in close relation to direct information content attached.

I have been face to a personal bad experience in an earlier application during an event where we bring some nice object specially built to interact with the application I was showing. At the beginning it was cool until the time you let visitors playing around the table and suddenly discover that some objects are not anymore present.


“A missing object should not avoid accessing to it’s initial content”

I do not know what I get in mind when I was preparing the application, but I get a strange feeling that I had to be able to access object content without the object. And this is what I have done and saved my demo at that time.


Rule 9 : Keep track of your main application information

PixelSense application goes to different state during the run and provide different useful information. Keeping track of those information will help you during the whole application level to get those states available at any time.

Here is below one sample class model we used to store such information during the whole application life time:

class

The main interesting thing here is that this class is a part of a MVVM architecture and its properties can be accessible at any time.


Rule 10 : Do not hesitate to disable unwanted touch input if needed

After releasing an application and deploy it to the customer location, I always spend time to clearly look at users and the way they interact with the application in order to bring fine tuning during validation process on site. During that phase, I write down all that works as well as all that does not work.

I was of course expected different type of people around my application but also people wearing different type of clothes.


“Why am I talking about clothes here as we are talking about designing an application? “

Well, have you ever present the result of your application to your manager and ask him to try it? I am sure yes and at that time you have also notice that his long tie touching the screen disturbed his interaction with requested content. This is what could happen in real as well, as you are not expected to wear only tee-shirt to use your application correctly.

At this time you need to adjust you application in the environment it is used. For such you simply need from your main application windows to disable all touch input that your application do not need.

The following sample is filtering only finger input and store Finger position and orientation in an application state as seen in Rule 9:


Private
void SurfaceWindow_PreviewTouchDown (object sender, TouchEventArgs e)

{

If (e.Device.GetIsFingerRecognized ())

{

this.contactPosition = e.Device.GetPosition (this);

this.contactDegreeOrientation = e.Device.GetOrientation (this) +90;

AppState.Instance.FingerOrientation = this.contactDegreeOrientation;

AppState.Instance.FingerPosition = this.contactPosition;

}

}

With this simple code, the big tie will not be seen any more as a touch input.

Rule 11 : Use a single SurfaceWindow for better portability and flexibility

In PixelSense application design and WPF in general we should forget the multi window approach and going more to a single Window showing different type of content as illustrated below:

content

This architecture will offer flexibility and easier portability. With such approach we have a single main input entry for the whole application and content type gets displayed based on different criteria and  directly activated through data binding. The power of the MVVM pattern here takes all it’s importance.

Let’s keep In mind as well that the Surface SDK 2.0 is compatible with Windows 7 platform.  By having one single input window for the whole application the compatibility change of the application for Windows 7 compatibility will be resume only by providing a Surface Window for PixelSense app and a standard WPF Windows for Window7.

The only change to switch between those environments will be done from the App.xaml.cs file by overriding the App_Startup as shown below:
Void App_Startup (object sender, StartupEventArgs e)

{

if (InteractiveSurface.PrimarySurfaceDevice.IsTagRecognitionSupported == true)

{

//define Surface environment

StartupUri = new Uri (@”/PropertySense;component/SurfaceMainWindow.xaml”, UriKind.Relative);

}

Else

{

//define Windows 7 environment

StartupUri = new Uri (@”/PropertySense;component/MainWindow.xaml”, UriKind.Relative);

}

}

Rule 12 : Drag & Drop operation are far from being natural if not shown

During whole application design where drag and drop was in place, I have notice how complicated it is to make such operation intuitive and evident for users when they are in front of your application for the first time.

By this I really mean that you let users manipulate your application and simply see them, without showing or saying anything. We make this simple test where we provide a simple view of photos equivalent as the one below:

propertydetails2
How do you think user does in front of such photos collection?

First they see an horizontal slider so they understand easily that sliding from left to right can give access to more content. The next thing they most all do is touching a picture and wait. All users were doing exactly the same thing until I decide to show them that they can drag.

From that simple test and great feedback results, you need to start to reconsider that interaction. It does not mean that the drag/ drop interaction is bad in this condition but more how to have it more intuitive to users.  I start to explore different approach as:

  • Add to the interaction the Tap gesture on all element in the list
  • Display a Notification message to the user at a certain time
  • Display text information inside the visual element being manipulated
  • Simulate the drag operation when no activities on the list

From those possible solution, I came from the fact that simulating the interaction with a small storyboard sequence was the most effective way to bring users to repeat that gesture

This scenario keeps me remember all time that people never do what you are expecting them to do in some situation.


Rule 13 : Users are the main source of inspiration

When you have completed the deployment of your application, after a certain time of use, always collect feedback of users manipulating your application. If you have the possibility to go at the location, do not hesitate and simply open your eyes. You will surely see something that you will have never think of or never seen.


“By looking at others using your app, you are already thinking of your next application design”

Rule 14 : Users are the main source of inspiration

When you build a Surface application for Microsoft PixelSense device and have in mind the multi user approach, cleaning teh screen is not necessary the first thing you are thinking of. This comes once again when you look at users the way they use the app. In all application I have made I could notice that users allways open content, but most of the time never close it after they have seen it. This is the reason why after some time of user your main application screen is full of content, the the next user coming to use your app will be confuse on where to start.

Base on that fact we need to try to follow those simple aspect of your content :

    1. Always provide a close button to be able to close manually your visual content
    2. Provide an inactivity time out on your content in order that after a certain time of no use, content will be closed automatically

Hoping that I could help you a bit in your reflexion based on my succesfull deployement so far where those rules gets extracted. Of course they engaged only my own personal opinion based on proven scenarios.

Categories: PixelSense, SURFACE, WPF Tags: , ,

Sample Nested ScrollViewer interaction design

November 18, 2012 4 comments

In one of my project on a Samsung SUR40 with Microsoft PixelSense, I was facing to an interesting interaction design based on different information groups needed to be logically displayed and accessible in a natural way.

The final idea is illustrated by the picture below:

The visual tree

The most external brown rectangle is a panorama panel which should be able to be scroll in both horizontal directions in order to have freely access to child group control. When the user will reach is interesting group of information represented by the blue rectangle, inside information, in my scenario was to be able to scroll vertically the information. And here comes the interaction logic from where you have nested scrollable control.

In order to reach my goal the layout has been define by the following tree, but you could adjust in according to your need.

Control architecture

The resulting XAML of the main panel :

 

The resulting XAML of the group control :

 

Interaction & design key point:

From the interaction that we have setup and as mentioned earlier, we need to get it smooth, logic and natural. If we look closer to the inside group control, they are nothing much that SurfaceList box control with have been define to handle vertical scroll only in that sample. So in each group control we will be able to naturally scroll for vertical data.

But then how can we scroll the panel horizontally?

The first answer to this question: Easy we just need to scroll the whole panel with the help of the most outside scroll viewer. That is correct. But asking the user to scroll the whole panel by placing is finger at the bottom narrow space of the control, do you thing user will all think of that? Absolutely not.

The best approach and most natural way would be that users initiating a horizontal scroll from any group control should be able to start the main panel to scroll, as illustrate bellow:

Apparently should be possible without any trouble but in fact the scroll does not happen.

As we have seen in our layout, each control group embedded a scrollviewer set for vertical scrolling. Horizontal scroll is disabling, but even if it would be enable it will scroll the group control content only. So why does this not work?

The reason came from the fact that the touch is capture by the scrollwiewer and the handle is set to true, so the event in not propagated up to the visual tree.

Nested scrollViewer behavior comes to the rescue

In order to allows that interaction, we have define a nested scrollviewer behavior class which pass to the parent scrollviewer control interaction which is not dedicated to its child.

By adding this behavior to each group control scrollviewer, the gesture will be passing to the parent scrollviewer which is the main panel and you will have a natural scrolling effect.

You can download the project from msdn sample here.
The sample include MVVM pattern

SAMSUNG SUR40 Black Screen Scenario

September 9, 2012 15 comments

I was recently browing the Surface forum and discover a topic about a guy which gets for no reason after 10 minutes, his SUR40 unit going to a black screen state.
I was quite surprise to see that a month after reading that post I sudenly face to the same issue. Now I am wondering why I have read it :-).

What was strange in all I could notice is that the scenario of the balck screen was occuring only when running the unit in pure Surface mode.
So as a normal use, I start to press the power button to get the menu and press Surface Exit Mode, but it has no effect at all. The only thing that work was switching of the power.
I start then to incriminate the Windows update service, which by default advise for a reboot after 10 mn when update gets downloaded. No luck that was not the case.
Then I start to increminate a kind of software problem, so in order to be sure I restore my unit to factory default. At this time it is a brand new unit based on softaware, and drivers..

But the problem still remains…
After all my test and validaded procedure, I decide to contact Samsung, which then decide to change the CPU board and trouble is gone.

So if one day you met such situation, and you made all test you can in order to be in a state as you would receive the unit, then you have a chance to be in such scenario.

I would like to thanks them anyway for they fast action to get my unit up and running again

Categories: PixelSense Tags:

How to debug remotely your Microsoft PixelSense application

July 11, 2012 1 comment

Recently I receive request on how to debug a Microsoft PixelSense application from Visual Studio, either from customers or forums.
You will find below experience I get and what works for me in such approcah

When debugging on Microsoft Surface there are 2 ways of doing it:

  • By having a developpement environment installed directly on the SUR40 for debugging on the machine
  • By using remote debuging

Developpement environement on the unit

Personnaly with past experience on the previous verison of Surface, I was expeimenting the first approach. This solution require some good organisation. Organisation in the sense that installing a complete developpement environement can result in an unstable environment. So of course I would not recommand that approach on a production unit. Before going to such solution you should take care to get in hand a clean backup system in a safe place in order to be able to recover a clean machine. The way I was using it is by creating a deployement image of the SUR40 machine at the desire state I want it. Then keep the generated .WIM file in a safe place. At anytime with that file and a WinPE boot disk I can recover my clean system. Once done you can install whatever you want on the unit and debug directly on it.

Remote debugging

I have spend some times to explore the remote debuggnig on Surface and what do we need for that.
The first thing to consider is the way you are going to identify remote users. In case you connect your unit to your company domain, then user rights will be much easier to manage and hande. In the oposite, you will need to have your developpement account on on your dev machine duplicated on the Surface unit.

Setup the SUR40 environment:

    • Ccreate the same account used on your dev machine or login in to your domain with same account
    • Install on the VS2010 remote debugger ( 64 bits or 32 bits based on your scenarios ). You can get those binaries from http://www.microsoft.com/fr-fr/download/details.aspx?id=475
    • Create a folder that will receive you compile debug bits ( for instance C:\RemoteAppDebug)
    • Be sure to get full right on that created folder from your dev account on your dev machine

Set up your Visual sudio environment:

  • Open your Surface project
  • Compile your project in Debug and copy all bits including symbole files to your RemoteAppDebug directory on your SUR40
  •  Create a new configuration setting and call it “Remote Debug SUR40”
  • Right click on your project and select properties

The following picture illustrate setting to define for your remote debugging :

 

As shown above the important thing is to set the Remote Folder of the SUR40 defined earlier as it would be on the local machine.
What I mean here is if you map for instance the C drive of the SUR to a drive on your machine in will not work.

The next step is to specify that you need to run the debug setting on a remote machine. As shown on above picture be sure to check the “Use remote machine” option and specify the name of your SUR40 in the text box.
In case you are not connected to a domain with your SUR40 unit, you will have to enter here the machine name by passing the account to be used like <username>@Machinename

Start debugging:

When you have all this in place you can now debug your application. For that you need to :

  • Start the Visual Remote debugger on the unit based on the target aplication you deploy ( x86 or x64 )
  • Be sure you can access to the RemoteAppDebug folder from your devlopment machine
  • Press F5 to start debuggin your application

Trouble you might face on :

Firewal settings which prevent the conection with your remote environment
– Authentification

For those , please refer to recommandation and check settings according to the following information :
http://msdn.microsoft.com/en-us/library/bt727f1t.aspx

IMPORTANT : This debugging scenario allows you to debug you application when the SUR40 is in administrative mode as this request need admnistration priviledge

Microsoft Surface and Microsoft PixelSense

June 20, 2012 Leave a comment

It’s done; Microsoft recently announced its own slate named Surface. Hmmm Surface, does its sounds familiar to you? You are right Surface was dedicated before to the famous Microsoft Surface unit and Samsung SUR40. Due to the introduction of the Microsoft Surface slate Microsoft has review his product branding to avoid confusion. Today the slate market is growing rapidly. Slate world, introduced by Apple now seems to have a serious competitor.  After releasing to market the famous Samsung SUR40, Microsoft has surprised many people at the announcement of its slate device. Surface will be powered by different version of Windows 8.This slate is offering both comfort, power and design.  As usual, on quite similar technology like Surface Computing, people start to be confused but with the time and expected product impact, I am quite sure that things will be clear after a while.

Microsoft PixelSense is the technology brand used on the SAMSUNG SUR40. The brand Microsoft Surface previously used for the SAMSUNG SUR40 is dedicated now to the new Microsoft slate which will run Windows 8

So for Surface slate stick to the new web site link :http://www.microsoft.com/surface/en/us/default.aspx

Old Microsoft Surface brand became Microsoft PixelSense and links are :

http://www.microsoft.com/en-us/pixelsense/default.aspx

http://www.microsoft.com/en-us/pixelsense/SoftwarePlatform.aspx

http://msdn.microsoft.com/en-US/windows/desktop/hh241326.aspx

Microsoft Surface :

Categories: PixelSense, SURFACE Tags: , ,