Virtual Reality vs. Augmented Reality – Impact on Your Business

pexels-photo-166055

Virtual reality (VR) and augmented reality (AR) have come out of nowhere to become ‘the next big thing’.

With the failure of Google Glass as an augmented reality platform for consumers, the seeming success of the Oculus Rift as a gaming-based virtual reality platform, and the weird novelty of Microsoft HoloLens as a resurgence in the augmented reality realm, it can be hard to understand the scope, purpose, and worth of these new ‘worn on the head devices’.

  • Are they toys?
  • Are they the ‘next big thing’?
  • How could these directly impact my bottom line?
  • Can they add efficiency to current day to day work across an enterprise?
  • Is this VR / AR stuff even worth paying any attention to?
  • How did these things rise up and become a multi-billion dollar business, when I just seem them as weird head mounted toys?

VR and AR – A History

Everything starts with MEMS. MEMS stands for Micro Electromechanical Systems. MEMS allowed for the at-scale manufacturing of tiny, easily integrated, highly accurate, and super cheap sensors such as accelerometers and gyroscopes.

These MEMS based sensors can fit within a housing that is no larger than a chip on a tiny motherboard. The sensors are super accurate, and easily integrated into anything with a breadboard. The first consumer breakthrough MEMS device is the accelerometer within the Wii Remote from Nintendo. An accelerometer can measure the relationship of the sensor to gravity, meaning an accelerometer can tell you if you are moving up (i.e. negative acceleration to gravity) or moving down (i.e. positive acceleration to gravity). As Nintendo found out, this is all you need to create a multi-billion dollar home gaming business and revive your whole business.

Technology moves on from the Nintendo Wii game console and the simple accelerometer. Soon we have MEMS based gyroscopes that could do more than measure the relationship of the sensor to gravity, but could measure the relationship of the sensor to all spatial dimensions. We have super small magnetometers which can measure the relationship of the device to the true magnetic North Pole of the Earth. Barometers which can be used to measure air pressure and hence altitude. GPS sensors which can measure the latitude / longitude / time / altitude of the device using the Global Positioning System. Soon, you get the iPhone 5 that incorporates all of these sensors, and the Apple CoreMotion and CoreLocation iOS app frameworks, which allow any iOS app developer to discover and log the relationship of an iPhone to all of these real-world spatial dimensions, by regulating, smoothing, and aggregating input from all those sensors in real time for easy in-app consumption.

All of this using sensors that cost less than $10 to make and are as easy as soldering their dangling pins to a motherboard.

In addition to sensor technology, we also have the ARM and SOC (System On a Chip) revolution. ARM Holdings is an intellectual property and design company that produces simple, low-power chip plans. ARM happily toils away shipping out amazingly fast, super low-power, instruction set, and chip designs that anyone can purchase off the shelf and manufacture. ARM doesn’t make the chips, they are just masters of chip and instruction set design. Suddenly, seemingly out of nowhere, the ARM revolution comes full force. Suddenly we have Apple, Nokia, and Samsung taking these ARM designs off the shelf and starting to manufacture super cheap, low power, super fast, highly efficient full systems on a chip.

Those ARM SOCs start to make their way into hugely profitable smartphones. The high margin of smartphones relative to their cost cause a virtuous cycle of ARM SOC manufacturing improvements + cost reduction + ARM design improvements. Suddenly, we have all this high-end sensor and CPU power at super low-energy cost and monetary cost. It is now that we can put high-end ARM hardware + high-end sensor packages into the tiniest of shells with off-the-shelf battery power. Through the natural cycles of profitability driving innovation, we are at the point where we can incorporate actual supercomputers into the smallest of form factors. Even wearables.

At the same time as we are getting all this high-end ARM CPU and high-end super low form factor sensor packages, we get the capacitive touch revolution and the blue LED revolution. Now we can also create screens that sip power and are instantly relatable via the human power of touch.

On the side as part of the SOC revolution we also get the PowerVR line of high-end graphics chips which sip power and have a full hardware accelerated 3D API via OpenGL and DirectX.

It is from the shoulders of PowerVR 3D chips, all those MEMS sensors, LED screens, the ARM SOC revolution, and the virtuous cycle of aggregation + capitalization via the smartphone, that we now have a platform from which to go at full head mounted wearables, which are viable for early adopters to experience virtual reality and augmented reality applications.

The first round of these head mounted wearables falls into 2 camps:

  • Virtual Reality (VR)
  • Augmented Reality (AR)

Virtual Reality

Virtual reality works off the basis that the only thing the user can see and hear are the outputs of the virtual reality device. No outside stimulus is meant to enter the user experience.

Augmented reality works off the basis that the user needs their real world sensory experience augmented with virtual data and graphics. The scope of human vision while using an augmented reality device is meant to be completely real world, while in real time a digital overlay will be presented into the main sensory experience of the user.

The current mainstream implementation of virtual reality exists in the following major consumer products:

  • Google Cardboard
  • Samsung Gear VR
  • Oculus Rift

Google Cardboard
Google Cardboard is exactly what the name implies. It is just a simple cardboard (yes, just cardboard) mount for a smartphone.

The primary use of Google Cardboard is to present stereoscopic, 360 degree / spherical, video.

There are existing SDKs for Android and iOS which allow easy division of a smartphone screen for stereoscopic viewing of a specially created 360-degree video.

360-degree videos are easily created using special multi-camera rigs that film the full sphere of a given location. Software tools then aggregate the input from all the cameras on the rig into a single video. The single video is then projected onto a smartphone screen using special 3D spherical transforms with the 3D support of a PowerVR chip. 360-degree video playback is augmented further by aggregating the all the gyroscope, accelerometer and other position sensors present on a cellphone, to make the user feel immersed in the environment of the 360-degree / VR video.

Almost any smartphone can provide a VR experience for a user via Google Cardboard. It is simple canned video playback, with a few 3D tricks, to make the filmed environment seem more real to the user.

Gear VR
Samsung Gear VR ups the ante just a little over the Google Cardboard experience by providing a more dynamic experience than just a canned 360-degree video.

Oculus Rift
The ultimate VR experience is the Oculus Rift. Oculus Rift is a full division within Facebook.

The Oculus Rift completely shields the user from the real world via a full headset. The headset is wired into a high-end Intel based PC + high end graphics card. Less than 1% of all PCs shipped can power the Oculus Rift. The full power of the PC is pushed into the Oculus Rift to create fully immersive virtual worlds.

The revenue model for Oculus Rift is primarily gaming to start.  One can easily envision more live social interaction, and/or some enterprise uses (i.e. simulated training) for the fully VR worlds that can be created with Oculus Rift.

Most developers of VR have stated that they are glad that they get to work on VR and not AR (or augmented reality). The ability of VR developers to not have to bring in live computer vision from the real world, in real time, and augment real time human perception, is seen as way easier.

Augmented Reality

Augmented reality largely needs to solve all the problems of virtual reality, in addition to bringing in the live sensor input of human vision.

It is my opinion that augmented reality systems are much more suited to future enterprise use than virtual reality systems. This value within enterprise has largely been vetted by the weird Beta release and consumer level failure of Google Glass.

Google Glass
Google Glass was a problem looking for a solution. Google sought a solution in the consumer space hoping for some level of mainstream attraction and adoption. Instead, Google Glass has found a second life in the enterprise.

For those of you who haven’t used Google Glass, all Google Glass did was project a 32” flat panel TV into the upper right portion of your vision.

It was that simple. Any time you wanted to interact with the digital world, all you had to do was look up and to the right and there was a full virtual screen showing you something.

In many enterprise applications, the simple implementation of augmented reality via Google Glass can be extremely powerful. Imagine being an aircraft mechanic, and you would like to look up the design, plans, or details of a given part you are working on, but you are stuck in a fuel tank. Now you can get that information by simply looking up and to the right. The very simple projection of a single rectangle overlaid into human vision can lead to huge efficiencies in many areas of real work. The initial $1500 dollar price tag for Google Glass may have been a small one to pay for many in-the-field jobs. If all Google Glass did was allow a technician to view a relevant page of a PDF-based manual in the upper right area of their vision during tasks such as aircraft maintenance, automobile fleet maintenance, or assembly line work, the value of the simple augmented reality model that Google Glass presented may be realized quite quickly in increased quality, reduced repair times, or even increased safety.

It was unfortunate that Google went after the consumer instead of the enterprise markets.

Google Glass was just the start of augmented reality. We now have Microsoft HoloLens, which goes beyond the simple projected rectangle in the upper right of your vision and can fully incorporate fully 3D overlays onto any object that you can see within your field of vision.

Microsoft HoloLens
Microsoft HoloLens starts with simple gaming models for mass-consumer based revenue, but one can see a much larger enterprise vision as a dramatic improvement over the possible gains from Google Glass.

Imagine a supervisor looking out over a manufacturing assembly line while wearing Microsoft HoloLens and seeing real time stats on each station.

Imagine UPS, Amazon, or FedEx workers being able to get guidance to where something is in a warehouse overlaid directly onto their vision without needing any physical signs.

Imagine software engineers that can place status and development windows anywhere in virtual space for their work.

Imagine DevOps staff that can look out over a data center and see which machines have possible future hard drive failures, or  the real time status of a software deploy and onto which servers.

Realization of all the above with Microsoft HoloLens is aided by Microsoft’s vision of the Universal Windows Platform (or UWP). UWP allows businesses to reuse much of their existing C# / .NET / C++ code across a myriad of devices: Windows 10 Desktops, Xbox One, Windows 10 Mobile, and Microsoft HoloLens. In essence, many enterprises may already have logic that they can integrate into an augmented reality device so they can realize certain efficiencies with minimal software development overhead.

Augmented reality holds boundless promise for efficiencies within an enterprise, and we are on the cusp of being able to actually realize those efficiencies especially with Microsoft HoloLens and the Universal Windows Platform.

Microsoft has also recently announced the launch of Windows Holographic which allows third parties to create their own augmented (or mixed) reality hardware using Microsoft’s software.

Virtual reality and augmented reality each have their future killer applications and killer niches. Hopefully you can find them within your business and start to realize greater efficiencies and value with augmented reality, and possibly virtual reality, based solutions.

Practical Data Cleaning Using Stanford Named Entity Recognizer

I enjoy learning about all of the events in and around World War II, especially the Pacific theater.

I was reading the book Miracle at Midway  by Gordon W. Prange (et. al.) and started to get curious about the pre- and post- histories of all the naval vessels involved in the Battle of Midway.

Historically, so many people, plans, and materials had to merge together at a precise moment in time in order for the Battle of Midway to be fought where it was and realize its radical impact on the outcome of World War II.

I got curious as to the pre- and post-history of all the people and ships that fought at Midway and wanted a way to visualize all of that history in a mobile application.

As a software guy, I started digging into possible data sources of ship histories. I was looking forward to constructing my vision of a global map view onto which I can overlay and filter whole historical data sets in and around naval vessels such as:

  • Date of departure
  • Date of arrival
  • Location in lat / long of source / destination
  • Other ships encountered during mission
  • Mission being performed
  • People who commanded and served
  • People who may have been passengers or involved in the mission some way

I am not anywhere near constructing my vision of fully time and space dynamic filtered ship history maps yet. The keyword being: Yet.

This story details one tiny step I undertook to try and get to my goal against a found data source.

 

Finding a data source

I found a whole set of plain text ship histories as part of the Dictionary of American Naval Fighting Ships (DANFS).

An awesome Pythonista had already ripped the DANFS site content into a SQLite database which contains all 2000+ ship histories. – Thanks jrnold!

The ship histories are interesting to read but are not organized in a machine friendly format. The histories are largely just OCR’d text + a little HTML styling markup which was embedded as a side effect of the OCR process. I fully acknowledge that beggars can’t be choosers when it comes to data sources of this type. DANFS is as good a place to start as I could find this side of petitioning the National Archives for all naval ship logs. As any data scientist will tell you: You will spend 90% of your time cleaning your data, then 10% of your time actually using your data.

In this case, most of the ship histories are written in chronological order from past to present (once you separate out some header text regarding the background of the naming of the ship). In theory, if we just had a way to markup all the location names in the text, we could process through the text and create an in-order list of places that the ship has visited. Please note: All this is largely naive. It’s actually much more complicated than this, but you have to start somewhere.

Needless to say: I don’t want to do location identification by hand across 2,000+ free text ship histories. We are talking about the entire history of the American Navy. Having to classify and separate out locations by hand, by myself, is a huge task!

It turns out that there is a library that will markup locations within the text via an already trained machine learning based model: The Stanford Named Entity Recognizer (or Stanford NER)

Note: I continue in this post by using Stanford NER due to its C# / .NET compatibility. You may also want to check out Apache openNLP. Also keep in mind that you can also pre-process your data using NER / Natural Language Processing (NLP) and feed the results into ElasticSearch for even more server side search power!

DANFS History Snippet Before being run through Stanford NER:

<i>Abraham Lincoln </i>returned from her deployment to NAS Alameda on
9 October 1995. During this cruise, the ship provided a wide variety of
on board repair capabilities and technical experts to 17 American and
allied ships operating in the Middle East with limited or non-existent
tender services. In addition, the Communications Department completed a
telemedicine video conference with Johns Hopkins Medical Center that
supported X-ray transfers and surgical procedure consultations.

 

DANFS History Snippet After Stanford NER:

<i>Abraham Lincoln</i>
returned from her deployment to
<LOCATION>NAS Alameda</LOCATION>
on 9 October 1995. During this cruise, the ship provided a wide
variety of on board repair capabilities and technical experts to
17 American and allied ships operating in the
<LOCATION>Middle East</LOCATION>
with limited or non-existent tender services. In addition, the
<ORGANIZATION>Communications Department</ORGANIZATION>
completed a telemedicine video conference with
<ORGANIZATION>Johns Hopkins Medical Center</ORGANIZATION>
that supported X-ray transfers and surgical procedure consultations.

Pretty cool, huh?

Notice the ORGANIZATION and LOCATION XML tags added to the text? I didn’t put those into the text by hand. Stanford NER took in the raw text from the ship history of the U.S.S. Abraham Lincoln (CVN-72) (Before) and used the machine learning trained model to markup locations and organizations within the text (After).

 

Machine Learning: Model = Sample Data + Training

From the documentation regarding the Stanford Named Entity Recognizer:

Included with Stanford NER are a 4 class model trained on the CoNLL 2003 eng.train, a 7 class model trained on the MUC 6 and MUC 7 training data sets, and a 3 class model trained on both data sets and some additional data (including ACE 2002 and limited amounts of in-house data) on the intersection of those class sets. (The training data for the 3 class model does not include any material from the CoNLL eng.testa or eng.testb data sets, nor any of the MUC 6 or 7 test or devtest datasets, nor Alan Ritter’s Twitter NER data, so all of these remain valid tests of its performance.)

3 class: Location, Person, Organization
4 class: Location, Person, Organization, Misc
7 class: Location, Person, Organization, Money, Percent, Date, Time
These models each use distributional similarity features, which provide some performance gain at the cost of increasing their size and runtime. Also available are the same models missing those features.

Whoa, that’s an academic mouthful. Let’s take a deep breath and try to clarify the Stanford Named Entity Recognizer pipeline:

NER

 

All the parts at the top of the diagram labelled ‘Machine Learning’ (with black borders) are already done for you. The machine learning + training output is fully encapsulated and ready for use via models available for download as a handy 170MB Zip file.

  • When you crack open the model zip file, you will see the classifiers sub directory which contains the 4 class Model, 7 class Model, and 3 class Model:
    • Classifiers
    • It is one of these 3 models that you will use to initialize the CRFClassifier in your client code depending on your needs.

The parts at the bottom of the diagram in the ‘Using NER Classifier’  space (with purple borders) are what you will do in your client code to get person, location, and organization markup placed within your text. Check out the instructions on how to do this in C# / .NET.

The key to the client code is the CRFClassifier from the Stanford.NER.NLP NuGet Package. The CRFClassifier takes in your text and the trained model output file, from the above zip file, to do classification of person, location, and organization within your text.

Sample C# code:

 // Loading 3 class classifier model
            var classifier = CRFClassifier.getClassifierNoExceptions(
                classifiersDirecrory + @"\english.all.3class.distsim.crf.ser.gz");

            var s1 = "Good afternoon Rajat Raina, how are you today?";
            Console.WriteLine("{0}\n", classifier.classifyToString(s1));

 

Cleaning the data

Cleaning a set of OCR’d free form text files and converting it to a format that makes it easy for a developer to process it using code can be difficult. Data cleaning is an ad-hoc process requiring use of every data processing, and possibly coding, tool you have in your tool belt:

  • Regular Expressions
    • String replacement
  • XML DOM traversal
    • Node + attribute add
    • Node + attribute remove
  • Standard string split and replacement operations
  • Log to console during processing and manually create filtered lists of interesting data.
    • These lists can also be fed back as training data for future machine learning runs.
    • Filter values on created lists from source data.
  • Use of trained models

All that said, Named Entity Recognition gives you a fun and solid starting point to start cleaning your data using the power of models from machine learning outputs.

I highly recommend using Stanford NER as one or more stages in a pre-production data cleaning pipeline (especially if you are targeting the data for rendering on mobile platforms).

Beware: your data may have a series of false positive PERSON, ORGANIZATION, or LOCATION tags written into it by Stanford NER that may have to be filtered out or augmented by additional post-processing.

In my case Stanford NER also marks up the names of people in the text with PERSON XML tags. You may notice that Abraham Lincoln above is part of a ship name and also the name of a person. In my first run of this ship history text through Stanford NER, the text Abraham Lincoln was surrounded by PERSON XML tags. Having 1,000 occurrences of the person ‘Abraham Lincoln’ in a ship history about the U.S.S. Abraham Lincoln is probably not very useful to anyone. I had to run a post-processing step that used the XML DOM and removed any PERSON, ORGANIZATION, or LOCATION tags if the parent of those tags was an ‘i‘ tag. I found that ‘i‘ tag was written into the original history text from DANFS to indicate that it is a ship name so it was the easiest (and only) data marker I had to aid in cleaning. The same problem would occur for the U.S.S. Arizona, U.S.S. New Jersey, U.S.S. Missouri,  and other text where locations, people, or organizations were used as ship names.

I fully intend on trying to use machine learning, and alternate training sets for Stanford NER, to ensure that PERSON, ORGANIZATION, and LOCATION tags are not written if the text is within an ‘i’ tag (but haven’t done this yet).

As I was cleaning the data using Stanford NER I ran across instances where the rank or honorific of a person was not included within the PERSON tag:

Capt. <PERSON>William B. Hayden</PERSON>

My initial implementation to include the honorific and/or rank of the person was problematic.

  • After the NER stage, I scan through the XML output using System.Xml.Linq (i.e. XDocument, XNode, XElement) looking for any PERSON tags.
  • Using the XML DOM order I went to the previous text node right before the start PERSON tag.
  • I then displayed the first 3 words, as split by a space, at the end of the preceding text node.

In my naiveté I figured that the honorific and/or rank would be at worst about 50 different 3 word variations. Things like:

  • Private First Class
  • Lt. Col.
  • Col.
  • Gen.
  • Maj. Gen.
  • Capt.

Well imagine my astonishment when I discover that the DANFS data doesn’t only contain the names and honorifics of military personnel; but of passengers and people in and around the history of the ship:

  • Senator
  • Sen.
  • King
  • Queen
  • Princess
  • Mrs.
  • Representative
  • Congressman
  • …. and many, many, many more

In addition I found out that almost every possible abbreviation exists in the ship history text for rank:

  • Lt.
  • Lieutenant
  • Lieut.
  • Commander
  • Cmdr.

My second pass at determining the honorific of a person may just involve a custom training stage to create a separate ‘Honorific’ model trained by the set of honorific data I have derived by hand from the whole DANFS text data set.

As stated above: Data scientists spend way too much time cleaning data using any technique that they can. On this project, I found that I was never really ever done extracting new things from the data, then cleaning the output data some more in a never ending loop.

I hope in the future to provide all the sample cleaned DANFS data, my data cleaning source code, and a sample mobile app that will interactively render out historical ship locations.

Coping with Device Rotation in Xamarin.Android

You think that you have your Android application in a state where you can demo it to your supervisor when you accidentally rotate your device and the app crashes. We have all been there before and the good news is that the fix is usually pretty simple even if it can sometimes take awhile to find.

This has always been an issue for Android developers, but I have found that, due to the unique interaction between your C# classes and the corresponding Java objects, it seems to be a little more sensitive with Xamarin.Android apps. In this post, we will discuss what happens when you rotate your device and cover the different techniques that you might choose to use to manage your application state through device rotations as well as the ramifications of each of them.

Configuration Changes

So, what happens when you rotate your device?  Your device’s orientation is considered to be a part of the configuration of your application and, by default, Android will restart your Activity whenever it detects a change to the configuration.  At first glance, this seems like a pretty heavy-handed approach to handling device rotations, but there is a reason behind it. To understand that reason, we need to go back and review a few basics about Android app development.

Android, and by extension Xamarin.Android, has a way for you to create resources that only apply for a particular configuration value. Resources can be added to a folder that is tied to a configuration value, and those resources will only be used if that configuration value exists in the current setup. This is seen most often with drawables when you see the various drawable-mdpi or other drawable-*dpi folders so that you can provide images that are scaled appropriately for the resolution of the device. The Android system will choose the proper drawable folder and fall back to the base drawable folder at runtime whenever a given image is requested. This system of coupling resources to a given configuration goes beyond drawables to include all of the resource types, so layouts, strings, values, colors, etc. These can all be restricted using the same configuration qualifiers.

Device orientation is one of those configuration values that can be used to conditionally load different resources through the use of the “*-port” or “*-land” qualifiers.  Though, this means there could potentially be different resources used when viewing the app in landscape mode than in portrait mode, and that needs to be accounted for when the device rotates.  The Android team decided that restarting the Activity would be the best way to handle this so the resources could be reloaded with the new configuration values when the Activity restarts.

That might all sound fine, except that it can cause problems if you have not correctly accounted for this behavior in your code.  There are several approaches that you can take to deal with this, although the simplest approaches can also be the most restrictive to your app.

Prevent Orientation Changes

The first approach is also going to be the easiest to handle, but it is the most limiting, because it involves telling Android that your Activity only supports a single orientation. You can do this by setting the ScreenOrientation property of the ActivityAttribute on your Activity class to the orientation that you want to force your Activity to use.

[Activity(ScreenOrientation = ScreenOrientation.Portrait)]
public class MyActivity : Activity
{
}

This approach has some obvious drawbacks, but if it works for your UX needs then it will be a simple way to ensure that an orientation change will not affect your application.

However, it is important to keep in mind that orientation changes are not the only configuration changes that might occur and cause your Activity to restart. For instance, a user can change their font/text size, which will also trigger your Activity to restart. So suppressing an orientation change is not a complete fix for any issues that your app would experience with restarting the Activity.

Manually Handling Configuration Changes

The second approach is a much more manual approach. It is possible to tell your app that you want to manually handle configuration changes in your code instead of restarting the Activity. To implement this approach you need to override the OnConfigurationChanged method in your Activity and manually process the new configuration object to do whatever needs to be done and then subscribe to specific configuration changes. Again, just like preventing an orientation change, this is as simple as tweaking your ActivityAttribute to specify which configuration changes will trigger calls to your method.

[Activity(ConfigurationChanges = ConfigChanges.Orientation | ConfigChanges.ScreenSize)]
public class MyActivity : Activity
{
    public override void OnConfigurationChanged(Configuration newConfig)
    {
        base.OnConfigurationChanged(newConfig);

        // perform actions to update your UI
    }
}

You might have noticed that I included the ScreenSize configuration change in my attribute. The reason is since API 13 (Honeycomb) the screen size also changes with the orientation, so you need to subscribe to both changes in order to keep your Activity from restarting.

This source code example will indeed prevent the configuration change from restarting your Activity, but it will not change which resources were used. If you were originally in portrait mode, then you would still be using your portrait resources even though you are now in landscape mode. If you want to also update your resources, then you will need to manually inflate the new resources and replace the old resources with them. However, since the Activity is not being restarted, that means the lifecycle events (OnCreate, OnResume etc.) are not executing, so any code that obtains references to views in your layout, or initializes values in your layout, will need to be performed again for your newly inflated views.

This can quickly become a lot of work if your UI is anything other than trivial and it is going to be prone to breaking if you forget a step or something changes in the future. As a result of that, I cannot recommend this approach unless your app does not utilizing orientation based resource overrides that would require inflation.

Retaining your Fragment Instance

If you are making use of fragments in your app, then they are also destroyed and recreated along with your Activity when a configuration change occurs. If your Activity class itself is just a thin wrapper around different fragments that actually contain most of your application state, then maybe it will be enough for you to persist your fragments and let the Activity still recreate itself. Like the previous examples, this is very simple to do since you just have to make one method call on your fragment to let Android know that the instance needs to be saved.

// from activity
var fragment = new MyFragment();
fragment.RetainInstance = true;

// OR in fragment
public override void OnCreate(Bundle bundle)
{
    RetainInstance = true;
}

Android will save the instance of your fragment and when it rebuilds your Activity it will reuse that fragment instance instead of creating a new one. This sounds pretty good, but if your fragment makes use of any resources that are orientation specific then you run into the exact same problem that you have with manually managing the orientation change. You will still need to inflate your new resources and initialize them manually, so other than saving your member variables for you this approach does not gain you a lot.

However, one place where this technique can be very useful is if you have some objects that may not serialize/deserialize well. As long as those objects do not retain references to the Context/Activity, such as other views or drawables, you can add those objects to a dummy fragment and retain that fragment. The fragment itself should just be a thin fragment that does not do anything else, and since there is no resource inflation happening in it you do not have to worry about reinflating anything.

Save and Restore your Application State

The Android designers knew that destroying and recreating the Activity was going to cause problems so they provided a mechanism for developers to save their state and then restore it after recreation.

Both the Activity and Fragment classes have a SaveInstanceState method that receives a Bundle where you can store serializable data. This method is called just prior to those objects being destroyed so the class states are still valid. You can use this bundle to store member variables from your class, or data that was retrieved and you do not want to have to retrieve it again, or anything else that is serializable.

protected override void OnSaveInstanceState(Bundle outState)
{
	base.OnSaveInstanceState(outState);

	outState.PutBoolean("someBoolean", someBoolean);
        outState.PutInt("someInt", someInt);

        // assume someObject is of type List<SomeModel>
        // I like to use Newtonsoft.Json to serialize to strings and back
        outState.PutString("someModels", JsonConvert.SerializeObject(someModels));
}

Activity classes have a RestoreInstanceState method that receives the Bundle containing the saved state and it has a chance to repopulate the class’s members with their data, although the same bundle is also passed to OnCreate so you could put your restore logic there as well depending on your need. RestoreInstanceState is called after OnStart, so if you need to initialize views before the Activity is started then you will want to use OnCreate. Keep in mind that the bundle in OnCreate can be null if the Activity is being launched so you will need to perform a null check.

protected override void OnCreate(Bundle savedInstanceState)
{
        base.OnCreate(savedInstanceState);

        // you must do a null check before referencing it here
        if (savedInstanceState != null)
        {
            someBoolean = savedInstanceState.GetBoolean("someBoolean", false);
            someInt = savedInstanceState.GetInt("someInt", 0);

            someModels = (IList<SomeModel>)JsonConvert.DeserializeObject<IList<SomeModel>>(savedInstanceState.GetString("someModels", null));
        }
}

protected override void OnRestoreInstanceState(Bundle savedInstanceState)
{
        base.OnRestoreInstanceState(savedInstanceState);

        // this method is only called when restoring state, so no need to do a null check
        someBoolean = savedInstanceState.GetBoolean("someBoolean", false);
        someInt = savedInstanceState.GetInt("someInt", 0);

        someModels = (IList<SomeModel>)JsonConvert.DeserializeObject<IList<SomeModel>>(savedInstanceState.GetString("someModels", null));
}

Fragments are a little different in that there are multiple methods that receive the Bundle with the saved state so you can restore your state in any of them. Generally speaking I would recommend using the OnActivityCreated method to restore your state since this happens prior to the UI views in your Fragment getting restored. If you needed to restore your state after the UI views being updated then you can use the OnViewStateRestored method.

public override void OnActivityCreated(Bundle savedInstanceState)
{
	base.OnActivityCreated(savedInstanceState);

	// load the data from the saved cache if it exists
	if (savedInstanceState != null)
	{
		someModels = (IList<SomeModel>)JsonConvert.DeserializeObject<IList<SomeModel>>(savedInstanceState.GetString("someModels", null));
	}
}

Android does not want you to have to do all of the work so it will automatically save the state of all views in your UI with IDs for you. Pieces of information like your scroll location are also saved and restored with the views in between OnActivityCreated and OnViewStateRestored, so if you want your scroll location to be correct then you will need to populate new adapters with your saved data and attach them to your lists in OnActivityCreated so that the scroll size is correct before Android sets the location.

One other piece that you will need to keep in mind is that Android will also attempt to restore your fragments and the back stack in the fragment manager. However, if your Activity keeps a reference to any of the fragments within it, you will need to save that fragment identifier so that Android can restore it with the correct instance. Fortunately they provide an easy way to do that.

protected override void OnSaveInstanceState(Bundle outState)
{
	base.OnSaveInstanceState(outState);

        // I am using the SupportFragmentManager here since I am using AppCompat with the support libraries.  This should also work with FragmentManager if you are not using AppCompat
        SupportFragmentManager.PutFragment(outState, "currentFragment", currentFragment);
}

protected override void OnRestoreInstanceState(Bundle savedInstanceState)
{
        currentFragment = SupportFragmentManager.GetFragment(savedInstanceState, "currentFragment") as MyFragment;
}

It might seem like a lot of work to save and restore your state, but all you really need to do is save off your Activity and Fragment’s instance variables and restore them at the appropriate moments. Most of the issues that I run into deal with forgetting to save/restore variables that I have added.

How does this affect async/await?

One other thing that you will need to keep in mind is that you will need to manage your async/await Tasks. You should try to implement your Tasks so that they can be cancelled if needed. If you have a pending Task when your Activity restarts, when the Task completes it will try to resume the original location which no longer exists. Ideally you should cancel any pending Tasks when the Activity or Fragment is stopped, or come up with an approach where the Task is running in some class instance that is not destroyed with the Activity.

Conclusion

As a developer, it can be annoying work to properly maintain your application’s state. If your application has a relatively simple UI that does not involve resource overrides, then you are probably going to be safe ignoring orientation changes. However, if your requirements change in the future, then that decision could give you a headache. This is one of those cases where it is probably easier to implement it properly from the beginning rather than ignoring it and refactoring it later once it has become an issue.

I hope that this article has helped you come to a better understanding of Android configuration changes and how you can take steps to make sure that your app is going to work properly when it is rotated.

Writing Node Applications as a .NET Developer – Getting Ready to Develop (Part 2)

In the previous blog post, I provided a general overview of some the key differences between the two frameworks. With this out of the way we’re ready to get started writing an application. However, there are some key decisions to make regarding what development tools to use as well as getting the execution environment set up.

Selecting an IDE/Text Editor

Before I could write a line of code, I needed to decide on an IDE/Text Editor that I wanted to use to write my application. As a C# developer, I was spoiled with the number of features that Visual Studio offered a developer that allowed for a frictionless and productive developing experience. I wanted to have this same experience when writing a Node application so before deciding on an IDE, I had a few prerequisites:

  • Debugging capabilities built into the IDE
  • Unobtrusive and generally correct autocomplete
  • File navigation via symbols (CTRL + click in Visual Studio with Resharper extension)
  • Refactoring utilities that I could trust; Find/Replace wasn’t good enough

While I love Visual Studio, I find that its JavaScript editor is more annoying than helpful.  Its autocomplete often gets in the way of my typing and it will automatically capitalize my symbols without my prompting.  Add to the fact that since I was working with a new framework and was already spreading my wings, I wanted to expose myself to another tool for learning’s sake.

Given my preferences above, I decided that JetBrain’s Webstorm would fit my needs:

  • Webstorm offers a Node debugging experience that rivals VS’s. One can set breakpoints, view locals and evaluate code when a breakpoint is hit.
  • The IDE’s autocomplete features (although not perfect) offer not only the correct symbols I’m targeting but often times would describe the signature of the function I was intending to call.
  • By indexing your project files on application start, Webstorm allows for symbol navigation via CTRL + click.  I was even able to navigate into node_modules files.
  • When refactoring code, Webstorm will search filenames, symbols and text comments, providing a safe way of refactoring code without (too many) headaches.

While not at the same level as Visual Studio’s C# development experience, Webstorm offers the user the next best thing, allowing for an environment that offers a familiar developer experience.  Although there are other (free) options available (Sublime Text, Atom, Visual Studio Code) I found that with these editors, I had to do more work to set up an environment that would allow me to develop at a productive pace.

Embracing the Command Line

Due to the power of Visual Studio as a tool and its ability to abstract away mundane operations, your average .NET developer tends to be a little wary of using the command line to perform common tasks.  Actions such as installing dependencies, running build commands and generating project templates are handled quite well in Visual Studio through wizards and search GUIs, preventing the user from having to know a myriad of tool-specific commands.

This is not the case when working with the Node ecosystem and its contemporary toolset.  Need to install a dependency via npm? A quick `npm i -S my-dependency` is required.  Want to run a yeoman generator to scaffold out an express application?  You only need to download the generator (if you don’t have it) using the same npm tool, run the generator with `yo my-awesome-generator` and walk through the prompts.  How about a build command?  Assuming you have an npm script set-up, typing `npm run build:prod` will do, (even though this is just an alias for another command line command that you will have to write).  In Node development, working with that spooky prompt is unavoidable.

While it might feel tedious and a step backwards as a developer, using the command line as a development tool has many benefits. You generally see the actions that a command line command is performing which gives you better insight into what is actually happening when you run `npm run build:prod`.  By using various tools via the command line, you have a better grasp of which tool is meant for what purpose.  This is in comparison to Visual Studio where at first blush, one equates Nuget, compiling via the F5 key and Project Templates to Visual Studio as a whole, not grasping that each of the commands you perform are separate toolsets and dependencies that Visual Studio invokes.  Having better insight into your toolchain can help in troubleshooting when a problem arises.

Running Your Code

The final step in writing a Node application is preparing your environment to run your Node code.  The only thing you will need to run your application is the Node runtime and the Node Package Manager (included in the same download and installed alongside Node).

Node.exe is the actual executable that will run your Node application.  Since Javascript is an interpreted language, the code you write is passed to this executable which parses and runs your application.  There is no explicit compilation step that a user must perform prior to running a Node application.  Furthermore, unlike applications written in a .NET language, Node programs are not dependent on a system-wide framework to be present. The only requirement for your code to run is to have the node.exe on the system path. This results in the deployment story of a Node application to be simpler and allows for cross platform deployment that is not yet readily available to .NET applications.

The neat thing about Node is that if you type in the `node` command without any file parameters, you get access to the Node REPL right in your console.  While this is great for experimentation or running/testing scripts, it’s a little lacking and I’ve only used it for simple operations and language feature tests.

While node.exe is what runs your application, npm is what will drive your application’s development.  Although the “pm” might stand for Package Manager to pull in your project dependencies, it’s more of a do-all utility that can run predefined scripts, specify project dependencies and provide a manifest file for your project if you publish it as an npm module.

Summary

Oftentimes with new frameworks and technologies, I have experienced frustration in getting my environment set up so that I could write code that runs at the click of a button. However with Node, the process is very straightforward, simply requiring one to install the runtime and package manager which are both available as an MSI that can be found on Node’s website.  From there, you can run your Node program by accessing the command line and pointing to your entry file.  In all honesty, the hardest part was deciding on an IDE that offered some of the features I became accustomed to when working in Visual Studio.

In the next and final post in this series, I will provide my overall experience with writing a Node application, detailing some questions I had surrounding application structure and testing, as well as giving a summary on my feelings on the runtime.

Writing Node Applications as a .NET Developer

As a .NET developer, creating modern web apps using Node on the backend can seem daunting.  The amount of tooling and setup required before you can write a “modern” application has resulted in the development community to display “Javascript Fatigue”; a general wariness related to the exploding amount of tooling, libraries, frameworks and best practices that are introduced on a seemingly daily basis.  Contrast this with building an app in .NET using Visual Studio where the developer simply selects a project template to build off of and they’re ready to go. [Read more…]

Universal Windows Platform – The Undiscovered Country for Windows Desktop Apps

It is unfortunate, but I think Dr. McCoy from Star Trek: The Original Series said it best: ‘Windows 10 Mobile is dead, Jim“…. and just to pile on with one more.

Buried underneath the red shirt like death of Windows 10 Mobile lies the amazing Universal Windows Platform (UWP).

The developer capabilities of the Universal Windows Platform have been documented in many a location.

UWP provides:

  • Full developer parity at the API and framework level across all flavors of Windows 10, Xbox One, HoloLens, and Windows 10 Mobile devices.
  • A simplified install model via APPX bundles.
  • A per-app separation of registry and file systems.
  • XAML controls ‘just work’ across all Windows 10 / UWP devices.
  • Targeting the UWP API set ensures that your app works across Windows 10 / UWP devices.

In my experience, there is nothing weirder than seeing your Windows 10 / UWP app just work on an Xbox One,  in a virtual projected rectangle via a HoloLens, and via mouse and keyboard on a standard Windows 10 PC.

 

NetToday-UWP

The above diagram is from the great Build 2016 presentation – .NET Overview given by Scott Hanselman and Scott Hunter.

UWP fits within a completely different vertical of development than the existing Windows Desktop technologies of Windows Presentation Foundation (WPF) and Windows Forms. As such UWP has access to the more modern feature set common to Windows Store applications. UWP has both duplication and subsets of API / XAML features that exist within WPF.

Right now may be a good time to take a look at your existing Windows Desktop apps and assess if they can be moved to UWP.

A UWP port of your legacy desktop app will allow access to modern UWP features:

  • Live tile support
  • Push notifications
  • Cortana
  • Background Downloader

There are 3 primary techniques to bring your current legacy desktop app to UWP:

  • Project Centennial
  • Straight UWP + Portable Library Breakout
  • Brokered Windows Runtime Components

Microsoft’s Project Centennial can be thought of as a UWP enabling wrapper for your existing Windows Desktop app. You may be able to repackage your existing Windows Desktop application and run it as a UWP app via a Project Centennial wrapper on the forthcoming Windows 10 Anniversary Update. As with all huge development efforts, there may be some caveats to using Project Centennial to package your Windows Desktop app as a UWP app (i.e. driver installation for custom hardware devices is not supported).

Update: Project Centennial based apps can go-live against the Windows 10 Anniversary Update as of August 2, 2016.

A straight UWP + Portable Library code breakout of your app could allow the features of existing Windows Desktop apps to be migrated in a straightforward way to Windows Store / UWP applications.

Note that the main UI layer of your existing Windows Desktop app is  not easily converted to UWP.

  • WPF XAML is not the same as UWP XAML with differences in control support, touch events, and lack of Triggers in UWP XAML.
  • UWP + XAML is a radical departure from WinForms. The headaches you may have encountered going from WinForms to WPF are only magnified in a WinForms to UWP conversion due to no real access to the underlying Win32 API set.

Brokered Windows Runtime Components allow a UWP app to tunnel between the UWP API and the Windows Desktop environment to run full .NET Framework 4.6 code on the Windows Desktop side. The downsides of this approach are many, the most limiting of which is that this technique can only be used on sideloaded UWP apps, not on UWP apps which are meant to be distributed via the Windows Store.

Interesting and non-obvious outcomes I have discovered during Windows Desktop app to UWP app code migrations:

  • You discover that there may exist possibilities for your app on Android and iOS with your existing code via third party Portable Libraries and your own code as Portable Libraries running via Xamarin.
  • I have found that having UWP compatible code, and Portable Libraries, allows you to create live prototypes that can bring your business services into an augmented reality future on Microsoft’s HoloLens, and a passable default experience on Xbox One.

We are at a true maturity point across the Windows, iOS, and Android ecosystems when it comes to repurposing managed code across all the .NET runtime variants via portable libraries. There is no better time than now to dust off that old-school Windows Desktop code and see how you may be able to put it to great new purposes on modern devices via UWP / .NET Core and/or Xamarin for iOS and Android.

I believe that the great things that Microsoft has shipped via their Universal Windows Platform have been lost in the 5 year sideways journey to catch up to Android and iOS. That sideways journey started with Windows CE variants in the form of Windows Phone 7; through the slight update of Windows Phone 8;  the flub of full Windows 8;  the long forgotten Windows RT / Windows On ARM strategy; through the full revamp (and necessary stepping stone) that was Windows Phone 8.1; and onto today’s real deal: Windows 10 / Universal Windows Platform.

Microsoft always nails something the third time they take a stab at it. In this case it feels like they have something real with UWP after 5 or 6 stabs and a whole bunch of missteps.

In a recent post to ArsTechnica, Peter Bright, master writer of all things Microsoft, wrote up a concise technical history regarding the path that Microsoft took in order to arrive at the current Universal Windows Platform destination.

With the maturity and solidity of UWP, and the positive mindshare linkage to Windows 10 via marketing and branding schemes, I think it is right to declare that UWP gets Microsoft out of the ‘Bull wandering through a china shop’ phase. With UWP, Microsoft is firmly into ‘We have something that is truly a firm third place contender, and primed for growth’ territory.

 

Windows Desktop app development has always been a major pain when it comes to:

  • Installation and Upgrades — The only clean way to do this was with Windows Installer, and the only clean way to do Windows Installer is Windows Installer XML (just look at that crazy XML schema, and bizarre otherworldly installer object model — Thank goodness for Votive though!)
  • COM + Win32 API / PInvoke — No matter what you do, it always seems like you have to resort to COM or  Win32 / PInvoke for something.
  • Device Access — It seems as if you need to know DIFx to install a driver, then have to know way too much about Windows Driver Model, Bluetooth sockets, or UMDF just to get read and write streams to your device from a program.
  • Security — How many times have you seen the ‘Red Banner Dialog of Doom‘ while installing something? Only to be told to click ‘Install this driver anyway’?

To solve the above platform deficiencies, the Windows platform needed a simplified app development model along the lines of Android and iOS. They have that scheme today in the form of UWP, with far superior tooling in Visual Studio 2015, and far easier app coding with C# / XAML.

On previous projects, the primary things holding back any sort of Windows Desktop to Windows Store / UWP migration were:

  • Third Party Software Libraries — Tied directly to full .NET Framework 4.x and usable only in Windows Desktop apps.
  • Specialized device access (i.e. USB / COM Port / Bluetooth) — Oft times these devices came from third parties as well as an access library written for the device that was only usable on Windows Desktop apps.

The third party software library realm has softened quite a bit. Many third party libraries that were directly tied to full .NET Framework 4.x have been open sourced (allowing you to pull needed bits out), or turned into Portable Libraries for use in UWP (and other mobile platform) applications.

If your existing Windows Desktop app requires integration with devices, you may have some issues with a straight Project Centennial wrap of your app. You may have to adapt and rewrite your device access layers against UWP APIs.

Specialized device access has matured greatly within UWP / Windows Store apps with the release of Windows 10 / UWP. There are now fully supported UWP API sets which allow:

A more friendly environment has been created in the world of Windows application development via a series of fortunate events:

  • The depths of despair that came with Windows 8 are gone with the rise of Windows 10.
  • The purchase of Xamarin by Microsoft.
    • Along with the support guarantees and savvy short term moves made since the acquisition.
  • Maturity of open source code + flexible licensing of that code
  • The rise of the UWP, iOS, and Android compatible Portable Library.

In general, the environment is such that it is a great time to look at a way to fracture and/or pivot your Windows Desktop apps onto UWP, and possibly even iOS and Android via Xamarin. Give it a try, you might be surprised how far your existing old-school code can stretch across modern devices.

Your pivot to get your Windows Desktop application working on Universal Windows Platform may even work on Surface devices from The Next Generation.

Apple WWDC – The Worth of Being There

In this era of travel budget crackdowns and higher than normal oversight into technology budgets, the iOS software engineer can get lost in the shuffle when it comes to training time and opportunities to talk with other members of the iOS development community.

Due to the rise of the popularity of Apple’s iOS platform, Apple WWDC (Worldwide Developer Conference) has become the central rallying point for iOS developers. It seems that the entire iOS community gathers in San Francisco over the week of WWDC. Many developers go to San Francisco over the week of WWDC even if they don’t have a ticket to the conference.

A ticket to WWDC costs $1599. Since 2014 you only get to purchase a ticket if you get picked during the lottery phase of enrollment. In my opinion, this is a very fair system considering the high number of developers who wish to attend.

Due to the high number of people going out to San Francisco over WWDC week, the hotel prices during WWDC week have almost tripled since 2011. John Siracusa,  of Mac OS X Review and Accidental Tech Podcast fame, has equated a night of sleep in a hotel during WWDC as $ equivalent to an Apple Watch. You sleep 1 day, that’s 1 Apple Watch (Sport). You sleep 2 days, that’s 2 Apple Watches… From Sunday through Friday you may have to plan on spending $1600 just to sleep.

The airfare remains affordable. Well, as affordable as airfare can be from your home location to SFO.

Before you have so much as sipped even one $7 Blue Bottle coffee (totally worth it, just do it!), the conference has already cost around $5000 dollars.

To a cost conscious manager $5K as a baseline price for a week spent at a conference can seem like a pretty steep price to pay.

I would submit that the $5K cost is a really good value if you know how to attack WWDC.

Summary:

  • WWDC is not anything like Microsoft Build.
  • You get insight into the hierarchy at Apple that you can’t get anywhere else. Especially the relationship between you (the developer), the Apple Evangelist, and the Apple Engineer.
  • The all-day technology-specific labs that Apple runs at WWDC may be the only way you are ever going to get face time with Apple personnel that have specific knowledge in your app subject area.
  • You get inspiration and insight from the special lunch sessions.
  • You can take advantage of the 1 to 5 ratio of Apple personnel to attendees.
  • You agree to a Non-Disclosure Agreement regarding session content. This turns out to be a great thing because you get exposed to stuff that you never experience otherwise.
  • There is now plenty of good content presented at other nearby venues at the same time as WWDC.
  • Sometimes you are at the epicenter of a major shift in Apple’s technology stack.
    • Those who attended WWDC 2014 got the boost of being at the center of the Swift rollout. The Swift rollout surprised most Apple employees, not to mention many attendees.

Note: During this blog post you may notice that I am being completely vague and only giving you an impression or insight with no detail. When you attend WWDC you agree to a Non-Disclosure Agreement. This means that you can’t talk about details or technical information related to content presented there. Because of this, I can only give you my impressions of my experience at WWDC.  This NDA has loosened in recent years, but I think it is still good to keep the attitude: ‘What happens at WWDC, stays at WWDC’.

 

WWDC – General Structure

WWDC starts with a day 1 keynote. This is a classic Apple keynote.

The keynote is usually at 10AM Pacific Time.

For WWDC attendees who want to be ‘in the room’ this means that you may have to line up at about 7AM. You are let in the building around 8:30AM. Then you wait. For almost an hour and a half. In a hallway with no chairs.

They open the doors to the main keynote area, everyone finds a seat, and the keynote begins.

The keynote ends about an hour later. Everybody leaves for lunch. Then the remainder of the first day is spent on a more engineering focused deep dive set of talks called ‘State of the Union’.

The rest of the week has standard conference tracks scattered around the many rooms all grouped by technology.

Many of the sessions are repeats of sessions from previous years with a little more content added and/or clarified.

I am pretty sure I am not divulging too much info here as all of these sessions and the schedule are available via the WWDC app.

 

WWDC != Microsoft Build

In order to really ‘grok’ how Apple’s WWDC is dramatically different from other technology conferences you really need to attend. I have been extremely fortunate in that I have been at 2 WWDCs and a series of Microsoft conferences. I would also point out that the WWDC differences also show how Apple is a dramatically different company (especially from Microsoft).

Apple WWDC is dramatically different from Microsoft conferences in so many ways.

  • Degree of Preparation
    • WWDC is highly polished. Every session. All the time. It feels like all the session speakers have rehearsed their material for weeks. It feels like all the presented material in all the sessions has been vetted by the Evangelist (and possibly marketing) hierarchy at Apple.
    • Microsoft Build has well polished keynotes, but the per-session material can feel a little ‘slap dash’. In this way Microsoft Build sessions feel more genuine but aren’t as watchable as Apple WWDC sessions.
  • Openness
    • Contrary to the initial impression you may get, WWDC is fairly open. It is more open once you understand the role of the Apple Evangelist as a gatekeeper. Do not try to get the time or attention of a presenter at a given session. Go to the labs. The labs are largely a free zone where engineering conversations flow freely between attendees, Evangelists, and Apple engineers.
      • Apple presenters do not usually hand out their e-mail or contact information as their public exposure seems closely regulated by the Evangelists.
    • Microsoft Build is completely open with engineers and members of the Microsoft hierarchy handing out Twitter handles and work e-mail addresses.

 

Insight into hierarchy and roles – You, Apple Evangelist, Apple Engineer

The Apple WWDC session structure is very rigid. If you analyze a series of Apple WWDC session videos you start to notice this rigid structure:

  • Speaker shows title slide of session with their name on it for about 5-15 seconds. — No e-mail address, Twitter handle, or other contact info of the engineer will appear on the slides.
    • Note: Sometimes the presenter exposes some contact info, but it is incredibly rare.
  • Speaker proceeds through their material.
  • Last slide shows links to online Apple SDKs, additional WWDC sessions, and the name of an Evangelist for the technology stack that the session was about
  • If the session has a Q&A you will notice that the Evangelist does triage of incoming questions from WWDC attendees. The Evangelist will usually do their best to answer the incoming question, frame the scope of the answer, then will either trust an engineer to answer on stage or redirect you to a lab time to talk more.

The well coordinated Developer to Evangelist to Apple Engineer communication path is unique. The role of Evangelists as gatekeepers to the time and energy of the Apple Engineer leaks through in almost every session you attend at WWDC. Once you understand the rules of the Apple hierarchy in regards to exposure of Apple engineer face time to the developer community,  you are now ready to head into the lab spaces at WWDC get your questions answered and have great conversations with Apple personnel.

 

The Labs are invaluable

The per-technology lab areas at WWDC are invaluable. If you have had electronic contact with an Apple engineer at any time the previous year, you should be able to get face to face contact with that engineer in a lab space. WWDC is, most likely, the only time that you will ever get real face time with an Apple engineer throughout the year.

If you are signed up for any of Apple’s side programs (i.e. Made for iPhone/iPod [MFi], AirPrint, …) then the labs are the one safe space for you to talk engineering specifics in and around that program. The NDAs you sign in order to participate in those programs may largely preclude you from even talking with your peers about what you are really working on. At the specific lab times, and only at WWDC, can you get one-on-one time to talk about the proprietary NDA’d specifics of your application.

When I was at WWDC 2010 there was mention of a series of technologies during the keynote that seemed to overlap with a non-Apple project I was working on. I went down to the labs to talk with Apple people about their project, and related my experience on my non-Apple project. From that experience I  gained a ton of insight and validation regarding my project that I never would have received otherwise.

 

Go to the lunch sessions – They are the hidden gem of WWDC

You just spent a whole morning attending hard technology sessions. You have a numb brain. You have some messages and work to catch up on. A lunch to get eaten. The last thing you may want to do is go to a session over lunch.

The WWDC lunch sessions are where Apple invites people from outside of the company to come in and share how they are applying a specific technology stack to a given creative problem. Every lunch session I have attended have been the best sessions I have attended at any conference. The speakers are engaging, the content is insightful and inspirational. Don’t miss the lunch sessions.

 

There is a 1 to 5 ratio of Apple personnel to attendees

This is one of those stats that I heard then just kind of dismissed. I mean: How could I ever possibly be able to pick out Apple personnel from the 5,000 other attendees at the conference?

It turns out that Apple gives employees specially colored badges which stand out. In addition, most of the Apple employees wear an ‘Apple Store’ like t-shirt. You do end up seeing and talking to quite a few Apple employees. Many Apple employees seek out and engage with developers at WWDC because that is one of the very few times that they can engage with the developer community.

Over breakfast one morning I had struck up a conversation with another engineer about App Store policies. An Apple employee responsible for App Store policies, and some of the technology, overheard our conversation and joined in. The conversation that was sparked was great. In this case the Apple employee had a series of questions that he asked, and largely listened as we talked about a series of things. The feedback we got was invaluable regarding how and why Apple has the App Store policies that they have.

At each session there is a roped off area at the entrance: One side where Apple employees line up, the other side where non-Apple employees line up. All non-Apple employed attendees get into the session first, then the Apple employees get any remaining seats.

I mention this because when you combine the waiting lines in front of a session with the specially colored badges + t-shirts you can get insight into which sessions may be the most interesting.  You get immediate visual feedback for what the Apple engineers think the most important technology topics are just by looking down a wide conference room hallway and looking for clusters of colored t-shirts.

Way back in 2011 I was at WWDC and I was waffling between 3 different sessions being held at the same time. One was about Bonjour networking, one was about Xcode enhancements, and one was really out there (for me at least): LLVM / Clang compiler futures. I walked past the networking and Xcode enhancement sessions and there were people in line, but only 1 or 2 Apple employees. Then I went past the LLVM / Clang compiler futures session, the line of Apple employees waiting to get in was down the hallway and had practically turned the corner. I immediately ducked into that LLVM / Clang compiler futures talk.

 

Non-Disclosure Agreement – Not a bad thing

I hope that I am allowed to tell the above story, because you sign an NDA that states that you cannot talk about any session content from WWDC. This actually turns out to be a good thing.

Apple has relaxed the NDA in recent years. However, that NDA still applies for some sessions, labs, and conversations that you have with Apple employees. If you are under NDA for other Apple programs, then those NDAs are fully binding in the lab areas as well. You may be asked by an Apple employee in a lab area if you are under NDA before they can talk about certain technologies.

I can’t get specific, because I signed an NDA when I attended WWDC 2011, but I think I can dance around enough stuff to give you insight into why that NDA was a good thing.

Over lunch Apple has special sessions given by people from other companies that highlight how they use certain Apple technologies. These special sessions are skewed way more to practical application of technologies for specific businesses or creative endeavors.

At one lunch session we were able to see unreleased creative content, as well as how technology enabled that content to be created. The speaker stated outright that without the NDA we agreed to at the beginning of WWDC that there was no way his management would have allowed him to show this special content and inside info during his presentation.

C# vs. Swift – Iron Man vs. Captain America

In Captain America: Civil War we get to see the ultimate battle between Iron Man and Captain America.

It is a battle of simple gutty defense vs. smart weapons and flashy offense, humility vs. brashness, down in the dirt vs. up in the clouds.

To totally geek it up, the same kind of battle exists in the languages that software engineers use today and I believe this is especially true in the battle of C# vs. Swift.

Don’t worry, this really isn’t a versus type write up. If anything I seek to point out each language’s unique strengths, then show how software engineers can get into the right superhero mindset to really use those strengths, and be aware of the weaknesses, to create great solutions.

 

C# = Iron Man – Strong Flashy Offense

C# is truly the Iron Man in this comparison. It is amazing how much showy weaponry C# can bring to bear to solve problems.

  • Have a problem with your UI freezing up? Bring in Task<T> / async – await, and seamlessly refactor years old code to be fully asynchronous, even within native UIs where thread synchronization is key.
  • Have a super large dataset that you need to tame? Bring in IEnumerable<T> / yield return or Lazy<T> from the deferred execution realm.
  • Have a need to produce arrays and collections that are the result of super smart functional filtering? Bring in LINQ.
  • Need to do smart filtering of XML? Bring XDocument and its magic LINQ over XML powers.
  • Need to do JSON processing? Head on over to NuGet, reference Newtonsoft.JSON ,and get stable, powerful JSON serialize / deserialize behavior and be done in no time.
  • Need to hit a REST service with super specific HTTP header formatting and authentication? System.Net.Http has you covered.
  • Need a cross-platform way to retrieve data from remote sites + Model / View / View Model architecture? Check out ReactiveUI.
  • Need async but inline immediate responses to collection changes? Check out Rx.NET.
  • C# is strongly typed, but you can also use reflection and loosen that strong typing as needed.
  • C# can even be used to generate itself.

You can even do most the above across all platforms. Windows, Windows Mobile, Android, iOS, Linux servers, … C# has you covered, in any way, any time, any where.

There is every gadget and gizmo that you can bring in and bolt on to existing C# code to get your job done. Even the very oldest C# code from 2002 is still supported and easily transformed using Visual Studio, or Resharper tooling into something shiny and modern.

C# has all these weapons, but the lifetime of these weapons are sometimes mere seconds before they are replaced with the next version via NuGet. The masters of C#, just like Tony Stark, have no issue with throwing out just created stuff in favor of the next great thing.

C#’s weaknesses are the dreaded NullReferenceException and the indeterminate mutability of most data structures. C# is great at attacking problems, and hitting at the 90% of the solution in a lightning quick way. However, C# doesn’t play defense very well. If you go too fast too far with C# you can out run your air cover, leaving you vulnerable when attacked from the flanks by null members, timeout errors, weird data changes, unhandled exceptions, or not connected to network scenarios.

C# is a highly flexible, multi-weapon language that you can use to attack any problem.

Swift = Captain America – Strong Defense

Swift is the ultimate boots-on-the-ground and in-the-mud defensive language. In Swift you don’t solve problems by attacking them head on with whole armories of shiny weaponry, you hunker down, put your shield up, and start slogging up that hill one step at a time.

Every line of Swift code written requires the author to think of the worst case:

The inventors of Swift must have looked at the metrics of existing Objective-C iOS applications and realized that the number 1 and 2 problems were:

  • Coders not realizing that the data they are changing on one screen leads to changes in data on other screens.
  • App crashes due to trying to access the value behind a null pointer.

Defense was needed. The power of Captain America’s vibranium shield was needed, only this time wielded by coders to stop these major in-app issues.

Coding in Swift is highly defensive in nature. To see this in action take a look at Swift Optionalsif .. let syntax, and the class initialization rules. As you use them, Swift optionals will permeate your entire thinking as you rev up your data structures. Every variable you use and/or create needs to be immediately thought of in terms of optional vs. non-optional.

Within Swift the use of specific reference and value types allows you to defend against unwanted mutation to your data structures.

Each line of defensive code is one more brick piled on top of other bricks of defensive code. When done you end up with applications that resemble the Great Wall of China and are truly written for the worst case. This layered defensive code coordination leads to super solid software structure that can take a real beating when attacked by the invaders (or users) of the software.

When done solving the problem in Swift you have a beautiful shiny shield that can repulse almost any mutability or null data attack. Because you play so much defense in Swift, coding up solutions to things seems to take longer. In reality, you have taken all that defense that C# out ran and have incorporated it into the structure of the solution.

The constant nullability slog, coupled with the constant mutability slog, while coding in Swift leads the developer to wanting more weapons to go at problems. Constant defensive coding slog can lead to developer fatigue quite quickly. However, when the problem solution is achieved, the engineer can see that their defensive coding effort has been like using Captain America’s shield as their primary weapon.

With the future potential of Swift existing across platforms, it could mean more solidly crafted software across the board, from server side to mobile applications. The underlying thought process behind Swift is already starting to leak into other languages, especially in the area of nullability.

I know that Swift isn’t without its share of flashy bolt-on weapon systems. Comparing C# to Swift when it comes to third-party library support is like comparing Iron Man’s crazy spin laser and hand repulsors to any weapon Captain America has ever wielded (at least in the movies).

 

But who wins in the ultimate battle?

If you ask me, I think Captain America and Iron Man are both good guys who should be augmenting each other’s strengths and covering their corresponding weaknesses.  I bet the  *SPOILER* real big bad *SPOILER*  is hiding behind the scenes and making them battle it out.

As with all things in software engineering there is never a clear winner. There are drawbacks and advantages to the structure, approach, and capabilities of all languages and constructs. These drawbacks and advantages tend to fit or not fit for solutions to given problems.

Good ideas tend to bubble to the top (i.e. delayed execution, in-language async, nullability enforcement, mutability enforcement), whereas bad ideas tend to settle to the bottom (i.e. garbage collection in Objective-C).

Swift is so new, the in-language concepts so hardened, the community so outspoken and willing to move quickly, that the verdict is not yet out.

C# has been around so long, has so much weaponry, and is currently so incredibly flexible across even dynamic and functional concepts. There is no way C# doesn’t continue to flex with some of the core concepts surfacing from the Swift and Kotlin worlds.

Who knows, one day you just might see Iron Man wielding Captain America’s shield. Just like C# currently wields dynamic and functional constructs, C# could well start to take on deep nullability constructs as they exist in Swift.

At the end of the day, we all win as these 2 languages battle, take on, and wield the best parts of each other to surface great solutions to real software problems.

My Trek Through MinneBar 11

minnebar.logoOn April 23, 2016 I attended my first minnebar conference: minnebar 11

I am amazed that content of this quality was presented by community members for free. It was totally worth being inside and roaming the maze-like halls of Best Buy Corporate Headquarters on one of the few sunny Saturdays we get up here in the frozen tundra of Minnesota. The exposure to new technical topics was great, but more importantly experiencing the energy of the people who are active in the Minnesota tech community was the real core of the experience.

I will try to mirror the energy and great themes of each of the sessions that I attended. The keyword is ‘try’. I apologize in advance if the energy and competence of each session I attended doesn’t shine through. Hopefully, some of the links I am adding to each session will help you navigate to additional resources.

Minnesota: The State of Tech [Startups]

This session consisted of a panel of people active in the Minnesota startup community.

The general takeaways that I got from the panel were:

  • Get involved with the community
  • Don’t be afraid to share your ideas. Sharing your idea will only make it stronger.
  • Double down when it seems like you should quit, double down again when it seems like you should stop, and when it gets hard and you think you are truly at the end, double down again
  • There is help for anything you want to do with your startup. From business, to community, to technology, there is an amazing community here in Minnesota that is dedicated to helping you get your business started in some way.
  • In general the panel is encouraged at the state of the startup and general business climate in Minnesota.
  • The depth of participation in the MN Cup Startup Competition was cited as a great sign of the vitality of the startup community within Minnesota.

University of Minnesota Research is Generating Disruptive Tech Sector Inventions

I can’t say enough about how eye opening this presentation given by Russ Straate and Chris Ghere was for me.

The process and overall maturity of the way in which the U of M incubates and manages the movement of their technology portfolio from the university into the business sector is astounding.

I can’t do the whole presentation justice, but for future reference you should watch the U of M Entrepreneurs site. Especially great is the University Startup Pipeline Google Doc which shows the stages of the startups as they move from the university onto business.

Running your startup with APIs (accomplishing more with less code!)

Ben Peter gave a great presentation on how to use the Zapier service to create custom data workflows from existing third party Web APIs.

His best demo was a code-free custom workflow that would send a free t-shirt to an attendee given an e-mail address contained in a text message. The demo used Zapier to aggregate SMS, Google Sheets, and Printfection, all without code to create a custom business promotion on-the-fly.

There is real power in being able to aggregate the features of entire businesses via their API surface. This power allows startups to create easy solutions for repetitive business tasks in a fraction of the time and resources of having to write custom code or rev up whole hosts and custom service silos.

Kotlin vs Swift : Which language is more Modern, Concise, Safe and Functional?

Andrew Rahn of Iconfactory gave a great side by side talk that compared Kotlin and Swift.

I really enjoyed his openness, energy, and passion for this topic. His openness really made the talk a true two-way discussion between himself and the audience.

Personally, I have done some Swift, and absolutely no Kotlin. His session served as a great jump start for me as I learn more about the Kotlin language.

The general take aways were:

  • Both Swift and Kotlin are really new languages, and as such are in a state of flux as their owners tweak the compilers, syntax, and environments
  • Both Swift and Kotlin are modern languages that force the developer to use the compiler to check nullability and mutability rules
  • Swift and Kotlin suffer greatly from being bolted on top of their respective legacy heritage (Swift — CocoaTouch / Objective-C / Automatic Reference Counting — Kotlin – Java / Android)
  • I would be incredibly lucky to ever get a chance to work with Andrew Rahn on any project.

Panel: Why YOU Should Enter a Data Science Hackathon

I went out of my comfort zone of front-end UI development and attended this panel.

Please see the minnebar session page for the panel members and full description of the session.

My takeaways:

  • When it comes to data science you spend over 90% of your time cleaning up your data sets before you can actually do any analysis. Things like different date formats, missing or null values, punctuation, and differing data entry techniques cause the data scientist to spend a ton of time cleaning up data.
  • There is a debate within the data science community between those who are going hunting through data looking for patterns, and more hard science statisticians who seek to apply statistical techniques to data to achieve a less biased / less ‘hunt and peck’ methodology to looking at mass data sets.
  • The Analyze This! competition seems like a really great thing to take a look at. The Analyze This competition is a long term (3+ months) competition that goes from raw data through to creating a possible predictive model.
  • Another interesting resource that was mentioned was Kaggle. Kaggle is a source of competitions and sample datasets.

Exploring Stateless UIs in Swift

At heart I am a native front-end developer, and this talk was 100% in my wheelhouse.

Sam Kirchmeier and Adam May from Livefront, and helper / organizers of TC HACK, presented their discoveries while creating a well factored, stateless UI, for iOS.

Their work brings the architectural ideas of React and the concepts in Flux to front-end iOS development using Swift.

Their presentation was done so well that the core concepts they presented could be applied to clean up and simplify almost any front-end development you may be doing on any platform.

Their work has inspired me to start pursuing a similar framework for Xamarin.iOS and Mono.Android development. In fact, the more I think about, the more I realize that their work may be more powerful when expressed in a cross-platform framework as it could lead to incredibly thick UI code sharing between platforms, and incredibly thin per-platform code. The possibilities for eliminating MVCs (Massive View Controllers) or huge overwrought Android Activity code are awesome.

If you do any UI work at all on any platform: Run, don’t walk, to their presentation slides and see the future.

Looking into the Crystal Ball: Using Google’s Prediction API

This presentation by Justin Grammens detailed his exploration of the Google Prediction API.

Justin had some great examples of applying the Google Prediction API against the Nice Ride data set to try and predict where a bike would be dropped off depending on when and where it was picked up.

The main question from the audience during Justin’s demos was: What is the underlying statistical model that the Google Prediction API uses? The answer, unfortunately, is that Google isn’t telling.

I would file the Google Prediction API away as a future tool for your toolbelt. At least until Google takes it away, never to be seen again.

Firebase – A Real Time Document Database

There are a plethora of document databases to choose from nowadays. The entire nature of storing data is changing, so how we work with data needs to change as well. Single page applications on the web need to be responsive, not just in layout but in communication as well. Users have come to expect a higher quality of data representation, and the landscape is quickly evolving.

[Read more…]