Android Market Immaturity

Whilst I’m on the war path of Android, there’s a number of major issues that the Android Market has which have been causing me great annoyance lately. It feels very much like Google rushed out a Market application that meets their major requirements, but haven’t put much thought into a lot of how the market will behave in the real world.

 

1. Application Update Management

My IT background has a large component of working with enterprise and corporate organisations, in particular, telecommunications companies. These companies are often known for their annoyingly slow habits of deploying new software:

  1. Determine new software version to use.
  2. Document installation, deployment procedures.
  3. Complete strict testing of applications.
  4. Deploy application.
  5. Test and ensure functionality. If a fault occurs, rollback using the documented procedures.

On the other end of the spectrum, the Android Market has the following behavior:

  1. Find updates. (automatic updating can be turned on/off per application).
  2. Install them.
  3. Don’t like the application following the update? Software bugs? Tough, deal with it.

Whilst I’m hardly going to advocate making test plans for deploying Android updates, I think Google need to take some lessons from the enterprise environments – software will always surprise you with bugs, so plan for rollback options.

There will be times when you update an android application, only to discover that it’s changed in some undesirable way, or that it’s developed a bug in a key feature that you use every day or maybe just doesn’t suit you as much as the older release.

I’ve experienced this issue in the past, where a twitter client update broke posting images via twitter for about a month, before a subsequent update fixed it. Whilst this was occurring, I had no means to be able to go and downgrade the application to the older version that had worked fine.

Sure, it’s not as scary an issue as 10,000 customers not having internet like the telco world, but for that user who’s suffering a bug that impacts something they use daily, it’s a big fucking deal.

Add versioning and rollback support. Seriously. Please. Linux has had this sorted for years (decades?), you can always downgrade a package on a distribution to an earlier version if so desired.

Whilst it is possible to downgrade an application on Android if you can locate the .apk file elsewhere, if the application is only available via the Android Market, there is no approach other than earlier phone application backups that you might have created.

 

2. Vanishing Applications

I’ve been using Android for some time now, since around Android 1.5, during this time I’ve used a lot of different applications and have experienced the annoying issue of applications that I like and use being removed from the Android market place.

What tends to happen is:

  1. User find a nice application that meets their needs, downloads and installs.
  2. Developer pulls the application from the market – this can be any number of reasons – trademarking, unhappiness at application quality, removing a free app and going commercial only, no longer any desire to maintain, or even due to removal by Google for malware.
  3. User ends up buying a new phone, or re-installs a new Android OS image and wants to install all their favorite applications again.
  4. User is unable to find their application on the market to download again.

Once this happens, the only option is to try and recover the application from an existing phone, find it floating around online or if it’s an open source application, find the public repository (even abandoned apps tend to keep the source around) and download and compile the application.

Otherwise the user is left with trying to find an alternative application (if one exists) that could be better or worse than what they previously had.

This particular problem has bitten me enough that I’m always actively seeking for open source options and choosing them, even if a proprietary application is slightly better – the knowledge that I can always build the app myself if it vanishes is a key point.

Unfortunately it’s not that easy to always tell which apps are open source or proprietary thanks to the Android Market’s unclear licensing information:

 

3. Clear licensing information.

Android Market will not report what license a particular application has when viewing the applications details or even when downloading the application.

This is a problem as there’s no way in the market application to tell whether an application is free as in freedom or free as in beer, which is a big problem for any users like myself wanting to choose software options that are under an open source license.

There have been numerous requests to Google to change this, something that surely must not be  a hard feature to add, but there’s been no visible progress on this issue.

For now I’m taking more efforts to research applications before installing them, and using F-Droid, the open source only repository as a first stop to find applications.

 

4. Freedom & Censorship

The use of the Google Market application offers some handy features such as the ability to remotely install software onto the phone via browsing the market website, a legitimate and useful function for some.

This connection to Google also allows Google to remove applications that are undesirable – the intent of this is to remove known malware and malicious content from devices, once again, a legitmate and valid use.

The downside, is that there is the capability for Google to use this connection to install or remove other software components in future, for either their own motives or that of a court order.

Consider something like a wikileaks application providing leaked data, or an application to bypass censorship which causes embarrassment or problems for the US governement. As a US company, Google could be ordered to remove that application from devices worldwide, a very plausible and concerning scenario – even if a user is confident about the ethics of Google, it wouldn’t stop a court order forcing software to be removed.

If this scenario seems far fetched, remember that Amazon removed a particular book from all their e-readers after a copyright dispute, removing not only the book, but all the user prepared notes for them.

I’m a strong supporter of computing freedom, having vendors like Google becoming gatekeepers and controllers of what we can and cannot run is concerning, particularly as the future of legislative policy appears to be tighter and nastier, particularly with the US.

 

Can it be fixed?

It would be pretty straightforwards for Google to fix issues 1-3:

  • Add version awareness to the market place, so a user could downgrade applications – even if it was limited so a user could only download to a version they previously had, I would be happy.
  • Keep pulled applications in the market place (with exclusion of apps removed for malware/malicious purposes – in that case, it should be removed and labelled as such) at least for users who have downloaded them in the past, so we can continue to use our favorite apps. A warning that this application has been abandoned or some other term would be fine.
  • Provide licensing information for applications, along with search abilities to find applications by license type. A link to the upstream source would be a nice touch too.

The 4th issue is a little more complex as the ability to remotely manage software has valid features and isn’t as simple as just removing.

Ideally I think the best approach would be to adjust the structure of Google’s Android integration, so the hooks into Google having control over the phone can be changed to allow/always prompt/disable approach.

This still allows for all the current functionality, but gives users with concerns about Google’s abilities to control how their phone behaves.

I’m pessimistic about Google actually going and fixing these things – they aren’t major selling points to attracting new users to Android, but I think they need to be addressed for Android to be more reliable and usable long term.

Android: the free-ish mobile platform

I’ve been using Android for a while now, starting with the somewhat underpowered HTC Magic (G2) before moving up to a much snappier Google/Samsung Nexus S which is now my current model.

Whilst I’m a fan of the platform overall, I’m encountering more and more issues every day with the fact that Android is being positioned as the poster child of open source in the mobile space (with other alternatives like Meego and WebOS way behind in terms of market share and consumer awareness), yet Android is only partially open source, still relying on large proprietary chunks.

With the recent release of Android 4 (Ice Cream Sandwich), I decided I would run through the steps of compiling Android from source code – I’m a firm believer of only running things that you have the ability compile yourself, and have gone through the exercise of building Linux From Scratch and custom distributions in the past to gain understanding of how the Linux OS is assembled and functions.

Android’s open source release (AOSP) is available from source.android.com which provides instructions for downloading the (large!) source tree, tools and building them into a functional device.

Doing so was an interesting experience – some of the first issues encountered are:

  1. AOSP is limited to working out-of-the-box with only a select number of devices – any official “Google” phones, like the Nexus S or Galaxy Nexus are supported, along with a couple additional vendor models (such as the Xoom tablet).
  2. If you have a non-google supported phone, you’re on your own – depending on your vendor, it may be a simplish, painful or maybe impossible task to obtain the required binary blobs. And that doesn’t cover whether or not the phones have locked or unlocked bootloaders.
  3. Even the Google AOSP supported phones can’t run a pure open source stack, proprietary downloads are supplied by Google for specific hardware components for each model and for a specific OS release. Should Google decide to stop supporting a device with future Android versions (as has happened with earlier devices, you won’t easily be able to support the hardware).
  4. The source is big, the build is hungry and you’ll want some performance to build it. I allocated around 40GB for the checked out source and build space and used most of it, along with 8GB of RAM and a few cores on my Phenom. On the plus side, the build only took a few hours, not the days-long efforts some online had predicted.
  5. Google’s build instructions are a bit lacking, given a week, even a Google intern would be able to make a massive improvement to it, I ended up finding many useful commands online that weren’t documented on AOSP’s home page – such as how to package a build into an OTA style .zip for deployment.
  6. The Linux kernel isn’t compiled as part of the Android build process. Instead, Android used a supplied pre-build binary kernel and just includes it into the finished OS image. If you need to adjust the kernel, it must be built separately and then placed into the correct location in the Android sources. This process isn’t documented anywhere on the AOSP homepage that I could find.

The base AOSP build provided me with core functionality including:

  • Functional base operating system and all hardware (thanks to binary blobs from Google for my Nexus S).
  • Communications – calls, txts, wifi, bluetooth, internet browsing
  • Contacts/Address Book.
  • Ability to install applications from direct .apk download or transfer using adb from PC.
  • A generally working and usable device overall.

But I didn’t have a number of needed functions that are typical of off-the-shelf Android devices:

  • No support for Google Account synchronization – without this, I was unable to download synced contacts and any other information from the cloud account.
  • No android market, the only way to install applications is from third party markets, direct download of .apk files or self-compiling applications.
  • No google service-based applications –  google maps, gmail, google talk, etc
  • No face unlock ability – I expected this would be part of ICS, but seems it’s part of Google’s application set…. this mix of having open source and proprietary components is one of the biggest problems with Android, you aren’t always sure what is or isn’t open source.

To get these other needed functions that are typical of an Android phone, you need the Google apps package (or at least the market application so others can be downloaded).

The killer part is that this package isn’t freely available. From what I’ve been able to find, it seems Google only provides their application package to vendors who pass their tests for Android compatibility to maintain quality.

Personally I think this is a lot of crap, considering the shocking quality of some Android devices that have come out – at the very least, the Android Market place application should be freely available, so users can at least choose to download applications, even if Google decides a particular vendor doesn’t deserve their Google apps.

In the end I managed to source a package of the Google applications for ICS thanks to the efforts of the Cyanogenmod team, but this is a shocking approach – not only is there an uncertainty about having the latest versions, but having users trawling through the internet to find tarballs on some forum is an easy avenue for attack and getting malware onto phones.

The fact that I’m so reliant on hackery to get key functionality back for my phone if I choose to build from source, rather than using my phone vendor’s build images is giving me solid reasons to consider the feasibility of dumping Google’s components from my phone and finding open source replacements for them all.

Whilst Google deserves credit for making the base OS comparability easy to build for users of Google-approved devices, the fact that they’ve allowed vendors to get away with binary blobbing drivers everywhere and keeping key functionality proprietary (market, etc) is pretty bad.

Chasing down binary blobs in order to get a device to work as expected is much more reminiscent of days spend pirating software, not of a healthy open source project and it makes Android feel much more hacky and crappy than it should be.

And the fact that the open source build will work with so few of the phones on the market out-of-the-box is just appalling for an OS that’s called open source – I should be able to go pickup any Android phone in the market and be able to compile AOSP for it and have all the hardware supported, not just select models.

Part of this is the fault of device manufacturers, but IMHO, Google should have put down some restrictions in the use of the Android trademark, so that a device could only actually be an “Android” device if it was fully open source and featured an unlocked bootloader.

I’d even accept a compromise where binary blobs are needed for specific hardware, as long as the blob wrapper can be compiled against different kernels and is free to redistribute (aka firmware style), so that I could buy a phone running Android 2 and happily go and build Android 4 for it and expect it to work.

Overall, I’m happy that I could build a functional image for my Nexus S without too much pain, but disappointed that so much of the feature set we are used to with Android isn’t actually open.

Custom CA certificates & Android

With the number of servers I have internally, I have setup my own Certificate Authority and sign all my internal SSL certificates against this private CA.

This offers the useful advantage of being able to import the one CA certificate into all my devices and then being able to validate all connections to remote systems – if you run more than one or two personal servers, I’d highly recommend this approach – certificate signing takes a little bit of getting used to, but it’s a good skill to have.

As I want to access a number of systems via my Android mobile, I needed to import this CA file – the following instructions were followed with ICS release 4.0.3, however it may apply to earlier/later releases as well.

If you’ve followed most typical instructions for building your CA, you will have an PEM encoded CA certificate file in ASCII format. This is fine for import into most browsers and desktop OSes, however Android is particularly fussy with it’s input and requires a binary format only.

You can convert the CA PEM format file with the following command:

openssl x509 -inform PEM -outform DER -in CA.pem -out CA.crt

Then transfer the generated CA.crt file to the sdcard – easiest is via adb:

adb push CA.crt /sdcard/

Once done, you will be able to tell Android to install the CA file via Settings -> Security -> Credential Storage and selecting “Install from storage” and following prompts.

To verify functionality, easiest test is to access an https website signed with your CA certificate via the browser.

Some commenters have had issues - here is me importing a valid CA cert in DER format.

Some commenters have had issues – here is me importing a valid CA cert in DER format.

2012 New Year’s Resolutions

It’s 2012, time to update my new years resolutions and make plans for this year! :-)

Firstly, what’s happened since the start of 2011?

  • I met Lisa, we’ve been together 11+ months now! :-O
  • I’ve packed up my life and moved to Auckland after Lisa started a journalism job up here, it’s the first time I’ve lived outside of Wellington, so it’s a big change for me!
  • I’ve been working for Prophecy for over a year now, these days working from home or from customer sites in Auckland, going pretty well.
  • I’ve done more travelling around NZ than ever before, various road trips around the North Island and seeing new places.

I think I’ve done a pretty good job of the resolutions I made last year:

  • I’ve reduced the amount of stuff I have, sold almost all my rackmount servers, sold all my beloved data centre components and cleared out huge piles of things from the flat.
  • I haven’t quite gotten down to the goals of my less, less, less life approach, now that I’m not single it’s not quite as easy, if I clear out some space, Lisa just fills it in with something.  ;-) Although I’ve done a pretty good job of holding myself off buying gadgets that I don’t really need this year.
  • I’m working sane hours and having a bit more of a work/life balance. Will be interesting to see how this lasts now that I’m working from home though.
  • My blogging is more frequent, although not as much as I think it should be.

What I haven’t achieved is as much projects or geeking as I wanted, so I will be making more time this year for doing so and getting into more fun projects. :-)

So, my new years resolutions for 2012 are:

  • Complete planned minimalism, selling off items I don’t need or use, reducing to things I actually care about. I still have too much cruft IMHO, but it’s hard to keep it low.
  • Getting fitter – I need to lose weight and want to level up my fitness to potentially tackle the Te Araroa walk in late 2012 or early 2013, and with the move to Auckland I’m not getting the level of exercise I used to have in Wellington.
  • Reduce amount of crap in my diet – reducing alcohol and snack foods will be the major difference, if I don’t buy it, I won’t eat it….
  • Get out and explore the Auckland region!
  • Get back up the mountain for some more snowboarding! I loved my first trip and want to do much more!
  • Blog more, social network less. Blogs take longer, but are much more meaningful posts and I waste far too much time on social networks. Target is at least one decent blog post per week.
  • Improve my saving situation and consider investment options for my savings.
  • Move off proprietary networks as much as possible – mostly twitter at this stage, but also interested in options to replace google search, maps, etc.
  • Complete many mostly done projects and release publicly! Lots of cool things I’d like to get out there. :-)
  • Self enhance – work on improving my computing work environment with better tools – I’m planning to try one new tool every week and then pick the best ones for my needs.
  • Learn new computing languages: Python, C++ and Java. I know bits of the above, enough to get by, but I want to be fluent in them.
  • Learn a new IRL language. I want something I can use and then go overseas and live there for a while, so thinking maybe something like French or German, both places I’d like to go. Maybe German, then I can sing along to Rammstein songs better ;-) I think the key bit will be having someone to practice with, ideally a native speaker somewhere.
  • Decide what my long term plans will be – Auckland, Wellington, or overseas. It’s a little more complicated now that I’m not as unattached as I once was.

Xmas Break

I’m now back in Auckland after a week and a bit of tripping down to see Lisa’s family in Hawkes Bay and mine in Wellington, as well as catching up with friends along the way.

In some ways it was nice to have the break, but (scarily) happy to be back in Auckland to spend my second week off just doing geeky things – planning to build Android from source, upgrade laptop and complete other fun tasks. :-)

Some pictures from the trip:

Raaaage at slow campervans not pulling over on the otherwise quite fun Taupo-Napier route.

Looking over the boats from one of the many drinking establishments on West Quay in Napier.

Neat sea wall formation in Napier.

OMG I got to pet a bunny! It's like a cat, for vegetarians!

Hers & his laptops snuggling ;-)

Best part of Hastings: an escape route!

Due to the Manawatu Gorge being blocked by slips, ended up on the Saddle Rd bypass track, right by the wind farms :-D

OMG it's geekflat! I could even still pickup the wifi (just) from the AP I setup there. (seen from the Mercue on Willis Hotel)

 

Delicious iced coffee from Maranui Cafe in Lyall Bay Wellington. Kinda overloaded with tasty bits ontop....

Wellington was a little wet..... for the 3 days we were there, it rained almost constantly :-(

Looked into the towering boxes of mine at my parents place..... decided the 500+ ethernet cables could just stay there for now...

Noooooo, how can you be so cruel lift??? There's *six* stories to go up :-/

All the people without death wishes pulled over at the road side thanks to huge thunderstorm right above flooding the road and reducing visibility. Could actually feel the thunder shaking the car!!

Xmas & New Years Plans

Time is rushing by in Auckland and already making xmas and new years plans. I’ll be heading  down to Hawke’s Bay for the Xmas period with Lisa’s family and then down to Wellington for a few days to see my family and catch up with a few friends.

Holiday plans are:

  • 24 Dec: Drive from Auckland to Hawke’s Bay
  • 25 Dec: Hawke’s Bay [Plans: Family Stuff]
  • 26 Dec: Hawke’s Bay [Plans: None]
  • 27 Dec: Hawke’s Bay [Plans: Family Stuff]
  • 28 Dec: Hawke’s Bay [Plans: None]
  • 29 Dec: Drive from Hawke’s Bay to Wellington. [Plans: Catch up with family]
  • 30 Dec: Wellington [Plans: catch up with friends]
  • 31 Dec: Wellington [Plans: catch up with friends]
  • 1st Jan: Drive back from Wellington to Auckland

If you’re in Wellington or Hawke’s Bay on any of those dates, let me know, always keen for drinks and a catch up. :-)

Other than that, I will have from 1st – 8th Jan (inclusive) in Auckland on leave, so if you’re passing through Auckland, come say hi. I expect to be sitting around coding and upgrading my laptop’s Linux build most of the time. Possibly from in the pool, considering I can barely handle the Auckland heat already. :-/

Process Monitor for Windows

From time to time I’m dragged into the murky world of Microsoft to debug a problem a customer is having, or to setup the occasional small business server.

My Windows skills are passable, but certainly nowhere near as good as my Linux capabilities – simple stuff like doing an strace on linux becomes a “ummmmm how?” on Windows.

Recently had an issue with an old Windows application refusing to launch on Windows Server 2008 due to a missing DLL file – found a very handy application published by Microsoft that works on XP SP2+ called “Process Monitor”.

It shows all calls being made by programs including what DLLs are being called and what registry values are being queried, along with the response from all these calls.

In my case, I was able to see what DLL file the problem was requesting and seeing that the OS was returning a NOT FOUND response – installed the require library, and fixed. :-)

Download from: http://technet.microsoft.com/en-us/sysinternals/bb896645

 

Walk Te Araroa?

I’m not a particularly outdoorsy kind of person, but there have been times when I’ve ventured out into the sun and been rewarded with painfully blistering skin instead of my delightful geek tan (as witnessed with my recent snowboarding efforts).

One thing that’s come to mind in past, is exploring more of New Zealand on foot, rather than it racing by at 100km/h – so I pondered the idea of whether there are proper walking tracks from the top to bottom of New Zealand.

Turns out there is – the Ta Araroa track goes from the top of the North Island to the bottom of the South Island and is almost complete with a completion date of end of 2011. :-)

It would be a pretty intense walk, according to google maps it would take 18 days if I walked 24×7, so in reality it would take a good 2 months minimum factoring in sleep time and the fact that I’m sure I’d want to stop over at some paces overnight.

And of course, I’d need to increase my fitness, upskill with learning how to camp and tramp properly, but there’s  certain appeal to doing it and to having that time just exploring NZ in a different way to what most experience.

Something to consider for late 2012 I think. :-)

I’m not the first person to consider this either, in the past few days have come across a guy who’s currently doing the walk, someone currently making plans, and a guy who biked the length of NZ.

Of course, I’d probably do it typical geek style, expect me to lug a laptop along with me and blog and tweet the whole way, but part of me is keen to have that time away from regular life, away from apartments, junk, work, people, debts and having those long periods of isolation to just ponder life.

And maybe I’ll be able to drag some friends and maybe Lisa into doing it to. Although I know she thinks I’m nuts for even thinking about this idea. ;-)

OMG Auckland

Goodbye Wellington (pic of my plane by @scottjpalmer at Wellington Airport)

So I’m now offically an Aucklander, after my trip down to Kiwicon, I can’t deny it any more – I’ve moved in, my server is back online and I’m slowly setting into regular patterns of work/social life with Lisa.

It’s been pretty weird so far, having days that are up (yay, omg new city, new exploring, new friends) and days that are down where I miss some of my great Wellington friends, family and reminiscing on past experiences.

I grew up in Wellington and whilst somewhat well traveled (in the AU/NZ region anyway) this is the first time I’ve actually truly lived away from home and it’s a big change – from living in the city where I know every other street, had visited friends/hookups in every other suburb, started a business, met some amazing people, obtained nemisies, to living in a strange city which I’m unsure about whether or not I’ll like.

To be fair to Auckland, most of my past experiences have been as a visitor here for business, so trips have basicly involved Airport – Traffic – Office – Traffic – Airport which isn’t really giving the city fair trial, so I’m prepared to go into this with an open mind, to look at what’s good in Auckland rather than focusing on the bad and to get out there to meet new people and do new things.

I’ll be in Auckland for certain until late June 2012, when Lisa and I will then be re-evaluating what we want to do – I’m still very strong on doing a number of years overseas, it’s more a case of deciding when the best time to do it is. And I expect to return to NZ eventually, particularly if I do decide to raise a family long term, I think I’d want to do that in Wellington.

Going to miss my family in particular – not being able to pop over whenever I feel like to chat and hang with my brother is certainly a downside to the move.

And I’ll also miss my very good friend Tom, who has been my long suffering flatmate, for a while employee and most importantly friend. Considering we’ve probably been flatting together for the past 4+ years, it’s kind of like leaving a partner or brother. :-/

It’s also a time of great change as I close a chapter on my single years – I now have a flat with just Lisa and I – no longer firing objects or having sword fights in a kitchen with rackmount servers all over the floor with my friends; dragging some random people home; heading to strange addresses late at night to meet snuggle buddies; collecting blade servers from weird places to setup in my startup company office and heaving it through the office alone or boozy twitter sessions in Wellington’s great bars.

Wellington is both full of great and bad memories… I love it more than anything and will be back, but in a way, it’s nice having this chapter away from it for a while.

Right now, starting a new chapter in life with Lisa <3 And it’s scary, but it’s good. :-)

Pretty Wellington (from #geekflat apartment I lived in for the past ~3 years)

Packing Nightmare.... I found I have 3 car loads of stuff and only had one car :-/

Found my first Linux whilst tidying up :-D

Squeezing everything into the boot of my car before driving up to Auckland. Managed to fit my big server in underneath everything and used clothes in all the odd gaps... even stuff under passenger seat. I need a bigger car. :-/

Goodbye curry with @thatjohn (John) and @macropiper (long suffering Tom)

Final night in Wellington, the fireworks farewell was a nice touch ;-)

Server online at new apartment. Pretty much #1 priority. Far cry from the days of the twin 42U racks tho!

Kiwicon in Wellington

Having only just moved to Auckland, I’ve only just started reassembling my computer setup and unpacking boxes, yet already back down to Wellington attend the 5th Kiwicon Hacking & Security conference.

Looking forwards to it, it’s always an interesting conference and I get to catch up with the guys at geekflat and my other Wellington mates for curry, geeking and other awesomeness.

I’m flying down Friday morning, spending the day working from the Wellington office, then catching up with friends in the evening.

Saturday will be conference and I’ll try to get along to the tweetup that’s taking place. Sunday will also be conference and probably a visit to parents place before returning to Auckland in the evening.

So it’s a pretty busy visit, totally need more time – missing Wellington and my mates heaps after the move. :-(