The latest Neilson report shows that mobile app downloads continue to steeply rise. The average number of apps on a phone is now 41 up from 32 last year.

The most interesting finding to me is that the top 50 apps that are downloaded (Angry Birds, etc) account for 58% of all app usage. That leaves 42% of all time spent on mobile apps to the Long Tail apps. This means a significant amount of user time is available to niche mobile applications. So if you have a great idea for a mobile app, it is likely you can find a market that will download and use your app (provided the idea really IS good and the execution and marketing is good as well).

So dream big!

I have used SearchEngineNews.com for Search Engine Optimization (SEO) help in the past. They offer very solid SEO advice and good tools. They occasionally offer classes. I have never personally taken a class, but I would trust that they are very thorough and helpful.

I saw this yesterday and thought many of you might be interested.

Note: this is NOT a commission based link.

Big Data’s first IPO has hit NASDAQ and I’m sure there will be plenty more to come. Why’s this a big deal? Big Data, defined as crunching numbers in the Giga, Tera and Petabyte levels, is what is taking business intelligence to the next level. Just for comparison, your PC or Mac probably has about 100 to 200 GigaBytes of storage capacity. Imagine 1000 of your computers filled to maximum capacity with data about your customers (not just music and family pictures). That would be 1 TeraByte. Now imagine 10,000 of your computers with your customer’s data. That would be 1 PetaByte. Now imagine what you could do with that data. That’s Big Data – pretty neat, huh?

Splunk is the Big Data company that IPO’d last Friday. Splunk has a system that most businesses can import their data into in order to crunch into meaningful numbers that can impact business decisions. The downloadable software (for a fee) is used by names as big and varied as Visa, LinkedIn, Aetna, Geico, Department of Defense (hm, wonder how’s that working), Raytheon, Bridgestone Tires, and many more.

Most of you reading this do not warrant a Splunk solution, but the Amazon Web Services can certainly provide the Big Data processing to turn your data into intelligence.

Etsy.com known for their ecommerce website for selling merchandise is also known to programmers as a top tier coding company.

They recently announced a grant program for supporting women in technology. Check out their hacker grant page and if you’re a woman into technology and want to be an amazing coder, apply for the program.

While at lunch yesterday with 2 women coders, 1 male coder and myself (male), we talked about grants and programs like this for women and people of minority background. The guy and one of the women argued that any kind of bias like this is inherently unfair and is bad for everyone.

I think there are 2 points to consider on this topic, one of which gets all the play and the other of which gets no popular publicity. The first point is the obvious one that everyone focuses on: fairness. Is life fair and are “social engineering” programs unfair? And of course the answer is: its not fair. Any grown up knows the rule: life is not fair. Its just not built to be fair – or perhaps it is built that way but at a much deeper level of fairness then we humans care to look at. Go for a hike in the woods and see the growth and decay. There is no apparent fair, moral order to what lives and dies.

But what are we going to do, whine about it? Someone is born into a wealthy family and gets the best education money can buy. Is that fair? Of course not, but its the way it is. So should we not have conscious programs built to give some folks an entry point to a better lifestyle? That’s the question most discussions get stuck on: fairness. I think we  can argue all day on that.

But I think there is a more pertinent, practical point that we all miss: quality. Is a technology department better off with a vast majority of well off white males? Or can the IT department actually benefit from having different genders and ethnic and social backgrounds? Technology is about ways of thinking about problems. Problems in software tend to have multiple good solutions and are not of the 2 + 2 = 4 type. My personal experience, says yes, different intelligent, experienced individuals with different social, gender, ethnic, geographic backgrounds create for a more balanced team. The quality is higher. Yet we live in a society that is very quantitative focused. SAT and GPA are often used to determine school placement, but hardly describe what a person can bring to the table.

What’s your feeling on this topic? Does quality matter? Should there be programs and grants, private and public, that support gender and minority background candidates?

Hive is a data warehousing software system that sits on top of Hadoop and facilitates querying by users not literate in MapReduce. Hive was originally developed by Facebook and now enjoys support by many companies after Facebook donated the software to the Apache Foundation.

Hadoop by itself has no notion of data types or formats, it is essentially just a storage mechanism using HDFS (Hadoop File System). Hadoop relies on MapReduce developers to know about the contents of the HDFS and be able to write intelligent MR programs. Hive supports a data model using standard tables with rows and columns. Importantly it supports partitioning your table on a particular dimension. For example you could store customer purchases and partition on month of purchase. This allows you to later create queries on an organized data model.

Hive is broken down into several components including:

• Metastore for storing metadata about the data model
• Query engine for executing sql like queries
• Client command-line
• Interfaces for creating common data types as well as custom data types

Hive is an interesting project because it allows you to expose the best parts of hadoop, namely MapReduce and data storage, to end users who may have no idea about map reduce or no interest in writing MR programs.

Today the Starwood SPG iPhone app version 3.0 launched. This was a huge improvement over past versions and is the best, most feature rich and easy to use mobile app for hotels. I have been working as a developer on this project since July of last year and its always nice to see efforts finally get out into the real world live production. The feedback has been fantastic.

The work I was involved in was the API to expose to mobile applications, the various functionality from property content to rate searches to booking requests on Starwood’s SOA platform. We worked through various requirements, design and development cycles and a final performance enhancement phase. All exciting stuff. The team at Starwood is top notch and it has been a pleasure working along side of them.

So if you have an iPhone, check it out! Android is on its way…

Starwood SPG iPhone App

ZooKeeper synchronizes machines across various processes within large clusters. It is one of the cloud based technologies that is gaining notable acceptance. The biggest public adopters include Rackspace, Yahoo and Zynga.

Imagine maintaining configuration information that each machine in a 1000 node cluster must be aware of and sensitive to any changes. ZooKeeper is specifically for solving these types of problems.

Often building applications within a cloud infrastructure can focus on the functionality and business requirements and short change the small details that large processes involve. For example, imagine a hotel company that had a large cluster designed to process business intelligence information about its customers and hotel stays. Within this BI application, perhaps there is a Booking Demographics Report process that depends on a sub process that runs a statistical algorithm on customer and booking information. Perhaps this process is close to real time so is constantly processing new data. The Booking Demographics Report though would have to wait or block on the sub process. If each of these processes involved the coordination of multiple machines, ZooKeeper would maintain that organization and make seamless integration of a hierarchy of processes possible.

Other uses of ZooKeeper include locks, barriers, leader election, configuration management, queues and more.

Cloudera has a good overview article on ZooKeeper. And Dzone has a good primer with overview diagram and Getting Started.

A few lessons learned on scale:

If you are working on a technology project (or any business project for that matter), know your scale! How much does each customer cost (both time and money) per purchase and how many of those do you need to make a profit. Then figure out if the number of users you need to make a profit is even possible. This seems simple, but Google just made a half billion mistake on a technology that does not scale. So if can happen to them, it can happen to you.

Most web and software businesses should be able to handle thousands of visitors without an enormous investment. Getting above that number does require a significant investment of technology time and money, but is still well within reach using CDNs and services like Amazon’s CloudFront and EC2. Know what your costs are if you want to scale to a larger customer base or what the costs are if your customer base shrinks.

Don’t forget about customer service, maintenance, book keeping, tax preparation, legal hiccups and all of the other miscellaneous items that will consume your time and money. Your idea for a web business might seem like an amazing idea, but make sure you enjoy your business enough that the time and financial investment in your web or mobile app has intrinsic value to you as a person, i.e. if your business flops make sure you will not regret having opened it! Don’t be afraid to charge enough for your service or product to meet your scale level. If you don’t think you can charge the amount you calculated that you need to operate comfortably… well don’t start the business or find a different approach or customer base for your services/products.

Competing on price is tempting because it seems like an easy sell. Competing on experience, quality or uniqueness is hard because it requires self-confidence and dedication to be persistent to learn how to do things better. But low price rarely scales efficiently without massive investments. Walmarts and Targets don’t just happen – they require a lot of money to build that level of scale.

To scale on a dimension other than price, make sure you focus on customers who can afford your services. Find out where they can be found; they will most likely not be in the same places as the poorer quality customers and if they are your approach to them will likely be much different. On that note: beware the hustle. If you have to hustle and bustle to find your customers, you’re likely competing on prices. If you enjoy this, have at it. If not, find out how to be more unique. Avoiding hustle doesn’t mean not working hard, it just means not being frantic. Also beware negotiating your vendors into obscurity. If you’re negotiating the best price on the planet, your vendors probably are not very good or experienced and/or they will resent you. But also beware of paying too much. If you cannot afford the best vendor in town, find a good but cheaper alternative and adjust your expectations of the end product (don’t be afraid of DIY).

Finally, make sure you enjoy your atomic level of scale or figure out what has to change to get to a different level to make you more profitable/happier.

Today the primary sponsor of the much maligned SOPA and PIPA bills abandoned their efforts. Yesterday sites like wikipedia went dark in oppositions to the bills. Google, Facebook and many major technology companies protested the bill as well as many of the NY Tech Meetup. The main point of contention was that the bills were extremely vague and many felt it would give undue power of regulation and control over an industry that needs openness and lack of restrictions like it needs oxygen.

Important Note: I am not a deregulationist. Lack of control of the internet does not hurt anyone. Lack of regulation of financial firms and markets, airlines, mining companies, etc can certainly hurt people.

Why does openness and lack of restrictions help technology?

Technology is based on ideas and creativity much the same as art and music. When art and music are confined to a particular style and modality, there is a limit to expression. For example the Dark Ages did not see radical development of creativity and styles. New art and music took centuries to develop. The Age of Enlightenment saw many new composers and artists and a divergence of art styles as different as Impressionism, Cubism and Surrealism.

Likewise, software and technology are based on creating from ideas and on the people who have gone before you. For example, creating a website with a user login is extremely easy now. Just install WordPress and bang, you have a new website with login capabilities. 10 years ago, it would have taken 6 weeks of development from an expensive programmer to create a website with a login system. So by using freely available software you can focus on business logic and what matters rather than re-inventing the wheel.

So instead of focusing on restricting the internet, we should be focusing on something far more important: education. Countries like India, China, Russia, and the Ukraine get it. They invest a ton of money in technology infrastructure and education. That is why so many tech jobs in the US are populated by H1 visa holders and not American citizens. Instead of wringing our hands about piracy, intellectual property and the like, we need to be jumping to the forefront and creating new technologies and businesses. And the foundation of those possibilities are educational opportunities and encouragement.

Finally I saw some heartening news. Mayor of NYC Michael Bloomberg is going to learn how to code! And if the Mayor can do it, so can you! CodeAcademy signed up almost 100K people for a New Years resolution of learning to code. As a society we should really focus how we educate kids and what we focus on. Last I checked, school is not cool. That is a reflection on the education process not on the kids. Our educational system is extremely outdated. Its still based on a 1950s mass production model where a single person (the teacher) had the answers and imparted them to as many people as possible (the students). But now the answers are everywhere and can be accessed at any time. Perhaps we can think of a better method of education.

Our economy is not going to survive off of manufacturing and agriculture like it did once. It has to be off of high intellect jobs. We do need more technology professionals. But tech skills can be brought to virtually any profession from law to medicine to art and music. And now you can learn at Code Academy as well.

Have some fun and check out Code Academy – it might show you that coding is not so mysterious and you can do it too!

For those of you that don’t know Hadoop yet, it is a distributed, parallel processing framework. What’s that mean? Its the same type of software that Google uses to process its web crawling to offer meaningful search results. The amount of data being produced has reached critical mass and needs a new methodology to process meaning out of it. Hadoop provides that and is the next biggest thing, supplanting the ubiquitous “Cloud Computing”. The number of developers with Hadoop knowledge and the number of new ventures based on Big Data will drastically rise.

Google and Amazon both have large offices in NYC. Even upstate is getting in on the tech deal with the 4.4 billion dollar investments. Its worth mentioning that Hadoop World was in NYC this year…

Evernote is my favorite business of 2011. It’s a simple service that fills a great personal and business need, organization of data or notes. It has many integration partners. For example, Shoeboxed.com allows you to scan in business cards and receipts and have them imported automatically to Evernote. That gives the customer searchable access to data in a very easy format.