The New Garage of Analytics

Analytical consulting requires more than a computer, some data, and some coding. It requires effective organization, proper communication, innovative methods, and what I call “meaning motivation” – the innate desire for insight, the curiosity for understanding, an appetite for intellectual exercise.

That having been said, I’d like to organize a new garage of analytics. These are some of the toolboxes necessary for solving complex problems in a ever complex world. These are the toolboxes we access for effective organization, proper communication, method innovation, and meaning motivation.

And in terms of process, math aligns us to break complex problems into those that are much simpler to understand and digest while still seeing the big picture and the end goal. This very much follows that concept. No problem will be exactly like another, but if we follow a similar digestion process to attack complex problems, well, at least our stomach will not be upset.

The Organizational Toolbox

The Sociological Toolbox

The Analytical Toolbox

The Visualization Toolbox

The Balance Toolbox – A step away is sometimes the best step forward.

  • Artistic Expression
  • Fitness/Exercise
  • Learning
  • Reading
  • Sleeping
  • Talking
  • Traveling
  • Writing

The Intersection Of Expertise

As I begin my job search (25 applications in 2 days so far!) I keep asking myself how to describe what I’m looking for in a job and in what realm do I wish to work? There is no specific job title that describes my experience and education (e.g. “doctor” or “software engineer”) and there is no one department in which I’ve worked or wish to work (e.g. “Operations” or “Logistics” ). Yes, I have an academic background in mathematics & statistics yet it’s difficult to communicate why I have that academic background. I do not necessarily want to become a statistician but rather I fully understand the quantitative nature of things and the power that numbers, math, and quantitative methods have in all aspects of business, government, and life.

So where does this leave me? Well, unemployed and confused, for one. But that’s okay with me. I’m confident that with my capabilities, no matter how hard they may be to communicate in an application or even to a recruiter, I’ll find the position that leverages my abilities and motivation.

That being said, I think I’m at least getting close to describing where I stand, and in real-world terms. It’s at an intersection of sorts – between quantitative methods, scientific and technological realms, and the human element. It’s interdisciplinary – can fit within any group or team or stand alone as an independent researcher or consultant. It’s also dynamic – parallels the speed with which modern business operates and the flexibility required to optimally support the needs and requirements of many types of personnel.

I’ve used a similar image a few times, in posts on knowledge innovation and math in 2010 and beyond. Here I’ve intersected three main topics while including some of my strengths in the middle. Now if I could only match those to a job title…

At what intersection do you operate?

The Ultimate Personal Dashboard

With some great technological advancements in the past decade, why am I still organizing my life in bookmarks and spreadsheets?

The next great technology needs to get more personal. We need to drop the rectangular web browser and think in higher dimensions. Let’s say iGoogle meets Macbook Dashboard meets a much better version of the new Yahoo! homepage meets the iPhone application platform. I’m talking about a secure, personal, customizable dashboard/portal through which one can live. It’s where I’ll track my information, both from the web and my mind to better organize and optimize my life. It’s where I’ll see and interact with my personal data in a comprehensively insightful yet very organized environment.

Right now, how do I track my information? Some is on the web, some is on my hard drive, and some is on paper. I have over 200 username and password combinations I use to login to various sites. I’ve got at least 250 bookmarks in 15 top-level categories. I’ve got spreadsheets that summarize my finances and visuals I’ve created to try and learn about them. For now, when I need to know something, I find the appropriate link, look up my account credentials (if not stored), and then investigate. But for those in a similar place in life, are my personal needs really that different?

If I list out all the things I do online, all the things I read online, all the information I organize on my computer, all the personal resources I access online, and all the questions I might have about myself, can I begin to minimize some clutter? Can I get Google Reader, Macbook Dashboard, iGoogle widgets, social network widgets, and personal spreadsheets in a secure, organized interface? Please?

Base

  • Accounts – Search logins by account, email, username, password, notes, date added, date updated
  • Address Book – Contact Info, birthdays, anniversaries
  • Links – Yahoo!, Google, GMail, CNN, Wolfram|Alpha
  • System Stats – Files/Folders, latest backup, storage space
  • Weather – Today’s weather, 7-day forecast, full interactive radar/satellite map

Financial

  • Bills – Due dates, billing cycles, average costs due
  • Energy Monitor – Monitor your home utilities, set “green” goals
  • Finances Monitor – Monitor stocks, IRAs, retirement, savings, checking, credit card
  • PayPal – Request/receive payments, see pending invoices
  • Subscription Management – Expected issues, renewal dates,

News/Events

  • Coming Soon – Movie releases, Tickets on sale, Upcoming concerts (Thrillist, Ticketmaster, Fandango)
  • Google Reader Tracker – Total unread, shared items, etc.
  • Local – Weekend Events (Going Out Guide, Eventful, etc.), Breaking News
  • News – CNN News Pulse
  • Sports – Scores/News

Social Media/Networking

  • Brand Monitor – See sentiment for desired keywords/terms
  • Discussion Board Monitor – Track your posts and comments, desired forums
  • Hot Topics – See trend topics and most searched items
  • Notifications – Facebook, LinkedIn, Twitter
  • Social Timeline – LinkedIn Updates, Twitter Lists, Status Updates
  • Web Analytics – Twitter Stats, Google Analytics

Entertainment

  • Movies – Times, upcoming releases, IMDB search, RottonTomatoes rankings
  • Music – Playlists, connect with Grooveshark albums, iTunes Radio, etc.
  • Photos – Flickr/Picasa portlet
  • Sports – Fantasy team tracker, favorites scoreboard, breaking news
  • TV – Guide, schedule of favorites, DVR control

Health

Lists

  • Map – Where I’ve Been, Where I want to go
  • Reading List – What I’ve Read, What I’m Reading, Connect to Amazon
  • Recipes – Saved links, suggested items, BigOven link
  • Shopping – Grocery (connect with PeaPod), Retail deals/coupons
  • Tasks / To-Do
  • Watch Lists – eBay Auction, StubHub
  • Wish List – Amazon, iTunes, Retail Stores

Utilities

  • Calculator
  • Currency Conversion
  • Dictionary/Thesaurus (Wordnik)
  • Flight Tracker
  • Job Tracker – Monster, USAJobs, search agents
  • Maps – My placemarks, directions, search locations
  • Shipment Tracker – UPS, USPS, FedEx, etc.
  • Translator

This is just a list of things I do, need, have, and want. Obviously there are a lot more to be added. It’s important to note that all of these widgets/portlets have a similar foundation that parallel the major dimensions (in light blue) I spoke about in my earlier post on the boundaries of the human condition:

Accounts – List of all companies/organizations. Information is tagged by the company and all info can be found with regards to that account, when needed.
Dates/Time – Many things are calendar-based and should be aggregated to a personal, customizable calendar view
People – Address Book is a foundational database. People can be searched throughout for linkages and notes.
Places – With the current technological trend, many needs are location-based (including news and tweets). Personal organization dashboards should leverage geo-tagging for contextualization of information to the user.

It’s also important to note that most people want information in 3 forms: a quick preview, an expanded summary, and an interactive tool. This follows closely with a recent social trend – high variability in the speed with which we move. Sometimes we want a snapshot of our current personal information because that’s all that we have – a few seconds of time. At other times, we may have a few minutes of free time, most likely coupled with a defined question or purpose:

“How much do I have in my checking account?”
“What will the weather be like this weekend?”
“Need to transfer rent money to roommate.”
“Did my package arrive safely?”
“Who has a birthday in the next month?”
“What are the hot news items of the day?”
“I want to buy a book from my Amazon wish list.”
“To which country should I travel next summer?”

And finally, this cannot be overwhelming. It needs to be there when you need it but not short circuit your mood if you don’t check it for three weeks. All charts and graphics need to be simple and interactive and customizable, but also intelligent in design to attract the most novice of digital users.

So what will the next decade bring us? Will personal desktop technology be able to fully leverage the vast amounts of data we have online, on our computers, and in our heads? Will the world become more stat-conscious, and learn to take insight from the graphical display of life data? Will the desire for a less-click lifestyle drive better personal dashboards for secure, centralized organization? I hope so.

Future Search Requires Human Intervention

The business of organizing information is a tricky one in that it’s a moving target. It’s hard to tell what will be the next hot topic or consumer need, and the contribution of digital information is not necessarily bound by the structure and rules of those needing that information. Second, the mechanisms by which we wish to access and use information are constantly changing. The world in 2050 won’t be PowerPoint presentations, heat maps and csv exports, just as 1950’s world wasn’t all emails, tweets, and search engines. Lastly, with the emergence and growth of social media and networking, information contribution has dramatically increased. Without providing the statistics, it’s well known that more people are online, and more people are hitting a “Submit”, “Post”, “Send”, or “Tweet” button than ever before. As a result, there are more pieces of information, more individual topics, more ways to categorize information, more uses for information, more ways to see information, and more unknowns about that information than ever before.

That being said, it’s important to not be unnerved by this. Collectively, we are smart enough to guide search technology in the right direction, or at least put it in the right place so that its path is optimized for future generations. And collectively, we are smart enough to ensure that the ways we use information will publicly provide guidance to the development of a optimal suite of available tools for effective visualization and communication. That being said, by focusing on the relative near-term, emerging trends can be detected, understood, and leveraged for advancement in the business of organizing information. Gaining an edge on future trends brings tremendous value and exactly that – edge. Among other methods, the detection of such trends requires quantitative analysis mixed with psychology, history, and good ol’ intuition.

In terms of finding information (and more importantly, relevant information), most people think simple search. Or maybe more specifically, they think Boolean search with some available advanced search options. It’s Google, Bing, Wolfram|Alpha, or some other engine – typing in some logical query, occasionally setting some supporting filters, scrolling through the first ten results, and settling on what looks best by some meta data. This certainly works today, and these search engines may be sufficiently wrapping their arms around the influx of new and changing information. But soon technology reaches a limit. The responsibility must shift back to the human. As the informational requirements become more complex and the underlying data become more specialized, relevancy must become more human-driven. We can’t rely completely on technology to provide us with answers.

This notion is two-fold:

  1. Future queries needs more human input. This means “advanced search” needs to become part of “normal search”. Spending 3 seconds to check a box could save 3 minutes in scanning results. You can obviously set defaults for your regular search, but I’d be interested to see stats on the use of type of search vs time spent scanning results. The distance between the two will only grow with the ever-accelerating growth of internet content.
  2. Future content needs more human organization. About.com, Wikipedia.org, and howstuffworks.com are great examples of this. They are sites that show up in search results that organize information. Wired Magazine had a good article last month on About.com as an “answer factory” (in fact, their motto is “Guidance. Not Guesswork” – I love that). These sites organize thought and mimic the human mind looking for an answer to a question, not a query. Websites and blogs need to organize information about information, and as a result we can increase transferable knowledge and decrease search time.

It’s obvious but important to note that people collectively drive relevancy through thought. Trend topics in Twitter, News Feed on Facebook, Google Hot Trends, etc. are driven by fingers on a keyboard, which are driven by synapses in the brain, which are driven by actions and observations, which are driven by circumstance – and maybe even fate. Technology should not drift too far away or we’ll lose access to the very knowledge we create.

shapes and squiggles

Armed with a pencil and paper, you can simplify about 99% of the world’s problems.

Despite a couple decades of extra-substantial technological growth, there are two things that can never be replaced: the pencil and paper. For the toughest analytical challenges, only so much can be done computationally to simply and digest such problems. For these challenges, the solutions should start with a pencil and paper.

The first step in breaking down a problem is the conversion of the problem from the brain’s three dimensional space to a the two dimensional space of paper. In mathematics, there are several examples of such similar breakdowns: matrix decompositions, polynomial factorizations, projections, transforms, etc. The breakdown is necessary to see things in a new light, a simplified light, and a light that otherwise may not have been turned on.

Step 1. Grab a pad of paper. Do not put boundaries on where you can write and draw.
Step 2. Grab a pencil. Sharpen it and keep the pencil sharpener close.

So now that we have pencil and paper in hand, what do we draw? Well here’s my point. There is a geometric toolbox that provides a valuable framework for the problem solving environment. These are the shapes and squiggles.

1. Matrices

Two-by-two matrices are especially valuable for initial sorting of qualitative data. Assign a binary variable to each axis, name the cells, and define the relationships. Categorizing concepts and attacking each cell independently can help find hidden relationships and provide insight for subsequent analyses. See my previous post on matrix power for more on matrices.

2. Graphs

For more quantitative and scaled concepts, draw a set of axes to start. Visualize relationships between variables by drawing lines or curves and then attack each extremum and graphical sector. Plot knowns and/or hypotheticals on the graph and decipher the meaning of specific coordinates. Jessica Hagy’s blog ‘Indexed’ is a good example of translating mind to graph.

3. Lists and Mind Maps
The proper organization of information is often the most valuable visual tool in solving complex problems. Of course there are technologies to assist in the visualization and organization of information (mind maps, spreadsheets, etc.) but it’s important to use pencil and paper as the primary stepping stone to using some software/web app. Check out a mind map on different mind mapping software and a post on five great uses of mind maps.

4. Circles

Circles have shape and have a shape that is unique. They overlap well, fill space comfortably, and are easy for the human mind to spatially interpret. Eulerian circles (or Venn diagrams) are the simple example of circles put to use on paper for analytical means. There are several other adaptations of circles for comparative reasoning, such as with GL Hoffman’s “gruzzles”.

5. Doodling

The mind works in mysterious ways. Drawing without bounds can release otherwise inexpressible thought. There’s the somewhat structured doodling such as with UI mock-ups, schemas, and decision trees, and very unstructured doodling that might look like an impossible maze of dots and lines. The importance lies in the fact that your brain knows most about the problem, and the pencil is driven by the brain. Any new representation put forth on paper, by your brain, is a new representation of that problem not previously seen. In other words, “doodling allows the unconscious to render in symbolic expression”.

The shapes and squiggles live on. And the shapes and squiggles will always live on because they are the simplest yet most powerful functional tools our mind can use to express our conscious, subconscious, and unconscious thoughts.

matrix power

How much of your life can you fit into rows in columns? Well, enough of it for you to cherish the matrix as a valuable organizational and analytical tool.

Spreadsheets, tables, and matrices are used in every aspect of life. We track finances, monitor tasks, plan our future, and analyze potential relationships with rows and columns. And we are surrounded by this information as individuals, as part of small social groups, and as part of large organizations such as classes, companies, or governments.

More simply, matrices and tables give a new structure to elements of our life that are not always so two-dimensional. From the new structure, we can glean new insights and inspire new visualization of those same elements to make best-informed decisions. To me, a matrix is a valuable analytical tool that helps organize information for insight and action.

(Note that I am using the term “matrix” to represent that much more than numerical arrays of the math world. I am including categorical mappings, tables, lists, and spreadsheets too.)

The University of Cambridge Institute for Manufacturing nicely defines the matrix as an essential decision support tool:
  

“A two by two matrix is a useful tool for initial sorting of qualitative data. The axes should be chosen so that, e.g., the data with the most desirable characteristics will fall into the upper left quadrant and the least desirable in the lower right quadrant. While groups may be unable or unwilling to assign absolute values to qualitative data, they usually find it relatively easy to come to a consensus as to which quadrant something belongs in.

Generally, the two by two matrix is a useful tool for categorising things that can be reduced to two simple variables, particularly when quantitative information is unavailable and qualitative judgments must be made.

It enables a rapid clustering (or separating) of information into four categories, which can be defined to suit the purpose of the exercise. It is particularly useful with groups as a way of visibly plotting out a common understanding or agreement of a subject.”

Authors Alex Lowy and Phil Hood describe the matrix as “the most flexible and portable weapon in the knowledge worker’s intellectual arsenal”.

What’s best about the matrix is that flexibility. Depending on need, you can get as much power out of a 2×2 matrix as you can from a 5×5 matrix. Increased dimension does not translate to increased power. The matrix is flexible and dynamic to the needs of your analysis. You control the path to discovery.

And although matrices do a nice job of pairing categorical relationships, you can also translate these pairs to numerous other visualizations to better contextualize the information at hand. Turn your row and column headers into scaled concepts, map them to some x- and y- axes, and try and fit your qualitative information to a line that describes the relationship between x and y. Is the relationship directly proportional, inversely proportional, linear, parabolic, or along some other path? What do each of these types of relationships mean for your categorical variables?

It’s important to note that there can be fuzzy lines too. Not all cells need to have values and not all relationships need any sort of defined continuity. Empty cells and undefined relationships provide insights that are just as valuable as the populated and defined ones. Lack of data is data in itself, and that’s a great thing.

In the end, the matrix is just one part of the analytical toolbox and can provide a wide range of insight for your personal and professional life. Box up your data, organize it, visualize it, and use new structure to optimize your life.

Examples

Business/Leadership: Gartner, an IT research and advisory company, has created the “Magic Quadrant” to analyze types of entities in the business world. By plotting the ability (or inability) to execute against the completeness (or incompleteness) of vision, businesses can be categorized with those sharing similar characteristics, as Leaders, Challengers, Visionaries and Niche Players. This is a useful example of turning abstract qualities into groups for targeted strategy and decision making.

Product Development/Management: For analyzing how to grow a business from the product side, one matrix shows how plotting types of markets vs types of products can help guide that growth strategy.

Math/Statistics: Type I and Type II error tables are used to describe possible errors made in a statistical decision process. This is a great example of mapping relationships between categories, naming the cells, and using the matrix to understand what each cell represents.

My Alltop – overview and feedback

My Alltop is a simple, intuitive way to manage lots of news headlines from a broad range of topics and sources. It reduces my clicks and browser windows and drives me to read more diversely. However, I am already beginning to feel overwhelmed as I’m up to about 30+ feeds in 5+ major categories. Although I’m happy with its current state, My Alltop needs a couple new features and functionality enhancements if it hopes to be one of my primary click recipients…

Click Here to Visit My Personal Alltop Page


About Alltop (from their site)

“The purpose of Alltop is to help you answer the question, “What’s happening?” in “all the topics” that interest you. You may wonder how Alltop is different from a search engine. A search engine is good to answer a question like, “How many people live in China?” However, it has a much harder time answering the question, “What’s happening in China?” That’s the kind of question that we answer.

We do this by collecting the headlines of the latest stories from the best sites and blogs that cover a topic. We group these collections — “aggregations” — into individual web pages. Then we display the five most recent headlines of the information sources as well as their first paragraph. Our topics run from adoption to zoology with photography, food, science, religion, celebrities, fashion, gaming, sports, politics, automobiles, Macintosh, and hundreds of other subjects along the way.

You can think of Alltop as the “online magazine rack” of the web. We’ve subscribed to thousands of sources to provide “aggregation without aggravation.” To be clear, Alltop pages are starting points—they are not destinations per se. Ultimately, our goal is to enhance your online reading by displaying stories from sources that you’re already visiting plus helping you discover sources that you didn’t know existed.”

Some Initial Feedback

1. Feed Management/Layout – The “Manage” window where you can drag and drop your feeds does not match up with My Alltop’s actual layout, due to a big advertisement in the top right of the My Alltop page. It’s not the easiest thing to order feeds how I would like them either. When I drag and drop a feed to a new position, it auto-shifts feeds that I had already positioned in proximal locations. It should be much simpler. How about a “My Feeds” list/column from which you can drag and drop or remove feeds from real estate boxes? Then over time I can always activate or deactivate my feeds, but still have the full set of feeds I am or have been interested in from which I can choose.

2. Feed Categorization – Let users categorize their feeds on the page. Then I can open/close categories to make the best use of my screen real estate at any time. This could then allow me to have a master feed for each category, sorted by date and listing the authoring site. I’m not saying to reproduce Google Reader, but I think there is some simple similar functionality that could and should be employed to make it easier to organize incoming information.

3. New/Old Links – Another similarity to Google Reader would be a functionality that shows new links versus those that have already been seen by the user. Google Reader marks a post or article as read as the reader scrolls past it. In My Alltop, it’d be simply useful to see which links are new on the page, whether or not they have been scrolled over for a preview or clicked on and opened in a new page. The easiest implementation would be something in conjunction with comment #2 above, where in a master feed the latest links are always at the top of the list, similar to Twitter’s homepage (but now for news articles from multiple sources).

Summary

–Google Reader certainly fits a different use case but provides a nice functional model for accessing and organizing lots of news information. So does Google Fast Flip, which was just released and allows for “fast-flipping” of articles from major news publishers, organized by topic, section, and source.

–Incremental developments are helpful for user retention and sustained engagement. With that, there are some simple features that, if implemented, would greatly improve usability and comfort for the My Alltop user.

–My Alltop should continue to become a nice complement to Google Reader and Twitter as primary, personal hubs of near real-time information. Check it out!