I had a great conversation the other night while at the Seattle Web Analytics Wednesday (#waw) with Carlos (@inflatemouse) and a dozen others.  @inflatemouse brought up the idea that an analytics provider using the cloud, increases or at least possibly increases the risk of security breach to the data.  This is, after all a valid point, but because of the inherent way web analytics works this is and is not a concern.

Web Analytics is Inherently Insecure

Web analytics data is collected with a Javascript Tag.  Omniture, Webtrends, Google, Yahoo, and all of the analytics providers use Javascript.  Javascript is a scripting language, which is not compiled, and stored in plain text in the page or an include, or passed into the URI when needed.  This plain text Javascript is all over the place, and able to be read merely by looking at it.  So the absolute first point of data collection, the Javascript tags, is 100% insecure.

The majority of data is not private.  So this insecurity isn't a huge risk or at least should not be.  If it is, you have larger issues before you even contemplate using an on-premise and cloud solution to bump up your compute and storage capabilities.  Collecting data that needs to be secure via web analytics is an absolute no.  Do NOT collect secure, private, or other important pieces of data this way.  If you have even the slightest legal breach in this context, your entire analytics provision could have this data scraped, possibly used in court in a class action suite, or in other ways even.

For the rest of this write up, I will assume that you?ve appropriately encrypted, or enabled SSL, or otherwise secured your analytics or data collection in some way.

Getting that Boost on Black Friday

eE-commerce has gotten HUGE over the last decade.  The last Black Friday sales and holiday season saw the largest e-commerce activity in history.  Omniture, Webtrends, and all of the other web analytics providers often see a ten fold increase in web traffic over this period of time.  Sometimes, for some clients, this traffic is handled flawlessly by racks and racks of computers sitting in multiple collocation facilities around the world.  However, for some clients that have exceedingly large traffic boosts, data is lost.  (yes, ALL the providers lose data, more so during these massive boosts)  The reason is simple, the machines can?t process in time or handle the incoming traffic because the extra throughput isn?t available to scale.

Enter the cloud.  The cloud has vastly more scalability, almost an infinite supply by comparison, to any of the infrastructure available to the analytics providers.  Matter of fact the cloud has more scale available than all of the analytics providers.  This is actually saying a lot, because Webtrends (and maybe some of the others) I know does an amazing job with their scalability and data collection, arguably more accurate and consistent than any of the other providers (especially since many of them just sample and "guess" at the data).

So when you extend your capabilities to the cloud for web analytics do you really increase your security vulnerability?  Most of the providers of web analytics have their own array of security measures, that I won't go into on levels of security.  However, does introducing the cloud change anything?  Does it alter the architecture so significantly as to introduce legitimate security concerns?

Immediately, from a functional point of view, assuming good architecture, intelligent system design, and good security practices are in use already, introducing the cloud should and is transparent to clients.  For the provider it should not increase legal concerns, functional concerns, or otherwise pending the aforementioned items are taken care of appropriately.  But that is just it, every single current provider has legacy architecture, various other elements that do not provide a solid basis for a migration to the cloud for that extra bump of power and storage.

So what should be done?  What if a provider wants that extra power?  Can the technical debts be paid to use the awesome promises of the cloud?  Is the security really secure enough?

Probably not.  Probably so.  But . . .

This provides a prospective opportunity for a new solution for web analytics to be provided.  It provides a great opportunity for a modern cloud based solution, that provides more than just a mere Javascript tag and insecure unencrypted data to be collected for analysis.  It provides the grand opportunity to design an architecture that could truly lead the industry into the future.  Will Webtrends, Omniture, Unica, or someone else step in to lead the analytics industry into the future?

At this point I'm not really sure, but it definitely is an interesting thought and a conversation that I have had a lot of people at #altnet meetings, cloud meetups, and with cloud architects, engineers, and others that have similar curiosities.  I await impatiently to see someone or some business take the lead!

I was looking at my analytics over the last week or so and am really annoyed.  I had a solid multiple hours of downtime over the weekend.  Of all times, while I am at ALT.NET Seattle #altnetseattle Conference for my hosting provider to go down.  I would have liked to make blog entries, but instead they got all crammed up.

Anyway, I just wanted to post my massive hit I took because of this.  I do hope that webhost4life improves this and it doesn't happen again.  My uptime has usually been remarkably good, but this was horrendous, easily breaking my 99% plus uptime promise.

Look at that dip!??!  Come on webhost4life, how about that not happening again!  73% down against regular trends, which if I had that 99% uptime I would have at least gotten the majority of the hits for that day.  Which would have been a decent percentage increase over the previous trended period since I was at a conference!

Harumph.  : (

Within Webtrends Insight one has the ability to check out the story, an automatically generated feature that gives a written overview of the activity on your site.  I really like this feature as it gives verbal perspective.  Below is a screen capture of my blog as of today for the last 7 days. Click on the image for a larger image for readability.

The other really great looking bit is the Visits Overview Report.  All Web Compliant, and I'd show you the page, but you have to have a Webtrends Analytics Account.  :P

That is it for this week.  I am out on vacation next week, so the blog will be nice and silent.

Sharepoint 2010 has some great new features.  Especially considering some of the disturbing necessities of previous Sharepoint Versions.

The first thing I noticed was that Visual Studio 2010 has default templates for building Sharepoint Projects.  This is huge compared to the monstrous and annoying effort required for development on previous versions.

Being able to build Sharepoint Projects from Visual Studio 2010 is a huge change, enabling IT & Development Groups to actually use regular Windows 7 Machines instead of needing Windows Server operating systems and licenses to build Sharepoint Applications.  To me, this was always a non-starter, and actively discouraged Sharepoint Applications in the past ? obviously most companies had not listened to the voice of reason.  But now, I have changed to actively encouraging Sharepoint 2010 Projects now.

The second thing is, it is more open and easier to build public facing sites finally.  This has been desperately needed.  I am glad that it has finally become a higher priority.  Not that I totally trust the Sharepoint 2010 Server to be fully web compliant, I have seen that they've taken leaps ahead.

The other thing I stumbled upon recently that I had not seen was the analytics capabilities of Sharepoint 2010.  The overview is that you can get traffic, search, and inventory reports based on how much traffic the site gets, who visits the sites, and other web analytics data.  Definitely check out the blog entry "Introducing Web Analytics in Sharepoint 2010" on the Enterprise Content Management (WCM) Blog.

Between the Sharepoint Analytics, Silverlight, and offerings we provide at Webtrends we now have direct extensibility points form the internal Sharepoint Data Sources, Analytics Data Stores, temp caching and other control mechanisms within Silverlight, and much more.  What does all that techno-babble mean?  It means more data, more available faster, easier access, decreased development times, more accurate reporting, more timely reporting, and the list goes on.  Pretty much ever metric about web analytics within Sharepoint 2010 has just gotten better.

The other architectural pieces I noticed is the Sharepoint Aggregation of data is now per web application in the farm, per site collection, per site, and per search service application.  This can be broken out even further with On Demand Webtrends Analytics by providing ETL between the Sharepoint and pulling the Sharepoint or extended analytics data Webtrends provides into the Business Intelligence (BI) framework within an enterprise.  At this point I can only imagine the scale and available data points for managing enterprise collaboration, social media, and other capabilities with this combination.  I'm looking forward to seeing what kinds of ROI we can see from these integrations, whatever it is, I already know it will be measurable and I suspect substantial.

For more information about Sharepoint 2010 check out the Sharepoint Blog and for more information regarding how Webtrends can get your Sharepoint Site Analytics extended check out Webtrends Professional Services!

Are there any data points, questions, thoughts, or ideas you have about connecting Sharepoint & Analytics Data?  Anything you haven't seen connected that should be?  Does your organization use Sharepoint, Webtrends Analytics, or other collaboration software?  Anyway, I am no shill, I am honestly interested in these connections outside of the fact I work for Webtrends.  If you have a minute, please let me know what your thoughts are on these technology connections.

I went to the session on Microsoft Silverlight Analytics Framework (MSAF) today while here at MIX 2010.  It was a great walk through the features, ideas, and what the end goal is.  Michael Scherotter did a great job of lining up the ideas, intentions, and the functional ideas behind the framework.

The framework is built around the Silverlight Behaviors.  If you aren't sure what behaviors are, check out these entries from Nikhilk.net Silverlight Behaviors, Silverlight 3 Drag Behavior, An Introduction to Behaviors, Triggers, and Actions, and of course the MSDN Documentation on the matter.

Some of the key features of the framework is to support out-of-browser scenarios, which works perfectly with out Webtrends DX Web Services.  Offline scenarios, which again, we have worked toward supporting at Webtrends DC Web Services via caching and other criteria as needed.  Another feature that I was really stoked about is the Microsoft Expression Blend integration that removes the need for coding, thus simplifying the addition of analytics components based on events or other actions within a Silverlight Application.  This framework also easily supports A/B Testing (again, something we do quit a bit of at Webtrends with Webtrends Optimize.

The last thing I really wanted to point out was the control support that this framework has support in already from Telerik RadControls, Smooth Streaming Media Element, and Microsoft Silverlight Media Framework Player 1.0.  These are implemented with behaviors and handlers exposed via MEF (Managed Extensibility Framework).

All in all, great second day, great analytics framework for Silverlight, and great presentation.  Until tomorrow, adieu.

Read Part 1 & Part 2 of this series if you haven't already. 

Last entry I worked up the basic origins and destinations for the data we have. Now we have to start turning this into something real, something concrete. The following processes include a whole soup of acronyms and other cryptic vocabulary. The most common thing I will probably use is the ETL acronym, which stands for Extract, Transform, and Load. This is the process of bridging the data to the various originations and destination and moving that data appropriately.

In the last entry I finalized the originations and destinations as shown below;

In my previous entry you may have noticed that I had posted SSIS with the Destination list.  Being one that corrects themselves when mistaken, I took it out, as it does not belong there.  The SSIS is our tool that will perform the ETL functionality for this project.

At this point we are finally going to get into the dirty bits of these pieces of technology, and how we need to tie them together.  I am going to attack them over the next few entries based on the order in the lists above.  The first item, is the Excel & Access 2007 customer relations listings from sales.  Here?s a description and a few shots of what this thing looks like.

The access database is setup with a very simple relational data schema.  This is shown below (click image for larger view).

You can see there are pretty standard pieces of data, in a generally normalized (3rd form for the most part) structure.  This is fine.

Next is a shot of the data entry screen for adding opportunities.  There are respective screens for customers and employees.  Everything needed for a basic customer list & tracking basic things.  Nothing too extravagant here either.  Again, all is fine.

Below is a simple report that shows the available opportunities that are open.

Another report showing the forecasts.

Below is another forecast sliced grid.

So all that seems normal enough.  But the processes are what makes things tricky.  If everyone just managed sales from the database, all would be right in the world.  The first thing that breaks this is that each sales person enters their sales during the day and other information in a spreadsheet that is not linked to the underlying database.  Someone each morning puts the previous days sales information.  This of course, breaks down data integrity.  Below is a sample sheet that is used each day.

One thing that Excel is used for, that doesn?t break reports is the lists of prospective customers to call.  As shown below.

Now that we have a break down of the Excel & Access Customer Relations Management Software, I will move onto the other pieces of technology in the next entry.  This is the data point that has the most prospective data risk, so I put it at the top of the list to cover first.  After I cover each of the systems, we will move into the architecture of the system overall.  So keep reading, more juicy bits to come.

So here is the lowdown on what I will be presenting, demoing, and showing to the attendees & anyone who is interested at Engage 2010.  If you are attending and have not signed up on the Engage Crowdvine Site, please do so.  There are some great conversations going on there.

One other thing I wanted to point out, is that you MUST attend the Ignite in New Orleans.  Ignite events are a blast!  Portland, Oregon regularly has some of the largest Ignite Events, and since we are bringing a whole lotta Portland love, and NOLA is well known for some great parties, I know the NOLA Ignite will be awesome!  So make sure to attend.

First off, Brian Gallagher and myself will be showing off some awesome technology for video tracking at the science fair break out sessions.  There will be a Tuesday Science Fair and a Wednesday Science Fair, and I will be there with Brian going over the tracking.  So if you miss us one day make sure to check it out the next day!

The next speaking session is the enterprise & business intelligence topic I have mentioned in some of my previous blog entries.  The session is titled Make Analytics Work across your Enterprise.  Our topic summarized is, The nerd of the Enterprise;  ETL, EPM, ERP, BI, CRM, and how the alphabet soup works together for real Enterprise Business Intelligence and Insight.

The final engagement I have is a topic near a dear to my work here at Webtrends.  Over a year ago a rock star team of developers; Rob D, James K, and myself started working on the Data Exchange, REST Web Services for Webtrends.  Of course now those have been in production and are now on v2.0 already!  Rob has gone on to rock the current iPhone Webtrends Application (Check out the Webtrends Blog Post Too) & James is coding away at some of the awesome infrastructure that keeps all the data flowing smoothly on the back end (to the web services, to the Insight UI, the whole thing ? James is awesome at that).  So we built these services many months ago and now I get to present them as a workshop session at Engage.  Open web services based on good clean REST Architecture Principles have always been an interest of mine and I have been and will continue to be stoked that we at Webtrends have these available for customers!  So be sure, if interested, to check out the Workshop:  The RESTful Way ? How to Use Webtrends REST Services.

Also for further reading, check out some of my past entries on REST Services;

• REST, So Many Nuances, The Self Signed SSL Cert Override
• REST Web Services Synchronous Request that points at the Webtrends Developer Network Blog Entry.
• Just a Bit More REST which has a really bad example of unit tests, which are more integration tests, running against some REST Web Services.
• File Extension ".svc" Annoyance
• RESTful .NET by Jon Flanders is probably one of the best books on the WCF based REST Services.  So if you are looking for a solid guide, this is a great place to get just that.

Alright.  Here I am working through some analytics trending of my Twitter, Facebook, www.adronbhall.com, LinkedIn, and other various sources of traffic.  Here's the frustrating part.  I don't mean a little frustrating, I am talking about a-grade I almost want to break something frustrating.  I have bit.ly, Webtrends Analytics, a database with cross-correlated results, a Twitter monitor in another window, and other little apps to track what is going on.  Where does most of this data come together?  In my head.  Why do I have to manage everything this way, it is damn frustrating!

So here is an idea, and I am just thinking outside the box here (as I always do since nobody will let me in the box anyway :p ).  How about these things get merged into a good solid single interface.  Maybe even some of the environments get rolled together in addition to the tracking analytics and reports.  How about a solid dashboard that allows drilldown into the various mediums to allow actual interaction with the data and platforms?  Imagine a dashboard, that reports the analytics for all these things but then lets you just kick in and start tweeting or posting the latest bits to Facebook.  Why have a half dozen or more applications and web sites open.

What we need here is some technology leadership.

Others have discussed this recently and are looking for some results.  Ok, some of you readers may be thinking, "do you not work for Webtrends Adron?  Go build this stuff!"  Well maybe others and I am, but I am still frustrated at not having it right now.  It seems this is something that is slowly slipping by in this industry and we need a centralizing effort for all these data points to help in actually making sense of them.

A fellow analytics & marketing professional, Rodica Buzescu writes in her blog Morphing Through Time a post titled Web Analytics ? Leaving a Lot to Be Desired.  She lays out the current analytics situation very clearly.  There is no tool that really bridges the data together easily.  Some BI Pros out there may say "oh but I have all that stuff together".  This may be true, but how much time and effort did that take?  What we need is a centralized management solution for this data.  Something to bridge the data together to make it actual, usable, and actionable information.

I have also had a number of conversations with Eric Peterson about various analytics on various platforms.  One such paltform is Twitalyzer (@twitalyzer), that he and @katzpdx have put together.  This is a prime example however, of a very useful tool, but something that lies outside of other tools.  The question is, how to integrate and what to integrate.

As an analytics advocate, practitioner, and developer myself I have one huge issue with a lot of my analytics that I use.  Webtrends Analytics & Data Mart help me bring most of those together.  However there are a number of things that are still disparate and segmented in the wrong areas.

So with this post as it stands, what do people want joined?  Where are all these data points that people want?  I think something good just may come out of this desire, but the ideas do have to come together solidly.  I have things I can?t disclose, but I know there are minds here at Webtrends thinking about bridging this gap right here and right now.

Part II of my Business/Enterprise Intelligence Series, check out part 1.

In the previous part of this ongoing series of getting the business intelligence and analytics pieces of the enterprise intelligence world figure out, I hit on the first step of the entire process. Collection of available data points. Not the desired data points, nice to haves, or might haves, but the available data points.

In this entry I will step past that and get into a little more of the technical depth of how these data points will be connected in the enterprise systems. More to the point, how they are not connected and will need to be in the enterprise.

A Quick Review, The Systems

• Point Of Sale (POS) w/ 300+ stores
• Webtrends Analytics tracking the Awe Widgets Inc Website
• Internal Accounting Software (IAS)
• In-house Built Customer Lists for Sales w/ Excel & Access

With the four systems we have a huge amount of actionable data points. So how do we connect all of these data points? That is the thorny question that comes up. Again, just like during the discovery of what data is available, we have a simple first step for this process.

Figure out the origin and destination!

That is really it. Of course, as anyone would concede, there is a lot in between. Without knowing where we are starting and where we need to end up though, finding the in between is in vain. Sure, one can make the mistake as many business intelligence projects do, and start building architecture before anyone knows or actually can use the architecture. This is a severe mistake, and I myself have literally seen millions of dollars get wasted only to have to start a project over, right in the middle of the original project. Do NOT become one of those projects, find the origin and destination!

In this case I am again, going to use my creative side and determine the operations of this company. Awe Widgets Incorporated is currently using SQL Server and has in house skills developed among their staff. That provides an easy option of moving towards SQL Server Reporting Services (SSRS) and working with SQL Server Integration Services (SSIS) for the ETL bits. Whatever other platforms are in between will be easily connected with these two tool stacks. So now I know my multiple originations, and my single point destination.

Moving on to the fun bits.

Now we have to figure out how we're going to get from our origination to our destination. This is where the trip becomes interesting. Just to make sure things stay clear, and list out what we have that we are working with.

So now we are starting to get somewhere.  We now where we are, what we have, and where we want to go.  Time to wire some things up, so stay tuned.  That will be in the next entry.  Also, if you are planning on attending Webtrends Engage in New Orleans, but sure to look up the Enterprise Intelligence session that I will be presenting with Heather Crince of Webtrends and Tony G. of Orbitz?

If you missed the previous entry in this series, check out Where is the Other Data Tracking?!  Where are My Acronyms?!?!

Where is the business intelligence?

This blog entry may instigate just a bit.  It will also be a little long for a blog entry. You've been warned.  I suppose though, if you know me & the work I do, that is not really something new.  I see something wrong, broken, or otherwise and I am likely to point it out and describe it in detail.

As I roll into 2010 coding, implementing, and rocking with Webtrends, I have noticed something lacking in the analytics industry.  I will add the clause that obviously Webtrends has people thinking about these things and actively working on this topic, but what I want to point out is a general issue.  Where is the other data, where is the existing data?

It seems, even though some company's kind of get to a certain point in connecting data points, not many really do.  The biggest reason is that most companies are just a few steps away from actually being able to do so.  The other even larger reason is, many do not realize what data should or should not be connected.

When someone starts pulling CRM (Customer Relationship Manager/Management), Analytics, POS (Point of Sale), ERP (Enterprise Resource Planning) data, and other sources into a single reporting repository we finally have real business intelligence.  Otherwise so many entities stumble through the land mines of data confusion.  I see this so much it really drives me crazy sometimes.

So how can a company or entity identify and connect these points of data?  It often starts with a ridiculously simple step.  At risk of oversimplifying things, let me just state the first step in getting out of the data confusion land mines is to first figure out your data.  Ask these things:

• What data does the business have?
• What data is currently used and available?

Do NOT ask what data you want, do NOT ask what may not be.  What you want to know first, and so many companies make this mistake, is to know what you know.  Do not, at the early stage of business intelligence information gathering start asking too many hypotheticals.  I promise the risk of failure increases exponentially for every hypothetical data point added.

Once you have identified what data is available, start figuring out how the data is related.  Once you understand the data you can then, and only then, make the huge leap to determining what data you want and how to get it to where you want.

Let me draw this out in a real world example.  Beware; I am using my creative mind now!

What we have so far, for Awe Widgets Incorporated, is several data points.

• Point of Sale/POS Systems in 300+ stores.
• Web Analytics (by Webtrends of course) tracking all sorts of great data points on the Awe Widgets Incorporated Website.
• Internal Accounting Software (Almost ERP, not really)
• In-house Built Customer Lists for Sales.

So there we go, four key pieces of tracking.  So how would they work together?  With a little further analysis (my key creative side now analyzes Awe Widgets Incorporated internal structure) and we find a few connections.

Correlation, POS to Webtrends Analytics

The POS System has a tracking identifier for customers which we can use to sync up with logged in users tracked via Webtrends Analytics.  This data can be used to derive who is and is not in stores purchasing.  In addition trending could follow the user flow to derive some actionable decisions on how to encourage online or store front shopping.  Just these two data points being connected add a lot of value.

Correlation, Internal Account Software ties to POS

Another data point tie in with the aforementioned POS & Webtrends data is the Internal Accounting Software (IAS).  The IAS holds information related to each sale, and other correlated information about how sales are going for the quarter, year, and other performance indicators.

Correlation, In-house Customer Lists for Sales

The sales department, in aggressive technical fashion has built a number of customer lists in Excel & Access.  The Access Application has a partially updated data store with a server based Excel file holding the updated piece of data about each of the sales person's current sales.  I know, I hear it now, every developer that is familiar with this scenario screaming, "OMG, you have your data in Excel AND Access, and it is supposed to have integrity, and be aaaaaaaaaaaaaaaaggggggggggggggggghhhhhhhhhhh noooooo!"  But you know, this @#$% happens.  : )  When things are like this, solutions get creative.

Tying Together the Pieces

Alright, this is when the awesome nerd bits start to happen.  But I have covered enough for this entry.  In the following entries on this topic I will step through this first data finding mission and start discussions on how to connect these sources and get that data mart, warehouse, or other middle tier piece into action.   I will continue on and lead into how the data can finally start telling a real story.  Because in the end, the real story is, somebody needs actionable data to act upon.  Does it really matter where it is?

Check out Part II of this series.