I had a great conversation the other night while at the Seattle Web Analytics Wednesday (#waw) with Carlos (@inflatemouse) and a dozen others.  @inflatemouse brought up the idea that an analytics provider using the cloud, increases or at least possibly increases the risk of security breach to the data.  This is, after all a valid point, but because of the inherent way web analytics works this is and is not a concern.

Web Analytics is Inherently Insecure

Web analytics data is collected with a Javascript Tag.  Omniture, Webtrends, Google, Yahoo, and all of the analytics providers use Javascript.  Javascript is a scripting language, which is not compiled, and stored in plain text in the page or an include, or passed into the URI when needed.  This plain text Javascript is all over the place, and able to be read merely by looking at it.  So the absolute first point of data collection, the Javascript tags, is 100% insecure.

The majority of data is not private.  So this insecurity isn't a huge risk or at least should not be.  If it is, you have larger issues before you even contemplate using an on-premise and cloud solution to bump up your compute and storage capabilities.  Collecting data that needs to be secure via web analytics is an absolute no.  Do NOT collect secure, private, or other important pieces of data this way.  If you have even the slightest legal breach in this context, your entire analytics provision could have this data scraped, possibly used in court in a class action suite, or in other ways even.

For the rest of this write up, I will assume that you?ve appropriately encrypted, or enabled SSL, or otherwise secured your analytics or data collection in some way.

Getting that Boost on Black Friday

eE-commerce has gotten HUGE over the last decade.  The last Black Friday sales and holiday season saw the largest e-commerce activity in history.  Omniture, Webtrends, and all of the other web analytics providers often see a ten fold increase in web traffic over this period of time.  Sometimes, for some clients, this traffic is handled flawlessly by racks and racks of computers sitting in multiple collocation facilities around the world.  However, for some clients that have exceedingly large traffic boosts, data is lost.  (yes, ALL the providers lose data, more so during these massive boosts)  The reason is simple, the machines can?t process in time or handle the incoming traffic because the extra throughput isn?t available to scale.

Enter the cloud.  The cloud has vastly more scalability, almost an infinite supply by comparison, to any of the infrastructure available to the analytics providers.  Matter of fact the cloud has more scale available than all of the analytics providers.  This is actually saying a lot, because Webtrends (and maybe some of the others) I know does an amazing job with their scalability and data collection, arguably more accurate and consistent than any of the other providers (especially since many of them just sample and "guess" at the data).

So when you extend your capabilities to the cloud for web analytics do you really increase your security vulnerability?  Most of the providers of web analytics have their own array of security measures, that I won't go into on levels of security.  However, does introducing the cloud change anything?  Does it alter the architecture so significantly as to introduce legitimate security concerns?

Immediately, from a functional point of view, assuming good architecture, intelligent system design, and good security practices are in use already, introducing the cloud should and is transparent to clients.  For the provider it should not increase legal concerns, functional concerns, or otherwise pending the aforementioned items are taken care of appropriately.  But that is just it, every single current provider has legacy architecture, various other elements that do not provide a solid basis for a migration to the cloud for that extra bump of power and storage.

So what should be done?  What if a provider wants that extra power?  Can the technical debts be paid to use the awesome promises of the cloud?  Is the security really secure enough?

Probably not.  Probably so.  But . . .

This provides a prospective opportunity for a new solution for web analytics to be provided.  It provides a great opportunity for a modern cloud based solution, that provides more than just a mere Javascript tag and insecure unencrypted data to be collected for analysis.  It provides the grand opportunity to design an architecture that could truly lead the industry into the future.  Will Webtrends, Omniture, Unica, or someone else step in to lead the analytics industry into the future?

At this point I'm not really sure, but it definitely is an interesting thought and a conversation that I have had a lot of people at #altnet meetings, cloud meetups, and with cloud architects, engineers, and others that have similar curiosities.  I await impatiently to see someone or some business take the lead!

I have recently decided to move on from Webtrends and wanted to send a public thank you to everyone.  I have served in three distinct departments for Webtrends over the years; VI SWAT Team Support, Engineering, and Consulting Services.  In each of these groups I have worked with some great individuals.  These team members have been good people, technical rock stars, and all around great company over the last few years.

• Neil | You are a rock star tier three leader.  Your boot camp training sessions are awesome!  You can shoot ok too.  Keep shooting, keep rocking the leadership, and I don't even have to suggest keep being a thinker.
• Jayson B | Thanks a million for the multitudes of conversations and random .NET tech discussions and such.  Those conversations always rocked, and introduced me to more than a few new ideas.  Keep thrashing the code and I am sure you will be rocking an elite team in no time.
• Jay | Aweosme ranting with ya.  Enjoyed working with you and hearing about the bus commute in.  If I ever get my transit sleuth blog running again I will be sure to
• Doyle | Thanks for sight tapping yourself in the eye, that was great entertainment.  If you ever need some help, I can teach you how to shoot those real guns.  Keep hacking that T-SQL Code, keep the stats up, yadda yadda yadda.  You rock, thanks for the off hand banter every day!
• Alexander | Your product knowledge is renown around the company and outside of the company.  Don't spill too much beer and I will be looking forward to our next communism versus capitalism discussion.  Maybe we will flip sides sometimes just to play devil's advocate.
• Gareth | It has been great fighting wars with you, you have also been a valuable point of knowledge and technical assistance with the product, the wars, and more.  I will miss the random banter throughout the day.  Keep kicking ass and I will look forward to our next pint or pitched battle.
• Heather C | Keep the integrations integrated and integrating, it has been fun.  Wish we could have worked together more.  Also enjoyed presenting with you in New Orleans at Engage, maybe we'll stumble into that chance again!
• Brian G | Great presenting in New Orleans, may video tracking be the bee's knees!
• Josh W | Keep it virtualized, I know you will.  Without you so much would just fall over itself, you honestly keep the whole of the company's progress moving forward.
• James K | You are an excellent developer, it was fun working with you back in my Webtrends engineering days.
• Rob D | Keep those awesome iPhone Apps coming.  Those suckers rock . . . of course that means I have to go get another iPhone.  But yeah. . .
• Tim | I hardly knew ye' and we only made it to Zack's Shack for dogs once!!  We were supposed to go every Tuesday!  Well maybe in the future we'll grab one of those Tuesdays.  Keep throwing down awesome PPC Stats and numerical goodness for your clients.
• Richard | OMG, the end of the world is coming.  Oh wait, that was just me missing my bus.  But seriously, watch out for all the crazies.  I hear there are a few.  Great conversations, keep having fun.
• Michael C | Thanks for being a great boss man, you rule.  Your insights into business and getting the services team working with other departments is top notch.
• Gary | Thanks for being a great boss man, fellow team member, and helping me keep my sanity in the desolation of Thief River Falls.
• Virginia | Will you ever return to Webtrends?  Eh, whatever the case, have fun make sure to keep kicking and enjoy your new married life.  I'm sure you will!
• Jeff P | Thanks for the random hallway chats, music suggestions, guitar discussions, practice space ideas, and million other things we've randomly discussed.
• Jeff J | Thanks for the insights, keep up the great job app man.  If you have any thoughts, questions, or otherwise feel free to shoot me an e-mail!
• Brad G | Keep hacking, make sure to use yourself an MSDN and get and stay ramped on the latest .NET bits.  I know you will and yes I will be working on the analytics frameworks in the future.
• Dash | Daper man, daper.  Keep it money, keep it rolling, keep it smooth.  I know I can trust you to do so.
• Eric G | Make sure the photons are loaded or Xander & Neil are going to take you out next round.  Also thanks for the random insightful e-mail thread comments and thoughts.
• Dan B | Dude!  Watch out for the Javascript, it looks like that Actionscript but it is broken!!??!?  Egads!  Great technical breadth, keep up the hacking and keep the Javascript under control.  I know you will.
• Sean B | Thanks for the political strategies, updates and the lowdowns on the info.  I appreciate it.  I'll definitely miss getting lunch and figuring out the markets. I'll have to break out and hit Portland sometime, we will have to grab Dan and get a Pita Pit.
• Michael G | You are the a totally ruling VP o' Sales, you have brought life to a zillion conversations.  I don't even need to suggest, because I know you'll be rocking the sales for some time.  I look forward to hearing Webtrends reach new records quarter after quarter.  Keep up the mischievousness!
• Jeremy D | Keep those clients happy (and in line) and keep those sales rolling in!
• Sandra | Neighbor, coworker, fellow nerd.  Keep having fun, and getting that knowledge to the clients.
• Perry | Mississippi here saying have a blast, keep the training flowing, and stay out of trouble.
• David S | You are the steady calm of the consulting services division.  Stay steady.
• Kelvin C | Dude, you have some of the best ideas, keep up the energy, and don't ever lose that awesome sense of humor!
• David S | Keep up the amazing work, and tear it up with the designs.
• Ryan P | HTML 5 RULZ!  FTW!  Yeah!
• Justin G | The UX is amazing, you have changed Webtrends for the better.
• Justin K | Beer, blogging, Webtrends, and more social media.  Keep up the good work, it has been a blast!
• Kamran A | Thanks for keeping the automations going.  I knew things happened with high quality points when you were on the job!
• Jeahna | You should just be in charge of everything.  Seriously!  You rule the house!
• Kelly K | You are keeping this ship from going aground on many an occasion, you rock.
• Carmen | Thanks for keeping me in line.  Somebody has to do it.  Great working with you, have a blast in ops.
• Dave C | Thanks for leading services and keeping the keel steady through good times and rough times.
• James M | Your leadership of partnerships is without peer.  You and Abed have been great team members and partners in business.
• Abed | Thanks for the discussions, the camaraderie, and helping kick off some great partnerships!
• Jascha Kaykas-Wolff | I didn't get to work directly with you, but you have revolutionized Webtrends Marketing Group, amazing, truly awesome job!
• Alex Yoder | Seriously, you should have taken over months before.  I'm honestly very glad you were at the helm during my time at Webtrends.  It wouldn't have been the same.  Your leadership really helps drive the company to new heights.

If I missed anyone, my apologies, you all have been rock stars!  I loved working with the whole crew of the Webtrends Ship!  Keep sailing on the high tech channel, I wish everyone success in their endeavors.

In this entry I just want to cover some examples of how to connect to Webtrends DX Web Services.  The DX Web Services use REST as the architecture, providing simple URI based end points to connect to.  With the Webtrends SDK you can connect to these services with your account information.  Here are the basic steps to retrieve a profile list, the reports from one of those profiles, and then the report you want from that report list.

First step is to create a Webtrends User.

WebTrends.Sdk.Account.User webtrendsUser = new Account.User();

webtrendsUser.UserName = username;

webtrendsUser.Password = password;

webtrendsUser.AccountName = account;

After you create the Webtrends User, simple request a profile list by getting list of ProfileDefinition Objects.

List<WebTrends.Sdk.Profile.ProfileDefinition> profiles = 

  WebTrends.Sdk.Factory.NavigationFactory.BuildListing(webtrendsUser);

Next you will want to grab a report based on the profile you are in and your credentials.

List<WebTrends.Sdk.Report.ReportDefinition> reports = 

WebTrends.Sdk.Factory.NavigationFactory.BuildListing(profiles[i], webtrendsUser);

In the code above, i would equate to the specific profile you want from the retrieved list of profiles in the profiles list.  The common scenario is that one has pulled the profiles into a drop down, combo, or list box that the user can select.  Then when the user selects the specific profile that profile object can then be used to pull the List of ReportDefinitions.

Once we have the report definitions, all sorts of criteria can be added together to query for a specific report.  This is also were things can get a little tricky.  For instance, take a look at the code below.

WebTrends.Sdk.Factory.ReportFactory.CreateDimensionalReport(

    report.ID.ToString(), profiles[i].ID.ToString(), "2010m01", webtrendsUser);

The CreateDimensionalReport takes 4 parameters for this particular overload.  The report ID, profile ID, the Webtrends Date Format, and the Webtrends User Object.  There are a number of other overloads available within this factory's method that allow for passing the specific REST URI, and other criteria to retrieve the report of your choice.  In the near future we will be adding some more to this method also, which will provide more flexibility without needing to use the full REST URI.

I will have more on this, so all you Coders out there using Webtrends DX Services, I hope this is helpful!  Enjoy.

Within Webtrends Insight one has the ability to check out the story, an automatically generated feature that gives a written overview of the activity on your site.  I really like this feature as it gives verbal perspective.  Below is a screen capture of my blog as of today for the last 7 days. Click on the image for a larger image for readability.

The other really great looking bit is the Visits Overview Report.  All Web Compliant, and I'd show you the page, but you have to have a Webtrends Analytics Account.  :P

That is it for this week.  I am out on vacation next week, so the blog will be nice and silent.

Sharepoint 2010 has some great new features.  Especially considering some of the disturbing necessities of previous Sharepoint Versions.

The first thing I noticed was that Visual Studio 2010 has default templates for building Sharepoint Projects.  This is huge compared to the monstrous and annoying effort required for development on previous versions.

Being able to build Sharepoint Projects from Visual Studio 2010 is a huge change, enabling IT & Development Groups to actually use regular Windows 7 Machines instead of needing Windows Server operating systems and licenses to build Sharepoint Applications.  To me, this was always a non-starter, and actively discouraged Sharepoint Applications in the past ? obviously most companies had not listened to the voice of reason.  But now, I have changed to actively encouraging Sharepoint 2010 Projects now.

The second thing is, it is more open and easier to build public facing sites finally.  This has been desperately needed.  I am glad that it has finally become a higher priority.  Not that I totally trust the Sharepoint 2010 Server to be fully web compliant, I have seen that they've taken leaps ahead.

The other thing I stumbled upon recently that I had not seen was the analytics capabilities of Sharepoint 2010.  The overview is that you can get traffic, search, and inventory reports based on how much traffic the site gets, who visits the sites, and other web analytics data.  Definitely check out the blog entry "Introducing Web Analytics in Sharepoint 2010" on the Enterprise Content Management (WCM) Blog.

Between the Sharepoint Analytics, Silverlight, and offerings we provide at Webtrends we now have direct extensibility points form the internal Sharepoint Data Sources, Analytics Data Stores, temp caching and other control mechanisms within Silverlight, and much more.  What does all that techno-babble mean?  It means more data, more available faster, easier access, decreased development times, more accurate reporting, more timely reporting, and the list goes on.  Pretty much ever metric about web analytics within Sharepoint 2010 has just gotten better.

The other architectural pieces I noticed is the Sharepoint Aggregation of data is now per web application in the farm, per site collection, per site, and per search service application.  This can be broken out even further with On Demand Webtrends Analytics by providing ETL between the Sharepoint and pulling the Sharepoint or extended analytics data Webtrends provides into the Business Intelligence (BI) framework within an enterprise.  At this point I can only imagine the scale and available data points for managing enterprise collaboration, social media, and other capabilities with this combination.  I'm looking forward to seeing what kinds of ROI we can see from these integrations, whatever it is, I already know it will be measurable and I suspect substantial.

For more information about Sharepoint 2010 check out the Sharepoint Blog and for more information regarding how Webtrends can get your Sharepoint Site Analytics extended check out Webtrends Professional Services!

Are there any data points, questions, thoughts, or ideas you have about connecting Sharepoint & Analytics Data?  Anything you haven't seen connected that should be?  Does your organization use Sharepoint, Webtrends Analytics, or other collaboration software?  Anyway, I am no shill, I am honestly interested in these connections outside of the fact I work for Webtrends.  If you have a minute, please let me know what your thoughts are on these technology connections.

I went to the session on Microsoft Silverlight Analytics Framework (MSAF) today while here at MIX 2010.  It was a great walk through the features, ideas, and what the end goal is.  Michael Scherotter did a great job of lining up the ideas, intentions, and the functional ideas behind the framework.

The framework is built around the Silverlight Behaviors.  If you aren't sure what behaviors are, check out these entries from Nikhilk.net Silverlight Behaviors, Silverlight 3 Drag Behavior, An Introduction to Behaviors, Triggers, and Actions, and of course the MSDN Documentation on the matter.

Some of the key features of the framework is to support out-of-browser scenarios, which works perfectly with out Webtrends DX Web Services.  Offline scenarios, which again, we have worked toward supporting at Webtrends DC Web Services via caching and other criteria as needed.  Another feature that I was really stoked about is the Microsoft Expression Blend integration that removes the need for coding, thus simplifying the addition of analytics components based on events or other actions within a Silverlight Application.  This framework also easily supports A/B Testing (again, something we do quit a bit of at Webtrends with Webtrends Optimize.

The last thing I really wanted to point out was the control support that this framework has support in already from Telerik RadControls, Smooth Streaming Media Element, and Microsoft Silverlight Media Framework Player 1.0.  These are implemented with behaviors and handlers exposed via MEF (Managed Extensibility Framework).

All in all, great second day, great analytics framework for Silverlight, and great presentation.  Until tomorrow, adieu.

I was listening away to some ear blistering metal, as I often do, and an ad really jumped out at me.

If you can?t see what is written in the HTC ad to the right, click on the image.  What is displayed is a cross-correlation of several points of analytics data.  Before I jump right in and start explaining each point, think about what is going on with this ad.  This is by no means just some simple ad, there are a number of things going on here.

First data point.  Sprint & HTC, or whoever it is that put this ad together, has retrieved my listening favorites from Pandora.  Just looking at the bands listed shows that to be self evident.  This also seems to be the most obvious piece of data they could have collected about me, since I am logged into Pandora.  This is probably achieved by some web services or other API that Pandora provides advertisers.

The second data point is not immediately noticeable.  I am still at a loss to explain where they retrieved this data point.  What is it?  Concert dates for bands.  Each of the bands listed in the HTC app that is displayed is a coming show.  Matter of fact, it almost seemed like they had shown me my own HTC, except I don't own one.  :)

Now my location data, I am suspecting probably came from Pandora too, but it is the third point regardless.  All together the ad utilizes geo-positional location, my Pandora music preferences, and pulls local concerts from another source (maybe a Pandora listing too?).  This is a perfect use of preferences to display things that are truly relevant to me.

In addition, they may have just helped to sell me on a new phone for my personal line.  I am up for a replacement and anything that runs Google Droid seems cool, but I?ll admit, with the sneak peaks at Windows 7 Mobile that I've seen and the proposed ability to use Silverlight ? I WILL BE switching from the iPhone when that is released.

Week before Mardi Gras, the Saints rocked and won the Super Bowl, and @webtrends threw an awesome #wtengage Conference.  Props out to the long list of Webtrends People there;  @DashLavine, @caseycarey, @webtrendspeter, @justinogarrity, @sullybridgetb, @kaykas, @justinkistner, @michelewarther, @mrdiggles, @yodera, @vkenkal, @derekfine, @ekrobi, @mccook, @robinoula, @benfogarty, @noexg, @thomschoenborn, @mediachick, and others!  The event was awesome, great sessions, great knowledge transfer, great food, just great ? period!

Also must say the meetings, meets, and introductions were great.  I'm glad to have met dozens and dozens of people and get those names and faces connected.  Finally caught up with; @seanpower, @johnlovett, @christineconley, @ed1chandler, @bullfrogmedia, @ebeane, @cgrantski, @drcasio, @nadolski, and many others.  Then there are others I met and caught up with and still got more catching up to do; @aknecht and @bosilytics.

Seriously, all of these individuals are rock stars in the analytics community.  The conversations, new ideas, thoughts, and general kicking around New Orleans was a great time.  If you are interested in the next Webtrends Engage (and you ought to be if you do anything with analytics), check out the 2011 Conference coming to San Francisco.

Code Camp Kick Off

In other news, Code Camp is kicking off real soon.  Specifically, the kick off meeting is tonight here at Webtrends.  This year Code Camp is going to be pretty huge.  I am guessing at around 400-500 people, more sessions, more geeking & nerding about, it will be an awesome time for all.  So keep reading and I will be posting more tidbits about the upcoming Code Camp.

Read Part 1 & Part 2 of this series if you haven't already. 

Last entry I worked up the basic origins and destinations for the data we have. Now we have to start turning this into something real, something concrete. The following processes include a whole soup of acronyms and other cryptic vocabulary. The most common thing I will probably use is the ETL acronym, which stands for Extract, Transform, and Load. This is the process of bridging the data to the various originations and destination and moving that data appropriately.

In the last entry I finalized the originations and destinations as shown below;

In my previous entry you may have noticed that I had posted SSIS with the Destination list.  Being one that corrects themselves when mistaken, I took it out, as it does not belong there.  The SSIS is our tool that will perform the ETL functionality for this project.

At this point we are finally going to get into the dirty bits of these pieces of technology, and how we need to tie them together.  I am going to attack them over the next few entries based on the order in the lists above.  The first item, is the Excel & Access 2007 customer relations listings from sales.  Here?s a description and a few shots of what this thing looks like.

The access database is setup with a very simple relational data schema.  This is shown below (click image for larger view).

You can see there are pretty standard pieces of data, in a generally normalized (3rd form for the most part) structure.  This is fine.

Next is a shot of the data entry screen for adding opportunities.  There are respective screens for customers and employees.  Everything needed for a basic customer list & tracking basic things.  Nothing too extravagant here either.  Again, all is fine.

Below is a simple report that shows the available opportunities that are open.

Another report showing the forecasts.

Below is another forecast sliced grid.

So all that seems normal enough.  But the processes are what makes things tricky.  If everyone just managed sales from the database, all would be right in the world.  The first thing that breaks this is that each sales person enters their sales during the day and other information in a spreadsheet that is not linked to the underlying database.  Someone each morning puts the previous days sales information.  This of course, breaks down data integrity.  Below is a sample sheet that is used each day.

One thing that Excel is used for, that doesn?t break reports is the lists of prospective customers to call.  As shown below.

Now that we have a break down of the Excel & Access Customer Relations Management Software, I will move onto the other pieces of technology in the next entry.  This is the data point that has the most prospective data risk, so I put it at the top of the list to cover first.  After I cover each of the systems, we will move into the architecture of the system overall.  So keep reading, more juicy bits to come.

So here is the lowdown on what I will be presenting, demoing, and showing to the attendees & anyone who is interested at Engage 2010.  If you are attending and have not signed up on the Engage Crowdvine Site, please do so.  There are some great conversations going on there.

One other thing I wanted to point out, is that you MUST attend the Ignite in New Orleans.  Ignite events are a blast!  Portland, Oregon regularly has some of the largest Ignite Events, and since we are bringing a whole lotta Portland love, and NOLA is well known for some great parties, I know the NOLA Ignite will be awesome!  So make sure to attend.

First off, Brian Gallagher and myself will be showing off some awesome technology for video tracking at the science fair break out sessions.  There will be a Tuesday Science Fair and a Wednesday Science Fair, and I will be there with Brian going over the tracking.  So if you miss us one day make sure to check it out the next day!

The next speaking session is the enterprise & business intelligence topic I have mentioned in some of my previous blog entries.  The session is titled Make Analytics Work across your Enterprise.  Our topic summarized is, The nerd of the Enterprise;  ETL, EPM, ERP, BI, CRM, and how the alphabet soup works together for real Enterprise Business Intelligence and Insight.

The final engagement I have is a topic near a dear to my work here at Webtrends.  Over a year ago a rock star team of developers; Rob D, James K, and myself started working on the Data Exchange, REST Web Services for Webtrends.  Of course now those have been in production and are now on v2.0 already!  Rob has gone on to rock the current iPhone Webtrends Application (Check out the Webtrends Blog Post Too) & James is coding away at some of the awesome infrastructure that keeps all the data flowing smoothly on the back end (to the web services, to the Insight UI, the whole thing ? James is awesome at that).  So we built these services many months ago and now I get to present them as a workshop session at Engage.  Open web services based on good clean REST Architecture Principles have always been an interest of mine and I have been and will continue to be stoked that we at Webtrends have these available for customers!  So be sure, if interested, to check out the Workshop:  The RESTful Way ? How to Use Webtrends REST Services.

Also for further reading, check out some of my past entries on REST Services;

• REST, So Many Nuances, The Self Signed SSL Cert Override
• REST Web Services Synchronous Request that points at the Webtrends Developer Network Blog Entry.
• Just a Bit More REST which has a really bad example of unit tests, which are more integration tests, running against some REST Web Services.
• File Extension ".svc" Annoyance
• RESTful .NET by Jon Flanders is probably one of the best books on the WCF based REST Services.  So if you are looking for a solid guide, this is a great place to get just that.