*Check out the 2015 update to this post

I’ve been answering a lot of questions on Team Foundation Server reporting over the last couple of weeks so I thought I’d put some of that information in a blog post.

Team Foundation Server is an amazing tool for storing information about your development projects and all that information can be retrieved if you know what the options are.

This post is targeted at TFS 2013 but most of it is relevant for TFS 2010 & TFS 2012

How does data flow in TFS?

This is probably the most concise reference of TFS reporting architecture and the various options that I can find: Create, customize, and manage reports for Visual Studio ALM

 

In a nutshell, when you work as normal in TFS - checking in source code, executing builds, running tests and completing work - all that information is being stored in transactional, operational stores within the Team Project Collection database (there is a separate collection database for each Team Project Collection that you create). 

These stores are designed to provide high transaction speed and data integrity in normalised tables meaning data isn’t duplicated.  As a result, it’s not very easy to work with these tables, it’s difficult to know which tables to use to get the data you want and queries can be slow because of all the joins required. 

A number of data adapters then populate a single, relational, data warehouse with the latest data from the stores.  Note that reportable data from all team project collections is published to that warehouse (Tfs_Warehouse) meaning that you can report across multiple collections (obviously, you should be careful who you give permissions to!)

Tfs_Warehouse has a schema that is much easier to understand because it is optimised for queries and reporting instead of transactions.  Additionally, this database can be on a completely different server so queries won’t slow down TFS.  The data warehouse is organised in a star-like schema into fact tables and dimension tables.

Finally, and every 2 hours by default, an Online Analytical Processing (OLAP) database (Tfs_Analysis) is populated to allow easy reporting on that information and this is where most of the standard reports will pull their data from.

You can read more about the architecture used for TFS reporting in Components of the TFS data warehouse

Types of Report

What types of reporting are available in TFS?

• Work Item Queries
• Work Item Reporting
• Out-of-the-box Excel Reports
• Out-of-the-box SSRS Reports
• Team Web Access Charting
• Team Web Access standard reports
• Custom Excel Reports
• Custom SSRS Reports
• SQL Queries
• TFS API
• REST API

Let’s look at them in a little more detail

Work Item Queries

Work Item queries are probably the simplest way of pulling information from TFS.  The data retrieved is always up to date as queries do not rely on the data warehouse. 

There are 3 types of query

• Flat Queries
• Direct Link Queries
• Tree Queries

For more information on Work Item Queries see Query for work items

TFS 2013 Update 2 adds the option to query on Work Item Tags.  If you have not installed TFS 2013 Update 2 or if you have TFS 2012 then  you cannot include tags in your Work Item Queries

Work Item Reporting

Work Item Reporting is the easiest route into custom reporting with TFS.  You can take a Flat List Query and within Excel click the New Report button on the Team menu

or in Visual Studio, click the Create Report in Microsoft Excel

From there you can choose to generate a report for the current state and/or the trend over time

TFS will generate a workbook with a number of reports on separate worksheets and a table of contents.  You are free to customise any of these reports and save the result for later use.

Here is an example of editing an Excel report Custom Burnup chart with TFS and Excel

For more detail, see Creating Reports in Microsoft Excel by Using Work Item Queries

Out-of-the-box Excel Reports

Depending on your chosen process template, and if you have installed SharePoint as part of your TFS deployment then you may have access to a number of standard Excel reports. 

These are included with both MSF templates (Agile & CMMI) but not with the Visual Studio Scrum template.  That said, with a little tweaking they could work with the Scrum template (eg, swap references to Story Points field used by MSF Agile to the Effort field used by Scrum)

They can be accessed through the SharePoint portal or from the Documents tab in Team Explorer.

Here is the Task Progress report as an example.

See Excel Reports for more detail

Out-of-the-box SSRS Reports

Again, depending on your chosen process template (and assuming you have installed Reporting Services) then you will have access to a number of reports automatically when you create your Team Project.  They can be accessed via Web Access, SharePoint and Team Explorer.

View, upload, and organize reports (Reporting Services Reports)

For example, this is the standard Backlog Overview report available with the Scrum process template

For a full list of available reports see Reports (SQL Server Reporting Services)

To view the reports, you must be assigned or belong to a group that has been assigned the Browser role in SQL Server Reporting Services. For more information, see Grant permissions to view or create reports in TFS.

Team Web Access Charting

Charts are new in TFS 2013 and are a bit like Work Item Reports but a lot simpler to create and not as customisable or detailed.  The big benefit is that you can pin these charts to your Home page in Team Web Access (you’ll need TFS 2013 update 2 for that) so they become very visible to your team

My good buddy, Colin, has an excellent write up on reports here although it pre-dates Update 2 so doesn’t mention pinning to the home page or customising colour on the reports (read about that here)

You can view charts with a Standard TFS Client Access License but to create them you must have a Full license which means Visual Studio Premium, Ultimate or Test Professional.  See more about Access Levels for Team Web Access here.

Team Web Access Charting

Team Web Access standard reports

Team Web Access has some useful reports which can be accessed very easily.  The reports are non-customisable but the good thing about them is that they do not rely on the data warehouse cube so they are never out of date.

Velocity

The Velocity diagram shows the number of Story Points (or whatever unit you are using to estimate User Stories/Backlog Items) scheduled and completed across iterations.  This will help with forecasting and release planning for your project.

Cumulative Flow Diagram

The CFD Diagram shows up to 30 weeks of data displaying the number of Work Items and their state over a date range.  If bulges are showing up on your CFD it could be an indicator of a bottleneck in your development process

TFS 2013 Update 2 adds the ability to specify a start date for your CFD which can be very useful if you have a long running project.

Sprint Burndown

Finally, your Sprint Burndown shows the number of hours (assigned to the Tasks) your team has committed to in the sprint.  As work is completed (or added) then the value for Remaining Work on the Y-axis will change and an ideal trend line will show you when you are likely to complete the work if the current work rate continues

TFS 2013 Update 2 finally added the much requested feature to be able to remove weekends from the TWA Burndown chart.  This means the burndown does not show the “steps” of a flat line when no work is completed at the weekend which could be a little demoralising.  Giles covers it in more detail here

Custom Excel Reports

Now you get into some proper custom reporting.  I still feel that the Work Item Query Reports covered earlier in the article are the easiest way to get started with Excel but you can connect to the Analysis Database manually if you like.

Take a look at Create a Report in Microsoft Excel for Visual Studio ALM which describes how to connect Excel to the TFS Analysis database.  Doing it this way means you are creating a report from scratch which is great if you know what you want but there is definitely a steeper learning curve.  Start with reading Perspectives and measure groups provided in the Analysis Services cube for Visual Studio

Custom SSRS Reports

If you have a SQL Server Reporting Services expert on your team then they may prefer to generate a report that way.  The benefit of using SSRS is that the report can be published to Team Foundation Server and it will be available alongside the OOB Reports in the Reports tab of Team Explorer or the Reports website.  It also allows for much more detailed reports than excel and allows you to plot multiple data points on the same report.

You can use:

• Report Builder
• Report Designer

Both report authoring tools can target Tfs_Analysis and Tfs_Warehouse although you may find Report Designer more complex to begin with.  You can view a comparison of the two tools here.

It may be easier to start with an existing report and customise it.  As well as the OOB reports, there are a number of sample reports available such as the Community TFS Report Extensions

Create a Detailed Report using Report Designer

Table reference for the relational warehouse database for Visual Studio ALM

Perspectives and measure groups provided in the Analysis Services cube for Visual Studio

If you create a custom report then you may want to add it to your Process Template so it is automatically available for any new projects created.

Add reports to the process template

SQL Queries

The majority of the reports we’ve discussed so far pull their data from the Analysis Services OLAP cube but although it’s generally more difficult, there is no reason why you can’t grab data directly from the relational data warehouse

The TFS Reporting Guide is generally a good resource for TFS reporting but it has some good examples of running T-SQL queries directly against the TFS Data Warehouse relational database.  For example, check out the Mean Time Between Failure (MTBF) Report which uses the CurrentWorkItemView to retrieve the current state of each work item from the dimension table.  The sample query fetches all Work Items of type Bug that are in a Closed state in the chosen Team Project and calculates the average time difference in hours between them.

You can also query the TFS operational stores but it’s not advised and you should really use the TFS API instead.  For example How to generate a report of active users who log onto the TFS server

TFS API

One of the great things about TFS has always been it’s customisability and there has always been a reasonably well documented API for pushing and pulling information to/from TFS.  For example you can create a TFS Work Item with about 3 lines of C# code.  You can also use the TFS API to create reports in a supported way.  If you have a look at the WCF Service sample for a Build Duration Report in the TFS Reporting Guide Data Warehouse section.

You’ll need to become familiar with Extending Team Foundation too.

REST API

A little while ago an OData Service was released for TFS to allow clients to be created for smartphones, tablets and non-Windows operating systems as long as they supported http requests. 

Team Foundation Server OData API

OData Service for Team Foundation Server v2

TFS Reporting Guide also has a whole section on this

This has now been enhanced as described in A new API for Visual Studio Online.  However, at this stage this API is only available for Visual Studio Online but it will likely be available for on-premise installations in the next version of TFS.

Some Questions

What about Visual Studio Online?

Visual Studio Online (formerly Team Foundation Service Preview) does not use SQL Reporting Services and so there is no data warehouse to connect the standard reports.  You still have the Team Web Access reports and the option to use the TFS API but in addition it is worth checking out the very cool REST API for Visual Studio Online which should make it easier to pull information from TFS Online

What permissions do I need to view and create reports?

Permissions for Excel Reports That Connect To the Analysis Services Cube

What is the permission needed to view and create a Excel report for TFS Analysis Services cube?

If you prefer to use Report Builder to edit or add Reporting Services Reports then it is slightly different, you’d need to have your account added to the Content Manager role for Reporting Services.

Grant permissions to view or create SSRS reports in TFS

I’m using Microsoft Test Manager with the Visual Studio Scrum template and the OOB reports are lacking.  What can I do?

There are some standard reports for testing such as the Test Plan Progress Report and the Test Case Readiness Report.  If that is not enough then it is worth checking out the (now slightly old but still relevant) blog posts here:

Test Case Management (TCM) Reporting - Frequently Asked Questions - Part 1

Test Case Management (TCM) Reporting - Frequently Asked Questions - Part 2

which gives a lot more information on custom excel reporting.  

Are there metrics associated with Builds and Continuous Integration and can we access historic build success ratios and failures etc.?

There is a full build log with diagnostics available for all builds (you can set a retention policy on them) and also the information is published to the data warehouse.  There are OOB reports (Build Summary, Build Success over Time) and options for custom reporting as discussed above.  See Analyze and report on build details and build coverage using the Build perspective for more detail.

I want to my reports to update more frequently

You can change the process control setting

Managing the data warehouse and analysis services cube

Change a Process Control Setting for the Data Warehouse or Analysis Services Cube

My TFS data warehouse is broken

Check your account permissions, review log files and rebuild the warehouse.

Manually process the data warehouse and analysis services cube for Team Foundation Server

TFS Admin Console - Rebuild the Data Warehouse and Analysis Services Cube

TFSConfig Comman Line - RebuildWarehouse Command

I don’t have Reporting Services or SharePoint installed

No problem, get them installed, start generating your reports or sharing your Excel workbooks with your colleagues.

Install Reporting services

Install SharePoint

Upload reports to a team project

Summary

Hopefully someone will find this post useful and I know I am likely to redistribute a lot of the information within it in the future.  You can see that there are many options for pulling data and generating reports from Team Foundation Server and you just need to figure out which best suits your needs.

Thanks,

Richard

Tweet.

I was doing a little custom reporting using Report Builder today and when I came to preview my report I received an error message.

The permissions granted to user 'TFS2013\user' are insufficient for performing this operation.

System.Web.Services.Protocols.SoapException: The permissions granted to user 'TFS2013\user' are insufficient for performing this operation. ---> Microsoft.ReportingServices.Diagnostics.Utilities.AccessDeniedException: The permissions granted to user 'TFS2013\user' are insufficient for performing this operation.
   at Microsoft.ReportingServices.Library.ReportingService2010Impl.CreateReportEditSession(String Report, String Parent, Byte[] Definition, String& EditSessionID, Warning[]& Warnings)
   at Microsoft.ReportingServices.WebServer.ReportingService2010.CreateReportEditSession(String Report, String Parent, Byte[] Definition, String& EditSessionID, Warning[]& Warnings)
----------------------------
The permissions granted to user 'TFS2013\user' are insufficient for performing this operation.

That seemed odd as I hadn’t messed around with security settings and it was working recently.  Still, just to be on the safe side, I reviewed the article:

Grant permissions to view or create reports in TFS

Everything looked fine so I spent some time playing around with security settings and doing a little web searching.  I found an old stackoverflow post which talked about deleting the encryption key and lots of other things that seemed unlikely.

So I grabbed a cup of coffee and went outside for some fresh air and the answer hit me!

Simple as that, and kind of obvious - Run Report Builder as administrator.

Doh!  Hopefully this blog post will save somebody a little time and at least it has broken my blogging drought. 

Cheers,

Richard

Tweet.

I was delivering a training course on TFS 2012 this week and one of the delegates came across some unexpected behaviour that I hadn’t noticed before.

We had added the Work Items Check-in Policy to version control earlier in the day and were looking at automated builds.

With TFS you can perform a private build (or buddy build) to ensure that your proposed changes integrate successfully before you check them in.  You can also select the option to Check in changes after successful build which turns it into a kind of voluntary Gated Check-in.

This is what my delegate did and as his build completed successfully, his shelveset was committed to version control.  Somebody then pointed out that as the changes were not associated with a Work Item there should have been a policy warning like the one below.

I can’t say I have ever noticed this behaviour before and it took me a minute to realise the reason behind it.  Check-in policies are evaluated on the client side so for example if you use the Code Analysis Policy then it is your responsibility to run the code analysis rules against your code before check-in.  The server will not evaluate the policy for you, it will simply check to see if you have passed the rules locally.  The check in that happens as part of the private build happens server side so it did not have to pass the client side check-in policies.

You can of course control permissions on builds or ensure that the build also runs a suite of unit tests but it’s one to be aware of.

Richard

Tweet.

Recently I’ve been working with a customer on migrating their existing Team Project to a new Process Template on Team Foundation Server 2012. 

We’ve decided to go with the TFS Integration Tools to move selected Work Items and code across to a new project in a new Team Project Collection.  There were a number of alternative options open to us but this is a large project that has been migrated through the versions of TFS and we’re going to take the opportunity to clean it up and consolidate.

I spent some time mapping their existing Microsoft Solutions Framework for Agile Software Development 6.1 project to Microsoft Visual Studio Scrum 2.2.  I had to take into consideration some custom fields that had been added to the template and also some legacy fields that had been left hanging around after previous migrations.

The customer is not a heavy Visual Studio user, doing most of the development through Eclipse and as a result they are licensed for Visual Studio Professional 2012.  In turn this means that their developers have “Standard” licenses for Team Web Access. 

As part of the work we have been doing to transform their development process we have a need to use both the Product Backlog view and the Kanban board, both of which are only available to users who are “Full” licensed users of TWA. 

The cost of upgrading the developers to Visual Studio Premium was prohibitive as they will not make use of the additional Visual Studio features and the current Microsoft licensing did not give us any other option.

I evaluated both Urban Turtle and the Eylean products and although they both have their plus points and are very economically priced, neither gave us exactly what we were looking for.

Last week at the Build Windows Conference, Microsoft released a Preview of Team Foundation Server 2013 which also has a Go-Live license. 

Ordinarily it would be a pretty gung-ho decision to move your business onto pre-release software but in this case it is worth considering.  The reason is that Microsoft have listened to the feedback and re-jigged the licensing for TWA. 

TFS2013 now provides both the Kanban Board and the Backlog & Sprint Planning Tools to “Standard” users (VS Pro) and rightfully provides some very cool additional features (Portfolio Management, Team Rooms, Test Case Management, Feedback Management) to “Full” users (VS Premium, Ultimate & Test Pro) 

   

TFS 2012 Team Web Access Levels

TFS 2013 Preview Team Web Access Levels

As a result, there is a strong case to move the project to TFS2013 and although the decision on which way to go has not been made yet, I was worried that the TFS Integration Tools mapping that I had written would need a lot of work to update it for the Microsoft Visual Studio Scrum 3.0 Template.

I downloaded the Scrum 2.2 and Scrum 3.0 Templates and performed a Diff on the folders.

There were some minor build changes and some changes to allow for the new Git support but I was particularly interested in the Work Item and Reporting changes. 

Release Burndown report – Order by Iteration Path

Minor change but hopefully a helpful one as I have run into this problem before and presumably this small change to the rdl file will resolve it.

New Category – Feature Category

New Work Item Type - Feature

Both of these have been added to support the new portfolio management functionality in TWA.

And that’s it, so there are no field changes to the core Work Items.  Obviously it is still a Preview version of the template but it shouldn’t change significantly and I can make any tweaks if it does.

If you’re in a similar position in that you would like some of the “Full” features from TWA 2012 but the Premium upgrade does not make financial sense then TFS 2013 could give you what you need.

Cheers,

Richard

Tweet.

Microsoft announced that there will be a new Visual Studio and Team Foundation Server this year along with some other cool updates.  Some links in case you missed them:

Brian Harry: Visual Studio 2013 – A good overview of what’s coming up in the next version

Introducing Cloud-based Load Testing with Team Foundation Service – No need to go looking for tin to run those Load Tests now.

Improving Communication with TFS Team Rooms – Nice feature if your teams are distributed

Video: Application Lifecycle Management with Visual Studio 2013 – Good demo from the two Brians at MS.

TechEd North America 2013 – Check out all the on-demand recordings from TechEd

Testing Improvements in Team Foundation Service Update for TechEd 2013 – Updates to web based test management

Extending Visual Studio Team Explorer 2012 – More extension points for VS.

Team Foundation Task Board Enhancer: version .7.6 released – No feature updates but still one of the very few TWA extensions available.

Microsoft Code Digger – Updated this week so it can run against code not contained in Portable Class Libraries. Thanks for the heads-up Col

Cheers,

Richard

Tweet.

Are you using TFS 2012? 

Are you using or adopting Scrum?

Then get the most out of your tools and your process by attending a 2-day Certified ScrumMaster course with a free one day Scrum with TFS course.

The CSM course is taught by our Certified Scrum Trainer, Colin Bird, and I will be helping out with the optional, free third day which will give you hands on experience with TFS 2012, putting into practice what you have learned in the first 2 days.

The course will be held on Microsoft Campus in Reading on the 21st May – 23rd May

More details available here or you can register here.

If you hurry you can still get the discounted early bird price!

Cheers,

Richard

Tweet.

I was putting together a demo of cross browser Coded UI testing with Visual Studio 2012 and ran into a minor problem.  Hopefully this post will save you some time.

I grabbed the very helpful Visual Studio 2012 Application Lifecycle Management Virtual Machine and Hands-on-Labs / Demo Scripts from Brian Keller and installed Chrome, FireFox and the Selenium components for Coded UI Cross Browser Testing from the Visual Studio Gallery. 

After that, one line of code can make your Coded UI tests run on a different browser.  For example, just stick this line at the beginning of your test method:

BrowserWindow. CurrentBrowser = “chrome”;

More details here: Using Different Web Browsers with Coded UI Tests

The problem I ran into was that the Selenium Components installer failed, which seems to be a reasonably common issue due to access permissions, but helpfully Microsoft provide the simple steps to manually install what is needed.

1. Download chrome driver from:

http://chromedriver.googlecode.com/files/chromedriver_win_26.0.1383.0.zip

2. Download selenium dot net bindings from:

http://selenium.googlecode.com/files/selenium-dotnet-2.29.1.zip

3. Right click on the downloaded zip files.

4. Select "properties".

5. Under "General" tab, click on the "Unblock" button.

6. Now unzip both the files and copy the contents to the following path (for the selenium-dotnet 2.29.1 binaries, copy the contents of net40 folder):

"%ProgramFiles%\Common Files\microsoft shared\VSTT\Cross Browser Selenium Components" (for 32 bit machines)

"%ProgramFiles(x86)%\Common Files\microsoft shared\VSTT\Cross Browser Selenium Components" (for 64 bit machines)

On the 64 bit VM I unzipped the contents of the zip files into the “Cross Browser Selenium Components” folder and started my test but received the following error:

Test Method CodedUITestProject1.CodedUITest1.CodedUITestMethod1 threw exception:

Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException:

The playback engine could not find the selenium binaries required to run cross browser coded ui tests.  This may be because the Selenium .Net bindings and the Selenium Chrome driver are not installed or an older version of the Selenium Components Installer was used.  For more information about installing the binaries, see http://go.microsoft.com/fwlink/?LinkID=267598

The important line in the manual installation instructions that I’d failed to follow was:

for the selenium-dotnet 2.29.1 binaries, copy the contents of net40 folder

So, to resolve the problem, simply copy the contents of the “net40” sub-folder into the “Cross Browser Selenium Components” parent folder

Happy testing.

Cheers,

Richard

Tweet.

Yesterday I presented at Microsoft Tech.Days Online 2013.  It was a slightly odd experience presenting in an almost empty auditorium to some cameras and a bunch of AV guys.

*Picture shamelessly borrowed from fellow presenter Colin Beales

If you attended the sessions, hopefully you enjoyed it and if you have any questions then please do get in touch.  The event continues today with some great content for IT Pros.

Cheers,

Richard

Tweet.

It’s my pleasure to be part of Microsoft UK Tech Days Online 2013 which is happening next Wednesday (24th April 2013).  It’s shaping up to be an excellent 3-day online event for both developers an IT Professionals with some really interesting areas being covered.  Day One is aimed at the development community with sessions on targeting Windows 8 and Windows Azure alongside Team Foundation Server and Application Lifecycle Management.  The day starts at 09:20 BST 

Introduction to Tech Days Online

Developing in Visual Studio, what's changed in the last 12 months?

Developing for Windows 8 with Visual Studio 2012

Interview with Planky/David Gristwood - Visual Studio Tools with Windows Azure

Being Agile with Team Foundation Server 2012

TFS for everyone – Using TFS in a heterogeneous project (Eclipse, Java and using Git-tf)

Upgrading to Team Foundation Server 2012

Testing enhancements in Visual Studio 2012 Update 1 and Update 2

“The Thrill of the Hunt” with Exploratory testing in Visual Studio 2012

I am hosting two of the sessions and I’ll be around to answer questions live on the day.

12:00  Being Agile with Team Foundation Server 2012

Team Foundation Server 2012 introduced a number of excellent features and capabilities to help enable your agile team to plan, manage and track their work. This session will introduce the tools and provide details on how to get the best out of them.

15:40  Testing enhancements in Visual Studio 2012 Update 1 and Update 2

Visual Studio 2012 Update 1 and the newly released Update 2 have introduced some exciting new features and functionality to the core testing tools.  This session will demonstrate the new web based Test Case Management, cross browser support for Coded UI Testing and Microsoft Test Manager improvements.

You can register for this event here and it’s worth doing even if you can’t make it on the day as we’ll follow up with resources and recordings of the sessions.

Cheers,

Richard

Tweet.