LabVIEW Testing Part 2 – Reliability Lab

In our second video interview installment about how we test LabVIEW and our driver software, Byron Radle takes us through our internal Reliability Lab.  When you watch it, the background noise is coming from 20 fully-loaded PXI chassis and 40 fully-loaded CompactRIO systems all running at once in our lab – raw (loud) power!  You can tell from the freeze frame below that Byron is excited about this lab. 

 

Once again, if you can think of specific scenarios that we need to add to our testing matrix, please respond.

What is our bug-fixing process?

Over the years, I’ve had many people ask me how we make decisions in the development team with regards to choosing features, fixing bugs, and testing new versions.  This week, I thought I would give you some insight into how we track and manage the bug reports and support issues that we learn about through our technical support department.  Roy Faltesek manages our LabVIEW Testing and Product Support Engineers group.  Part of their job is to be the interface with and be the escalation path for our Application Engineers who talk to you on the telephone or through email to answer your questions.

What happens when a customer reports a bug?

Roy Faltesek:  Almost all bugs get reported to us via a call or email to our Application Engineering department. An Application Engineer (AE) works with the customer to confirm the problem and then files a Corrective Action Report (CAR), which is essentially a work order for Engineering to investigate and fix the bug. Our CAR database allows us to track and manage bug reports from reporting to their final resolution.

We encourage AEs to provide customers with the CAR number associated with their reported bug, which not only reassures them that the problem has been reported properly to engineering, but is also useful if anyone wants to inquire about the issue later, perhaps to check for additional workarounds or to see if the problem has been fixed in a subsequent release.

Do we fix all bugs that we get reports about?

RF: We don’t fix all bugs that are reported, but we fix a lot.  For example, over a thousand bugs were fixed in LabVIEW 8.5. We prioritize and classify all bugs that are reported and, in general, we fix the highest priority bugs first. Factors that go into setting the priority are the number of users who would encounter the problem, the pain level associated with encountering the problem, and the ability of users to avoid the problem.  Crashes and hangs are obvious high priority problems. Controls not aligned properly are obvious low priority problems.

Engineering reviews all reported CARs, assigns them to the right developer and tracks their status.  Each developer decides how to designate their assigned CARs.  Some may be closed as Not Reproducible, some as Not a Bug (meaning the behavior will not be changed) and some as Rejected (mostly used for what is essentially an enhancement request reported as a bug). Of the ones that are fixed, the delay between the time it is reported and the time the fix is available to our users can sometimes seem interminable. A bug that is reported shortly before release might be fixed relatively quickly, but even if the problem is minor, that fix might not be available in a released product until the next major or minor release, which could be a year later.

What can customers do in the interim?

RF:  When a customer reports a bug, their immediate need is not for a product fix, but for a workaround. The user has a task they are trying to accomplish or a function they need to perform. If the bug they have encountered is keeping them from performing their job, our priority is to work with the user to develop an acceptable workaround. It’s extremely rare that we are not able to do this relatively quickly, but occasionally it can take a few days or even weeks and may involve working with engineering to develop a workaround. Customers should expect that the AE and possibly Engineering will establish a workaround every time a customer submits a CAR.

How can customers track their bugs?

RF:  If the problem is of interest to numerous users or would be especially painful when encountered, we’ll document the problem in the Known Issues List for that release, a live web document which is updated as we become aware of new issues. We have been maintaining these documents since LabVIEW 8.5 and we’ll continue to do so for new versions of LabVIEW and other NI software in the future.

The LabVIEW 8.6.x Known Issue List

The LabVIEW 8.5.x Known Issue List

When the CAR is fixed in a new release, the CAR number will then be documented in the Bug Fix List for that release if we think the CAR would be of interest to users.

The LabVIEW 8.6.1 Bug Fix List

The LabVIEW 8.6 Bug Fix List

The LabVIEW 8.5.1 Bug Fix List

The LabVIEW 8.5 Bug Fix List

We are continuing to look into ways to make more information available to our users.

Roy plays a very important role in our development team – you can bet we’ll hear from him again.  In talking with users, I know there are more things you would like to see from us regarding bug tracking and reporting.  We are always discussing ways we can make this information available directly to you on the Web.  For now, here are a few questions to spur some comments from you (I have a pretty good idea of what you want next, but let’s open it for comment):

1.  Are you satisfied with our CAR tracking process and fix rate?  Our current approach is to make sure you have a workaround and then you then work the fix into a future release.  Do you feel like we do a good job at resolving the right issues?  How would our being more transparent help you?

2.  Do we document the right known issues?  Is that a document that you refer to while developing?  How would you like to see that documentation improve?

3.  Once a bug you reported is fixed, do you want us to contact you?  With our annual release cycle it can take several releases before your CAR gets implemented.  Are you satisfied with just a workaround, or do you see value in us “closing the loop.”

30-Second User Poll: LabVIEW Distribution

The economic situation is pretty daunting right now.  I am sure many of you are feeling it directly in your businesses.  NI is feeling it too, and we are looking for ways to be more efficient.  In past downturns, and again today, we have prioritized our investments in R&D – to ensure we can continue innovating and improving the products you depend on – and in Field Sales Engineers – to continue building our team of experts who consult with you on the solutions you are building. 

With that in mind, we are looking for ways we can save without negatively impacting the customer experience with our products.  One area where I think we can do a better job is in the distribution of our software.  Rather than spend a lot of money and effort in packaging and shipping LabVIEW boxes to you, we could invest in our infrastructure to make downloading LabVIEW faster, better, and easier.  Moving in this direction would not only be more “green” but would also save some shipping and freight that could be redirected back into product development.  I thought I would experiment with a poll to capture your thoughts. 

What are your opinions on LabVIEW documentation and distribution:

Resolution #1: Get More Involved in the LabVIEW User Community

For years, NI’s LabVIEW leadership (Marketing and R&D – yes, that would be me and my compadres) have taken a fairly hands-off approach when it comes to commenting in the LabVIEW community – in particular, forums on external sites like LAVA or INFO-LV. I am not talking about the typical technical questions, but more about the posts that are controversial – like when people start questioning our intentions or believe that we have made an outright wrong decision on some policy or other. In the past, our approach has been to sit back and rely on our very active and supportive user base to jump to our defense rather than ourselves because it might come across as NI “tampering” or being “heavy handed.” In reality, that is an “old school” approach that we need to change. I believe that today, we need to be more open to jumping in and sharing our perspective on these discussions where we can (sometimes we can’t comment on revenue or strategy issues that are not public yet). Many users often wonder aloud in these discussions if we (NI management) are listening. We are – many people monitor INFO-LV and Lava forums here at NI. When flames spark, we sometimes fail to act. I don’t expect to change anyone’s mind, but just the fact that we are listening and sharing our perspective is a good start. I hope to use this blog to share these perspectives – whether in reaction to a particularly spirited discussion we see on the forums or in discussions with customers, or as a premptive strike on topics where we are looking for feedback.

I shouldn’t be too critical. About four years ago, we made some pretty significant changes to our End User License Agreement to accommodate home use and use on multiple computers that was spurred on by an INFO-LV discussion. So we try to do what’s right most of the time – we just don’t always close the loop with those of you who are chirping on the forums.

The truth is, for most of our customers, it’s not about the big policy decisions or strategy and positioning that is interesting – its mainly about the technology and how you can get the most out of it. This is another, more relevant example of where we are planning to get more involved. We have recently developed our own “blog college” training session to get some of our R&D engineers prepped and ready to enter the blogosphere. You will see new blogs in 2009 authored by some of our engineers in areas like developing large applications in LabVIEW, Embedded Control with LabVIEW, Advanced math, signal processing, and simulation with LabVIEW, and general LabVIEW development techniques. We want to show you techniques to maximize your investment in LabVIEW and expose you to some of the newer areas you can use LabVIEW, directly from the desks of our experts.

Stay tuned.

 

2009 LabVIEW New Year’s Resolutions

(Confessions of an NI LabVIEW Guy)

I, like many of you, are taking some time at the end of the year to reflect on the successes of 2008 as well as the gaps that still exist.  In thinking about next year’s goals, I thought it might be fun to share some of these with the LabVIEW community at large to get your feedback or commentary about.  Most of these address areas where I believe we have some shortcomings today, hence my “confessions” sub-title.  

For now, I will simply list my resolutions.  I will use a series of subsequent posts to explain in more detail each of these and what I hope to accomplish in 2009.

1. Get NI more active in the user community
2. Provide better support for third-party tools
3. Be more transparent with our roadmaps
4. Clearly document our “support” and product lifecycle policies
5. Get serious about addressing upgrading and deployment pain
6. (an extra bonus) Develop new, more effective ways to capture user input and feature requests (and act on it!)

I will follow up with more details about what each of these means in the near future.  For now, have a great holiday season.