Hi gang.
We’ve been considering features that will enable out of the box ease of use for the first time Isolator developer. The big question is -
“How do you make that first time after-installation experience usable, discoverable, and actually make the devloper use and “grok” Isolator in 10 minutes or less. just to get the essence of it. The rest can be then handled by docs and forums and blogs.
Here are the top five suggestions we came up with and we need to choose one to start with that will bring the most value:
- “Fake this” : Be able to select a piece of production code in the code editor (usually a line or two) and then right click and select a new menu item “Fake this”. This will generate Isolator code that fakes that specific call(s). Then put it in a new test or put in the clipboard.
- “Generate test from Breakpoint” : you are in a debugging session and you arrive at the method where a bug exists. You right click in the method during debugging and select “Recreate this scenario using Isolator” or something of that nature. You will get an isolator test that has all the code needed to get to that piece of code with all the needed parameters and the system state in that matches what the code in the method exists. the test also invokes the method (alternatively – it invokes the first public method in the call stack that leads to this scenario)
- “Quick Fix for tests”. This will use resharper style quick fixes for your tests for test conventions as well as isolator conventions (like “you’re better of faking a static method here”)
- “Dependency walkthrough” (we need a much better name for this). The developer writes a standard unit tests calling the APIs they need, which could may well be an integration test but they don’t know it yet. Then right click on the test and select “analyze and generate dependency behavior”. This makes isolator repeatedly run the test in question. everytime isolator catches an exception, it will:
- show the current callstack to the developer and highlight the last call that threw the exception with the parameters sent to it
- allow developer to decide on one piece of the callstack what it’s fake behavior will be/ (for example – they will select that call and right click it, and select “ignore call”
- After this isolator re-runs the test with the new behavior saved in memory. (go to step 1)
- if the test finishes OK, the developer can then “save” the behavior they defined in the current test (the code will be generated)
- “Debugger fake notification” : When running an isolator test, during debugging, everytime you debug through a faked method a popup icon will say “you just stepped over a fake method” with some details about the method. maybe this only happens the 2nd time they debug through that method.. we don't’ know yet.
- If you were to choose just one of these, which do you think would add the most value to new users? (please answer the poll below)
- Can you think of other features which could be even more valuable? (feel free to add comments!)
The poll below is in Flash. If you can't see it, here is a direct link to the poll page.
Could you explain about #3, what will Isolator know?
ReplyDeleteEli, how about these quick-fix suggestions:
ReplyDelete- This test is using Isolator but is missing the [Isolated] attribute - quick fix is to add it
- The name of the test does not contain the name of the method under test - quick fix is to rename and suggest MUT_OldTestName()
- The code under test does not contain a call to the method you are faking - quick fix: remove the redundant WhenCalled()
I think we could build a ton of useful rules around this. These do not have to be Isolator specific but also provide unit testing guidance as well.
I think that by putting, Resharper Style, we are selling Resharper and not really the features/benefits that we are giving our first-time users. There is only so much that can be done with static code analysis.
ReplyDeleteEven the third rule that you suggested is not possible to do, unless the test is run already.
#4 is actually #3 but with Dynamic Code Analysis, ie. Run the code and bring up a wizard to help fix the test.
Doron's suggestions all rock - I would LOVE to see those. I also agree with what Eli said in second post.
ReplyDeleteHere's what have been my biggest hurdles:
1. Lack of examples in AAA in general - only the most basics are there and they are more of a 'cook book' rather than a 'here's why you would do it this way' - which would be nice addition to the end of those; along with (more) 'alternatives' when applicable. In my mind everything available in Reflection and Natural should be in AAA and those two should be deprecated. This would allow you to focus on 1 style (as well as users) - that was very confusing for me trying to learn two syntax's at once (Natural/AAA)
2. The 'videos' are what helped me grasp the concept when I first started quickly. Do more of these; ASP.NET web site rocks with all the videos they have.
3. Forum is very difficult to find things on. Occasionally when I have found something relevant there would be no follow up or a 'let's take this off-line' which is frustrating - taking it off-line doesn't help the rest of us - once an answer is found the solution should be posted. I know that's a nature of a forum, but it's still frustrating none the less.
4. Advertise Roy's book which will has helped my testing style/structure more than anything ever has in the past 5 years. When you are trying to write good tests to start with, Isolator becomes much easier to use. I don't know if that would be a conflict of interests, but that's my thought.
I'm all for the "test is using Isolator but is missing the attribute" fix that Doron proposed. I'd vote for that.
ReplyDeleteThat said, such a fix would be huge for existing Isolator users, but wouldn't accomplish the goal of helping a new user figure out Isolator sooner; finding missing attributes is more something helpful after that first 10 minutes and ongoing.
For the brand new user, I think the "fake this" item would be the best. The rest of the items are cool but wouldn't help that first time user as much as they would help someone who's already into it.