Testing.Platform, the new way to run .NET tests

Summary

Highlights

• Testing.Platform is designed to replace the older vs test for .NET testing. 🎯
• Key goals include making the platform deterministic, ensuring test consistency across different environments. 🏗️
• Enhances portability, making it easy to reproduce tests irrespective of the surrounding environment. 🌎
• Future-proof design integrates easily with new form factors and architectures, minimizing adoption effort. 🔍
• Extensible platform allows easy integration of additional functionalities and custom testing frameworks. 🔧

Key Takeaways

• Testing.Platform is the new way to run .NET tests, set to replace vs test. 🚀
• Built on six pillars: deterministic, portable, future-proof, extensible, performant, and secure. 🛠️
• Offers better cross-platform support and integrates with Docker without needing a Docker file. 🐳
• Supports native AOT compatibility and offers extensions for enhanced functionality. 🔌
• MsTest, NUnit, and xUnit are supported in preview versions for seamless transition. 👥

Overview

Testing.Platform, the new way to run .NET tests Transcription

• 00:00 - 00:30 [Music] hello welcome to my talk this think platform the new way to Run net test I'm yakob and I'm happy to be here on behalf of the net testing team and to be able to show you what we've been working on so testing platform is a new lightweight platform to eventually replace vs test vs test was created in 2012 and was
• 00:30 - 01:00 introduced in Visual Studio 2012 together with test Explorer and it was created as a platform to offer unified experience for testing framework and tests and running tests in visual studio so you can create the same reports you can run tests the same way in Azure devops through the vs test task no matter which framework you are using it being nunit xunit Ms test or something else at that time Visual Studio was on
• 01:00 - 01:30 Windows we had doet framework that was on Windows we didn't have a cross platform experience that came in 2017 with net core and at the beginning we probably didn't know how amazing change that will be for theet ecosystem for net core support for vs test was added and we got net test and net vs test commands but there was never a proper rewrite where everything would be done done in a way that it should be done in
• 01:30 - 02:00 net core in 2019 our team was established and we saw some cool additions to the net ecosystem such as arm64 or native aot and when we were adding support for those into vs dust we found ourselves struggling with that a little bit and so we started thinking how we can change the test platform to be more aligned with doet test and we started by writing one from scratch
• 02:00 - 02:30 which we now released in 2024 and called it testing platform and we released it together with Ms test 3.2 when we were creating this new testing platform we built it on six pillars to be deterministic portable future proof extensible performant and secure let's look into each point one by one so
• 02:30 - 03:00 deterministic for us that means that running the same tests with the same configuration should produce the same result it should produce it on your machine on build server on your colleague's machine that is helping you figure out the problem with your test or maybe on our machines when you are sharing a Repro of your problem this was driven by customer complaints of course and so one such complaint might be my tests are crashing because they pick up wrong version of
• 03:00 - 03:30 test adapter dll from new get cache and so this kind of mix and matching of different dlls inside of Bin folders inside of other places where they are picked up by the S test is a common source of problems and so we got rid of that totally now we have no such thing we only have static extension registration that we do through Ms belt and we don't have any reflection lookup we we don't have folder scanning
• 03:30 - 04:00 we don't look for files based on naming convention everything is very static and we also don't have any custom assembly resolver inside of the platform which is another source of lot of problems we want the platform to be portable so when the project or the tests are portable we want them to be detached from their surrounding environment so for example a complaint would be my are failing on the server
• 04:00 - 04:30 but locally they work and I cannot debug the problem and I don't know about you but I really hate that situation when it only reproduces on server and so when you are downloading your test from your server you want to just grab the bin folder and that should be enough to reproduce locally you don't want to be thinking what exact version ofet SDK I have installed on the server what exact version of Visual Studio I have installed there what exact packages I have restored in the new get cache and
• 04:30 - 05:00 all of those can influence the run with the current test platform also in some Legacy or in some backwards compatibility scenarios we are sharing adapters between projects with the current platform but not with the new one so everything should be more portable and much easier to reproduce on different systems we want our platform to be future proof and so because we saw a lot
• 05:00 - 05:30 of those amazing improvements in net and so we want test projects to not be special in the ecosystem anymore we want them to be easily manipulated like if you would be manipulating a console application and that way new form factors and new architectures should require minimal effort to adopt both on our side so we don't have to do a lot of work and on your side so you can very easily start dog fooding new things you can use them as soon as they are
• 05:30 - 06:00 available and you can also test them as soon as they are available and so for example with Native aot with the new platform we created it from scratch to be native aot compatible and we also added support into Ms test for native aot in a preview another complaint from a customer was doet Trace is the net Trace is a performance investigation tool in net and it's very hard to to use with
• 06:00 - 06:30 tests and so now the test projects inside of the new testing platform are just simple x executables and what you can do with a console you can do with tests and that's really really important point and we will demo that soon then we want the platform to be extensible and so internally it's built around extensibility a lot of the things it does are extensions internally and we also have extens that are external such
• 06:30 - 07:00 as hang dump that we will see which is similar to the blame functionality inside of vs test and so we want us and framework ERS and service providers to have easy time providing additional functionality that will be added to the current testing platform so for example a customer complain was my tests are flaky how do I retry them and so we created an extension for retrying tests and other extensions that help
• 07:00 - 07:30 also we don't want the new platform to depend on common libraries such as in vs test we rely on Newtons of Json and so when someone wants to try the auor of Newtons of Json tries to test Newtons of Json how does he do it well it can to be done with with vs test because it already depends on the version of that library and so the gore platform the new testing platform doesn't have any dependencies upon part from what ships inside of the
• 07:30 - 08:00 runtime so that way you can test hopefully everything we want our platform to be performant because you shouldn't be spending time running the platform itself you should be spending CPU and memory on running your tests and so the overhead of the testing platform of the new testing platform is minimal so a complaint might be test running is slow vs do console takes too
• 08:00 - 08:30 much memory and so the new platform reduced the number of processes that we start and the serialization between processes to a minimum and so now teams see 30% increase in performance just by moving to the new testing platform and doing nothing else and then last but not least we want the platform to be secure because security is very important and this of course needs to be
• 08:30 - 09:00 attacked from multiple places but we minimize the dependencies so detecting or having vulnerable dependencies is difficult when you don't have any and if we do have dependencies that are vulnerable maybe in some extension or if there is a problem found directly in the platform we ship everything the whole platform as net packages so we can go to net we can mark it as having a security problem problem and you will get a
• 09:00 - 09:30 warning in your build and also we can very fast ship fixes maybe even the next day so that's a lot of talking let's go to showing some demos so let's switch for that into my visual studio code and here in Visual Studio code I have a new project that I created through a template net new Ms test and nice p by testr Runner Ms test by saying D-
• 09:30 - 10:00 testrunner Ms test this created this project and it has two new properties enable Ms test Runner Ms test Runner is a new Runner that is based on the testing platform and is meant for Ms test as the name suggests and then the output type of the project is set to xce to executable and this is very important because then you can take the application and you can do what you can do with a console
• 10:00 - 10:30 application you can do with a test application so let's go and let's build it so we do do net build that builds the project any second now and we got a dll we can copy it here and we can change death to xce run it and it runs my tests very fast T I didn't have to look up any
• 10:30 - 11:00 other tool I didn't have to run with net test I didn't have to even have the SDK available once I have this application so I can take this app I can move it to a system that doesn't have net SDK and I can run it there I can also use net run to run this app and I'm not going to specify the parameters just yet and this is a convenient way of rebuilding your
• 11:00 - 11:30 application and running it as well so if you are changing stuff inside you can just run your app this way it will be rebuilt and you will run your test but again we are not bringing in any additional console we are not bringing any additional software this just runs as a console application and it runs the test inside of it so to show this point even further I want to show you how we can move those
• 11:30 - 12:00 tests to run inside of a Docker container for that I will switch into my my folder 02 Docker rise I have a project that looks totally the same as before I just added two properties based on the documentation and I will be using a functionality that was added into net build and it wasn't thinking about how to use it with test but I was thinking how to use it with console application
• 12:00 - 12:30 so now I can use this target publish container and that will without any knowledge of Docker without any creating of a Docker file it will just create an image for me that will have my app and I will be able to just run it directly in Docker so documentation tells me use those two properties is publishable and enable SDK container support I added both of them copying them from the
• 12:30 - 13:00 documentation and then I can run net publish publish container that's the Target that I need to call and then I'm calling ddl false which will temporarily just disable my terminal logger so I can see more information on the screen about what's going on and what I want you to see is that we're basing this on net runtime not net SDK net runtime and runtime is just barely enough it's just enough to
• 13:00 - 13:30 run an application but you cannot use runtime to develop and build the application so that's what we are basing the container on we have just the runtime and it created this hello world test for me and then I can go and call Docker run hello world tests image and just like that it runs my test inside of a dark container you can see here that we are using Linux so we're running running inside of a Linux
• 13:30 - 14:00 container directly here on windows so this easy it is to take the new testing platform test app move it inside of Docker I didn't have to write any Docker file I didn't have to have any knowledge and now I can just go and run my test on Linux very easily somewhere on a server so what I want you to remember from this demo is that the test projects in testing platform are now built as executable and so you can manipulate them you can
• 14:00 - 14:30 combine them with other tests the with other applications with other applications like like you would combine a console application so you can use doet trays you can put them inside of a container and so on now let's demo some extensibility and Ms test project as thek so for that I will move into this folder I will just quickly run this to
• 14:30 - 15:00 switch to the correct folder and then we will again look at the project so in this case I'm having the same kind of project and I have two extensions maybe you also notice them in the previous demos I have Microsoft testing extensions code coverage and TRX report and extensions for the new platform are packaged as new get packages and ship as them so you can very easily install them into your project project but I don't want to be using
• 15:00 - 15:30 Code coverage or TRX report I want to be using hang dump and hang dump is an extension that allows you to wait for a timeout if a test passes this timeout that I'm specifying hang jump will kill the process and it will help me find tests that are hanging and so what I want to show you here is that I have this test class which has a test that runs for 30 seconds and that's longer than 5 Seconds so for me that's a hanging test or test
• 15:30 - 16:00 that's being too slow and then I have a bunch of other tests like 300 of them that are running quite fast they are just sleeping for 100 milliseconds and so here I want to be calling this command so net run Dash Dash and then you have to specify a space this is a kind of a quirk of net run so you do net run space
• 16:00 - 16:30 dash dash space and then you specify the parameters for your app and so I'm saying hang dump and d d hang dump time out 5 seconds because I want the longest test to take 5 Seconds otherwise I want to kill the process and so my tests are running and maybe notice this really nice progress bar that we have now I finished the 300 test and now I'm
• 16:30 - 17:00 waiting for the 302nd test to finish and it didn't finish it expired the time and it was CAU by The Hang dump dumper which dumped the pro process and now I have a dump and so I can take this memory dump and I can take it down from the CI server to my local and see exactly what happened in the process or I can use it locally maybe to see what exactly was hanging and this way I can determine why
• 17:00 - 17:30 my tests are hanging so this one of the extensions that we have and you might be familiar with it already from vs test there it's called blame blame hang but maybe let's now look at the project so we have an extension and we installed it through a package and now we have like three of them and you might be thinking if I have 20 of those extensions that's going to
• 17:30 - 18:00 be very difficult to maintain like there's this version now suddenly code coverage has a different version how do I align them seems kind of difficult and we were thinking the same thing and we were getting also the same feedback and so we came up with Ms test SDK so let's compare those two projects here in the new one we have project and we specify Ms test. SDK and then the version of the SDK I want and I can also generate it
• 18:00 - 18:30 from this template net newms test-- SDK and in the old one on the top I only had the standard Microsoft NET SDK so in the new Ms SDK driven project I have the emus SDK the version I can also specify it in global Json and then I have only a bunch of
• 18:30 - 19:00 properties so the target framework and so on the standard stuff and then I am using a property to just enable the extension which is called enable and then the name of the extension Microsoft testing extensions hang dump and I set it to true and that will install my extension with the correct version inside of the project and then there's nothing else so that's the whole project file and so then I have in the test class
• 19:00 - 19:30 totally the same thing as before I just want to be switching into into this folder and then I will be going here and I will run the same command as before and exactly the same thing as before should happen we run the 300 tests we finish them and then we went then we wait for 5 seconds for that last test to finish be killed collect a hang dump and exactly that thing
• 19:30 - 20:00 happened so msus SDK really cool way of managing your dependencies your extensions inside of your project but I want to show you one more extension and that extension is called hot reload and for that I will have to switch into my visual studio and here in Visual Studio I have a solution which is using the SDK again and I'm using a different version to
• 20:00 - 20:30 enable the extension so maybe if you're still not happy if it's still too complicated to add the enable for every single extension you can use something that's called a test extension profile so by default we now ship three of them we have the default we have all Microsoft and we have none so by this by just a single property you can select multiple extensions to be installed and so in this case I'm using the All
• 20:30 - 21:00 Microsoft which will install all the extensions that we're shipping for the testing platform and that includes the hot reload extension to further enable the hot reload extension I need to specify this environment variable testing platform hot reload enabled and then I can go and I can run my ad just like I would with a console application it's building now for a
• 21:00 - 21:30 second and I want to move it here and move this to the other half and also Zoom this up a little bit so you can see it better and then the last thing that I want to do is that I want to check if I have this hot reload on file safe enabled and I do so all is good I will go into my test class and you can see there is an assertion that says hello
• 21:30 - 22:00 net 9 and I will make it little outdated and say hello net 8 and I will save right now and just when I saved the test refreshed and the test ran and now it's broken because the actual value is outdated so I will delete it set it back to nine and press contrl s and I couldn't even leave the key and the test ran and updated my results so
• 22:00 - 22:30 hot reload is integrating with the hot reload capability it makes running test interactively quite easy you can use it together with filters and it's really good experience of debugging and running your tests so that's hotf let's switch back into the presentation and so we have some cool extensions like hot relo rryy the amazing Cod coverage we have crash dump hang dump we have support Port fakes we have output with progress we have DRX
• 22:30 - 23:00 there is lot of to impac you can write you can read all of that in our documentation but your extensions don't have to limit themselves to reports or some basic stuff you can also alter a completely new testing framework like Tom did with dunit and he was a great driver of the design and of the stuff that we were adding into the test plat into the new testing platforms so definitely go check out that new testing
• 23:00 - 23:30 framework talking about Community provided Frameworks we cannot omit xunit and N unit so let's see how those work together with testing platform and for that I will switch into my visual studio code again and here I have my solution in this folder so I will go here I will
• 23:30 - 24:00 again run this to switch the correct folder and I have this testing platform nvs solution so I will do do net that's not do do belt that will build my solution and I will get those DLS and now I can show you how they run like I did before so we will go and we will um change this to
• 24:00 - 24:30 XZ thank you very much and so here inside of this project we enable an unit Runner just like we did with Ms test so here we say enable and unit Runner we also say output type XE and uh you should be using this preview version of an unit 3 test adapter to be able to do this so to be able to activate the new mode those
• 24:30 - 25:00 packages are just available on net.org so it's very easy you just have to update the version and then we can go and we can run a unit I don't know what I did and that doesn't work oh I have it to false why do I have it to false
• 25:00 - 25:30 so we build one more time we run it and now it runs through the testing platform and it executes my test just like it did before with Ms test Runner so nunit supports in the preview version the new testing platform and we are super happy about that and we can move to xunit so for xunit I will copy this I will
• 25:30 - 26:00 again rewrite that to xce run it and you can see that it ran my test through the testing platform Runner as well so it also works together with xunit and let's look at the project and how we enable it in here the property name is slightly different so it's called use Microsoft testing platform Runner and so that's the property that you need
• 26:00 - 26:30 to use and you also need to use those pre uh preview packages on net again they're available on net.org as you would expect and so nunit and xunit they both now support testing platform but they both require so far the preview versions packages to be used and so if you're using nunit or xunit please go and try those versions and post feedback either
• 26:30 - 27:00 to us or to the respective Frameworks on their respective repost we and them we are very grateful for feedback and for maybe potential fixes and PRS from you so now for the last demo we have to look at Visual Studio because a lot of us use visual studio for writing code and then net test because that's how people usually run tests these days
• 27:00 - 27:30 so let's switch to that and for this I will stay in this folder and the whole time I was telling you don't use net test use net run but let's see what happened if we actually run that test so I run it it builds my projects and all of the projects execute and this is what I was hoping for because we are working hard at keeping backwards compatibility and the testing framework authors are as well and so we are
• 27:30 - 28:00 leaning back on vs test when we running net test so you can migrate to testing platform you can reap the benefits of having testing platform the portability the ability to just easily pack your test projects into a doer container for example but then you don't have to break everything you don't have to break every single flow that you have you can in some cases again rely on the current vs test platform we are working on a new
• 28:00 - 28:30 solution for net test that will be more integrated that will be more native to testing platform but we are not there yet but when we will be we will be replacing again this experience very cautiously so stay tuned then let's switch to visual studio and so here in Visual Studio I have test Explorer I will go and run my test and again and as we will hopefully see in a second the tests pass they are
• 28:30 - 29:00 all green and also in this case we are leaning back onto the current experience with vs test but again we are working on a new experience that will be more native to the testing platform that will have less overhead and so on inside of visual studio and so we will be shipping that under a preview flag maybe in 17713 visual Studio or maybe earlier let's
• 29:00 - 29:30 see so in summary the testing platform enabled projects are still usable in Visual Studio they are still usable in net test you can migrate to the new testing platform right now try it out and uh then fall back onto vs test under those other scenarios to run your test so you don't have to keep you don't have to break everything just to be able to try the new thing and that's it that's all my demos so in summary we would love your
• 29:30 - 30:00 feedback we would love your ideas improvements your issues you can file all of them on the Microsoft SL test FX this is the same repo where Ms dest is living and where now the new platform is living and uh we have discussions enabled we have issues enabled of course and so you do you you can choose whatever works for you uh share the feedback share the ideas with us we are very very happy to listen so that's everything that I have
• 30:00 - 30:30 thank you very much for listening and please go and download do net 9 today because it's released and it's amazing thank you