How I might put a check into the build to make sure an invalid VM size is not accidentally uploaded?
Background: I recently became aware of just how easy it is to inadvertently configure an Azure WebRole instance VM size to an 'expensive' option on Azure.
At the risk of appearing very stupid, and for those who are interested, this is how I did it... I was exploring the settings of my WebRole in Visual Studio, and I must have changed the default single 'ExtraSmall' (AO) VM size to multiple 'ExtraLarge' (A4) which when next deployed, resulted in my usage jumping to many dollars a day, and running out of Azure credit quite quickly ... Doh!!
It got me thinking about how I might put a check into the build to make sure an invalid VM size is not accidentally uploaded?
After considering various powershell, and azure api approaches, I decided the quickest and easiest way was to add a pre-build event on the Cloud Service project (the one that has the .csdef file) as follows...
for /F "tokens=5 delims=<>= " %%n in ('findstr "vmsize=" $(ProjectDir)ServiceDefinition.csdef') do (if "ExtraSmall"==%%n (exit 0) else (exit 1))
That's it!
For those who want to know how it works... this pre-build event is a command line/batch file command that can be placed directly in the pre-build events of the project, on a single line. It opens the ServiceDefinition.csdef file and finds the line that contains the text 'vmsize='. This returns the following line...
<WebRole name="WCFServiceWebRole1" vmsize="ExtraSmall">
Then, using the following characters as delimiters (<,>,=,{space}), it takes the 5th token from that line, which in this case gives us...
"ExtraSmall"
...It then compares that value against the literal string "ExtraSmall", and returns errorlevel 0 if it matches, or errorlevel 1 if not. returning errorlevel 1 fails the build and stops any subsequent package and deployment from resulting in the wrong VM size from being deployed in Azure.
Related
I have an c# wpf application and an additional installer project to create an installer of the application. Among many other files, the installer project has a installer.cmd file (I took over it from someone else) and a Product.wxs file. I was able to create an installer by running msbuild tools\installer.cmd /P:Library=net461 /P:Platform=x64 command on the developper comand prompt.
I was trying to create a desktop shortcut in all user's desktops and was playing with *.wxs file for that. I know you guys will roast me because of the following sentence but I don't know what else can I say. Something I did break something and somehow the msbuild starts to give error. I cannot say what I did because I tried many different things.. I went back to original project but it didn't help. So what is done is done. I will talk about the error now.
The error itself doesn't tell much:
C:\workspace\QToolInstaller\tools\installer.cmd(117,5): error MSB3073: The command "nuget Source Add -Name PkgSrc_638107674121812897 -Source C:\workspace\QToolInstaller\exports" exited with code 1.
I checked online to understand this error code but there wasn't much. However there is another line just before this erorr line:
The source specified has already been added to the list of available package sources. Provide a unique source.
Well, this wasn't in usual warning color or marked as a warning. But I think it is worth to investigate.
So the "source" mentioned here is PkgSrc_638107674121812897. I scanned the installer.cmd to see where is this coming form and saw that this file is named by using time ticks as:
<LocalSource>PkgSrc_$([System.DateTime]::UtcNow.Ticks)</LocalSource>
Although I ran the msbuild command multiple times, the name of the source is always different. I ran the command in diagnostic mode by using -verbosity:diag option. And I checked the file name among all the logs. There is no match. I tried to change the name to something ridicolous that cannot be listed anywhere but got the same error. Removed alll the files, created them again, restarted the PC etc. Nothing have changed. This name is really unique and but still I am getting this message. I wish I had logs form previous successfull runs. I cannot tell if this is the cause of the failure or not. Does any of these ring any bell fellas?
It is solved! Luckily a colleague of mine have faced with this before. I hope no one will have to deal with this problem again.
So apparently there is a bug(?). Altought that name is unique it thinks the opposite. When I do list the sources with thw followng command
nuget source
among the one that should be there, there is one extra line as PkgSrc_638107643440982987. Although my installer.cmd suppose to remove the source every time after it is done, it didn't for once. And although the new source is named differently, it doesn't care and gives that non-warning, non-error message followed by a meaningless error code 1 message. How to solve it?
We tried to remove the source by command prompt like this:
nuget source delete -Name PkgSrc_638107643440982987
But it is not deleted. So we went to the Visual Studio->Tools->NuGet Package Manager->Package Manager Settings->Package Sources. Found the guilty source and remove it. And then it is resolved.
I divide this to two issues , but solving any one of them will solve the other.
Issue 1:
When I use the section at the bottom of the following picture and add "configurations to release", the build is not triggering release management.
I queried the build logs for hours, and saw that it stops the release when it founds out that the configuration "does not match current"
If ConfigurationsToRelease Matches Current
Initial Property Values
Condition = False
Final Property Values
Condition = False
Final Property Values
Condition = True
before more words, I will show a picture that sums it up:
the build definition and ms build logs
in a default case, where configuration to release is blank , release management would continue from this point and write in the logs "Release the build" (as a command that happened) and the build would trigger release management.
If you look at my tfs build configurations, you can see it's exactly the same as the upper regular ms build configurations , but still I get this error of a mismatch.
In any case I only have only one dialog to fill my configurations at in build definition as shown in the 'configuration dialog' in the above pic.
I succeeded to release once or twice this way in this same project the other day by adding release configurations, it somehow worked, but then stopped working (worked once or twice as a glitch I think like something is cached there) but 99% of the other attempts failed, it was always stopping my trigger from tfs to release since the first time I tried it a few months ago.
Has someone here experienced it? I looked in a lot of places and spotted only one guy that complained about it. His solution was to remove it (not exactly a solution)
Is there a build argument that can fix this? (/p:something=something)
Issue 2: if anyone can solve it in a way different than rm configurations , then I don't need issue 1 to be solved.
For any one who is interested why I even messed with release configuration section of build definition, it's because I want rm to wait for all transformations to happen before rm intervention, and this seems like a way to tell rm , ok dude , see there? you got 2 configurations to wait for their build.
The thing is that by default when that configuration section is blank, rm is getting in the way of tfs build and tfs build is getting in the way of rm, something like a circular wait. rm expects both transformed folders to exist in build outputs when tfs build is waiting for rm to finish it's run after the first build, tfs wants to continue building the second configuration (and transform) but rm is involved already , seeking for it's second configuration, breaking the build when fails to find it, hence the second configuration will never be created, while rm is still waiting for it and tfs build waits for rm and build breaks. confusing? read again and see the pic below cause it's interesting enough.
more info for clarity:
the next stage of RM is trying to get something from build outputs folder before it's already there.
for example , if i set Release build to true , it will only build the first configuration (a folder is created, picture below) , rm will succeed in first step (QA.Release), and continue strait to try and grab Release for its next stage, but it's not built yet by TFS Build , which waits for rm to finish it's weird intervention in the middle of tfs's build work . and like i said above, I'm sure I've seen it work once or twice in one of my attempted builds.
the tfs output folder whith release build flag on(only one transform happens) \ when off (all two transforms work) + rm error when on (circular wait)
If I understand your situation correctly I think your problem is that you are building multiple configurations. This breaks one of the core tenets of continuous delivery which is that for a given release you should build only once and deploy that same build to each stage in your pipeline, in sequence.
In order to do this your build needs to be stage agnostic, which in practice means all your configuration (eg database connection string) needs to be tokensised so that the correct value can be swapped in for the particular stage (QA, Release etc). I have a blog post series here that explains the full process (for a sample web application) in great detail.
i managed to solve issue 2 , and by doing so, issue 1 is no longer relevant .
i switched to ReleaseTfvcTemplate.12.xaml as a build template (found on your C:\Program Files (x86)\Microsoft Visual Studio 12.0\Release Management\Client\bin)
and then the build finished all configurations before RM intervention .
truly seems like a bug in ReleaseDefaultTemplate.11.1.xaml , or this might be because i was using some additional msbuild arguments (which were needed) , or the fact i'm using slow cheetah to create transforms on a windows service (transformations are only introduced in web applications).
either way , i'm now able to perform advanced tasks , like using the transforms to add \ remove tags that should be different in production , for example , i can leave the use of diagnostics in the configuration file for qa use , and remove it for production to make the verbosity lower there . still am using RM PlaceHolders technique in conjunction to transformation technology , to enjoy both worlds where it comes to changes related to environments , but still keeping the principles by passing the same build (dlls) through all environments .
There are some registry keys that are owned by System, and I can't edit them unless I run as System. I want to be able to edit these keys from my VB application. Any tutorials I've found suggest using PSTools, the now deprecated At command, or schtasks. PSTools seems great, but its license seems to restrictive (about redistributing). At and schtasks are too messy, and require scheduling delays.
This is related to my previous question. I want to take what I discovered, and implement it via VB.
Modify audit policy (group policy)
I also found this, but am getting error 1314. I realized despite running VS2010 "requireAdministrator", and even Running my EXE as Administrator, it still says my username in Task Manager (running as me, not Administrator). I logged in as Administrator, and got Error 5 instead. Even after ensuring I had the rights set, as suggested by this post (CreateProcessAsUser error 1314), it still gave me Error 5.
Using a vb.net application running as SYSTEM, how do I start a detached process for each logged on user?
I changed the example to "TokenAccessLevels.Read and Duplicate" rather than MaximumAllowed.
If Not DuplicateTokenEx(hToken.DangerousGetHandle,
TokenAccessLevels.Read & TokenAccessLevels.Duplicate,
Nothing,
SECURITY_IMPERSONATION_LEVEL.SecurityImpersonation,
TOKEN_TYPE.TokenPrimary,
phNewToken) Then
I get Error 5 from CreateProcessAsUser. This is while running VS2010 as Administrator.
http://forum.sysinternals.com/tip-run-process-in-system-account-scexe_topic16714.html
This post was the best I could do. Basically, install a service, run it, then delete it. According to the OP, this is his explanation.
So how does it work?
The & symbol tells cmd.exe to parse treat the text that follows as if it were a new line in a batch file (It basically is a new line
delimiter which allows multiple commands to be combined into 1 line)
To break it down: Create Service - sc create -- binPath= "cmd /c start
calc" type= own type= interact Start Service net start -- (This could
also be done with: sc start --) Delete Service sc delete --
Variables
Service Name: --
App to run: calc
Variables:
How it works:
cmd /c - allows us to pass in parameters to cmd (Without this initial
part, it will work if you copy & paste in cmd.exe, but not in the Run
Dialog. So this fixes it) sc create binPath= - Since cmd.exe does not
respond to service commands, when the SCM runs the app in binPath (Reg
= ImagePath) then it will terminate it, when not responding in a timely fasion, therefore cmd.exe cannot be used. It has to call
something else which in this case is calc.exe sc create type= This one
took a while to figure out. The inital problem is the Window Station
in which cmd.exe is launched in (which in turn is inherited by
calc.exe (its child process)). Luckily after reading Mark's Windows
Internals e4, I was able to solve it by specifying the service as
being Interactive. (Experimenting, it actually has to be BOTH
interactive(256) and own(16) (256|16 = 272) . Basically what this
allows is for the windows to run in \WinSta0\Default (The current
user's desktop, allowing the window to be displayed.) After some
research from being frustrated that sc would not accept type
=own|interact, I found out that it allows us to specify it again, and instead of overwriting Type (dword) it bitwise-ORs it (Adds it).
Problems Solved! net start - start the service (probably calls
StartService) cmd.exe runs with the command line (CL) of start [File]
in which start probably calls ShellExecute (Its ashame that MS didn't
allow start to specify a SW_* commands (like hide). Although it does
allowing us to min/max windows. cmd.exe opens the app/file, the SCM
terminates cmd.exe for not responding in a timely fashion to its
commands, and the window is now shown to the user. sc delete - Finally
we clean up our path by removing the service
For me, this worked.
cmd /c sc create -- binPath= "cmd /c start app.exe" type= own type= interact & net start -- & sc delete --
Now, I had trouble with getting absolute paths to work. I had to put my .EXE in System32 and SysWOW64, so I didn't have to use an absolute path. According to the site, this is supposed to allow absolute paths to work.
cmd /c sc create -- binPath= "cmd /c start \"\" \"C:\windows\regedit.exe\" " type= own type= interact & net start -- & sc delete --
It never worked for me, as it would hang for some time and never start the app. It should almost instantly complete.
I have managed to do Continuous deployment for my Web project using TFS Msbuild.
I have goggled for few hours but couldn't find a relative link to achieve Continuous Deployment for windows service.
Possible to do CD for windows service using TFS Build Definitions? i.e for every check in below steps should be performed, I am using TFS2010 with Windows Server 2008 R2
1] Stop Service,
2] Copy respective Project folder from (Source) Build server to (Destination Server)'staging server1' or 'staging server2'
3] Start Services (willing to do this step manually)
Any blog,tutorial references to achieve this? My guess is need to use Power shell scripts but not sure.
Should be ok, you'll need to install an agent on the box you're deploying to. And you'll need to be able to exit the XAML templates (you'll probably want to copy your existing template that does your build and just add the stop/copy/start stuff onto the end of it).
After your CI build, you'll need to edit it (the XAML template) to start and stop the service you can use the "invoke process" activity (you'll probably want to do something like make it generic and pass in the service name as an argument - note you can change the display names etc in the Metadata argument so it appears meaningful in your build definition).
As far as copying stuff across goes, you can do this fairly easily by accessing properties like the drop location.
Should be fairly straight forward - once you get your head round modifying the templates!
Edit:
Sorry for not responding sooner, I'd have to revise my earlier comment, this isn't as straight forward as it seems unless you really know what you want, I have been thinking about this and like skinning cats, there are more than one ways to achieve this... I've rewritten this a few times so I hope the edit's make sense :)
Boils down to the following:
1) Pass into your template the build agent/machine you want to run this on (this can be done as a simple string, or as an AgentReservationSpec - up to you), since it's unlikely to be the machine that you run your actual CI build on. This is done in the Arguments section of the XAML, as noted before, if you want to edit the display name/description you can edit the Metadata Argument. This machine needs a TFS agent installed of course.
2) Run the task on the remote machine, this is done by adding the Agent Scope activity into your template, you will have to use the info from step 1 to get the ReservationSpec (so would be easier if you add the argument as an AgentReservationSpec or you'll need to resolve this in the template)
2.1) Run the stop/uninstall, this is done via dropping in a (two actually) Invoke Process activity, Invoke Process can take arguments and you need to point it to the executable you're executing, so you'll want to use this, one for the NET command (i.e. NET STOP ), and one for InstallUtil.exe.
2.2) Copy the files from your CI to the remote server, you can use the Copy Directory activity for this, it needs a couple of parameters, the main one is the source location, you should be able to drop in a GetBuildDetail activity, give it a name then reference .DropLocation to get this, destination is wherever you like you're installing to.
2.3) Install the new service as step 2.1, you need to use Invoke Process to install the service, then you can use another to start the service up.
I haven't covered everything, but I haven't set this up myself so I'm sure there are a few pitfalls or things I haven't though of. Off the top of my head this makes sense, but maybe someone that knows better can poke a few holes in it :)
I have a build process setup to build a managed solution using team build. This solution requires an unmanaged component be registered with the server before we build the solution as we interface with it via COM.
The activity (InvokeProcess) I use to register the ComObject looks something like this
regsvr32.exe /s ComObject.ocx
The activity (InvokeProcess) I use to unregister it looks something like this
regsvr32.exe /u /s ComObject.ocx
I have also added the WriteBuildMessage and WriteBuildError to both InvokeProcess activities using stdOutput and errOutput as the message for each action. I also make sure to set the build message importance to high.
It's my understanding that this should redirect the standard output and error output into these logging activities.
The registrion, build, and unregistrion works just fine as long as I do not set the /maxcpucount argument of MSBuild to anything greater than 1.
Once I set it to something greater than 1, our cleanup script at the end of the process fails with this error message when attempting to delete the file in question.
Access to the path '...\ComObject.ocx' is denied.'
I think what's happening is that the unregister activity is failing to unregister ComObject.ocx from the server because MSBuild isn't done with it when it's running across multiple cpu's. Then when I get down to try and delete it from the build server, it's still registered with the system and will fail with the access denied error.
So how do I get regsvr32 output to the stdOutput and errOutput so that the WriteBuildMessage and writeBuildError activities will properly display it in the build logs. If I call regsvr32 with a file that doesn't exist, I see nothing in the build log.
I hope this is makes sense.
Update
The solution to this problem came from a minor tweak of pantelif's solution. What I did was in the if block that throws an exception, I do a WriteBuildError using this as the message:
String.Format("ErrorMessage: {0}", New System.ComponentModel.Win32Exception( System.Runtime.InteropServices.Marshal.GetLastWin32Error() ).ToString() )
This allows me to get the error from regsvr32.exe and write it to the build log.
As for your first question:
I believe it's possible to capture the output, by applying the technique presented by E.Hofman here.
More specifically, if you construct within your 'InvokeProcess' a sequence like presented (pic also from Ewald's post):
& then assign 'WriteBuildMessage' to stamp the 'ErrorMessage', you shall probably end up with getting the output you desire in the generated build log.