Are you responsible for a temperamental Windows Service? If so, you should definitely check out the latest version of Service Protector — the easiest way to achieve 100% uptime today.
Here’s what’s new in this release:
Email alerts include recent activity
Customers who have configured email alerts will notice that messages now contain the service’s last five events from the Windows Event Logs. The idea is to provide helpful context when something unusual happens, to avoid you having to log on to interrogate Service Protector’s reports.
Here is what an email with the new Recent Activity section looks like:
Delay the initial Sanity Check when detecting service problems
A customized sanity check is an excellent way to extend failure detection and automatically restart a faltering service. With a sanity check, you can probe network connectivity, check for a “stale” output file, and much more — whatever you like!
Service Protector version 7 allows you to delay the first sanity check. This is useful when your service takes a while to get ready — either at boot or after it has been restarted.
The new delay settings appear on the Configure Sanity Check window:
From the release notes, 20H2 doesn’t include significant changes to the Windows Services infrastructure. The update focused mostly on end-user improvements for the Edge browser, task tray notifications and the like.
Nevertheless, our team tested Service Protector 7.0 extensively on the new version of Windows 10. We’re pleased to report that no problems were detected and Service Protector remains fully compatible with all versions of Windows 10.
As usual, please review the release notes for the full list of features, fixes and improvements included in this release.
Upgrading to Service Protector 7
If you purchased Service Protector version 6 (after February 2019) you can upgrade to version 7 for free. Simply download and install “over the top” to preserve your existing services and all settings. Your registration code will continue to work.
My team inherited a cluster of 4 Windows Server 2012 R2 machines running a legacy finance application. At a time only one instance of the application should be up and running as the active/primary instance.
The application is cluster-unaware so we set it up with the Generic Application type. Most days it works fine but about once or twice per month the EXE stops responding and no failover happens. Someone has to log in and kill it to trigger failover, which is just plain silly.
I see that your AlwaysUp may be able to better manage the program. My question is, can your product work to control and maintain the supervision of these instances?
— Andrei
Hi Andrei.
Several of our corporate customers have deployed AlwaysUp in Windows clusters. They tell us that, despite having no specific features that target cluster management, AlwaysUp works very well in that context.
In situations like yours where 100% uptime of a particular application is important, AlwaysUp adds an efficient line of defense — one that complements traditional multi-machine failover. Here’s how that works.
Application resilience: AlwaysUp protects against application failures
AlwaysUp’s job is to ensure that your application is always running. If your application crashes, AlwaysUp will automatically restart it.
But AlwaysUp provides much more than basic crash protection. It can operate proactively, rooting out problematic situations before they metastasize into full blown failures.
For example, you can have AlwaysUp quickly recycle your application if it:
Monopolizes the CPU for too long;
Consumes too much RAM;
Fails to respond properly to network/web requests;
Stops writing to a log file.
In it helps, you can even have AlwaysUp restart your finance program once a week during off-hours — to fend off mysterious lock-ups and other unpleasant instabilities.
Less downtime when your application fails
Most importantly, your system/service will likely experience less downtime when AlwaysUp is the first line of defense. Instead of waiting for the cluster failure to be detected and the switchover to the backup server to occur, your application will quickly bounce back on the active server. That rapid resolution can shave many precious seconds off your recovery time!
System resilience: Clustering protects against catastrophic failures
While AlwaysUp is able to cure many application failures, there are a range of deeper problems that it cannot solve. For example, when:
The machine loses power;
The server’s operating system crashes;
The network experiences an outage;
A critical hardware component (motherboard, hard drive, etc.) malfunctions.
For those dicey situations — where a server has been compromised — your failover cluster setup will save the day.
Configure a “Generic Service” instead of a “Generic Application”
Since you set up a Generic Application resource type to monitor your important program, you should remove that and replace it with a Generic Service that monitors the Windows Service created by AlwaysUp:
That change will enable your cluster to fail over whenever AlwaysUp protection stops — not when your legacy application fails. That is an important distinction.
If your application is named “Legacy Finance App” in AlwaysUp, select the Windows Service called “Legacy Finance App (managed by AlwaysUpService)”. You can find out more about the service created by AlwaysUp on the Frequently Asked Questions (FAQ) page.
Posted onJanuary 25, 2021 (Revised April 19, 2022)
Not all Outlook functions work from a Windows Service
Calling Outlook from a Windows Service can be problematic. Even though many operations work fine, Microsoft has issued some pointed advice for customers looking to run any Office application in the background in Session 0:
All current versions of Microsoft Office were designed, tested, and configured to run as end-user products on a client workstation. They assume an interactive desktop and user profile. They do not provide the level of reentrancy or security that is necessary to meet the needs of server-side components that are designed to run unattended.
Microsoft does not currently recommend, and does not support, Automation of Microsoft Office applications from any unattended, non-interactive client application or component (including ASP, ASP.NET, DCOM, and NT Services), because Office may exhibit unstable behavior and/or deadlock when Office is run in this environment.
If you are building a solution that runs in a server-side context, you should try to use components that have been made safe for unattended execution. Or, you should try to find alternatives that allow at least part of the code to run client-side. If you use an Office application from a server-side solution, the application will lack many of the necessary capabilities to run successfully.
Very disappointing!
So instead of calling Outlook, which may be unreliable when run in the context of a service, look to one of these alternative solutions instead:
Solution #1: Have your Windows Service call PowerShell to send basic email
If you don’t want to install any third-party utilities, you can leverage Microsoft’s ubiquitous PowerShell utility to deliver your messages. And to help, we’ve created a simple script that, given eight required parameters, will send an email to any address:
Your email provider will determine the SMTP server and port number you should use. For example, if your provider is Gmail, the SMTP server is “smtp.gmail.com” and the port is 587.
To invoke the script from an application, run the PowerShell executable with the -File option. Specify the full path to the script along with the eight required parameters.
For example, if you’ve saved the script in C:\Utilities and you’re sending via Gmail, your command line will look like this:
powershell.exe -File "C:\Utilities\send-email.ps1" from@coretechnologies.com to@coretechnologies.com "Server Down Alert" "Server 'FileServer1' is down!" smtp.gmail.com 587 from@coretechnologies.com "PWD8581JG$"
Watch out for quotes in the subject and body and escape accordingly!
Solution #2: Use SwithMail to deliver complex messages from your Windows Service
SwithMail is a free, no-nonsense utility that can send very detailed email messages. It supports all the important messaging options, including:
HTML formatting
Multiple attachment files
CC and BCC recipients
“ReplyTo” configuration
After downloading the SwithMail zip file and extracting its contents to a suitable location, double-click SwithMail.exe to reveal the comprehensive command line:
Simply specify the options you need. For example, here is a sample command line that sends the same message as the PowerShell script above:
If you’re having trouble configuring SwithMail, add the /Log parameter and a path to a log file. Check the file for error messages after a failed run.
Solution #3: Update your Windows Service code to send email directly
If you have access to the service’s source code, your best option may be to include code to send email using the SMTP classes built into your programming language.
Here is some sample C# code (error handling omitted for clarity):
// Compose the message.
MailMessage mailMessage = new MailMessage();
mailMessage.From = new MailAddress("from@coretechnologies.com");
mailMessage.To.Add(new MailAddress("to@coretechnologies.com"));
mailMessage.Subject = "Server down";
mailMessage.Body = "Server <b>FileServer1</b> is down!";
mailMessage.BodyFormat = MailFormat.Html;
// Construct the SMTP object that will send the message.
smtpClient = new SmtpClient("smtp.gmail.com", 587);
smtpClient.EnableSsl = true;
smtpClient.Credentials = new System.Net.NetworkCredential("from@coretechnologies.com", "PWD8581JG$");
// Send the message!
smtpClient.Send(mailMessage);
Get in touch if you need help sending email from your Windows Service
Hopefully one of these three solutions, which don’t involve Outlook, will work from your service. If not — or if you have questions about the methods outlined above — please don’t hesitate to reach out to our support team. We’re here to help!
Posted onJanuary 10, 2021 (Revised September 24, 2021)
How can I tell if someone updated the services on our Windows 2019 server? Do you have any tools for that?
— Sheldon P.
Hi Sheldon.
Since Windows Services run with high privileges, it’s very important to keep an eye on them. And because of their inherent power, services are a prized target for bad actors looking to hack your system.
Indeed, 2020’s SolarWinds supply chain exploit — one of the worst attacks in the past decade — featured a rogue Windows Service depositing malware in the background. A periodic review of the list of services could have identified the compromise months earlier.
Anyway, our free Windows Service Auditor is an excellent monitoring tool that can help you in your situation. Follow these instructions to keep a watchful eye on your mission-critical servers.
1. Download & run Windows Service Auditor
Windows Service Auditor is portable application, meaning that you don’t need to install it. Simply download the executable file and place it in a folder where you can easily find it.
Double-click the file to start it. In a few seconds, a window listing all your Windows Services will appear:
2. Update your computer’s security policy to allow advanced auditing
By default, Windows does not keep track all changes made to Windows Services. That capability must be enabled via advanced security audit policies. Specifically, you need to watch for:
Windows Service Auditor makes it easy to enable that auditing in your local policy. To do so, open the Application menu and ensure that the Enable Local Audit Policy entry is checked:
3. Enable auditing for important Windows Services, to track who starts/stops/changes them
Do you care about the activities of a specific Windows Service? Even though we have enabled advanced auditing in step 2, you must enable auditing for each service that you would like to monitor.
To enable auditing of a service in Windows Service Auditor, highlight the service and check the Selected Service > Enable Auditing menu entry:
With auditing in place for a service, the Windows Event logs will record an event whenever someone attempts to start, stop or modify the service. And to save you from hours of digging through the Event Viewer, Windows Service Auditor will collect those records in the lower Events panel:
4. Capture a baseline snapshot of all services running on your machine
This short video shows how to capture a snapshot of all the services running on your computer:
To summarize:
Start Windows Service Auditor;
Select All Services > Export (XML);
Choose a file name where the services should be saved.
The file will contain an XML record for each service installed on your computer:
5. Compare future snapshots to the baseline, to identify changes
Whenever you want to check if any services have changed, you should:
Create a new snapshot XML file, as described in the previous section;
Using your favorite text comparison tool, compare the new snapshot to the baseline you established in the previous section.
The text comparison tool will highlight all changes that have taken place in between the snapshots.
We recommend using WinMerge — a free, mature text differencing tool for Windows.
For example, we established a baseline snapshot on December 29. On December 31, we wanted to see what changed with services so we took another snapshot. Afterwards, comparing the two snapshots with WinMerge identified 8 differences, including one showing that the TrustedInstaller service was stopped:
Best of luck managing your system!
UPDATE — September 24 2021: Now find out when a service’s executable has been modified
Windows Service Auditor version 3 includes a few vital improvements that will help you identify when your Windows Services have changed unexpectedly.
In the new version, Windows Service Auditor now captures the following fields that uniquely identify the executable file supporting the service:
path: The full path to the executable file started by the service.
date-modified: The date and time that the service’s executable file was last modified.
size: The size of the service’s executable file, in bytes.
hash: The SHA-256 hash value of the contents of the service’s executable file. This is a fingerprint that uniquely identifies the executable.
product-name: The “Product name” set in the service’s executable file (visible on the “Details” tab of the file’s properties)
company: The “Company” set in the service’s executable file (visible on the “Details” tab of the file’s properties)
file-description: The “File description” set in the service’s executable file (visible on the “Details” tab of the file’s properties)
file-version: The “File version” set in the service’s executable file (visible on the “Details” tab of the file’s properties)
For example, here is the XML captured for the “Print Spooler” service (which was compromised in June 2021):
With those fields included in the XML, the “diff” outlined above will highlight when the service’s executable has changed. No one will be able to swap out (or impersonate) the underlying file without it showing up on your radar!
Posted onDecember 28, 2020 (Revised April 3, 2022)
Do you run Box Drive as a Windows Service, to ensure that file synchronization starts as soon as your server boots? If so, please be aware that Box Drive version 2.18 may not start properly in the background!
Version 2.18 was released in October 2020. We took it for a test drive with AlwaysUp and here is what we discovered.
Box Drive starts but quickly exits
AlwaysUp started Box.exe several times. Each time, the process would run for a second and then shut down. After 5 attempts (as configured on the Restart tab), AlwaysUp gave up.
The activity report shows the back and forth:
Unfortunately the Box.exe process exited with a generic return code, with no indication of what happened. Puzzled, we decided to investigate the logs.
The log files report a strange problem
Box Drive spools its logs to a location in the AppData folder. You can find the files here:
A file named “Box-2.18.117.log” records the program’s activities when it starts. We noticed these telltale lines inside:
[36;49m2020-12-27 18:45:23.667 7596 INFO MainThread windows_sync_app_dele Calling start_application: [‘C:\\Program Files\\Box\\Box\\Box.exe’] [36;49m2020-12-27 18:45:23.667 7596 INFO MainThread windows_sync_app_dele Current memory usage: 90279936 bytes [36;49m2020-12-27 18:45:23.667 7596 INFO MainThread windows_sync_app_dele A Box Drive installation is in progress. Box Drive cannot continue to run
So for some bizarre reason, Box Drive thinks that an installation is in progress!
Thinking that it may be causing a conflict, we stopped and disabled the Box Update Service (which is responsible for automatic updates):
However, that didn’t make a difference. The “installation is in progress” error continued to thwart us.
So clearly Box Drive is having trouble running with AlwaysUp. But was the problem caused by AlwaysUp? We decided to investigate with an independent, third-party utility.
The problem isn’t limited to AlwaysUp — PsExec fails too
Microsoft’s free PsExec utility can start any executable in Session 0 — the only session available when your machine boots. PsExec is useful when troubleshooting applications that have trouble running at boot.
We ran this command line to launch Box as a specific user in Session 0:
But the run ended with the same result in the logs as before.
So the issue is not unique to AlwaysUp. For some strange reason, Box Drive version 2.18 simply refuses to run in the background, on the isolated Session 0.
Box Drive starts properly in the current, interactive session
As it does when started normally on your desktop, Box Drive runs fine as a Windows Service outside of Session 0.
If you select Start “Box Drive” in this session from the Application menu in AlwaysUp, Drive (and its familiar tray icon) will appear on your screen:
Of course, that will not be possible unless you log in, which defeats the purpose of launching file synchronization automatically at boot! Fortunately, AlwaysUp can help you get around that.
Configure automatic logon with AlwaysUp to start Box Drive at boot
Since Box needs a “normal” session to run at boot, you can:
Setup Windows autologon, to automatically sign in to Windows when your machine boots;
Have AlwaysUp launch Box Drive in that user’s session.
A few weeks ago, our developers reported the problem to Box Support. The detailed message documented all our key findings, including the log messages/files and the two methods of easily reproducing the failure.
The response from the Box Customer Success team was blunt, though not entirely unexpected:
Even so, it’s a bit short sighted of them to dismiss a problem that is impacting their paying customers!
Box Drive version 2.19 is “coming soon”
Apparently version 2.19 of Box Drive will be available very soon, probably in early 2021. Hopefully it will fix this “phantom installation” problem when running as a Windows Service.
(888) 881-CORE
