Hyper-V production checkpoints not being deleted after Veeam replication job on same Windows guest OS - backup

I am running two physical Windows 2016 servers in a clustered environment. The backup solution is Veeam Backup and Replication V11.
I have a replication job that has been failing regularly because the host server isn't merging the production checkpoints after.
I have to manually shut the guest down, wait for the checkpoints to merge, then power it back on. THe replication job then runs successfully but within 1-2 days the same issue occurs and I have to go through the same process again.
The guest OS is Server 2019 core.
The Hyper-V Worker event logs for the FAILED replications only show that the virtual hard disk is being used by another process.
I logged a ticket with Veeam, but they confirmed nothing more than it not being an issue with their software - they can see that post replication, the command is sent to merge the checkpoints.
I would really appreciate any ideas, help, suggestions.
Thank you.

Related

The concurrent snapshot for publication xxx is not available because it has not been fully generated

I'm having trouble running replication on sql server 2019. On replication monitor in Distributor to Subscriber history section I get action message:
The concurrent snapshot for publication xxx is not available because
it has not been fully generated or the Log Reader Agent is not running
to activate it. If the generation of the concurrent snapshot was
interrupted, the Snapshot Agent for the pub.
Is this message the cause of the replication I'm working on not running? I have tried various ways that I found on the internet, and nothing worked.
Does anyone have a solution?

Redis user uses high cpu

My server configuration is 4 GB Memory / 80 GB Disk / SGP1 - CentOS 7.6 x64.
My redis version Redis server v=4.0.11 sha=00000000:0 malloc=jemalloc-4.0.3 bits=64 build=4caa563e40a30492
This server is dedicated to Redis only
From the picture you can see there is a user called Redis and under is user there are two processes which is causing high CPU uses. I have no idea what is the purposes of these two commands. Is this harmful to my server should I keep them??
sysupdate and networkservice seem to be malicious services running in your system. Maybe somehow hackers got into your system and upload some scripts which are taking too much of your capacities. Most probably they are mining. So follow the below steps
stop that services
backup your redis file
restart your server
you can reinstall your redis
better to hire some consultant for hardening your redis server

Can I backup Hyper-V replica Virtual Machines instead of main VMs?

I backup my Hyper-V machines on my main server periodically. I have also turned on replication on different machine where I have also more storage space so my backup VM images go to this secondary machine.
Question is - when I backup replica will I have problems with restoration of VM if I would like to restore replica VM on main hyper-v server?
Can I just backup my replica VMs and avoid unnecessary file transfer between servers?
Can I just backup my replica VMs and avoid unnecessary file transfer between servers?
You can backup replica VMs , but the result might be not as you expected .
Because :
"
-Only crash-consistent backup of a Replica VM is guaranteed.
-A robust retry mechanism needs to be configured in the backup product to deal with failures. Or ensure that replication is paused when backup is scheduled.
"
This artilce may guide you in the right direction .

Live "Backup" or Disk Mirroring. Windows Server 2012

We are in position to decide to have 2 sites so when our Main Systems (site1 -- which is Primary location for our businesses) is down, so we have some virtual machines, file servers/file shares, SQL and Exchange in standby on other site (site2 -- secondary location). So we have some sort of backup so we could possibly have whole company up and running so fast we can.
What i want to ask you guys is about "live backup" Servers/file shares.
Do Windows Server have some tools so we can create exactly same copy of file shares on site2. Like fail-over cluster or something? We want that site1 and site2 file shares will communicate and have some sort of contact so when user copy some pictures to Primary file share (//fileshare1), then the secondary fileshare (//fileshare2) or Server, will now that there was been some changes in primary server/fileshare, and it'll copy that picture to site2. Some sort of "live backup" or mirroring.
Do Windows Server have some options like this?
Thanks for all your help!
If you want to use replica of VM as a solution to start replicated VM on the second site during a Disaster Recovery Plan, then this is possible, available in Windows Server (2012 or later) with Hyper-V Replica. This feature has no additional costs, it's included in Windows Server Licence.
This feature allows to replicate VM with a RPO of 30 seconds / 5 minutes or 15 minutes (depending on what you need, your network speed..). VM on recovery site (site 2) are shutdown until you want to use Site 2.
More information on Hyper-V Replica : https://technet.microsoft.com/en-us/library/jj134172%28v=ws.11%29.aspx
More over, in Windows Server 2016 (that will be release in H2 CY16) there is also a feature to replicate a Windows Volume between 2 serveurs (or 2 clusters) : Storage Replica
More information on Storage Replica : https://technet.microsoft.com/en-us/library/mt126104.aspx

BizTalk connectivity issue to SQL during VM snapshot

We have one VM for BizTalk and a separate VM for the SQL backend. We are using Veeam for backups which basically kicks off a snapshot of the VM. When this snapshot is being finalized on the SQL VM, BizTalk services on the application server fail. Usually they restart automatically but sometimes this requires manual intervention to start the services. The error below is logged on the BizTalk server.
Is there any timeout setting or config changes that will allow BizTalk services to stay up during the snapshot process?
An error occurred that requires the BizTalk service to terminate. The most common causes are the following:
1) An unexpected out of memory error.
OR
2) An inability to connect or a loss of connectivity to one of the BizTalk databases.
The service will shutdown and auto-restart in 1 minute. If the problematic database remains unavailable, this cycle will repeat.
Error message: [DBNETLIB][ConnectionRead (recv()).]General network error. Check your network documentation.
Error source:
BizTalk host name: BizTalkServerApplication
Windows service name: BTSSvc$BizTalkServerApplication
We experienced the same situation and error with both BizTalk 2009 and BizTalk 2013, each set up with two App servers and one SQL DB server.
When our VMware does the final step of the Snapshot backup on the Application servers, it freezes the application server for about 10 seconds, preventing it from receiving packets. On SQL Server 2008 and 2012, it by default will send out keep-alive packets to the clients every 30 seconds (30,000 ms). If the SQL server fails to receive a response back from the App server, it will send out 5 retries (default setting) of the keep-alive request 1 second (1,000 ms) apart. If SQL still does not receive the response back, it will terminate the connection, which will cause the BizTalk hosts on the App server to reset, and in our case, when our German-made ERP system sends its EDI documents over to BizTalk during that reset period, the transmission will fail.
We trapped the issue by running NetMon on the DB and App servers, waiting for the next error message. Upon inspection, we see the five SQL keep-alive packets being sent to the App servers 1 second apart, and at the same time there were NO packets at all received on the Application server. At first guess, one might think they were "just dropped network packets", which is rarely the case. We then made the correlation to the timing of the VM Snapshots, and now confirm each time the snapshot finishes each day, the App servers freeze.
As a Short-to-mid-term workaround, we raised the number of retries SQL attempts before declaring a connection dead, (5 by default), by adding the registry value TcpMaxDataRetransmissions and setting it to 30 (thus 30 seconds before SQL declares the client unresponsive). This has masked the problem for now for us, and use at your own discretion.
We are also looking at an Agent-based version of the VM Snapshot, which may alleviate the condition of freezing the server.
Is there any timeout setting or config changes that will allow BizTalk services to stay up during the snapshot process?
Not that I am aware of, however you might want to Google config options in the btsntsvc.exe.config file which is located in your BizTalk installation directory.
All messages that pass through BizTalk are written to the BizTalkMsgBoxDb and its other databases are involved if you are running tracking, BAM etc. The only service that can cache 'stuff' and handle a database outage is the Enterprise Single Sign-On (ESSO) Service. BizTalk therefore needs a persistent connection to the database server to remain 'up', hence why your Host Instance (BizTalkServerApplication) is stopping - it simply wouldn't be able to process messages if the database wasn't there.
I would add that your approach to back-ups probably isn't supported by Microsoft and I would further suggest that you seriously consider whether an approach that takes your database server offline during the backup is viable?
BizTalk has a pretty robust backup solution for its various databases built into the product, and I would recommend that you take a look at using this supported method.
If you do need to take snapshots of the database system - say once a night - you might want to consider stopping the BizTalk Host Instances, performing the snapshot, and then re-starting the Host Instances through some scripted task.
You might also want to consider checking whether there are any hotfixes for your version of BizTalk Server included in a Cumulative Update that might help address your problem.