The Azure Security Center (ASC) analysts team reviews and investigates ASC alerts to gain insight into security incidents affecting Microsoft Azure customers, helping improve Azure Security alerts and detections. ASC helps customers keep pace with rapidly evolving threats by using advanced analytics and global threat intelligence.
Although we have come a long way as far as cloud security is concerned, even today security factors are heavily discussed as companies consider moving their assets to the cloud. The Azure Security Center team understands how critical it is for our customers to be assured that their Azure deployments are secure, not only from advanced attacks but even from the ones that are not necessarily new or novel. The beauty of ASC lies in its simplicity. Although ASC uses machine learning, anomaly detection, and behavioral analysis to determine suspicious events, it still addresses simple things like SQL brute force attacks that Bad Guys/Script Kiddies are using to break into Microsoft SQL servers.
In this blog, we’ll map out the stages of one real-world attack campaign that began with a SQL Brute Force attack, which was detected by the Security Center, and the steps taken to investigate and remediate the attack. This case study provides insights into the dynamics of the attack and recommendations on how to prevent similar attacks in your environment.
Initial ASC alert and details
Hackers are always trying to target internet connected databases. There are tons of bad guys trying to discover IP addresses that have SQL Server running so that they can crack their password through a brute force attack. The SQL database can contain a wealth of valuable information for the attackers, including personally identifiable information, credit card numbers, intellectual property, etc. Even if the database doesn’t have much information, a successful attack on an insecurely configured SQL installation can be leveraged to get full system admin privileges.
Our case started with an ASC Alert notification to the customer detailing malicious SQL activity. A command line “ftp -s:C:zyserver.txt” launched by the SQL service account was unusual and flagged as by ASC Alerts.
The alert provided details such as date and time of the detected activity, affected resources, subscription information, and included a link to a detailed report of the detected threat and recommended actions.
Through our monitoring, the ASC analysts team was also alerted to this activity and looked further into the details of the alert. What we discovered was the SQL service account (SQLSERVERAGENT) was creating FTP scripts (i.e.: C:zyserver.txt), which was used to download and launch malicious binaries from an FTP site.
The initial compromise
A deeper investigation into the affected Azure subscription began with inspection of the SQL error and trace logs where we found indications of SQL Brute Force attempts. In the SQL error logs, we encountered hundreds of “Audit Login Failed” logon attempts for the SQL Admin ‘sa’ account (built-in SQL Server Administration) which eventually led up to a successful login.
These brute force attempts occurred over TCP port 1433, which was exposed on a public facing interface. TCP port 1433 is the default port for SQL Server.
Note: It is a very common recommendation to change the SQL default port 1433, this may impart a “false sensation of security”, because many port scanning tools can scan a “range” of network ports and eventually find SQL listening on ports other than 1433.
Once the SQL Admin ‘sa’ account was compromised by brute force, the account was then used to enable the ‘xp_cmdshell’ extended stored procedure as we’ve highlighted below in a SQL log excerpt.
The ‘xp_cmdshell’ stored procedure is disabled by default and is of particular interest to attackers because of its ability to invoke a Windows command shell from within Microsoft SQL Server. With ‘xp_cmdshell enabled, the attacker created SQL Agent jobs which invoked ‘xp_cmdshell’ and launched arbitrary commands, including the creation and launch of FTP scripts which, in turn, downloaded and ran malware.
Details of malicious activity
Once we determined how the initial compromise occurred, our team began analyzing Process Creation events to determine other malicious activity. The Process Creation events revealed the execution of a variety of commands, including downloading and installing backdoors and arbitrary code, as well as permission changes made on the system.
Below we have detailed a chronological layout of process command lines that we determined to be malicious:
A day after the initial compromise we began to see the modification of ACLS on files/folders and registry keys with use of Cacls.exe (which appears to have been renamed to osk.exe and vds.exe).
Note: Osk.exe is the executable for the Accessibility On-Screen Keyboard and Vds.exe is the Virtual Disk Service executable, both typically found on a Windows installation. The command lines and command switches detailed below, however, are not used for Osk.exe or VDS.exe and are associated with Cacls.exe.
The Cacls.exe command switches /e /g is used to grant the System account full(:f) access rights to ‘cmd.exe’ and ‘net.exe’.
A few seconds later, we see the termination of known Antivirus Software using the Windows native “taskkill.exe”.
This was followed by the creation of an FTP script (c:zyserver.txt ) which was flagged in the original ASC Alert. This FTP script appears to download malware (c:stserver.exe) from a malicious FTP site and subsequently launch the malware.
A few minutes later, we see the “net user” and “net localgroup” commands used to accomplish the following:
a. Activate the built-in guest account and add it to the Administrators group
b. Create a new user account and add the newly created user to the Administrators group
A little over 2 hours later, we see the regini.exe command which appears to be used to create, modify, or delete registry keys. Regini can also set permissions on the registry keys as defined in the noted .ini file. We then see, regsvr32.exe silently (/s switch) registering dlls related to the Windows shell (urlmon.dll, shdocvw.dll) and Windows scripting (jscript.dll, vbscript.dll, wshom.ocx).
This is immediately followed by additional modification of permissions on various Windows executables. Essentially resetting each to default with the “icacls.exe” command.
Note: The /reset switch replaces ACLs with default inherited ACLs for all matching files.
Lastly, we observed the deletion of “Terminal Server” fDenyTSConnections registry key. This is a registry key that contains the configuration of Terminal Server connection restrictions. This led us to believe that malicious RDP connections may be the next step for the attacker to access the server. Inspection of logon events did not reveal to us any malicious RDP attempts or connections, however:
- Disabling of Terminal Server connection restrictions by overwriting values in the “Terminal Server” registry key
reg.exe ADD “HKLMSYSTEMCurrentControlSetControlTerminal Server” /v fDenyTSConnections /t REG_DWORD /d 00000000 /f”
We also noticed a Scheduled task created. This task referenced a binary named “svchost.exe” to be launched out of the C:RECYCLER folder, which is suspicious.
Note that the legitimate “svchost.exe” files are located in the “WindowsSystem32” and “WindowssysWOW64”. Svchost.exe running from any other directory should be considered suspicious.
- Persistence mechanism – Task Scheduler utility (schtasks.exe) used to set a recurring task
C:WindowsSystem32schtasks.exe /create /tn “45645” /tr “C:RECYCLERsvchost.exe” /sc minute /mo 1 /ru “system
Recommended remediation and mitigation steps
Once we understood the extent and the details of the attack, we recommended the following remediation and mitigation steps to be taken.
First, if possible, we first recommended the backup and rebuild the SQL Server and reset all user accounts. We then implement the following mitigation steps to help prevent further attacks.
- 1. Disable ‘sa’ account and use the more secure Windows Authentication
To disable ‘sa’ login via SQL, run the following commands as a sys admin
ALTER LOGIN sa DISABLE GO - To help prevent attackers from guessing the ‘sa’ account, rename the ‘sa’ account
To rename the ‘sa’ account via SQL, run the following as a sys admin:
ALTER LOGIN sa WITH NAME = [new_name]; GO - To prevent future brute force attempts, change and harden the ‘sa’ password and set the sa Login to ‘Disabled’.Learn how to verify and change the system administrator password in MSDE or SQL Server 2005 Express Edition.
- It’s also a good idea to ensure that ‘xp_cmdshell’ is disabled. Again, note that this should be disabled by default.
- If TCP port 1433 if it is not required to be opened to the internet, ensure that there are no rules in Network Security Group that allows 1433. From your Azure Portal, take the following steps to configure a Rule to block 1433 in Network Security Group.
a. Open the Azure portal
b. Navigate to > (More Services) -> Network security groups
c. If you have opted into the Network Security option, you will see an entry for <ComputerName-nsg> — click it to view your Security Rules
d. Under Settings click “Inbound security rules” and then check for the existence of a rule that allows 1433 on the next pane (example detailed below):
e. If a rule allowing access to the SQL port 1433 is present, you’ll want to edit the rule to change the Action from “Allow” to “Deny”. Then click “Save”.
f. Then apply the newly created rule to the subscription
- Inspect all stored procedures that may have been enabled in SQL and look for stored procedures that may be implementing ‘xp_cmdshell’ and running unusual command. For example, in our case, we identified the following commands:
- Lastly, we highly recommend configuring Azure subscription(s) to receive future alerts and email notifications from Microsoft Azure Security Center. To receive alerts and email notifications of security issues like this in the future, we recommended upgrading from ASC “Free” (basic detection) tier to ASC “Standard” (advanced detection) tier.
Below is an example of the email alert received from ASC when this SQL incident was detected:
