Analyzing Clearswift log files with WebSpy
To anaylze and report on your Clearswift log files with WebSpy you need to:
- Setup Clearswift Logging
- Import your log files into a Vantage storage
- Download and Open the Clearswift Report template and Aliases
- Report on your storage
- Analyze your storage
Setup Clearswift Logging
Clearswift SECURE Web Gateway appliances store log files internally for 30 days. These logs can be manually exported from the appliance at any time by going to System > Proxy Settings - Transaction Loggings and clicking “Export now” in the green menu on the left.
However, you can configure the Clearswift SECURE Web Gateway to automatically export transaction logs to a separate FTP server each day. As these log files are stored off the appliance, they can be kept forever.
To configure automatic FTP log exports:
- Install an FTP server (for example Filezilla, or the FTP module in Microsoft IIS) on a machine in your network, preferably on the same machine your running WebSpy Vantage on, or a machine that Vantage has fast access to.
- In the web interface for your Clearswift SECURE Web Gateway, go to System > Proxy Settings - Transaction Loggings and enable the transaction logs and configure the FTP settings for the automatic transaction log export.
On the Web gateway a new log is created every hour but they are exported to the ftp server once a day. It is only the new logs files, i.e. those not already on the FTP server that are copied. Log files is deleted from the web gateway but they can be kept forever on the FTP server.
Importing into a Storage
Before you can start analyzing and reporting on your Clearswift logs, you need to import your log file data into a storage. Storages are optimized for quick data access so you can analyze and report on the data you are interested in faster.
The Input Dialog wizard is used to import log files. This wizard can also be launched by clicking Import logs on the Inputs pane.
- On the 'Storages' page, enter a name for a new storage or select an existing storage to import to.
- On the 'Input Type' page select Local or networked files and folders
- Select the Clearswift format on the 'Loader Selection' page.
- On the 'Input Selection' page, click Add | Add Folder.
- The Clearswift access log files will then be displayed
- Click OK to begin importing your data.
As Vantage imports your Clearswift logs, you can view the progress of the import on the Storages dock. The Storages dock displays the size of the log file (illustrated as size imported / total size), the number of records imported, and the percentage complete (shown in the progress column).
Download and Open the Clearswift Report template and Aliases
WebSpy has created a report template and a set of aliases for Clearswift. The aliases convert raw log data such as numbers and codes into more meaningful names for your reports. The report template extracts information for each of your top users and each of your top websites, along with overview information regarding categories, threats, file types and other traffic activity.
Download the template and aliases here:
‘Clearswift Template and Aliases.zip’
This zip file contains two files:
To open the aliases file, go to the Aliases tab, click ‘Open aliases’ and navigate to the “Clearswift Aliases.” file. Select Merge when prompted. The following aliases have now been added to your Aliases list: Action (Clearswift), Cache Descriptions (Clearswift), Cache Values (Clearswift), Categories (MRS), Error Info (Clearswift), Productivity (Clearswift), and Status Code Names (Clearswift).
- Clearswift Summary Report.Template
To open the report template, go to the Reports tab in Vantage, select ‘Open Templates’, and navigate to the “Clearswift Summary Report.Template” file. Select Merge when prompted. You will now see a report template called ‘Clearswift Summary Report (Web Proxy)’. You can now select this template and click the ‘Generate report’ link to run the report on your Storage (see ‘Reporting on your Storage’ below)
Reporting on your Storage
Ensure you have downloaded and opened the Clearswift Templates and Aliases as above.
To generate a report on your Clearswift log data:
- Click the 'Reports' tab at the top of the screen. This takes you to the Reports dock.
- Select the Clearswift Summary Report (Web Proxy) template.
- Click Generate report. This launches the Generate Report dialog.
- On the 'Storages' page, check the storage that contains your Clearswift log data. Click Next.
- On the 'Format' page, select the desired format for the report. Click Next.
- On the 'Publish' page, enter a name for the report, and select Display the report using the default viewer if you would like the report to open after it has been generated.
- Leave the Filters, File Selection and Partition pages as default and then click OK to generate the report.
Your report will now be generated.
Analyzing your Storage
Each field in your Clearswift log files can be reported on using WebSpy Vantage. Vantage produces ‘Summaries’ for each field in your logs. Sometimes, Vantage produces more than one summary per field. For example, Vantage produces several summaries from the URL field in Clearswift log files, such as Site Domains (e.g. google.com), Site Names (e.g. images.google.com) and Site Keywords (e.g. My search term).
To get an idea of the range of Summaries that you can use in your reports and filters, run an ad-hoc analysis on a small amount of data (such as one day). To do this:
- Click the Summaries tab at the top of the screen.
- This takes you to the Summaries dock.
- Click the New Analysis link in the 'Summaries' task pad to launch the Create Analysis dialog.
- Select your Clearswift storage from the Storage list.
- Select Forefront Threat Management Gateway 2009 Web from the Schema list and then click Next.
- On the 'Analysis Type' page, select the 'Ad-hoc Analysis' radio button and ensure 'Use pre calculated analysis if available' checkbox is checked.
Once the analysis is complete, all the available summaries are displayed on the left hand side. Click a Summary, such as Site Domains to see the list of websites that have been accessed, or Usernames to see the list of users accessing the web. You can analyse a specific user or site by right-clicking them and selecting Drilldown from the popup menu along with another Summary. For example, right-click your top user and select Drilldown | Site Domains to view the sites they have accessed.
Note: When Analyzing data on the Summaries screen, it is best to use a small amount of data. It is good practise to create a separate test storage containing only one day’s worth of data for Ad-hoc analysis on this screen, then use this information to build your desired report template(s) and run these reports on your full data set.