In this page you find:
In the logs and reports section of the Endian Hotspot Appliance there are different possibilities to look at and to analyse the log files.
The sub-menu on the left-hand side of the screen contains the following items:
/var/log/messages) filtered by source
and dateIn a nutshell, there are two modalities to access the log from the GUI: Live and “by-service”: In the live mode the log files are visualised as soon as they are created, while in the “by-service” mode only the logs produced by one daemon or service are displayed.
In a nutshell, the reporting module shows events happened on the Endian Hotspot Appliance using different widgets and graphs. All events occurring on the system and the information concerning them recorded by the syslog daemon are parsed and used to populate a sqlite3 database. From here, data are gathered according to the options and to the filters applied in the GUI and are displayed by the widgets.
Note
This module is loosely coupled with the Event notifications located in . All events recorded there, and for which email or SMS alerts are sent, appear also here, but the vice-versa is not true.
This page is divided into tabs: Summary, System, Web, Mail and Viruses. Except for the first tab, which shows an overview of all events, each of them is dedicated to a precise service running on the Endian Hotspot Appliance.
All the tabs share the same design: Below the tabs, on the left-hand side there are a date selector on the the left-hand side and a Print button on the right-hand side. Then, a line chart at with an horizontal slider right below, atop one informative boxes (Summary Grid) and a pie-chart. At the bottom, there are one or more tables, depending on the tab and the data shown. The table that is always present is the one displaying the syslog messages related to the events shown.
More in detail, here is a description of all the widget present in the reporting module.
Line Chart and Time Slider.
The line chart shows the event happened on the Endian Hotspot Appliance during the selected time span in a two dimensional graph, in which the x-axis shows the time interval and the y-axis shows the number of occurrences. A coloured line connects events of the same type.
Hint
Different types of event are denoted with different colours.
The time slider is located underneath the chart and allows, within the selected time span, a more fine-grained view of the events, depicted here as histograms. Indeed, the two grey handles on the left and right limits of the slider can be clicked and dragged to reduce the time span shown in the line chart. When reduced, the slider can also be moved by clicking in its middle and dragging it to the left or the right.
Summary Grid
The summary grid has a twofold purpose: On the one hand to show the number of occurrences of the various types of events that took place on the Endian Hotspot Appliance in the selected period, whereas on the other end to filter which type of events are shown in the line chart. Its content changes according to the tabs it is located, i.e., to the types of events logged. The summary grid is not present in the Mail, Attacks, and Virus tabs, in which is replaced by a number of tables with details about the events.
Pie Chart
The pie chart diagram shows graphically the number of event that took place in the selected time span. When in the Summary tab, each slice can be clicked, to open the tab corresponding to the type of event and show a more detailed representation.
Syslog table
A table that shows the syslog messages extracted from the log files and related to the events shown in the charts. When the table carries lot of messages, these are divided into many pages and can be browsed using the buttons and number at its left bottom. At the right bottom there is an icon that allows to refresh the table’s content.
The Summary tab gives an overview of all categories of events recorded on the Endian Hotspot Appliance. The summary grid allows to filter the following types of events:
Each category can be shown separately, with more information and a higher level of details in the other tabs of the page, see further on.
The System tab displays all events that are related to the system efficiency and to system administration. These are all the events shown:
A click on the small icon on the left-hand side of each event category causes the other categories to not be shown, while the current is further detailed and the pie chart is updated.
The Web tab displays the number of pages that have been accessed or blocked by the URL filter engine. The summary grid is composed by two tabs: Access report and Filter report.
This tab shows the domains that have been accessed, grouped into three tables showing respectively the Source IP Address, Domain, and Users with the total count for every item.
Note
The Access report tab is not present in all appliances.
This tab shows to which domains the access has been blocked. In the first table, the following categories are shown, that are those found in the Web filter (See ).
Like in the case of the System tab, a click on the small icon on the left-hand side of each event category causes the other categories to not be shown, while the current is further detailed and the pie chart is updated.
The other tables at the bottom show the counts of each the blocked objects: The Source IP Addresses, the URLs, and the Users.
The Mail tab displays all e-mails blocked as spam.
There is no summary grid in this tab, replaced by three tables, displaying counts for:
The Viruses tab displays all viruses intercepted by the anti-virus engine (See ).
The tables at the bottom show counts of the following information:
The Connections tab displays the average number of connections started by the users of the Endian Hotspot Appliance, grouped into:
In few words, the abilities of the traffic monitoring interface powered by ntopng can be summarised as follows:
The ntopng GUI is organised into four tabs: Dashboard, Flows, Hosts, and Interfaces. Moreover, there is also a search box to quickly display information about a given host.
In the footer of each tab, a couple of information are shown: Besides a copyright notice and a link to the ntop home page, there is a chart showing the network traffic over the last 20 seconds, updated in real time, and some numerical data about the current bandwidth used, the number of hosts and flows and the Endian Hotspot Appliance‘s uptime.
The dashboard shows all connections that interest the Endian Hotspot Appliance, that is, all established Flows in which the Endian Hotspot Appliance is involved.
The page is divided into several diagrams, with the first one -a so-called Sankey diagram showing all flows moving on the Endian Hotspot Appliance, updated in real time. The horizontal flows show the traffic between two hosts, while the vertical width of each flows is proportional to the bandwidth used by that flows, i.e., to the amount of data flowing. The connections -and therefore the direction of the data sent- are shown left to right: Hosts on the left hand-side of the diagram send data to hosts on the right-hand side and are identified by either their IP address or hostname. A click on one host leads to the Overview page in the Hosts tab, which shows several information about that host.
Below the Sankey diagram, four informative-only pie charts show in percentage the items that that generate the most traffic, divided into: Total by host (top left); application protocols (top right), ASNs (bottom left), and live flow senders (bottom right).
The active flows tab contains a big table with a number of information about the active flows:
Info. A click on the icon opens a new page in which more detailed information about that flow is shown.
Application. The application causing the flow. nDPI is used to recognise the application, therefore it might be necessary to wait for a couple of packets to see the correct application displayed: In this case, the (Too Early) message appears instead of the application name.
L4 Proto The network protocol used by the flow, which is usually TCP or UDP.
Client. The hostname and port used by the flow on the client side. Clicking on either the hostname or port, more information will be shown in a new page about the network traffic flowing that host or port.
Server. The hostname and port used by the flow on the server side. Like for the Client above, more information is shown when clicking on the hostname or port.
Hint
By clicking on the hostname or port, the table shows detailed information about it, opening a sub-tab in the Hosts tab.
Duration. The length of the connection.
Breakdown. The percentage of traffic generated by the client and by the server.
Throughput. The amount of data currently exchanged between the client (on the left, in black) and server (on the right, in green).
Total Bytes. The total data exchanged since the connection was first established.
At the bottom of the table, on the left-hand side it is shown the total number of rows shown , while on the right-hand side it is possible to browse the various pages in which the table is split, when the number of rows is higher that the pagination.
A click on the Info icon will give detailed information about that particular flow. Besides those already described above, these additional data are displayed.
It is possible to go back to the list of flows by clicking on the Flows hyperlink on the left, right above the table.
The Hosts tab allows to view several details about the involved parties of a flow: Host, port, application, flows and their duration, data exchanged, and so on.
Two representation are available: Host List and Top Hosts (Local)
The Hosts List representation shows information about all the hosts involved in some flow with the Endian Hotspot Appliance and the following data about them:
A click on the IP address opens an overview of the host, showing several information about it, besides those listed above:
From here it is also possible to open additional informative tabs about that host. Each tabs contains one or more pie charts (except for the Contacts and Historical tabs) above a textual summary of the data displayed.
Traffic. The network protocol used by the host. (TCP, UDP and ICMP being the most common).
Packets. The length in packets of each flow. (note: just my guess)
Protocols. The application protocol used by the host.
Flows. The table with all the network flows from the hosts.
Talkers. The Sankey diagram of the connections, very similar to the one shown in the Dashboard, which however shows only the most active flows.
Contacts. This tab is slight different from the others. It shows on top an interaction maps and on the bottom a list of connection that have the host as client or receiver.
traffic flown form and to the host in a given timespan (up to one year), that can be selected above the graph.
The Top Hosts (Local) representation shows a real-time graphic of the hosts that have active connections to the host. It displays the last 30 minutes.
The Interfaces tab allow to select the network interface, among the active ones, whose traffic should be displayed.
Note
It is currently not possible to select flows and/or hosts from different interfaces
When entering in the Logs section, or clicking on the Live entry on the sub-menu, the Live log viewer is shown, a box showing the list of all the log files available for real time viewing. Any number of logs to see can be chosen by ticking the corresponding checkboxes, that are displayed in a new window upon clicking on the Show selected logs button. To watch all the log files at once, simply tick the Select all checkbox right above the Show selected logs button and then click on the latter button. Otherwise, to view only one log file, simply click on the Show this log only link.
The window that opens contains two boxes, Settings at the top and Live logs at the bottom.
Warning
The list of log entries can become nearly unreadable if many logs are showed, due to the possible high number of log entries produced (especially by the firewall or proxy log, which can generate several log entries per second in case of heavy traffic). In this cases, the logs to be displayed can be configured in the Settings box.
This box allows to modify the settings of the log viewer, including which of the log files to show, their colour and options to highlight or find specific keywords.
On the right-hand side of the box appears the list of the logs that are currently displayed, and the colour with which they are highlighted, while on the left-hand side some additional control elements are shown, that help limit the output:
To add or remove some log from the display, click on the Show more link right below the list of the log files on the top right. The controls will be replaced by a table from which the desired log files can be selected by ticking or unticking their respective checkboxes. To change the colour of a log file, click on the colour palette of that log type and then choose a new colour. To show the controls again, click on one of the Close links below the table or below the list of the displayed log files.
The logs chosen for viewing are shown in this box, which consists of a table divided in three columns.
The actual message generated by the service or daemon and recorded in the log files.
Note
Some log messages -especially Firewall entries- span
more than one line, denoted by the 
button at
the right of the message. To show the whole message,
click on it or on the button.
Finally, there is also the chance to increase or decrease the window size by clicking on the Increase height or Decrease height buttons, respectively, which are situated on the heading of the box.
The sub-menu entries System, Service, Firewall, and Proxy show log files for different services and daemons, grouped by similar characteristics. Several controls are available to search within the log, or view only some entries of the log, many of which are the same in all the services and daemons, with only the System menu item and the HTTP report tab under Proxy that have some additional control. These sub-menu entries have also a common structure of their pages, organised in two boxes: Settings at the top and Log at the bottom.
Note
A message at the top of the page informs if on a given date there are no logs available: This can happen either if the daemon or service were not running, or if they did not produce any message.
In the remainder of this section, all the services and their peculiar settings are presented.
This page presents summaries for the logs produced by the Endian Hotspot Appliance, separated by days and generated by the logwatch log monitoring software. Unlike the other parts of the log section, it has its own settings to control the level of details shown. The following control elements are available in the first box at the top of the page.
Below the Settings box, a variable number of boxes appears, depending on the running services that have log entries. The Disk Space box should at least be visible, showing the available disk space on the chosen date, while other boxes that can show up include Postfix (mail queue) and Firewall (accepted and dropped packets)
Note that the summaries are not available for the current day, as they are generated every night from the log files generated the day before.
In this section appears the log viewer for the various system log files. The upper box, Settings, defines the criteria to display the entries in the lower box. Besides the common actions, one additional control is available:
Following the choice of the section, click on the Update button to refresh the logs displayed in the Log box at the bottom of the page, in which the Older and Newer buttons allow to browse the pages.
In this section appear the log entries for three of the most important services provided by the Endian Hotspot Appliance: OpenVPN and the anti-virus, each in its own tab. Only the common actions are available.
The firewall log viewer contains the messages that record the firewall’s activities. Only the common actions are available.
Information shown in the table are:
The proxy log viewer shows the logs for the four daemons that use the proxy. Each of them has its own tab: squid (HTTP), icap (Content filter), sarg (HTTP report), and smtpd (SMTP, email proxy).
In addition to the common actions, the log viewer for the HTTP proxy and confent filter allow these values to be specified:
The HTTP report tab has only one option: To enable or not the proxy analysis report generator, by ticking the Enable checkbox and clicking on the Save button afterwards. Once the report generator is activated, a click on the Daily report, Weekly report, and Monthly report links shows detailed HTTP reports.
Only the common actions are available in the tab of the postfix daemon.
This page contains all the global configuration items for the Endian Hotspot Appliance‘s logging facilities, organised into four boxes: Log viewing options, Log summaries, Remote logging, and Firewall logging
Growing Logging and disk space management
The standard policy for storing log files on Endian Hotspot Appliance has been the following. Every night, log files are rotated and saved as daemonname.nnn.gz, while newer messages are written in a new log file. nnn is a progressive number, starting from 1. On some appliances, disk space may be quickly filled up, especially if many daemons are actively logging.
The logging policy is to delete older log files to make room for newer ones, when the partition storing the logs is about to run out of space.
See also
More information about the policies about logging can be found in this article.
Trusted timestamping is a process that log files (but in general any document) undergo in order to track and certify their origin and compliance to the original. In other words, trusted timestamping allows to certify and verify that a log file has not been modified in any way by anyone, not even the original author. In the case of log files, trusted timestamping proves useful for example, to verify the accesses to the system or the connections from the VPN users, even in cases of independent audits.
Trusted timestamping is not enabled by default, but its activation only requires a click on the grey switch. When it turns green, some configuration options will show up.
The URL of the timestamp server (also called TSA) is mandatory, since it will be this server that signs the log files.
Note
A valid URL of a valid TSA is needed to be able to use trusted timestamping. Several Companies can supply this kind of service.
After clicking on the Save button, the settings are stored and, on the next day, a new button will appear in the Logs section, on the right-hand side of the Settings box:
See also
The official OpenSSL timestamping documentation and RFC 3161, the original definition of the Time Stamp Protocol.