Auditing

Sooner or later you'll want to know exactly what passed through your gateway and when. Request traffic logging is built for that moment: it captures per-request detail (timing, payloads, headers, and response data) and writes it to file or SQLite storage so you can query it later at your own pace.

Configuration ​

Basic Settings ​

{
  "RequestTrafficLogging": {
    "Enabled": false,
    "QueueCapacity": 10000,
    "StorageType": "file",
    "SqlitePath": "log/traffic_logs.db",
    "LogDirectory": "log/traffic",
    "MaxFileSizeMB": 50,
    "MaxFileCount": 5,
    "FilePrefix": "proxy_traffic_",
    "BatchSize": 100,
    "FlushIntervalMs": 1000,
    "IncludeRequestBodies": false,
    "IncludeResponseBodies": false,
    "MaxBodyCaptureSizeBytes": 4096,
    "CaptureHeaders": true,
    "EnableInfoLogging": true
  }
}

Configuration Options ​

Storage Types ​

File Storage ​

When StorageType is set to "file", logs are stored as JSON files with automatic rotation:

log/traffic/
├── proxy_traffic_20240120_103015.json
├── proxy_traffic_20240120_083045.json
└── proxy_traffic_20240119_154530.json

File Format:

• Each line contains a JSON object representing one request
• Files are rotated based on size (MaxFileSizeMB)
• Old files are deleted when count exceeds MaxFileCount
• Filenames include timestamp for easy identification

SQLite Storage ​

When StorageType is set to "sqlite", logs are stored in a SQLite database:

Database Schema:

CREATE TABLE TrafficLogs (
    Id INTEGER PRIMARY KEY AUTOINCREMENT,
    Timestamp TEXT NOT NULL,
    Method TEXT NOT NULL,
    Path TEXT NOT NULL,
    QueryString TEXT,
    Environment TEXT,
    EndpointName TEXT,
    TargetUrl TEXT,
    StatusCode INTEGER,
    RequestSize INTEGER,
    ResponseSize INTEGER,
    DurationMs INTEGER,
    Username TEXT,
    ClientIp TEXT,
    TraceId TEXT NOT NULL,
    RequestHeaders TEXT,
    RequestBody TEXT,
    ResponseBody TEXT
);

CREATE INDEX idx_timestamp ON TrafficLogs (Timestamp);

Log Entry Format ​

Each traffic log entry contains:

{
  "Id": 12345,
  "Timestamp": "2024-01-20T10:30:15Z",
  "Method": "GET",
  "Path": "/api/500/Products",
  "QueryString": "?$top=10",
  "Environment": "prod",
  "EndpointName": "Products",
  "TargetUrl": "http://localhost:8020/services/Exact.Entity.REST.EG/Product",
  "StatusCode": 200,
  "RequestSize": 0,
  "ResponseSize": 2048,
  "DurationMs": 45,
  "Username": "api-user",
  "ClientIp": "192.168.1.100",
  "TraceId": "a1b2c3d4",
  "RequestHeaders": {
    "Accept": "application/json",
    "Authorization": "[REDACTED]",
    "User-Agent": "MyApp/1.0"
  },
  "RequestBody": null,
  "ResponseBody": null
}

Field Descriptions ​

Security Features ​

Header Sanitization ​

Sensitive headers are automatically redacted:

• Authorization
• Cookie
• X-API-Key
• API-Key
• Password
• X-Auth-Token
• Token
• Secret
• Credential
• Access-Token
• X-Access-Token

Body Capture Controls ​

Request and response bodies are:

• Disabled by default
• Limited by MaxBodyCaptureSizeBytes
• Truncated with "..." suffix if exceeding limit
• Only captured for JSON/XML content types

Access Control ​

• Log files/database should be protected from web access
• Consider using separate storage with restricted permissions
• Implement log rotation to manage sensitive data retention

Performance Considerations ​

Traffic logging adds I/O overhead. The queue-based design minimises impact on request latency, writes happen in background batches, but high-volume deployments should tune the settings below.

Querying Traffic Logs ​

File Storage Queries ​

PowerShellBash

powershell

# Find slow requests
Get-Content "log/traffic/proxy_traffic_*.json" | 
    ConvertFrom-Json | 
    Where-Object { $_.DurationMs -gt 1000 } |
    Select-Object Timestamp, Method, Path, DurationMs

# Count requests by endpoint
Get-Content "log/traffic/proxy_traffic_*.json" | 
    ConvertFrom-Json | 
    Group-Object EndpointName | 
    Select-Object Count, Name | 
    Sort-Object Count -Descending

# Find failed requests
Get-Content "log/traffic/proxy_traffic_*.json" | 
    ConvertFrom-Json | 
    Where-Object { $_.StatusCode -ge 400 } |
    Select-Object Timestamp, Path, StatusCode

bash

# Find slow requests
cat log/traffic/proxy_traffic_*.json |
    jq 'select(.DurationMs > 1000) | {Timestamp, Method, Path, DurationMs}'

# Count requests by endpoint
cat log/traffic/proxy_traffic_*.json |
    jq -r '.EndpointName' | sort | uniq -c | sort -rn

# Find failed requests
cat log/traffic/proxy_traffic_*.json |
    jq 'select(.StatusCode >= 400) | {Timestamp, Path, StatusCode}'

SQLite Queries ​

-- Top 10 slowest requests
SELECT 
    Timestamp,
    Method,
    Path,
    DurationMs,
    StatusCode
FROM TrafficLogs
ORDER BY DurationMs DESC
LIMIT 10;

-- Request count by endpoint
SELECT 
    EndpointName,
    COUNT(*) as RequestCount,
    AVG(DurationMs) as AvgDuration,
    MAX(DurationMs) as MaxDuration
FROM TrafficLogs
GROUP BY EndpointName
ORDER BY RequestCount DESC;

-- Error rate by hour
SELECT 
    strftime('%Y-%m-%d %H:00', Timestamp) as Hour,
    COUNT(*) as TotalRequests,
    SUM(CASE WHEN StatusCode >= 400 THEN 1 ELSE 0 END) as Errors,
    ROUND(CAST(SUM(CASE WHEN StatusCode >= 400 THEN 1 ELSE 0 END) AS FLOAT) / COUNT(*) * 100, 2) as ErrorRate
FROM TrafficLogs
GROUP BY Hour
ORDER BY Hour DESC;

-- User activity summary
SELECT 
    Username,
    COUNT(*) as RequestCount,
    COUNT(DISTINCT EndpointName) as UniqueEndpoints,
    AVG(DurationMs) as AvgDuration
FROM TrafficLogs
WHERE Username IS NOT NULL
GROUP BY Username
ORDER BY RequestCount DESC;

Troubleshooting ​

Diagnostic Commands ​

PowerShellBash

powershell

# Check if logging is enabled
Get-Content "appsettings.json" | ConvertFrom-Json | Select-Object -ExpandProperty RequestTrafficLogging

# Monitor log directory size
Get-ChildItem "log/traffic" -Recurse | Measure-Object -Property Length -Sum

# View recent traffic logs
Get-Content "log/traffic/proxy_traffic_$(Get-Date -Format 'yyyyMMdd')*.json" | Select-Object -Last 10 | ConvertFrom-Json

bash

# Check if logging is enabled
cat appsettings.json | jq .RequestTrafficLogging

# Monitor log directory size
du -sh log/traffic

# View recent traffic logs
tail -n 10 log/traffic/proxy_traffic_$(date +%Y%m%d)*.json | jq .