Troubleshooting

When something goes wrong with your gateway, the fastest path to a fix is usually understanding what the system is trying to tell you. This guide walks you through the most common issues you will encounter in production, explains what is actually happening behind each error, and shows you where to look for answers. Rather than just listing fixes, the goal is to help you build intuition about the underlying causes so recurring problems become rare.

[[toc]]

Common Issues ​

The issues below represent the most frequent problems reported from production deployments. Each section explains how the problem shows up, why it tends to happen, and how to resolve it.

Authentication Failures ​

Authentication issues are the most common problems reported by API consumers, and fortunately they are usually quick to diagnose. They come in two flavors: requests that cannot be verified at all, and requests with credentials that lack the right permissions.

Missing or Invalid Tokens ​

When users receive 401 Unauthorized responses, or you spot "Authentication required" and "Invalid or expired token" messages in your logs, the request is arriving without a token the gateway can verify. This usually traces back to a missing Authorization header, an expired or revoked token, or a Bearer header that is formatted incorrectly.

A good first step is checking what the client is actually sending. The header should look like this:

Authorization: Bearer YOUR_TOKEN

If the header looks right, open the Web UI and navigate to Tokens to confirm the token exists and has not been revoked or expired. Should the token turn out to be stale, creating a replacement there and revoking the old one resolves the issue immediately.

When several users are affected at once, it is worth reviewing the authentication logs for patterns. A cluster of failures from one integration often points to a deployment that shipped with an outdated token, while failures spread across many clients may suggest a configuration change on the gateway side.

Insufficient Token Permissions ​

A 403 Forbidden response tells a different story: the token is valid, but it is not allowed to do what the request asks. You will see "Access denied to endpoint" or "Access denied to environment" messages in the logs. The token may lack the scope for the requested endpoint, or the user may be reaching for an environment their token does not cover.

You can inspect what a token is allowed to do directly from its file:

PowerShellBash

powershell

# View token file content
Get-Content ".\tokens\username.txt" | ConvertFrom-Json | Format-List

bash

# View token file content
cat ./tokens/username.txt | jq .

Then compare that against what the endpoint configuration expects:

{
  "AllowedEnvironments": ["prod", "dev"],
  "AllowedScopes": "Products,Orders"
}

If the user legitimately needs the access, editing the token in the Web UI under Tokens to add the missing scopes or environments is all it takes. While you are there, it is a good moment to confirm the endpoint's access rules still match your business requirements, and to note somewhere which scopes different integrations actually need. That documentation pays for itself the next time this question comes up.

Rate Limiting Issues ​

Rate limiting protects the gateway from being overwhelmed. When users report 429 Too Many Requests responses, or you find "Rate limit exceeded" and "IP blocked" messages in the logs, someone is sending more requests than the configured thresholds allow. Sometimes that is legitimate high-volume usage; sometimes it is a development team hammering the API during integration testing, or an automated script with an aggressive retry loop.

Start by checking what the current limits actually are:

{
  "RateLimiting": {
    "Enabled": true,
    "IpLimit": 100,
    "IpWindow": 60,
    "TokenLimit": 1000,
    "TokenWindow": 60
  }
}

Then look at who is hitting them:

PowerShellBash

powershell

# Search for rate limit events
Select-String -Path ".\log\*.log" -Pattern "Rate limit" |
    Sort-Object -Property LastWriteTime -Descending |
    Select-Object -First 20

bash

# Search for rate limit events
grep -h "Rate limit" ./log/*.log | tail -n 20

If the pattern is isolated to one user or IP, a conversation about retry logic with exponential backoff usually fixes it at the source. If legitimate usage has simply outgrown the thresholds, raising the limits in configuration is the right long-term answer.

For immediate relief during an incident, restarting the application pool resets all counters:

# Restart IIS Application Pool
Restart-WebAppPool -Name "PortwayAppPool"

Connection Issues ​

Connection problems can be frustrating because they often indicate issues with underlying services that your gateway depends on. Let's walk through the two main types you'll encounter.

When Your Database Won't Connect ​

Database connection failures will typically show up as 500 Internal Server Error responses when you try to access SQL-based endpoints. These errors happen when the gateway can't reach your SQL database or when the connection is dropped unexpectedly.

Your first step should be verifying that your connection string is correct and complete:

{
  "ConnectionString": "Server=YOUR_SERVER;Database=500;Trusted_Connection=True;Connection Timeout=15;TrustServerCertificate=true;"
}

Before diving into complex troubleshooting, test whether you can connect to your SQL database at all from your gateway server:

PowerShellBash

powershell

# Test SQL connection
$conn = New-Object System.Data.SqlClient.SqlConnection
$conn.ConnectionString = "Server=YOUR_SERVER;Database=500;Trusted_Connection=True;"
try {
    $conn.Open()
    Write-Host "Connection successful"
} catch {
    Write-Host "Connection failed: $_"
} finally {
    $conn.Close()
}

bash

# Test SQL connection (requires sqlcmd)
sqlcmd -S YOUR_SERVER -d 500 -Q "SELECT 1" && echo "Connection successful"

If basic connectivity works but you're still having issues, the problem might be with connection pooling settings. The gateway manages a pool of database connections to improve performance, but if these settings are misconfigured, you might see intermittent failures:

{
  "SqlConnectionPooling": {
    "MinPoolSize": 5,
    "MaxPoolSize": 100,
    "ConnectionTimeout": 15,
    "CommandTimeout": 30,
    "Enabled": true
  }
}

When Proxy Endpoints Stop Responding ​

Proxy endpoints act as intermediaries between your API consumers and your backend services. When these fail, you'll typically see timeout errors, "Error processing endpoint" messages, or 503 Service Unavailable responses. This is pretty common with legacy applications, where high availability of an API isn't guaranteed.

Start by testing whether the target service is actually available. Try accessing it directly:

PowerShellBash

powershell

# Test endpoint connectivity
Invoke-WebRequest -Uri "http://localhost:8020/services/Exact.Entity.REST.EG/Account" -UseDefaultCredentials

bash

# Test endpoint connectivity
curl -I http://localhost:8020/services/Exact.Entity.REST.EG/Account

If the direct connection works, check your proxy configuration to ensure the URL and settings are correct:

{
  "Url": "http://localhost:8020/services/Exact.Entity.REST.EG/Account",
  "Methods": ["GET", "POST"],
  "AllowedEnvironments": ["prod", "dev"]
}

Sometimes proxy issues are related to environment-specific configurations. Review your environment settings to make sure they match what the backend service expects:

PowerShellBash

powershell

# Check current environment settings
Get-Content ".\environments\500\settings.json" | ConvertFrom-Json

bash

# Check current environment settings
cat ./environments/500/settings.json | jq .

Health Check Failures ​

The gateway includes built-in health monitoring to help you identify problems before they impact your users. When health checks fail, it's usually indicating a resource constraint or connectivity issue that needs immediate attention.

When You're Running Out of Disk Space ​

One of the most critical health issues you can encounter is low disk space. When the system detects critically low storage, health checks will show "Unhealthy" status with warnings about remaining disk space. This can lead to log write failures and eventually cause the entire application to stop functioning.

Start by checking exactly how much space you have available:

PowerShellBash

powershell

# Check available disk space
Get-PSDrive -PSProvider FileSystem |
    Select-Object Name, @{Name="FreeGB";Expression={[math]::Round($_.Free/1GB,2)}},
                  @{Name="UsedGB";Expression={[math]::Round($_.Used/1GB,2)}},
                  @{Name="TotalGB";Expression={[math]::Round(($_.Free + $_.Used)/1GB,2)}}

bash

# Check available disk space
df -h

If you're running low on space, the quickest relief usually comes from cleaning up old log files. The gateway can generate substantial logs over time, especially with traffic logging enabled:

PowerShellBash

powershell

# Remove logs older than 30 days
Get-ChildItem ".\log" -Recurse -File |
    Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-30) } |
    Remove-Item -Force

bash

# Remove logs older than 30 days
find ./log -type f -mtime +30 -delete

For ongoing space management, configure automatic log rotation to prevent this problem from recurring:

{
  "RequestTrafficLogging": {
    "MaxFileSizeMB": 50,
    "MaxFileCount": 5
  }
}

When Your Backend Services Aren't Responding ​

Sometimes health checks will report that "one or more proxy services are not responding properly." This indicates that while your gateway is running fine, some of the backend services it depends on are having problems.

To get detailed information about which specific services are failing, request a detailed health report:

GET /health/details
Authorization: Bearer YOUR_TOKEN

Once you know which endpoints are problematic, test them individually to isolate the issue:

PowerShellBash

powershell

# Test specific endpoint
$headers = @{
    "Authorization" = "Bearer YOUR_TOKEN"
}
Invoke-RestMethod -Uri "https://your-gateway/api/500/Products" -Headers $headers

bash

# Test specific endpoint
curl -H "Authorization: Bearer YOUR_TOKEN" https://your-gateway/api/500/Products

If specific endpoints are consistently failing, review their error logs to understand what's happening:

PowerShellBash

powershell

# Find endpoint-specific errors
Select-String -Path ".\log\*.log" -Pattern "endpoint: Products" |
    Where-Object { $_ -match "ERROR" }

bash

# Find endpoint-specific errors
grep "endpoint: Products" ./log/*.log | grep "ERROR"

Performance Issues ​

Performance problems can be subtle at first but significantly impact user experience as they worsen. The gateway includes monitoring capabilities to help you identify and resolve these issues before they become critical.

When Everything Feels Slow ​

If you're experiencing high latency on API calls, timeout errors, or seeing duration measurements over 1000ms in your logs, you're dealing with performance degradation. This can stem from various causes, including database bottlenecks, network issues, or resource constraints.

First, enable detailed traffic logging to get visibility into exactly where time is being spent:

{
  "RequestTrafficLogging": {
    "Enabled": true,
    "EnableInfoLogging": true
  }
}

With logging enabled, you can analyze which requests are taking the longest to complete. If you're using SQLite for traffic logging, you can query this data directly:

-- Find slow requests (using SQLite logging)
SELECT Path, QueryString, DurationMs, StatusCode
FROM TrafficLogs
WHERE DurationMs > 1000
ORDER BY DurationMs DESC
LIMIT 20;

Often, performance issues are related to database connection management. If your connection pool is too small or configured incorrectly, requests may wait for available connections. Try optimizing these settings:

{
  "SqlConnectionPooling": {
    "MinPoolSize": 10,
    "MaxPoolSize": 200,
    "ConnectionTimeout": 30
  }
}

Diagnostic Tools ​

Effective troubleshooting is mostly about knowing where to look. The gateway generates extensive diagnostic data, and once you know which source answers which kind of question, most investigations become short.

Understanding Your Log Files ​

The gateway creates several different types of logs, each serving a specific purpose in helping you understand what's happening in your system. Knowing which log to check for which type of problem will save you significant time during troubleshooting.

Where to Find Your Logs ​

Your logs are organized in a logical structure, with different types of information stored in different locations:

Handy Commands for Log Analysis ​

When you're troubleshooting an active issue, these commands will help you quickly find relevant information.

To find recent errors across all log files:

PowerShellBash

powershell

# Find all errors in last hour
$oneHourAgo = (Get-Date).AddHours(-1)
Get-ChildItem ".\log\*.log" |
    Where-Object { $_.LastWriteTime -gt $oneHourAgo } |
    Select-String -Pattern "ERROR|EXCEPTION" |
    Format-Table -AutoSize

bash

# Find all errors in log files modified in the last hour
find ./log -name "*.log" -mmin -60 -exec grep -HnE "ERROR|EXCEPTION" {} +

To understand what types of errors are most common:

PowerShellBash

powershell

# Count errors by type
Get-Content ".\log\portwayapi-$(Get-Date -Format 'yyyyMMdd').log" |
    Select-String -Pattern "ERROR.*?:" |
    Group-Object -Property Line |
    Sort-Object Count -Descending |
    Select-Object Count, Name -First 10

bash

# Count errors by type
grep -oE "ERROR[^:]*:" "./log/portwayapi-$(date +%Y%m%d).log" |
    sort | uniq -c | sort -rn | head -n 10

For real-time monitoring during active troubleshooting:

PowerShellBash

powershell

# Monitor log file in real-time
Get-Content ".\log\portwayapi-$(Get-Date -Format 'yyyyMMdd').log" -Wait -Tail 50

bash

# Monitor log file in real-time
tail -n 50 -f "./log/portwayapi-$(date +%Y%m%d).log"

Database Diagnostics ​

The gateway uses SQLite databases to store authentication and traffic data. These databases contain valuable information for troubleshooting authentication issues and analyzing usage patterns.

Checking Authentication Status ​

When users report authentication problems, start by verifying their token status in the database:

-- Using SQLite browser or command line
SELECT Id, Username, CreatedAt, ExpiresAt, AllowedScopes, AllowedEnvironments
FROM Tokens
WHERE RevokedAt IS NULL
ORDER BY CreatedAt DESC;

This query shows you all active tokens, when they were created, when they expire, and what permissions they have.

Understanding Traffic Patterns and Errors ​

The traffic logs database is particularly useful for identifying patterns in errors or performance issues:

-- Error distribution by endpoint
SELECT EndpointName,
       COUNT(CASE WHEN StatusCode >= 400 THEN 1 END) as Errors,
       COUNT(*) as TotalRequests,
       ROUND(CAST(COUNT(CASE WHEN StatusCode >= 400 THEN 1 END) AS FLOAT) / COUNT(*) * 100, 2) as ErrorRate
FROM TrafficLogs
WHERE Timestamp > datetime('now', '-24 hours')
GROUP BY EndpointName
HAVING Errors > 0
ORDER BY ErrorRate DESC;

This query helps you identify which endpoints are experiencing the highest error rates, giving you a clear starting point for investigation.

Network and Connectivity Diagnostics ​

Sometimes the issue isn't with the gateway itself, but with the network connections it depends on. These commands help you verify connectivity to essential services:

PowerShellBash

powershell

# Test connectivity to SQL Server (adjust host/port for other providers)
Test-NetConnection -ComputerName "YOUR_SERVER" -Port 1433

# Test proxy endpoint
Invoke-WebRequest -Uri "http://localhost:8020/services/Exact.Entity.REST.EG/Account" `
    -UseDefaultCredentials -Method Head

# Check listening ports
Get-NetTCPConnection -State Listen |
    Where-Object { $_.LocalPort -in @(80, 443, 8080) }

bash

# Test connectivity to SQL Server (adjust host/port for other providers)
nc -zv YOUR_SERVER 1433

# Test proxy endpoint
curl -I http://localhost:8020/services/Exact.Entity.REST.EG/Account

# Check listening ports
ss -tlnp | grep -E ':(80|443|8080)\b'

These tests will quickly tell you if the problem is a basic connectivity issue versus something more complex within the application itself.

Understanding Error Messages ​

When troubleshooting issues, the specific error codes and messages you encounter provide valuable clues about what's going wrong. Rather than just memorizing these codes, understanding what they actually mean will help you diagnose problems more effectively.

Common Error Codes and What They Really Mean ​

Recognizing Log Message Patterns ​

The gateway uses a familiar logging pattern to help you quickly identify different types of events:

[INF] Rate limit enforced for {Identifier} - Someone hit the rate limits
[WRN] Tokens detected in the tokens directory. Relocate them to a secure location - Warning, take action
[ERR] Error processing endpoint {EndpointName} - Backend service issue
[DBG] SQL Query Request: {Url} - Database query being executed

These patterns help you quickly scan logs and identify the types of issues you're dealing with.

Emergency Procedures ​

Sometimes things go seriously wrong and you need to get the system back online quickly. These procedures are for emergency situations when normal troubleshooting isn't sufficient.

When the Application Won't Start at All ​

If your gateway won't start, the problem is usually at the infrastructure level rather than within the application code itself. Start with the most basic diagnostics:

First, check the Windows Event Viewer for any critical startup errors:

Get-EventLog -LogName Application -Source "IIS*" -Newest 20

Next, verify that IIS and your application pool are in the correct state:

# Check application pool status
Get-WebAppPoolState -Name "PortwayAppPool"

# Restart application pool
Restart-WebAppPool -Name "PortwayAppPool"

If IIS appears to be working but the application still won't start, check the application logs for startup errors:

Get-Content ".\log\portwayapi-$(Get-Date -Format 'yyyyMMdd').log" |
    Select-String -Pattern "Application start|FATAL|ERROR" |
    Select-Object -First 50

Complete System Reset (Use with Extreme Caution) ​

Before doing anything drastic, create a complete backup of your critical configuration:

# Create backup directory
$backupDir = ".\backup_$(Get-Date -Format 'yyyyMMdd_HHmmss')"
New-Item -ItemType Directory -Path $backupDir

# Copy important files
Copy-Item ".\tokens\*" "$backupDir\tokens\" -Recurse
Copy-Item ".\auth.db" "$backupDir\"
Copy-Item ".\environments\*" "$backupDir\environments\" -Recurse
Copy-Item ".\endpoints\*" "$backupDir\endpoints\" -Recurse

Once you have a backup, you can reset the application state:

# Stop IIS
iisreset /stop

# Clear logs (this removes diagnostic history)
Remove-Item ".\log\*" -Recurse -Force

# Start IIS
iisreset /start

After performing a reset, monitor the application logs carefully to ensure it starts up properly and test a few basic endpoints to verify functionality.

Keeping It Healthy ​

Prevention is always better than cure when it comes to gateway operations. By following these practices, you can avoid many of the common issues described in this guide and catch problems before they impact your users.

Proactive Monitoring and Maintenance ​

Regular maintenance doesn't have to be complicated, but it does benefit from consistency. Here are the key activities that will keep your gateway running smoothly:

• Keep an eye on your storage space. Disk space issues are one of the most common causes of gateway failures, but they're also completely preventable. Set up monitoring to alert you when disk space drops below 20%, and establish a routine for cleaning up old log files. The gateway can generate substantial logs, especially with detailed traffic logging enabled.

• Monitor your health endpoints regularly. Rather than waiting for users to report problems, set up automated health checks that call your /health endpoint and alert you to issues. Consider setting up simple monitoring scripts that test both the basic health endpoint and a few key API endpoints to confirm end-to-end functionality. The Telemetry guide shows how to feed gateway metrics into your existing monitoring stack.

• Test connectivity to your backend services. The gateway is only as reliable as the services it connects to. Regularly verify that your SQL database connections are working and that proxy endpoints can reach their target services. This is especially important after any network changes or server maintenance.

• Keep audit logs of configuration changes. When you modify endpoint configurations, token scopes, or other settings, document what you changed and why. This information becomes invaluable when troubleshooting issues that appear after configuration updates.

• Rotate your tokens periodically. Authentication tokens deserve the same care as passwords: change them regularly and immediately revoke any tokens that are no longer needed. This reduces your security exposure and ensures that only current, authorized integrations have access to your gateway.

Security Considerations ​

Security isn't just about preventing attacks. It's also about maintaining clean diagnostic information and ensuring you can trust your troubleshooting data.

• Keep detailed error messages away from external clients. While detailed error information is crucial for troubleshooting, it can also reveal sensitive information about your internal systems to potential attackers. The gateway returns generic error messages to clients by default while logging detailed information internally, and it is worth keeping it that way.

• Monitor failed authentication attempts. Keep track of repeated authentication failures, especially from the same IP addresses. This can indicate either misconfigured integrations that need attention or potential security threats that need investigation.

• Secure your diagnostic tools. The same database queries and log analysis tools that help you troubleshoot can also reveal sensitive information. It is a good idea to restrict access to logs, databases, and diagnostic endpoints to authorized personnel only.

Where to Go From Here ​

This troubleshooting guide covers the most common issues you'll encounter, but every environment is unique. As you become more familiar with your specific gateway configuration and usage patterns, you'll develop intuition about where to look first when problems arise.

For deeper information about specific aspects of gateway operation and configuration, these additional guides will provide more detailed guidance:

• Monitoring Guide - Set up comprehensive monitoring and alerting for your gateway
• Security Guide - Implement robust security practices and threat monitoring
• Deployment Guide - Best practices for deploying and configuring your gateway
• API Endpoints Guide - Detailed information about configuring and managing your API endpoints

Remember that troubleshooting is a skill that improves with practice. The more familiar you become with your gateway's normal operation patterns, the more quickly you'll be able to identify and resolve issues when they occur.