Advanced Labs
JA4H Intermediate

JA4HTTP (JA4H - HTTP Client Fingerprinting)

Short Name: JA4H

Description:

JA4H identifies and classifies HTTP clients based on their request patterns, headers, and behaviors. By analyzing HTTP methods, headers, user agents, and payload sizes, JA4H generates unique fingerprints that aid in distinguishing between different types of clients, detecting automated bots, and identifying malicious web traffic.

Understanding JA4H Components

Components of a JA4H Fingerprint:

  1. HTTP Request Methods:
    • Description: The HTTP methods used by the client (e.g., GET, POST, PUT).
    • Purpose: Different clients may favor specific HTTP methods based on their functionality.
    • Example: GET, POST
  2. HTTP Headers:
    • Description: The headers included in HTTP requests, such as User-Agent, Accept, Content-Type.
    • Purpose: Headers provide insights into the client’s software, preferences, and capabilities.
    • Example: User-Agent: Mozilla/5.0, Accept: text/html,application/xhtml+xml
  3. User Agents:
    • Description: Identifies the client software making the HTTP request.
    • Purpose: Helps in distinguishing between different browsers, bots, and custom clients.
    • Example: Mozilla/5.0, curl/7.68.0
  4. Payload Sizes:
    • Description: The size of the payload in HTTP requests.
    • Purpose: Variations in payload sizes can indicate different client behaviors or intentions.
    • Example: 512 bytes, 2048 bytes
  5. Request Patterns:
    • Description: The sequence and frequency of HTTP requests made by the client.
    • Purpose: Identifies behavioral patterns unique to specific clients or automated tools.
    • Example: Rapid successive GET requests to the same endpoint.
  6. Connection Properties:
    • Description: Properties related to the HTTP connection, such as persistent connections or keep-alive settings.
    • Purpose: Indicates the client’s approach to managing connections and resource utilization.
    • Example: Connection: keep-alive
  7. TLS Configurations (if HTTPS):
    • Description: TLS handshake properties when using HTTPS, similar to JA4.
    • Purpose: Provides additional attributes for fingerprinting secure HTTP clients.
    • Example: TLSv1.2, Cipher Suites: ECDHE-RSA-AES128-GCM-SHA256

Measuring HTTP Client Attributes

Techniques:

  1. Using Web Server Logs:

    • Description: Parse web server logs to extract HTTP request attributes.
    • Example:
    • Apache Log Entry with HTTP Headers:
    127.0.0.1 - - [12/Oct/2024:15:12:17 +0000] "GET /api/data HTTP/1.1" 200 512 "http://example.com" "Mozilla/5.0" "-"
    • Explanation: Extract method (GET), endpoint (/api/data), status code (200), payload size (512 bytes), referrer, and user agent.

  2. Using Zeek:

    • Description: Use Zeek’s http.log to capture HTTP requests and extract relevant attributes.
    • Example Zeek Script:
    event http_request(c: connection, method: string, original_URI: string, unescaped_URI: string, version: string) {
    print fmt("HTTP Request: %s %s %s from %s", method, original_URI, version, c$id$orig_h);
}

event http_header(c: connection, name: string, value: string) {
    if (name == "User-Agent") {
        print fmt("User-Agent: %s", value);
    }
}

  3. Using Python Scripts:

    • Description: Analyze HTTP traffic using libraries like Scapy or requests to extract and measure attributes.
    • Example Python Script:
    from scapy.all import sniff, TCP, IP
from scapy.layers.http import HTTPRequest
import hashlib
 
def extract_http_info(packet):
    if packet.haslayer(HTTPRequest):
        http_layer = packet[HTTPRequest]
        method = http_layer.Method.decode()
        host = http_layer.Host.decode()
        uri = http_layer.Path.decode()
        user_agent = http_layer.User_Agent.decode() if http_layer.User_Agent else "Unknown"
        payload_size = len(packet[TCP].payload)
        fingerprint_str = f"{method}_{host}_{uri}_{user_agent}_{payload_size}"
        fingerprint_hash = hashlib.sha256(fingerprint_str.encode()).hexdigest()[:8]
        print(f"JA4H Fingerprint: {fingerprint_hash}")
 
sniff(filter="tcp port 80", prn=extract_http_info, store=0)

  4. Using Browser Developer Tools:

    • Description: Manually inspect HTTP requests made by clients to understand their behavior.
    • Example:
      • Open Developer Tools in a browser.
      • Navigate to the Network tab.
      • Observe the HTTP methods, headers, and payload sizes of requests.

Calculating HTTP Client Metrics

  1. Calculating Payload Size Statistics:

    • Description: Analyze the distribution of payload sizes to understand client behavior.
    • Example Calculation in Python:
    payload_sizes = [512, 1024, 2048, 512, 1024]  # in bytes
average_size = sum(payload_sizes) / len(payload_sizes)
print(f"Average Payload Size: {average_size} bytes")
    • Output:
    Average Payload Size: 1024.0 bytes

  2. Identifying Common User Agents:

    • Description: Extract and analyze User-Agent strings to identify popular clients or bots.
    • Example Python Script:
    user_agents = ["Mozilla/5.0", "curl/7.68.0", "Python-urllib/3.7"]
common_agents = set([ua.split("/")[0] for ua in user_agents])
print(f"Common User Agents: {common_agents}")
    • Output:
    Common User Agents: {'Python-urllib', 'curl', 'Mozilla'}

  3. Analyzing Request Patterns:

    • Description: Identify repetitive or unusual request patterns to detect automated clients.
    • Example:
    Request Pattern: GET /api/data, POST /api/update, GET /api/data, GET /api/data

  4. Comparing TLS Configurations:

    • Description: Compare TLS handshake properties to differentiate between secure clients.
    • Example:
    Client 1: TLSv1.2, Cipher Suites: ECDHE-RSA-AES128-GCM-SHA256
Client 2: TLSv1.3, Cipher Suites: TLS_AES_256_GCM_SHA384

  5. Measuring Connection Properties:

    • Description: Analyze connection headers to understand client preferences for connection management.
    • Example:
    Connection: keep-alive

  6. Analyzing HTTP Methods:

    • Description: Identify the distribution of HTTP methods to determine client behavior.
    • Example:
    HTTP Methods: GET: 80%, POST: 15%, PUT: 5%

  7. Analyzing Request Patterns:

    • Description: Identify sequences and frequencies of HTTP requests.
    • Example:
      • Count the number of GET vs. POST requests.
      • Detect rapid succession of requests indicating automated tools.

  8. User-Agent Analysis:

    • Description: Categorize clients based on their User-Agent strings.
    • Example:
    user_agents = ["Mozilla/5.0", "curl/7.68.0", "Mozilla/5.0", "PostmanRuntime/7.26.8"]
unique_user_agents = set(user_agents)
print(f"Unique User-Agents: {unique_user_agents}")
    • Output:
    Unique User-Agents: {'Mozilla/5.0', 'PostmanRuntime/7.26.8', 'curl/7.68.0'}

Constructing the JA4H Fingerprint

Steps:

  1. Create the Fingerprint String:

    • Format:
    method<methods>_headers<headers>_ua<user_agent_hash>_payload<average_payload_size>
    • Example:
    methodGET_POST_headersMozilla/5.0_curl=7.68.0_ua_a1b2c3d4_payload1024

  2. Hashing Components:

    • Hash sensitive components like User-Agent strings to ensure privacy.
    • Example Hash Function in Python:
    import hashlib
 
def generate_hash(component):
    """Generates a SHA-256 hash of the given component and truncates it."""
    return hashlib.sha256(component.encode()).hexdigest()[:8]
 
user_agent = "Mozilla/5.0"
ua_hash = generate_hash(user_agent)  # e.g., 'f3a1b2c3'

  3. Example Fingerprint Construction:

    • Sample Client Data:
      • HTTP Methods: GET, POST
      • Headers: Mozilla/5.0, curl/7.68.0
      • User-Agent: Mozilla/5.0
      • Average Payload Size: 1024 bytes
    • Fingerprint Construction:
    ua_hash = generate_hash("Mozilla/5.0")  # e.g., 'f3a1b2c3'
    methodGET_POST_headersMozilla/5.0_curl=7.68.0_ua_f3a1_payload1024

Usage Example

Scenario:

You aim to differentiate between legitimate browser traffic and automated bot traffic accessing your web application.

Steps:

  1. Collect HTTP Client Metrics:
    • Use web server logs or Zeek to extract HTTP methods, headers, User-Agent strings, and payload sizes.
  2. Generate Fingerprint:
    • Construct the fingerprint string using the collected metrics.
    • Hash sensitive components like User-Agent strings.
  3. Store Fingerprint:
    • Save the fingerprint in your centralized analysis database for future comparisons.
  4. Classify Traffic:
    • Compare incoming fingerprints against known profiles to classify traffic as legitimate or suspicious.
    • Identify patterns indicative of bots or malicious activities.

Example Output:

JA4H Fingerprint: methodGET_POST_headersMozilla/5.0_curl=7.68.0_ua_f3a1_payload10244
Intermediate Lab 1JA4S Intermediate