Data Collection Methods & Integrations

CodeXray supports multiple methods for collecting API data, allowing you to choose the approach that best fits your workflow. Each method feeds into the same analysis pipeline, providing consistent security insights regardless of collection source.

Supported Collection Methods

Method	Type	Best For	Setup Time
HAR Files	Manual Upload	Browser-based APIs, one-time analysis	2 minutes
Postman Collections	Manual Upload	Documented APIs, testing workflows	2 minutes
Postman API Key	Automated Sync	Continuous Postman workspace monitoring	5 minutes
L7 Traffic Streaming	Network-level	Infrastructure-wide capture	15-30 minutes
SDK Instrumentation	Application-level	Production runtime monitoring	10-15 minutes

1. HAR File Upload

HTTP Archive (HAR) files capture complete HTTP transactions including headers, request bodies, and response data. This method is ideal for analyzing browser-based applications or capturing API traffic from web sessions.

How to Upload HAR Files

Interactive videos: HAR file upload interface

What is a HAR File?

A HAR (HTTP Archive) file is a JSON-formatted log of web browser interactions with websites. It contains detailed information about:

HTTP requests and responses
Request/response headers
Query parameters and payloads
Response times and status codes
Cookies and session data

How to Generate HAR Files

Chrome/Edge:

Open Developer Tools (F12 or Ctrl+Shift+I)
Navigate to the "Network" tab
Perform the actions you want to capture
Right-click in the network log → "Save all as HAR with content"

Firefox:

Open Developer Tools (F12)
Go to "Network" tab
Click the gear icon → "Save All As HAR"

Safari:

Enable Develop menu (Preferences → Advanced → Show Develop menu)
Develop → Show Web Inspector → Network tab
Export as HAR

HAR File Structure Example

{
  "log": {
    "version": "1.2",
    "creator": {
      "name": "WebInspector",
      "version": "537.36"
    },
    "pages": [],
    "entries": [
      {
        "_connectionId": "69447",
        "_initiator": {
          "type": "script",
          "stack": {
            "callFrames": [
              {
                "functionName": "",
                "scriptId": "16",
                "url": "https://app.example.com/static/js/chunk-vendors.54ee57c6.js",
                "lineNumber": 352,
                "columnNumber": 1245709
              }
            ]
          }
        },
        "startedDateTime": "2025-12-11T10:30:45.123Z",
        "time": 142.5,
        "request": {
          "method": "POST",
          "url": "https://api.example.com/v1/users",
          "headers": [
            {
              "name": "Content-Type",
              "value": "application/json"
            },
            {
              "name": "Authorization",
              "value": "Bearer eyJhbGciOiJIUzI1NiIs..."
            }
          ],
          "postData": {
            "mimeType": "application/json",
            "text": "{\"email\":\"user@example.com\",\"name\":\"John Doe\"}"
          }
        },
        "response": {
          "status": 201,
          "headers": [
            {
              "name": "Content-Type",
              "value": "application/json"
            }
          ],
          "content": {
            "mimeType": "application/json",
            "text": "{\"id\":\"12345\",\"email\":\"user@example.com\"}"
          }
        }
      }
    ]
  }
}

Key Fields Extracted:

request.method → HTTP method (GET, POST, PUT, DELETE)
request.url → API endpoint
request.headers → Authentication tokens, content types
request.postData → Request payload (checked for PII)
response.status → HTTP status code
response.content → Response body (analyzed for sensitive data)

Use Cases

Testing web applications during development
Capturing API traffic from third-party integrations
One-time security assessments
Debugging API issues
Documenting undocumented APIs

Security Note: HAR files may contain sensitive data like passwords and tokens. Review files before sharing and use CodeXray's automatic redaction features.

2. Postman Collection Upload

Postman collections are structured API definitions that include endpoints, request examples, and test cases. Uploading collections allows CodeXray to analyze your documented APIs.

How to Upload Postman Collections

Interactive videos: Postman collection upload interface

What is a Postman Collection?

A Postman collection is a group of API requests organized into folders. Collections include:

API endpoints with methods and URLs
Request headers and authentication
Request body examples
Environment variables
Pre-request scripts and tests

How to Export from Postman

Postman Desktop:

Open your workspace in Postman
Click the three dots (•••) next to your collection
Select "Export"
Choose "Collection v2.1" format (recommended)
Save the JSON file

Postman Web:

Navigate to your collection
Click "..." menu → Export
Download the JSON file

Collection Structure Example

[
  {
    "startedDateTime": "2025-09-28T11:26:06.131Z",
    "durationMs": 58.773,
    "request": {
      "method": "GET",
      "url": "http://100.91.223.26/",
      "headers": [
        {
          "name": "Accept-Language",
          "value": "en-GB,en;q=0.9"
        },
        {
          "name": "Upgrade-Insecure-Requests",
          "value": "1"
        },
        {
          "name": "User-Agent",
          "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
        }
      ],
      "body": null
    },
    "response": {
      "status": 200,
      "headers": [
        {
          "name": "Accept-Ranges",
          "value": "bytes"
        },
        {
          "name": "Access-Control-Allow-Origin",
          "value": "*"
        },
        {
          "name": "Cache-Control",
          "value": "public, max-age=0"
        },
        {
          "name": "Content-Encoding",
          "value": "gzip"
        },
        {
          "name": "Content-Type",
          "value": "text/html; charset=UTF-8"
        },
        {
          "name": "X-Content-Type-Options",
          "value": "nosniff"
        },
        {
          "name": "X-Frame-Options",
          "value": "SAMEORIGIN"
        }
      ],
      "body": null
    }
  }
]

Analyzed Elements:

Endpoint paths and HTTP methods
Authentication schemes (Bearer, API Key, OAuth2)
Request/response headers (security headers, CORS)
Request body schemas (PII detection)
Response status codes and error handling

Use Cases

Analyzing API specifications before deployment
Security review of documented endpoints
Validating API design against security best practices
Team-wide API inventory management
Integration testing workflows

Best Practice: Keep your Postman collections up-to-date with production APIs for accurate security analysis.

3. Postman API Key Integration

Automatically sync Postman workspaces with CodeXray for continuous API monitoring. This method eliminates manual uploads by connecting directly to your Postman account.

How to Configure Postman API Key

Interactive videos: Postman API key configuration

What is Postman API Key Integration?

Postman API keys provide programmatic access to your Postman workspaces, allowing CodeXray to automatically fetch collections without manual exports.

Benefits:

Automatic synchronization with Postman workspaces
Real-time updates when collections change
No manual export/upload required
Continuous monitoring of API definitions

Setup Instructions

Step 1: Generate Postman API Key

Log in to Postman web (https://web.postman.co)
Click your avatar → Account Settings
Navigate to "API Keys" section
Click "Generate API Key"
Provide a name (e.g., "CodeXray Integration")
Copy the generated key (shown only once)

Step 2: Configure in CodeXray

Navigate to CodeXray Dashboard → Integrations
Select "Postman API Key"
Paste your API key
Select workspaces/collections to monitor
Set synchronization frequency (hourly, daily, weekly)
Click "Save Configuration"

Step 3: Verify Connection

CodeXray will test the connection
Review discovered collections
Confirm synchronization is working

Permissions Required

Your Postman API key needs:

Read access to collections
Read access to environments (optional)
Workspace visibility permissions

Security Considerations

API keys grant read-only access to collections
Keys can be revoked anytime from Postman settings
CodeXray stores keys encrypted at rest
Audit logs track all API key usage

Use Cases

Continuous monitoring of evolving APIs
Automated security testing in CI/CD pipelines
Multi-team API governance
Centralized API inventory management

4. L7 Traffic Streaming

Layer 7 (application layer) traffic streaming captures API calls directly from your network infrastructure, providing complete visibility without application-level instrumentation.

L7 Traffic Configuration

Interactive videos: L7 traffic streaming setup

What is L7 Traffic Streaming?

L7 streaming captures HTTP/HTTPS traffic at the network layer by mirroring traffic from load balancers, API gateways, or network taps. This provides infrastructure-wide visibility.

Architecture:

Client → Load Balancer → Mirror Port → CodeXray Collector → Analysis
              ↓
         Application

How It Works

Traffic Mirroring: Configure your load balancer or network device to mirror HTTP/HTTPS traffic
Collector Deployment: Deploy CodeXray collector to receive mirrored traffic
TLS Decryption: Optional TLS termination for HTTPS inspection
Stream Processing: Real-time parsing and forwarding to CodeXray platform
Analysis: Immediate security analysis and risk scoring

Supported Infrastructure

Load Balancers:

NGINX (mirror module)
HAProxy (traffic mirroring)
AWS ALB/ELB (access logs)
Azure Application Gateway
Google Cloud Load Balancer

API Gateways:

Kong Gateway
AWS API Gateway
Azure API Management
Apigee

Network Devices:

Cisco ASA
F5 BIG-IP
Port mirroring (SPAN/TAP)

Configuration Example (NGINX)

location /api/ {
    # Forward to backend
    proxy_pass http://backend;
    
    # Mirror traffic to CodeXray collector
    mirror /mirror;
    mirror_request_body on;
}

location /mirror {
    internal;
    proxy_pass http://codexray-collector:8080;
    proxy_set_header X-Original-URI $request_uri;
}

Performance Impact

Minimal latency (typically <5ms)
Asynchronous mirroring doesn't block requests
No application code changes required
Scales horizontally with traffic volume

Use Cases

Infrastructure-wide API discovery
Legacy application monitoring (no code changes)
Multi-application environments
Shadow API detection
Compliance monitoring

Note: L7 streaming requires network-level access and may need coordination with your infrastructure team.

5. SDK Instrumentation

SDK instrumentation provides the deepest level of integration by embedding directly into your application code. The SDK intercepts all API requests and responses in real-time.

How SDK Instrumentation Works

The CodeXray SDK acts as middleware in your application, capturing API telemetry without modifying business logic.

Request Flow:

Client Request → SDK Intercept → Your API Handler → SDK Intercept → Response
                      ↓                                    ↓
                 Send to Collector                  Send to Collector

What the SDK Captures

Request Data:

HTTP method and full URL
Request headers (with configurable redaction)
Request body (analyzed for PII)
Authentication tokens (hashed)
Client IP and user agent
Timestamp and request ID

Response Data:

HTTP status code
Response headers
Response body (analyzed for sensitive data exposure)
Response time and duration
Error messages and stack traces

Metadata:

Application version
Environment (dev, staging, production)
Service name and instance ID
User context (when available)

Supported Languages & Frameworks

Node.js

const CodeXray = require('codexray-sdk');

// Initialize SDK
CodeXray.init({
  apiKey: 'your-api-key',
  serviceName: 'payment-api',
  environment: 'production',
  sampleRate: 1.0 // 100% of requests
});

// Express middleware
app.use(CodeXray.middleware());

Python (Flask/Django)

from codexray import CodeXrayMiddleware

# Initialize
codexray = CodeXrayMiddleware(
    api_key='your-api-key',
    service_name='user-service',
    environment='production'
)

# Flask
app.wsgi_app = codexray.wrap_wsgi(app.wsgi_app)

# Django - Add to MIDDLEWARE settings
MIDDLEWARE = [
    'codexray.middleware.CodeXrayMiddleware',
    # ... other middleware
]

Java (Spring Boot)

@Configuration
public class CodeXrayConfig {
    
    @Bean
    public FilterRegistrationBean<CodeXrayFilter> codexrayFilter() {
        FilterRegistrationBean<CodeXrayFilter> registration = 
            new FilterRegistrationBean<>();
        
        registration.setFilter(new CodeXrayFilter(
            "your-api-key",
            "order-service",
            "production"
        ));
        
        registration.addUrlPatterns("/api/*");
        return registration;
    }
}

import "github.com/codexray/go-sdk"

func main() {
    // Initialize SDK
    codexray.Init(codexray.Config{
        APIKey:      "your-api-key",
        ServiceName: "inventory-api",
        Environment: "production",
    })
    
    // Wrap HTTP handler
    http.Handle("/api/", codexray.Middleware(apiHandler))
}

Configuration Options

Basic Configuration:

api_key: your-api-key-here
service_name: payment-api
environment: production

Advanced Configuration:

api_key: your-api-key-here
service_name: payment-api
environment: production

# Sampling
sample_rate: 1.0  # 100% of requests

# Data redaction
redact_patterns:
  - "password"
  - "credit_card"
  - "ssn"

# Performance
buffer_size: 1000
flush_interval: 5s
async_mode: true

# Filtering
include_paths:
  - "/api/*"
exclude_paths:
  - "/health"
  - "/metrics"

# Security
tls_verify: true

How Data Flows

Interception: SDK intercepts request before it reaches your handler
Data Collection: Captures headers, body, and metadata
Redaction: Automatically redacts sensitive patterns (credit cards, passwords)
Buffering: Stores telemetry in memory buffer
Transmission: Sends batched data to CodeXray collector (async)
Analysis: Real-time security analysis and risk scoring
Response: Original request proceeds normally (no blocking)

Performance Characteristics

Latency Impact: <1ms per request (async processing)
Memory Usage: ~50MB for buffer and SDK runtime
CPU Usage: <2% additional overhead
Network: Batched transmission (minimal bandwidth)

Security Features

Automatic PII Redaction:

Credit card numbers
Social security numbers
Email addresses (configurable)
Phone numbers
Passwords and tokens

Secure Transmission:

TLS 1.2+ encryption
Certificate pinning
API key authentication
Compressed payloads

Privacy Controls:

Configurable field redaction
Request/response body size limits
Header blacklisting
User data anonymization

Deployment Best Practices

Development Environment:

Start with sample_rate: 0.1 (10% of traffic)
Test with non-sensitive endpoints first
Validate data redaction rules
Monitor SDK performance impact

Staging Environment:

Increase sample_rate to 0.5 (50%)
Test with production-like load
Verify all endpoints are captured
Review PII detection accuracy

Production Deployment:

Deploy to small percentage of instances first
Monitor error rates and performance
Gradually roll out to all instances
Set sample_rate to 1.0 for complete coverage

Troubleshooting

SDK Not Sending Data:

Verify API key is correct
Check network connectivity to collector
Review application logs for SDK errors
Confirm firewall allows outbound HTTPS

High Memory Usage:

Reduce buffer_size configuration
Decrease flush_interval
Enable request body size limits

Missing Endpoints:

Check include_paths/exclude_paths filters
Verify SDK middleware is registered correctly
Review sample_rate configuration

Use Cases

Production runtime monitoring
Real-time security analysis
Shadow API discovery in live traffic
User behavior analysis
Performance monitoring alongside security
Compliance auditing

Recommended: SDK instrumentation provides the most comprehensive API visibility and is the preferred method for production environments.

Comparison Matrix

Feature	HAR Upload	Postman Upload	Postman API Key	L7 Streaming	SDK
Setup Complexity	Very Low	Very Low	Low	High	Medium
Continuous Monitoring	❌	❌	❌	✅	✅
Real-time Analysis	❌	❌	❌	✅	✅
Code Changes Required	❌	❌	❌	✅	✅
Shadow API Discovery	✅	❌	❌	✅	✅
Production Ready	❌	❌	⚠️	✅	✅
Captures Runtime Behavior	✅	❌	❌	✅	✅
Infrastructure Access Required	❌	❌	❌	✅	❌

Integration Recommendations

For Quick Testing

Start with HAR Files or Postman Collections for immediate results without any setup.

For Development Teams

Use Postman API Key integration to automatically sync your evolving API specifications.

For Infrastructure Teams

Deploy L7 Traffic Streaming for organization-wide visibility without touching application code.

For Production Monitoring

Implement SDK Instrumentation for comprehensive, real-time API security monitoring.

Hybrid Approach (Recommended)

Combine multiple methods:

SDK for production runtime monitoring
Postman for documented API specifications
HAR for ad-hoc testing and debugging

Next Steps

Getting Help

Integration Support:

Documentation: https://docs.codexray.com
Support Email: support@codexray.io
Community Forum: https://community.codexray.com

Start collecting API data today and gain immediate visibility into your security posture.

Supported Collection Methods​

1. HAR File Upload​

How to Upload HAR Files​

What is a HAR File?​

How to Generate HAR Files​

HAR File Structure Example​

Use Cases​

2. Postman Collection Upload​

How to Upload Postman Collections​

What is a Postman Collection?​

How to Export from Postman​

Collection Structure Example​

Use Cases​

3. Postman API Key Integration​

How to Configure Postman API Key​

What is Postman API Key Integration?​

Setup Instructions​

Permissions Required​

Security Considerations​

Use Cases​

4. L7 Traffic Streaming​

L7 Traffic Configuration​

What is L7 Traffic Streaming?​

How It Works​

Supported Infrastructure​

Configuration Example (NGINX)​

Performance Impact​

Use Cases​

5. SDK Instrumentation​

How SDK Instrumentation Works​

What the SDK Captures​

Supported Languages & Frameworks​

Configuration Options​

How Data Flows​

Performance Characteristics​

Security Features​

Deployment Best Practices​

Troubleshooting​

Use Cases​

Comparison Matrix​

Integration Recommendations​

For Quick Testing​

For Development Teams​

For Infrastructure Teams​

For Production Monitoring​

Hybrid Approach (Recommended)​

Next Steps​

Getting Help​

Supported Collection Methods

1. HAR File Upload

How to Upload HAR Files

What is a HAR File?

How to Generate HAR Files

HAR File Structure Example

Use Cases

2. Postman Collection Upload

How to Upload Postman Collections

What is a Postman Collection?

How to Export from Postman

Collection Structure Example

Use Cases

3. Postman API Key Integration

How to Configure Postman API Key

What is Postman API Key Integration?

Setup Instructions

Permissions Required

Security Considerations

Use Cases

4. L7 Traffic Streaming

L7 Traffic Configuration

What is L7 Traffic Streaming?

How It Works

Supported Infrastructure

Configuration Example (NGINX)

Performance Impact

Use Cases

5. SDK Instrumentation

How SDK Instrumentation Works

What the SDK Captures

Supported Languages & Frameworks

Configuration Options

How Data Flows

Performance Characteristics

Security Features

Deployment Best Practices

Troubleshooting

Use Cases

Comparison Matrix

Integration Recommendations

For Quick Testing

For Development Teams

For Infrastructure Teams

For Production Monitoring

Hybrid Approach (Recommended)

Next Steps

Getting Help