TunnelHub SDK

TunnelHub SDK is a robust TypeScript library for implementing automated integrations with logging and tracing capabilities on the TunnelHub platform.

Overview

The SDK provides a foundation for building reliable data integrations with features including:

• Delta detection and synchronization
• Batch processing capabilities
• Comprehensive logging and monitoring
• Error handling and retry mechanisms
• AWS infrastructure integration

Installation

npm install @tunnelhub/sdk

Core Concepts

Integration Flows

The SDK provides three main types of integration flows:

1. Delta Integration Flow (DeltaIntegrationFlow)

   • Tracks changes between source and target systems
   • Handles insert, update, and delete operations
   • Maintains state between executions
   • Best for synchronization scenarios

2. Batch Delta Integration Flow (BatchDeltaIntegrationFlow)

   • Extends Delta Integration Flow
   • Processes items in configurable batch sizes
   • Optimized for large datasets
   • Supports bulk operations

3. No Delta Integration Flow (NoDeltaIntegrationFlow)

   • Simple one-way data transfer
   • No state tracking between executions
   • Available in single and batch variants
   • Ideal for one-time or streaming data transfers

Key Components

Automation Logs

The SDK automatically handles logging through the AutomationLog class, capturing:

• Operation type (INSERT, UPDATE, DELETE, NODELTA, TRANSFER)
• Status (SUCCESS, FAIL, NEUTRAL)
• Timestamps
• Detailed error messages
• Operation payloads

Delta Tracking

AutomationDelta manages state between executions:

• Stores previous execution state
• Enables change detection
• Persists in both DynamoDB and S3
• Handles large datasets efficiently

Parameters & Configuration

• Environment-specific configurations
• Custom parameter management
• Secure credential storage
• System connection details

Usage Examples

Creating a Delta Integration

class MyDeltaIntegration extends DeltaIntegrationFlow<MyDataType> {
    protected async loadSourceSystemData(): Promise<MyDataType[]> {
        // Implement source system data loading
    }

    protected async loadTargetSystemData(): Promise<MyDataType[]> {
        // Implement target system data loading
    }

    protected async insertAction(item: MyDataType): Promise<IntegrationMessageReturn> {
        // Implement insert logic
    }

    protected async updateAction(oldItem: MyDataType, newItem: MyDataType): Promise<IntegrationMessageReturn> {
        // Implement update logic
    }

    protected async deleteAction(item: MyDataType): Promise<IntegrationMessageReturn> {
        // Implement delete logic
    }

    protected defineMetadata(): Array<Metadata> {
        return [
            {
                fieldName: 'id',
                fieldLabel: 'ID',
                fieldType: 'TEXT'
            }
            // Add more metadata fields
        ];
    }
}

Creating a Batch Integration

class MyBatchIntegration extends BatchDeltaIntegrationFlow<MyDataType> {
    constructor(event: ProcessorPayload, context?: LambdaContext) {
        super(event, ['id'], ['name', 'value'], context);
        this.packageSize = 100; // Set batch size
    }

    protected async batchInsertAction(items: MyDataType[]): Promise<IntegrationMessageReturnBatch[]> {
        // Implement batch insert logic
    }

    protected async batchUpdateAction(oldItems: MyDataType[], newItems: MyDataType[]): Promise<IntegrationMessageReturnBatch[]> {
        // Implement batch update logic
    }

    protected async batchDeleteAction(items: MyDataType[]): Promise<IntegrationMessageReturnBatch[]> {
        // Implement batch delete logic
    }
}

Additional Features

Data Store

• Conversion table management
• Sequence generation
• System configuration storage

API Integration

• Built-in middleware for API Gateway
• Request/response logging
• Error handling

AWS Integration

• DynamoDB integration
• S3 storage
• Lambda context handling
• ECS task tracking

Best Practices

1. Error Handling

   • Implement proper try-catch blocks
   • Use appropriate error statuses
   • Provide meaningful error messages

2. Performance

   • Use batch operations for large datasets
   • Implement proper indexing in database queries
   • Monitor memory usage

3. Logging

   • Include relevant context in logs
   • Use appropriate log levels
   • Monitor execution statistics

4. Delta Management

   • Choose appropriate key fields
   • Implement proper change detection
   • Handle data consistency

Environment Variables

The following environment variables are required only when using the API integration features:

• TH_TENANT_ID: Tenant identifier
• TH_ENVIRONMENT_ID: Environment identifier
• TH_EXPIRATION_PERIOD: Log retention period in days

Contributing

Please read our Contributing Guidelines for details on submitting pull requests.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For detailed documentation, visit https://docs.tunnelhub.io

For support queries, contact [email protected]