@fnet/files-to-gcs

@fnet/files-to-gcs is a straightforward utility designed to facilitate the uploading of files from a specified directory to a Google Cloud Storage bucket. It offers an efficient way to transfer files, making it suitable for users needing to manage files in cloud storage environments.

How It Works

The utility connects to a Google Cloud Storage bucket using your specified project credentials. It takes a pattern to match files within a directory, and uploads these files to a destination directory in the specified GCS bucket. You have the option to perform a dry run to preview the files and their destinations without actually uploading them.

Key Features

• Upload files from a local directory to a Google Cloud Storage bucket.
• Specify a pattern to filter which files to upload.
• Customize the destination directory within the GCS bucket.
• Option for a dry run to see what would be uploaded without making changes.
• Set custom metadata for uploaded files.
• Verbose mode to log the details of each file upload.

Conclusion

This utility is a useful tool for users needing to transfer files from their local directories to Google Cloud Storage. Its simplicity and support for dry runs help ensure that users can confidently manage their file uploads without unintended changes.

Developer Guide for @fnet/files-to-gcs

Overview

The @fnet/files-to-gcs library facilitates the process of uploading files to Google Cloud Storage (GCS). It is designed to help developers transfer files from a local directory to a specified GCS bucket efficiently. With features such as custom metadata, dry runs for testing, and optional logging, this library provides a straightforward solution for managing file uploads.

Installation

To use the @fnet/files-to-gcs library, install it via npm or yarn:

npm install @fnet/files-to-gcs

or

yarn add @fnet/files-to-gcs

Usage

Here's how you can utilize the library to upload files to GCS:

import uploadFilesToGCS from '@fnet/files-to-gcs';

(async () => {
  try {
    const result = await uploadFilesToGCS({
      projectId: 'your-gcp-project-id',
      keyFilename: 'path/to/your-service-account-key.json',
      bucketName: 'your-bucket-name',
      dir: 'path/to/local/files',
      pattern: '**/*.jpg', // Pattern to match specific files
      destDir: 'uploads',  // Destination directory in GCS
      metadata: { cacheControl: 'private, max-age=0, no-transform' },
      domain: 'your-custom-domain.com', // Optional custom domain
      verbose: true
    });

    console.log(result);
  } catch (error) {
    console.error('Error uploading files:', error);
  }
})();

Examples

Basic File Upload

Upload all .png files from a local directory to the root of a GCS bucket:

import uploadFilesToGCS from '@fnet/files-to-gcs';

uploadFilesToGCS({
  projectId: 'my-project-id',
  keyFilename: '/path/to/keyfile.json',
  bucketName: 'my-bucket',
  dir: '/path/to/files',
  pattern: '**/*.png',
}).then(result => {
  console.log('Files uploaded:', result.files);
});

Dry Run

Simulate an upload without actually transferring any files:

import uploadFilesToGCS from '@fnet/files-to-gcs';

uploadFilesToGCS({
  projectId: 'my-project-id',
  keyFilename: '/path/to/keyfile.json',
  bucketName: 'my-bucket',
  dir: '/path/to/files',
  pattern: '**/*.txt',
  dryRun: true, // No files will be uploaded
}).then(result => {
  console.log('Dry run result:', result.files);
});

Verbose Mode

Enable verbose logging to see detailed output during the process:

import uploadFilesToGCS from '@fnet/files-to-gcs';

uploadFilesToGCS({
  projectId: 'my-project-id',
  keyFilename: '/path/to/keyfile.json',
  bucketName: 'my-bucket',
  dir: '/path/to/files',
  pattern: '**/*.js',
  verbose: true, // Logs each file's upload details
}).then(result => {
  console.log('Uploaded files with verbose output');
});

Acknowledgement

This library leverages the @google-cloud/storage package for interacting with Google Cloud Storage. It also uses @fnet/list-files for listing files based on specific patterns.

Input Schema

type: object
properties:
  projectId:
    type: string
    description: Project ID for Google Cloud
  keyFilename:
    type: string
    description: Path to the service account key file
  bucketName:
    type: string
    description: Name of the Google Cloud Storage bucket
  dir:
    type: string
    description: Directory to search for files to upload
  pattern:
    type: string
    description: Glob pattern to match files
  destDir:
    type: string
    description: Destination directory in the bucket
    default: /
  dryRun:
    type: boolean
    description: If true, simulates the upload process without executing
    default: false
  metadata:
    type: object
    description: Additional metadata to set for each file
  domain:
    type: string
    description: Domain to use instead of the default Google Cloud Storage domain
  verbose:
    type: boolean
    description: If true, logs details of uploaded files
    default: false
required:
  - projectId
  - keyFilename
  - bucketName
  - pattern