npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@warren-bank/hls-proxy

v3.5.4

Published

Node.js server to proxy HLS video streams

Downloads

171

Readme

HLS Proxy : HTTP Live Streaming Proxy

Basic Functionality:

  • proxy .m3u8 files, and the video segments (.ts files) they internally reference
  • to all proxied files:
    • add permissive CORS response headers
  • to .m3u8:
    • modify contents such that URLs in the playlist will also pass through the proxy

Advanced Features:

  • inject custom HTTP headers in all outbound proxied requests
  • prefetch video segments (.ts files)
  • use a hook function to conditionally decide which video segments to prefetch
  • use a hook function to conditionally redirect URLs in the playlist (before and/or after they're modified to pass through the proxy)
  • use a hook function to conditionally rewrite URLs after they're received by the proxy

Benefits:

  • any video player (on the LAN) can access the proxied video stream
    • including Chromecast
  • prefetch and caching of video segments ahead-of-time makes playback of the video stream very stable
    • solves buffering problems
  • the proxy can easily be configured to bypass many of the security measures used by video servers to restrict access:
    • CORS response headers (to XHR requests)
      • used by web browsers to enforce a security policy that limits which website(s) may access the content
    • HTTP request headers
      • Origin and Referer are often inspected by the server
        • when these headers don't match the site hosting the content, a 403 Forbidden response is returned (in lieu of the requested data)
    • restricted access to encryption keys
      • often times the encrypted video segments (.ts files) are readily available, but the encryption keys are well protected
        • if the keys can be obtained from another source, then a hook function can be used to redirect only those URL requests

URL Format:

  • [example Javascript]: construction of URL to HLS Proxy for video stream
      {
        const proxy_url      = 'http://127.0.0.1:8080'
        const video_url      = 'https://example.com/video/master.m3u8'
        const file_extension = '.m3u8'
    
        const hls_proxy_url  = `${proxy_url}/${ btoa(video_url) }${file_extension}`
      }
  • [example Javascript]: construction of URL to HLS Proxy for video stream w/ "Referer" request header
      {
        const proxy_url      = 'http://127.0.0.1:8080'
        const video_url      = 'https://example.com/video/master.m3u8'
        const referer_url    = 'https://example.com/videos.html'
        const file_extension = '.m3u8'
    
        const hls_proxy_url  = `${proxy_url}/${ btoa(`${video_url}|${referer_url}`) }${file_extension}`
      }
  • [example Bash]: construction of URL to HLS Proxy for video stream
      proxy_url='http://127.0.0.1:8080'
      video_url='https://example.com/video/master.m3u8'
      file_extension='.m3u8'
    
      hls_proxy_url="${proxy_url}/"$(echo -n "$video_url" | base64 --wrap=0)"$file_extension"
  • [example Bash]: construction of URL to HLS Proxy for video stream w/ "Referer" request header
      proxy_url='http://127.0.0.1:8080'
      video_url='https://example.com/video/master.m3u8'
      referer_url='https://example.com/videos.html'
      file_extension='.m3u8'
    
      hls_proxy_url="${proxy_url}/"$(echo -n "${video_url}|${referer_url}" | base64 --wrap=0)"$file_extension"
notes:
  • adding a file extension to the base64 encoded video URL is highly recommended
    • the following file extension values have important significance to indicate the type of file being requested:
      • .m3u8HLS manifest
      • .tsmedia segment
      • .keyencryption key
      • .jsonJSON data
    • though currently,
      • .m3u8is the only file extension that receives special treatment
      • all other file types (including those without any file extension) are piped directly to the HTTP response
high-level tools that automate this task:

Installation and Usage: Globally

How to: Install:

npm install --global "@warren-bank/hls-proxy"

How to: Run the server(s):

hlsd <options>

options:
========
--help
--version
--tls
--host <host>
--port <number>
--copy-req-headers
--req-headers <filepath>
--origin <header>
--referer <header>
--useragent <header>
--header <name=value>
--req-options <filepath>
--req-insecure
--req-secure-honor-server-cipher-order
--req-secure-ciphers <string>
--req-secure-protocol <string>
--req-secure-curve <string>
--hooks <filepath>
--prefetch
--max-segments <number>
--cache-timeout <number>
--cache-key <number>
--cache-storage <adapter>
--cache-storage-fs-dirpath <dirpath>
-v <number>
--acl-ip <ip_address_list>
--acl-pass <password_list>
--http-proxy <http[s]://[user:pass@]hostname:port>
--tls-cert <filepath>
--tls-key <filepath>
--tls-pass <filepath>
--manifest-extension <ext>
--segment-extension <ext>

Options:

  • --tls is a flag to start HTTPS proxy, rather than HTTP
    • used as shorthand to automatically configure the following options:
      • --tls-cert
      • --tls-key
      • --tls-pass
    • the values assigned to these options enable the use of a self-signed security certificate that is included in both the git repo and npm package, within the directory:
    • when all of these option are properly specified:
      • the https: protocol is used by all URLs in modified HLS manifests
  • --host is an IP or hostname with an optional port number that can be resolved and is reachable by clients
    • ex: 192.168.0.100:8080
    • used to modify URLs in .m3u8 files
    • when this option is specified without a port number:
      • the value of the --port option is appended
    • when this option is specified and the port number is 443:
      • the https: protocol is used by all URLs in modified HLS manifests
    • when this option is not specified:
      • the value of the "Host" HTTP request header is used
  • --port is the port number that the server listens on
    • ex: 8080
    • when this option is not specified:
      • HTTP proxy binds to: 80
      • HTTPS proxy binds to: 443
  • --copy-req-headers is a flag to enable the duplication of all HTTP request headers sent to the proxy → to the request made by the proxy to the video server
  • --req-headers is the filepath to a JSON data Object containing key:value pairs
    • each key is the name of an HTTP header to send in every outbound request
  • --origin is the value of the corresponding HTTP request header
  • --referer is the value of the corresponding HTTP request header
  • --useragent is the value of the corresponding HTTP request header
  • --header is a single name:value pair
    • this option can be used multiple times to include several HTTP request headers
    • the pair can be written:
      • "name: value"
      • "name=value"
      • "name = value"
  • --req-options is the filepath to a JSON data Object
  • --req-insecure is a flag to override the following environment variable to disable certificate validation for secure https requests:
  • --req-secure-honor-server-cipher-order is a flag to set the following key in the request options Object to configure the context for secure https requests:
    • {honorCipherOrder: true}
  • --req-secure-ciphers is the value to assign to the following key in the request options Object to configure the context for secure https requests:
    • {ciphers: value}
  • --req-secure-protocol is the value to assign to the following key in the request options Object to configure the context for secure https requests:
    • {secureProtocol: value}
  • --req-secure-curve is the value to assign to the following key in the request options Object to configure the context for secure https requests:
    • {ecdhCurve: value}
  • --hooks is the filepath to a CommonJS module that exports a single JSON Object
    • each key is the name of a hook function
    • each value is the implementation of the corresponding Function
    • hook function signatures:
      • "add_request_options": (url, is_m3u8) => request_options
        • conditionally add HTTP request options
        • inputs:
          • url
            • string URL
          • is_m3u8
            • boolean that indicates whether url will request an HLS manifest
        • return value:
          • Object having attributes that are combined with --req-options and used to send the outbound request to url
      • "add_request_headers": (url, is_m3u8) => request_headers
        • conditionally add HTTP request headers
        • return value:
          • Object containing key:value pairs that are combined with --req-headers
            • each key is the name of an HTTP header to send in the outbound request to url
      • "modify_m3u8_content": (m3u8_content, m3u8_url) => new_m3u8_content
        • conditionally modify the content of .m3u8 files before they are parsed to extract URLs
      • "redirect": (url) => new_url
        • conditionally redirect the URLs encountered in .m3u8 files before they are modified to pass through the proxy
      • "redirect_final": (url) => new_url
        • conditionally redirect the URLs encountered in .m3u8 files after they are modified to pass through the proxy
      • "rewrite": (url) => new_url
        • conditionally rewrite the URLs requested by clients before they are proxied
      • "prefetch": (url) => boolean
        • conditionally decide whether to prefetch video segments on a per-URL basis
        • return value must be a strict boolean type (ie: true or false)
        • otherwise, the default behavior supersedes
          • to only prefetch .ts files
      • "prefetch_segments": (prefetch_urls, max_segments, is_vod, seg_duration_ms, perform_prefetch) => new_prefetch_urls
        • conditionally filter the list of video segment URLs that are pending prefetch, when more than --max-segments are contained in an HLS manifest
        • inputs:
          • prefetch_urls
            • array of string video segment URLs
          • max_segments
            • integer that denotes the max length of the return value
          • is_vod
            • boolean that indicates whether the HLS manifest is for video-on-demand
              • if true:
                • the video is not a live stream
                • the HLS manifest is complete and contains URLs for every video segment that would be needed to play the entire stream from start to finish
          • seg_duration_ms
            • integer that represents the duration (ms) of each video segment in the HLS manifest
          • perform_prefetch
            • function that accepts an array of string video segment URLs, and immediately begins to prefetch all corresponding segments
        • return value:
          • array of string video segment URLs that is a subset of prefetch_urls
            • can be emtpy (ex: when using perform_prefetch)
        • pre-conditions:
          • the length of prefetch_urls is > max_segments
        • post-conditions:
          • the length of the return value array is <= max_segments
      • "request_intervals": (add_request_interval) => {}
        • enables the use of a cookie jar for all outbound HTTP requests
        • adds any number of timers that each execute at individually specified intervals
        • when each timer executes, it is passed an HTTP request client that is preconfigured to:
          • include the request headers that are specified by other relevant options
          • utilize the same cookie jar as all other outbound HTTP requests
            • this allows the implementation of custom logic that may be required by one or more video hosts to periodically refresh or update session cookies
        • an example would better illustrate usage:
            module.exports = {
              request_intervals: (add_request_interval) => {
          
                add_request_interval(
                  0, // run timer once at startup to initialize cookies
                  (request) => {
                    request('https://example.com/login', {username: 'me', password: '12345'})
                  }
                )
          
                add_request_interval(
                  (1000 * 60 * 5), // run timer at 5 minute interval to refresh cookies
                  (request) => {
                    request('https://example.com/heart-beat')
                  }
                )
          
              }
            }
        • more advanced configuration of the call to the HTTP request client is possible
          • the 1st parameter is required, and must be a URL string
          • the 2nd parameter is optional, and can contain POST data
          • the 3rd parameter is optional, and can be used for more advanced configuration options
        • usage of this HTTP request client is documented here
          • specifically, pay careful attention to the signatures for:
            • the latter two input parameters
            • the attributes of the Object that is resolved by the Promise in the return value (if the content of the response is needed)
  • --prefetch is a flag to enable the prefetch and caching of video segments
    • when .m3u8 files are downloaded and modified inflight, all of the URLs in the playlist are known
    • at this time, it is possible to prefetch the video segments (.ts files)
    • when the video segments (.ts files) are requested at a later time, the data is already cached (in memory) and can be returned immediately
  • --max-segments is the maximum number of video segments (.ts files) to hold in each cache
    • this option is only meaningful when --prefetch is enabled
    • a cache is created for each unique HLS manifest URL
      • all of the video segments (.ts files) associated with each distinct video stream are stored in isolation
    • when any cache grows larger than this size, the oldest data is removed to make room to store new data
    • when this option is not specified:
      • default value: 20
  • --cache-timeout is the maximum number of seconds that any segment cache can remain unused before its contents are cleared (to reduce wasted space in memory)
    • this option is only meaningful when --prefetch is enabled
    • when this option is not specified:
      • default value: 60
  • --cache-key sets the type of string used to represent keys in the cache hashtable when logged
    • this option is only meaningful when --prefetch is enabled
    • scope:
      • v0.16.0 and earlier
        • keys in the cache hashtable used this string representation
      • v0.16.1 and later
        • keys in the cache hashtable are full URLs
          • the data structure to cache video segments (.ts files) was updated
          • each unique HLS manifest is associated with a distinct FIFO list that holds --max-segments
          • when a video segment is requested
            • the proxy needs to search every FIFO list for a match
            • when keys in the cache hashtable lose fidelity, collisions can occur and the wrong video segment can be returned
            • full URLs are unique and guarantee correct behavior
    • 0 (default):
      • sequence number of .ts file w/ .ts file extension (ex: "123.ts")
        • pros:
          • shortest type of string
          • makes the log output easiest to read
        • cons:
          • in the wild, I've encountered video servers that assign each .ts file a unique filename that always terminate with the same static sequence number
            • this is a strange edge case, but this option provides an easy workaround
    • 1:
      • full filename of .ts file
    • 2:
      • full URL of .ts file
  • --cache-storage selects a storage adapter that is used to hold the cache of prefetched video segments
    • this option is only meaningful when --prefetch is enabled
    • memory (default):
      • uses RAM
    • filesystem:
      • each video segment is written to a new file within a specified directory
      • filenames are random and unique
  • --cache-storage-fs-dirpath specifies the directory in which to save video segments when using a filesystem-based cache storage adapter
    • this option is only meaningful when --prefetch is enabled and --cache-storage is filesystem
  • -v sets logging verbosity level:
    • -1:
      • silent
    • 0 (default):
      • show errors only
    • 1:
      • show an informative amount of information
    • 2:
      • show technical details
    • 3:
      • show an enhanced technical trace (useful while debugging unexpected behavior)
    • 4:
      • show the content of .m3u8 files (both before and after URLs are modified)
  • --acl-ip restricts proxy server access to clients at IP addresses in whitelist
    • ex: "192.168.1.100,192.168.1.101,192.168.1.102"
  • --acl-pass restricts proxy server access to requests that include a password querystring parameter having a value in whitelist
    • ex: "1111,2222,3333,4444,5555"
  • --http-proxy enables all outbound HTTP and HTTPS requests from HLS-Proxy to be tunnelled through an additional external web proxy server
  • --tls-cert is the filepath to a security certificate to use for HTTPS
  • --tls-key is the filepath to the private key for the --tls-cert security certificate
  • --tls-pass is the filepath to a text file containing the security passphrase for the --tls-key private key
    • optional, not required when the --tls-key private key was created without a security passphrase
  • --manifest-extension is the file extension associated with HLS manifests
    • default value: m3u8
  • --segment-extension is the file extension associated with media segments
    • default value: ts

Examples:

  1. print help hlsd --help

  2. print version hlsd --version

  3. start HTTP proxy at default host:port hlsd

  4. start HTTP proxy at default host and specific port hlsd --port "8080"

  5. start HTTP proxy at specific host:port hlsd --host "192.168.0.100" --port "8080"

  6. start HTTPS proxy at default host:port hlsd --tls

  7. start HTTPS proxy at specific host:port hlsd --tls --host "192.168.0.100" --port "8081"

  8. start HTTPS proxy at default host:port and send specific HTTP headers hlsd --tls --req-headers "/path/to/request/headers.json"

  9. start HTTPS proxy at default host:port and enable prefetch of 10 video segments hlsd --tls --prefetch --max-segments 10

  10. start HTTPS proxy using a non-generic security certificate hlsd --tls-cert "/path/to/cert.pem" --tls-key "/path/to/key.pem" --tls-pass "/path/to/pass.phrase"


Installation and Usage: Working with a Local git Repo

How to: Install:

git clone "https://github.com/warren-bank/HLS-Proxy.git"
cd "HLS-Proxy"
npm install

How to: Run the server(s):

# ----------------------------------------------------------------------
# If using a port number >= 1024 on Linux, or
# If using Windows:
# ----------------------------------------------------------------------
npm start [-- <options>]

# ----------------------------------------------------------------------
# https://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html
#
# Linux considers port numbers < 1024 to be privileged.
# Use "sudo":
# ----------------------------------------------------------------------
npm run sudo [-- <options>]

Options:

Examples:

  1. print help npm start -- --help

  2. start HTTP proxy at specific host:port npm start -- --host "192.168.0.100" --port "8080"

  3. start HTTPS proxy at specific host:port npm start -- --host "192.168.0.100" --port "8081" --tls

  4. start HTTP proxy at default host:port with escalated privilege npm run sudo -- --port "80"

  5. start HTTPS proxy at default host:port with escalated privilege npm run sudo -- --port "443" --tls

  6. start HTTP proxy at specific port and send custom "Referer" request header for specific video stream

npm start -- --port "8080"

h_referer='http://XXX:80/page.html'

URL='https://httpbin.org/headers'
URL="${URL}|${h_referer}"
URL=$(echo -n "$URL" | base64 --wrap=0)
URL="http://127.0.0.1:8080/${URL}.json"
# URL='http://127.0.0.1:8080/aHR0cHM6Ly9odHRwYmluLm9yZy9oZWFkZXJzfGh0dHA6Ly9YWFg6ODAvcGFnZS5odG1s.json'
curl --silent "$URL"
  1. start HTTP proxy at specific port and send custom request headers
headers_file="${TMPDIR}/headers.json"
echo '{"Origin" : "http://XXX:80", "Referer": "http://XXX:80/page.html"}' > "$headers_file"
npm start -- --port "8080" --req-headers "$headers_file"

URL='https://httpbin.org/headers'
URL=$(echo -n "$URL" | base64 --wrap=0)
URL="http://127.0.0.1:8080/${URL}.json"
# URL='http://127.0.0.1:8080/aHR0cHM6Ly9odHRwYmluLm9yZy9oZWFkZXJz.json'
curl --silent "$URL"
  1. start HTTPS proxy at specific port and send custom request headers
headers_file="${TMPDIR}/headers.json"
echo '{"Origin" : "http://XXX:80", "Referer": "http://XXX:80/page.html"}' > "$headers_file"
npm start -- --port "8081" --req-headers "$headers_file" --tls -v 1

URL='https://127.0.0.1:8081/aHR0cHM6Ly9odHRwYmluLm9yZy9oZWFkZXJz.json'
curl --silent --insecure "$URL"
  1. start HTTPS proxy at specific port and send custom request headers
h_origin='http://XXX:80'
h_referer='http://XXX:80/page.html'
h_useragent='Chromium'
h_custom_1='X-Foo: 123'
h_custom_2='X-Bar: baz'
npm start -- --port "8081" --origin "$h_origin" --referer "$h_referer" --useragent "$h_useragent" --header "$h_custom_1" --header "$h_custom_2" --tls -v 1

URL='https://127.0.0.1:8081/aHR0cHM6Ly9odHRwYmluLm9yZy9oZWFkZXJz.json'
curl --silent --insecure "$URL"

Observations:

  • when playing the proxied HLS video stream in an HTML5 player in a Chromium web browser (ex: THEOplayer)
    • if the page hosting the HTML5 video player is served from HTTPS:
      • when running only the HTTP proxy server:
        • the XHR requests from the player to the HTTP proxy server raise a security warning (insecure content)
        • the XHR requests get elevated to HTTPS, which are unanswered (since the HTTPS proxy server isn't running)
      • when running only the HTTPS proxy server:
        • the XHR requests from the player to the HTTPS proxy server will silently fail
        • this is because the HTTPS proxy server is using a self-signed security certificate
        • this certificate needs to be (temporarily) allowed
        • once it is, the video stream works perfectly
          • to allow the certificate:
            • browse to a URL hosted by the proxy server ( example )
            • you should see the warning: NET::ERR_CERT_AUTHORITY_INVALID Your connection is not private
            • click: Advanced
            • click: Proceed to 127.0.0.1 (unsafe)
            • done
  • when playing the proxied HLS video stream on a Chromecast
    • the HTTP proxy server works perfectly
    • the HTTPS proxy server doesn't begin playback
      • not sure why..
      • probably has something to do with the Chromecast's browser security policies
      • a more respectable security certificate (ie: more expensive) would probably fix it

Summary of (Rarely) Observed OpenSSL Connection Errors:

  • error: ssl3_check_cert_and_algorithm:dh key too small

    1. attempted fix: --req-secure-ciphers "AES128-SHA"
  • error: SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

    1. attempted fix: --req-secure-protocol "SSLv3_method"
      • result: Error: SSLv3 methods disabled
      • issue:
    2. attempted fix: --req-secure-curve "auto"

Other Projects:

(directly related, but very loosely coupled)
  • Webcast-Reloaded
    • consists of 2 parts:
      1. a Chromium web browser extension (.crx)
        • on each browser tab, it's silently watching the URL of all outbound requests
        • every requested URL matching a regex pattern that identifies it to be a video file is displayed in the modal window that toggles open when the extension's icon is clicked
        • links in this modal window open to URLs of component #2
      2. a static website
        • there is a selection of several HTML5 videos players
          • each is better at some things and worse at others
          • each integrates with a different Chromecast receiver app
        • there is a page to help redirect the intercepted video URL through a local instance of HLS Proxy

Other Projects:

(unrelated, but somewhat similar in scope and purpose)
  • Streamlink
    • notes:
      • this project has way more features, and is way more polished
      • though its main purpose is to transcode online video with ffmpeg and pipe the output into another program, it can be configured to not load a video player and instead start a web server
      • it can strongly support individual websites through single-purpose plugins
      • it can also support streams via direct URLs
        • using URLs from the wild will have mixed results, since cookies and headers and authentication aren't being managed by any plugin
    • docs:
    • binaries:
    • usage test:
      • streamlink --player-external-http --player-external-http-port 8080 --default-stream best --http-ignore-env --http-no-ssl-verify --url "https://XXX/video.m3u8"
    • usage test result:

Major Versions:

  • v1.x
    • commit history is in branch: v01
    • summary:
      • m3u8 manifest parser uses regex patterns to identify all URL patterns without any special knowledge of the m3u8 manifest specification
      • internal proxy module exports a function that accepts an instance of http.Server and adds event listeners to process requests
    • system requirements:
  • v2.x
    • commit history is in branch: v02
    • summary:
      • m3u8 manifest parser uses regex patterns to identify all URL patterns without any special knowledge of the m3u8 manifest specification
      • internal proxy module exports an Object containing event listeners to process requests that can be either:
        • added to an instance of http.Server
        • added to an Express.js application as middleware to handle a custom route
          • important limitation: since / is a valid character in a base64 encoded URL, the path for a custom route needs to end with a character that is not allowed in base64 encoding (ex: '/proxy_/*')
    • system requirements:
  • v3.x
    • commit history is in branch: v03
    • summary:
      • m3u8 manifest parser uses special knowledge of the m3u8 manifest specification to contextually identify URLs
      • internal proxy module exports an Object containing event listeners to process requests that can be either:
        • added to an instance of http.Server
        • added to an Express.js application as middleware to handle a custom route
          • important requirement: the path for a custom route needs to include exactly one unnamed parameter that matches the base64 encoded URL and (optionally) a file extension (ex: '/proxy/*')
          • the use of nested routers is supported
    • system requirements:

Legal: