npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@fluid-tools/fetch-tool

v2.12.0

Published

Console tool to fetch Fluid data from relay service

Downloads

1,915

Readme

@fluid-tools/fetch-tool

Connection using ODSP or routerlicious driver to dump the messages or snapshot information on the server. In order to connect to ODSP, the clientID must be set as the environment variable fetch__tool__clientId. If you have access to the keyvault this can be done by running this tool. Beware that to use fetch-tool on documents in the Microsoft tenant, you will need to follow the fetch tool usage instructions on the "Debugging Tools" page of the internal Fluid wiki.

NOTE: This package is a library intended for use within the microsoft/FluidFramework repository. It is not intended for public use. We make no stability guarantees regarding this library and its APIs.

Installation

To get started, install the package by running the following command:

npm i @fluid-tools/fetch-tool -D

Usage

Usage: fluid-fetch [options] "URL"
URL: <ODSP joinSession URL>|<Routerlicious URL>
Options:
  --dump:rawmessage               : dump all messages
  --dump:rawmessage:overwrite     : dump all messages and overwrite existing        messages.json
  --dump:snapshotVersion          : dump a list of snapshot version
  --dump:snapshotTree             : dump the snapshot trees
  --dump:snapshotBlob             : dump the contents of snapshot blobs
  --stat:message                  : show a table of message type counts and size
  --stat:snapshot                 : show a table of snapshot path and blob size
  --stat:dataType                 : show a table of data type
  --stat:channel                  : show a table of channel
  --filter:messageType <type>     : filter message by <type>
  --jwt <token>                   : token to be used for routerlicious URLs
  --numSnapshotVersions <number>  : Number of versions to load (default:10)
  --snapshotVersionIndex <number> : Index of the version to dump
  --saveDir <outdir>              : Save data of the snapshots and messages
  --loginHint <user>              : Username to prefill for interactive authentication against ODSP

Tips

  • If not done already run npm run build
  • Example command - run node lib/fluidFetch.js --saveDir example 'get-versions-fetch-url' in the fetch-tool directory
    • Look at the getVersions call in the relevant driver to build the right url
    • If you have access to the document, you can also look at the standard browser devtools and look for the url there as well.
    • This command creates an example directory (if it doesn't exist) in the fetch-tool folder.
    • If run multiple times without clearing the example directory, the snapshot will overwrite any old folders or files.
  • Looking at the example directory:
    • Go to 1-XYZ/decoded/tree.json to see the snapshot tree.
    • Each '#-XYZ' string in the tree.json correlates to a file in the decoded folder. These files are essentially blobs.
    • 0-XYZ/decoded/tree.json is an older snapshot tree.
    • The messages.json is a list of ops/messages that are stored.
  • For 401 authentication errors, as stated above, check that getkeys has been run.

Example Output

Messages Stats

--stat:message

105 total messages (103 delta storage, 2 initial ws messages, 0 dup)
Message Type (All)                                                       | Count      Bytes
----------------------------------------------------------------------------------------------------
join                                                                     |    19       7588
propose                                                                  |     1        269
noop                                                                     |    15       2806
attach                                                                   |     1        278
op                                                                       |    53      23854
leave                                                                    |    16       3810
----------------------------------------------------------------------------------------------------
Total                                                                    |   105      38605

--stat:dataType

107 total messages (105 delta storage, 2 initial ws messages, 0 dup)
Data Type (Operations only)                                              | Count      Bytes
----------------------------------------------------------------------------------------------------
map                                                                      |    28      13172
mergeTree                                                                |    20       7450
----------------------------------------------------------------------------------------------------
Total                                                                    |    48      20622

--stat:channel

109 total messages (107 delta storage, 2 initial ws messages, 0 dup)
Channel name (Operations only)                                           | Count      Bytes
----------------------------------------------------------------------------------------------------
[defaultDataStore]/root (map)                                            |     3       1232
[defaultDataStore]/0fb26504-369f-4234-ad97-0a303d3ec81f (map)            |     0          0
[defaultDataStore]/74577601-3af5-49a1-9ca5-db5d9ee128a8 (mergeTree)      |    20       7450
[defaultDataStore]/91ac6df2-dda6-409a-b5e3-be84ce9ab138 (map)            |    25      11940
[defaultDataStore]/92ebb388-68a0-4fc2-859e-5c01f12e992d (map)            |     0          0
----------------------------------------------------------------------------------------------------
Total                                                                    |    48      20622

Snapshot Stats

--stat:snapshot

Blob Path                                                                  | Bytes
----------------------------------------------------------------------------------------------------
!CONTAINER!/.attributes                                                    | 156
!CONTAINER!/.blobs                                                         | 4
!CONTAINER!/.gitmodules                                                    | 140
!CONTAINER!/deltas                                                         | 872
!CONTAINER!/quorumMembers                                                  | 220
!CONTAINER!/quorumProposals                                                | 4
!CONTAINER!/quorumValues                                                   | 184
[defaultDataStore]/.component                                              | 32
[defaultDataStore]/0fb26504-369f-4234-ad97-0a303d3ec81f/.attributes        | 64
[defaultDataStore]/0fb26504-369f-4234-ad97-0a303d3ec81f/header             | 4
[defaultDataStore]/74577601-3af5-49a1-9ca5-db5d9ee128a8/.attributes        | 72
[defaultDataStore]/74577601-3af5-49a1-9ca5-db5d9ee128a8/content/header     | 448
[defaultDataStore]/74577601-3af5-49a1-9ca5-db5d9ee128a8/content/catchupOps | 4
[defaultDataStore]/74577601-3af5-49a1-9ca5-db5d9ee128a8/header             | 180
[defaultDataStore]/91ac6df2-dda6-409a-b5e3-be84ce9ab138/.attributes        | 64
[defaultDataStore]/91ac6df2-dda6-409a-b5e3-be84ce9ab138/header             | 168
[defaultDataStore]/92ebb388-68a0-4fc2-859e-5c01f12e992d/.attributes        | 64
[defaultDataStore]/92ebb388-68a0-4fc2-859e-5c01f12e992d/header             | 4
[defaultDataStore]/root/.attributes                                        | 64
[defaultDataStore]/root/header                                             | 300
----------------------------------------------------------------------------------------------------
Total snapshot size                                                        | 3048

Debugging

If you would like to debug fetch-tool, you can create a unit test. Remember to add tsconfig.json and list mocha in devDependencies.

In the unit test, you can use setArguments() from fluidFetchArgs to pass in arguments you want to test. Then call the methods you want to run and you will be able to set breakpoints in vscode.

Example

describe("fetch tool", () => {
	it("can fetch messages", async () => {
		fluidFetchArgs.setArguments(your_args);
		const documentService = await fluidFetchInit(your_paramURL);
		await fluidFetchMessages(documentService, your_saveDir);
	});
});

Contribution Guidelines

There are many ways to contribute to Fluid.

Detailed instructions for working in the repo can be found in the Wiki.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

This project may contain Microsoft trademarks or logos for Microsoft projects, products, or services. Use of these trademarks or logos must follow Microsoft’s Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.

Help

Not finding what you're looking for in this README? Check out fluidframework.com.

Still not finding what you're looking for? Please file an issue.

Thank you!

Trademark

This project may contain Microsoft trademarks or logos for Microsoft projects, products, or services.

Use of these trademarks or logos must follow Microsoft's Trademark & Brand Guidelines.

Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.