neogrok

Neogrok is a frontend for zoekt, a fast and scalable code search engine. Neogrok exposes zoekt's search APIs in the form of a modern, snappy UI. Neogrok is a SvelteKit application running on Node.js and in the browser.

There is a demo deployment of neogrok and zoekt running together at https://neogrok-demo-web.fly.dev/. Try it out!

Features and usage

Together, neogrok and zoekt provide:

• Fast, live code search with a syntax based on regular expressions.
• Easy deployments. A single deployment of zoekt can performantly index and serve thousands of source repositories, using one of the many available indexers like zoekt-git-index to produce binary index files (called "shards") that are served by zoekt-webserver. Neogrok is just a veneer on top of zoekt-webserver; the only necessary configuration for neogrok is the URL of a running zoekt-webserver.
• Low resource utilization. The demo (which just indexes the neogrok and zoekt repos) happily runs on the smallest instances Fly can provision. Indexing the Linux kernel produces about 2.7GiB of index shards, and serving those shards uses just under 1 GiB of RAM.

Beyond that, neogrok aims to be self-documenting. You can find details on usage and features on the demo site's about page, and a full description of the zoekt search syntax on the site's syntax page.

Installing

Neogrok is packaged for installation on NPM. Simply npm install -g neogrok to install an executable.

Alternatively, building from source is easy. Clone the repository, yarn install && yarn run build && yarn run start. You can of course run the server without intermediation by yarn, by doing whatever yarn run start does directly; but the relevant commands may change in the future, whereas yarn run start will not.

Deploying

The demo deployment is configured in this repository. This configuration can serve as a guide for your own deployments of neogrok together with zoekt.

Configuration

Neogrok may be configured using a JSON configuration file, or, where possible, environment variables. Configuration options defined in the environment take precedence over those defined in the configuration file.

The configuration file is read from /etc/neogrok/config.json by default, but the location may be customized using the environment variable NEOGROK_CONFIG_FILE. The file is entirely optional, as all of the required configuration may be defined in the environment. If it is present, the file's contents must be a JSON object with zero or more entries, whose keys correspond to the option names tabulated below.

Configuration options

| Option name | Environment variable name | Required Y/N | Description | | :------------------------ | :------------------------ | :----------- | :--------------------------------------------------------------------------------------------------------------------------------- | | zoektUrl | ZOEKT_URL | Y | The base zoekt URL at which neogrok will make API requests, e.g. http://localhost:6070 | | openGrokProjectMappings | N/A | N | An object mapping OpenGrok project names to zoekt repository names; see below |

SvelteKit environment variables

Note that you can also configure, among other things, which ports/addresses will be bound, using SvelteKit's Node environment variables. See the list here.

Prometheus metrics

Neogrok exports some basic Prometheus metrics on an opt-in basis, by setting a PROMETHEUS_PORT or PROMETHEUS_SOCKET_PATH, plus an optional PROMETHEUS_HOST. These variables have the exact same semantics as the above-described SvelteKit environment variables, but the port/socket must be different than those of the main application. When opting in with these variables, /metrics will be served at the location they describe.

/metrics is required to be served with a different port/socket so as to not expose it on the main site; serving one port to end users and another to the prometheus scraper is the easiest way to ensure proper segmentation of the neogrok site from internal infrastructure concerns, without having to run a particularly configured HTTP reverse proxy in front of neogrok.

OpenGrok compatibility

As an added bonus, neogrok can serve as a replacement for existing deployments of OpenGrok, a much older, more intricate, slower, and generally jankier code search engine than zoekt. Neogrok strives to provide URL compatibility with OpenGrok by redirecting OpenGrok URLs to their neogrok equivalents: simply deploy neogrok at the same origin previously hosting your OpenGrok instance, and everything will Just Work™. To the best of our ability, OpenGrok Lucene queries will be rewritten to the most possibly equivalent zoekt queries. (Perfect compatibility is not possible as the feature sets of each search engine do not map one-to-one.)

Renaming OpenGrok projects to zoekt repositories

If your OpenGrok project names are not identical to their equivalent zoekt repository names, you can run neogrok with the appropriate openGrokProjectMappings configuration, which maps OpenGrok project names to zoekt repository names. With this data provided, neogrok can rewrite OpenGrok queries that include project names appropriately.