tinypg
v7.0.1
Published
Easy way to call sql files using postgres.
Downloads
2,234
Readme
Quick Start
npm i tinypg
import { TinyPg } from 'tinypg'
// Options default to PG environment variables
const db = new TinyPg()
async function main() {
const result = await db.query(`SELECT 'Hello, World'`)
console.log(result)
}
main()
// Prints:
// { row_count: 1,
// rows: [ { '?column?': 'Hello, World' } ],
// command: 'SELECT' }
Introduction
TinyPg makes it possible to use objects as the source for parameters in a query. For example:
db.query(`
SELECT *
FROM customer
INNER JOIN address ON address.customer_id = customer.customer_id
WHERE address.state = :state
AND customer.first_name = :first_name
AND customer.last_name = :last_name;
`, {
first_name: 'Joe',
last_name: 'Andaverde',
state: 'Kansas',
})
SQL files over embedded strings
Now that we're past the mess of managing parameters the next step is organizing our SQL statements. It's kind of ugly to embed SQL in your JavaScript syntax. TinyPg allows for specifying a directory for which it will load all files with the .sql
extension. The path to the file will normalized into a key for which you can look up and execute the SQL in the file as a prepared statement. For our projects we have hundreds of different SQL queries. It's worth noting that we don't subscribe to tools that generate SQL on your behalf. Many of our queries require use of Postgres features that no SQL generator can provide. Therefore, we don't bother trying to force these libraries to be smarter and stick to the language that's best suited for retrieving data from relational data stores: SQL.
Consider the following directory structure:
/app
/db
/address
create.sql
/customer
create.sql
fetch.sql
search.sql
If you provide /app/db as a root directory to TinyPg it will load and parse parameters from all sql files beneath that directory. These files are keyed using the path to the file e.g. address.create or customer.fetch. Here's an example usage:
db.sql('customer.search', {
first_name: 'Joe',
last_name: 'Andaverde',
state: 'Kansas',
})
TinyPg checks for the existence of required parameters when each query is executed. Instead of placing db null in it will fail with an error message describing the missing parameter. I highly recommend using an object literal to specify parameter in order to use some of the static analysis tools like tslint or the VS Code plugin.
Transaction support
If you've ever looked at handling transactions with node-postgres you'll quickly realize that it's easy to get into deadlock. Tiny handles the re-use of the same connection for all queries performed within the same transaction provided you use the new database object provided by the call to .transaction. Here's how to create a customer and associate an address in the same transaction.
db.transaction(async transaction_db => { // BEGIN
const create_result = await transaction_db.sql('customer.create', { // INSERT
first_name: 'Joe',
last_name: 'Andaverde',
})
const customer = create_result.rows[0]
await transaction_db.sql('address.create', { // INSERT
customer_id: customer.customer_id,
street: '123 W SomeStreet',
city: 'SomeCity',
state: 'SomeState',
zip: 12345,
})
return customer.customer_id
}) // COMMIT
.then(async customer_id => {
const fetch_result = await db.sql('customer.fetch', { // SELECT
customer_id: customer_id,
})
return fetch_result.rows[0]
})
Whenever your promise succeeds it'll automatically execute COMMIT and in the event of failure it will execute ROLLBACK. Calling transaction on a Tiny instance that's already in a transaction will be a no-op. It is important that the lambda given to the transaction function return a promise. Otherwise COMMIT may be called at an unexpected time. Synchronous errors thrown in a transaction lambda will be caught and result in a ROLLBACK.
Here's the sequence of SQL statements that would be executed:
BEGIN
INSERT INTO customer ...
INSERT INTO address ...
COMMIT
SELECT * FROM customer ...
Notice the select AFTER the transaction has been committed. This is very important in order to return data that's actually persisted in the database.
Events
Events are emitted for the beginning and end of a query. This has been helpful for us to diagnose slow running queries. Our practice is to create a single instance of TinyPg per process and attach handlers to the query and result events to log all database queries.
Sometimes you need to associate several database calls with some context e.g., a web request. TinyPg provides a way to create a brand new event emitter that can emit events separately from the global handlers. This functionality isn't thoroughly flushed out and may not fit all use cases but works great for us thus far. Here's an example:
const db = new TinyPg(options)
function ApiRequestHandler(request, reply) {
const isolated_db = db.isolatedEmitter()
isolated_db.events.on('query', context => {
console.log(request.request_id, context.name)
})
return new UserService(isolated_db).list()
.then(users => reply(users))
}
In the above example isolated_db is the same instance of TinyPg except with an overridden events property and dispose method to remove all listeners. The UserService can create other services and pass its reference to isolated_db to other services. In doing so, you can track all database queries executed as the result of every API request.
Hooks
Hooks enable user defined functions to be called before (or after depending on the hook) TinyPg functions are executed. Most hooks allow user defined context to be passed throughout the hook lifecycle. Different hooks defined in the same TinyHooks
object maintain a shared context across each hook call. The following hooks are currently supported:
preSql
- Called at the start of
TinyPg.sql
before the db call is performed. (tiny_ctx: TinyCallContext, name: string, params: TinyPgParams) => HookResult<[string, TinyPgParams]>
- Context for the given
HookSet
is set to thectx
field of the object returned
- Called at the start of
preRawQuery
- Called at the start of
TinyPg.query
before the db call is performed. (tiny_ctx: TinyCallContext, query: string, params: TinyPgParams) => HookResult<[string, TinyPgParams]>
- Context for the given
HookSet
is set to thectx
field of the object returned
- Called at the start of
onQuery
- Called when the
Pg.PoolClient
is obtained (start of db call). The query event is also emitted at this point. (ctx: any, query_begin_context: QueryBeginContext) => any
- Context for the given
HookSet
is set to the return value
- Called when the
onSubmit
- Called when the query is [submitted]. The submit event is emitted at this point. See Pg Query.
(ctx: any, query_submit_context: QuerySubmitContext) => any
- Context for the given
HookSet
is set to the return value
onResult
- Called when the query promise is resolved or rejected. The result event is emitted at this point.
(ctx: any, query_complete_context: QueryCompleteContext) => any
- Context for the given
HookSet
is set to the return value
preTransaction
- Called at the start of
TinyPg.transaction
. (transaction_id: string) => any
- Transaction context for the given
HookSet
is set to the return value
- Called at the start of
onBegin
- Called immediately after the
BEGIN
promise is resolved to begin the transaction. (transaction_ctx: any, transaction_id: string) => any
- Transaction context for the given
HookSet
is set to the return value
- Called immediately after the
onCommit
- Called immediately after the
COMMIT
promise is resolved to commit the transaction. (transaction_ctx: any, transaction_id: string) => any
- Transaction context for the given
HookSet
is set to the return value
- Called immediately after the
onRollback
- Called immediately after the
ROLLBACK
promise is resolved to abort the transaction. (transaction_ctx: any, transaction_id: string, error: Error) => any
- Transaction context for the given
HookSet
is set to the return value
- Called immediately after the
Note: a HookResult
follows the form:
interface HookResult<T> {
args: T
ctx: any
}
There is only one rule about using hooks: HOOKS MUST BE SYNCHRONOUS
Hooks can be created via the hooks
field on the TinyPgOptions
passed to the constructor OR by calling withHooks(hooks: TinyHooks)
on an instance of TinyPg
. An example TinyHooks
object can be found below. Notice how well hooks play with tracing tools such as StackDriver trace.
{
preSql: (tiny_context, file_name, params) => {
const tracer = TraceAgent.get()
const span = tracer.createChildSpan({
name: `${file_name}_sql`,
})
_.forEach(buildSqlLabels(tiny_context, params), label => {
span.addLabel(label.label_key, label.label_value)
})
return {
ctx: {
sql_span: span,
},
args: [file_name, params],
}
},
onResult: (ctx: { sql_span: TraceAgent.PluginTypes.Span }, query_complete_context) => {
const tracer = TraceAgent.get()
if (tracer.isRealSpan(ctx.sql_span)) {
_.forEach(buildResultLabels(query_complete_context), label => {
ctx.sql_span.addLabel(label.label_key, label.label_value)
})
ctx.sql_span.endSpan()
}
return ctx
},
preTransaction: transaction_id => {
const tracer = TraceAgent.get()
const span = tracer.createChildSpan({
name: 'tinypg_transaction',
})
span.addLabel('transaction_id', transaction_id)
return {
transaction_span: span,
}
},
onCommit: (transaction_ctx: { transaction_span: TraceAgent.PluginTypes.Span }, _transaction_id) => {
const tracer = TraceAgent.get()
if (tracer.isRealSpan(transaction_ctx.transaction_span)) {
transaction_ctx.transaction_span.endSpan()
}
return transaction_ctx
},
onRollback: (transaction_ctx: { transaction_span: TraceAgent.PluginTypes.Span }, _transaction_id, error) => {
const tracer = TraceAgent.get()
if (tracer.isRealSpan(transaction_ctx.transaction_span)) {
transaction_ctx.transaction_span.addLabel('error', error)
transaction_ctx.transaction_span.endSpan()
}
return transaction_ctx
},
}
VS Code Plugin Support
If you're using TypeScript in your project (which I highly recommend) you can get an extra level of validation and editor integration by using the TinyPg VS Code plugin. This plugin can statically analyze (why I suggest using object literals) your code to ensure you've referenced sql files that exist and have provided all required parameters.
API
constructor(options: Partial<T.TinyPgOptions>)
- root_dir: string[] - a list of directories. All directories must be specified using the full path.
- connection_string: string - The database connection string in URL format. e.g. postgres://user:password@host:port/database?options=query
- error_transformer: Function - Allows transforming all errors from TinyPg to your domain.
- capture_stack_trace: boolean - Opt-in to capturing stack trace to give a better indication of what function in your domain caused an error.
- tls_options - TLS options passed to the underlying socket.
- pool_options: (See node-pg-pool - only difference is casing)
- hooks: TinyHooks - TinyHooks object see Hooks for details about hooks.
Example error_transformer
const error_transformer = (error) => {
const parseErrorByCode = () => {
const pg_error = error.queryContext.error
const code = pg_error.code
switch (code) {
case '22P02': // Invalid text representation
return new E.InvalidArgumentError(error.message)
case '23502': // Constraint error
return new E.InvalidArgumentError(`Invalid Argument: ${pg_error.column}`)
case '23503': // Foreign key violation
return new E.ForeignKeyViolationError('Foreign Key Violation', pg_error)
case '23505': // unique violation
case '23P01': // exclusion constraint violation
return new E.ConflictError('Data Conflict Error', pg_error)
case '23514': // Check Violation
return new E.InvalidArgumentError(`Invalid Argument: ${error.message}`)
default:
return new E.UnknownPostgresError(error.message)
}
}
let new_error
if (error.queryContext && error.queryContext.error && error.queryContext.error.code) {
new_error = parseErrorByCode()
} else {
new_error = new E.UnknownPostgresError(error.message)
}
new_error.stack = error.stack
return new_error
}
See Pg Error Codes Documentation
query<TResult,TParams>(raw_sql: string, params?: Object): Promise<T.Result>
- raw_sql: string - The SQL query to execute.
- params: Object (optional) - parameters for the query.
sql<TResult,TParams>(name: string, params?: Object): Promise<T.Result>
- name: string - The key of the sql file. This is the path to the file substituting
.
for path delimiter. e.g.users.create
formattable(name: string): T.FormattableDbCall
Select a SQL file that has formattable parts. See node-pg-format for format strings. This is useful when needing to build dynamic queries.
- name: string - The key of the sql file. This is the path to the file substituting
.
for path delimiter. e.g.users.create
formattable example usage
database/users/retrieve.sql
SELECT *
FROM users
WHERE last_name = :last_name
ORDER BY
-- Custom ordering
%s
user_id DESC;
Usage in code
db.formattable('users.retrieve')
.format('last_name ASC,')
.query({ last_name: 'Andaverde' })
Resulting Query
SELECT *
FROM users
WHERE last_name = :last_name
ORDER BY
-- Custom ordering
last_name ASC,
user_id DESC;
transaction<T = any>(tx_fn: (db: TinyPg) => Promise): Promise
Starts a database transaction and ensures all queries executed against the provided TinyPg instance use the same client.
- tx_fn: (db: TinyPg) => Promise - Provides db to perform transacted queries.
withHooks(hooks: T.TinyHooks): TinyPg
Returns a new instance of TinyPg with the given TinyHooks
added to the end of the instance's hook collection. See Hooks for more details about each hook.
interface TinyHooks {
preSql?: (tiny_ctx: TinyCallContext, name: string, params: TinyPgParams) => HookResult<[string, TinyPgParams]>
preRawQuery?: (tiny_ctx: TinyCallContext, query: string, params: TinyPgParams) => HookResult<[string, TinyPgParams]>
onQuery?: (ctx: any, query_begin_context: QueryBeginContext) => any
onSubmit?: (ctx: any, query_submit_context: QuerySubmitContext) => any
onResult?: (ctx: any, query_complete_context: QueryCompleteContext) => any
preTransaction?: (transaction_id: string) => any
onBegin?: (transaction_ctx: any, transaction_id: string) => any
onCommit?: (transaction_ctx: any, transaction_id: string) => any
onRollback?: (transaction_ctx: any, transaction_id: string, error: Error) => any
}
isolatedEmitter(): T.Disposable & TinyPg
Provides an isolated event emitter so that query
, submit
, and result
events (in that order) can be monitored for all queries related to the new TinyPg instance.
close(): Promise
Shuts down the postgres client pool.
Development
You should have a local development Postgres server running. This server must allow connections from the postgres
user without password. If this isn't the behavior your want change the connection string in src/test/helper.ts
.
npm install
npm test