pg_tiktoken

Overview

PIGSTY 3rd Party Extension: pg_tiktoken : tiktoken tokenizer for use with OpenAI models in postgres

Information

• Extension ID: 1870
• Extension Name: pg_tiktoken
• Package Name: pg_tiktoken
• Category: RAG
• License: Apache-2.0
• Website: https://github.com/kelvich/pg_tiktoken
• Language: Rust
• Extra Tags: pgrx
• Comment:

Metadata

• Latest Version: 0.0.1
• Postgres Support: 17,16,15,14,13
• Need Load: Shared library do not need explicit loading
• Need DDL: Need CREATE EXTENSION DDL
• Relocatable: Can be installed into other schemas
• Trusted: Untrusted, Require Superuser to Create
• Schemas: N/A
• Requires: N/A

RPM / DEB

• RPM Repo: PIGSTY
• RPM Name: pg_tiktoken_$v
• RPM Ver : 0.0.1
• RPM Deps: N/A
• DEB Repo: PIGSTY
• DEB Name: postgresql-$v-pg-tiktoken
• DEB Ver : 0.0.1
• DEB Deps: N/A

Availability

Installation

Install pg_tiktoken via the pig CLI tool:

pig ext install pg_tiktoken

Install pg_tiktoken via Pigsty playbook:

./pgsql.yml -t pg_extension -e '{"pg_extensions": ["pg_tiktoken"]}' # -l <cls>

Install pg_tiktoken RPM from YUM repo directly:

dnf install pg_tiktoken_17;
dnf install pg_tiktoken_16;
dnf install pg_tiktoken_15;
dnf install pg_tiktoken_14;
dnf install pg_tiktoken_13;

Install pg_tiktoken DEB from APT repo directly:

apt install postgresql-17-pg-tiktoken;
apt install postgresql-16-pg-tiktoken;
apt install postgresql-15-pg-tiktoken;
apt install postgresql-14-pg-tiktoken;
apt install postgresql-13-pg-tiktoken;

Create pg_tiktoken extension on PostgreSQL cluster:

CREATE EXTENSION pg_tiktoken;